Research Questions
- How can LLMs make more rational decisions in complex real-world scenarios involving uncertainty (e.g., corporate earnings calls)?
- How can uncertainty be quantified using factor profiles?
- How can analogical reasoning based on similar past cases improve LLM decisions and make them more transparent?
Results
- The DEFINE framework achieved higher accuracy and F1 scores than the alternative methods (accuracy 29.6%, F1 23.7%).
- Summarising long transcripts into structured factor profiles improved decision accuracy.
- Decisions were more evenly distributed across five categories (Strong Buy → Strong Sell), with particularly strong performance on “Strong Buy” predictions.
- Training with cross-sector data outperformed training on a single sector or a single company.
- The analogy approach correctly transferred insights from similar historical cases 69% of the time.
Findings
- Performance:
- DEFINE outperformed DeLLMa and standard LLM + chain-of-thought approaches.
- Efficiency Through Structure:
- Using structured factor summaries (15 factors across 3 groups) yielded higher accuracy than processing full-length transcripts.
- Balanced Decision Distribution:
- Model decisions did not cluster around “Buy”; outputs were more balanced across all five categories.
- Analogy-Based Reasoning:
- When KL divergence was used to find similar past examples, 69% of decisions matched the closest historical analogue.
- Unexpected Insights:
- In some cases, the model recommended “Buy” even when the probability of a positive outcome was low, reflecting the rational paradoxes inherent in investment decisions.
Scores
- LLM Models: 5
- Synthetic Data: 1
- Method: 5
- Speed: 3
- Ethics: 1
- Accuracy: 5
- Demographics: 0
If you would like to explore this research in more detail, click here to read the full paper.