Research Questions
- How do LLMs behave in social interaction scenarios?
- Which mechanisms (fairness, reciprocity, competition) shape their responses?
- Can chain-of-thought (CoT) reasoning be used to predict LLMs’ social behaviour?
Results
- LLMs do not act solely out of self-interest; they also lean towards social welfare and reciprocity.
- Behavioural patterns vary by model family and capacity: as capacity increases, GPT and Mistral models become less self-interested, whereas LLaMA models become more self-interested.
- Larger models (GPT-4, LLaMA-3-70B) show strong group identity effects.
- CoT-based justifications predict decisions with 90%+ accuracy.
Findings
- Social Orientation:
- Most models produced prosocial, welfare-oriented responses, with relatively little competitive behaviour.
- Model Differences:
- GPT & Mistral: Higher capacity → less self-interest
- LLaMA: Higher capacity → more self-interest
- Group Identity Effects:
- High-capacity models showed a clear tendency towards social welfare when interacting with in-group members.
- Reciprocity:
- Except for Mistral 7B, all models showed both direct and indirect reciprocity, but unlike humans, they did not differentiate strongly between the two.
- Predictive Power of CoT:
- Emphasis on social welfare increased the likelihood of prosocial behaviour by ~30 percentage points.
- Emphasis on self-interest reduced prosocial behaviour by 15–25 percentage points.
- High AUC scores and >90% predictive accuracy indicate that CoT traces are powerful tools for understanding LLM social decision-making.
Scores
- LLM Models: 5
- Synthetic Data: 2
- Method: 5
- Speed: 1
- Ethics: 3
- Accuracy: 4
- Demographics: 0
If you would like to explore this research in more detail, click here to read the full paper.