Research Questions
- Do LLM agents align more closely with human behaviour when given only demographic information, or when supplemented with human belief networks?
- Can providing a single belief “seed” improve human–LLM alignment across related topics?
- How does the structure of belief networks influence the accuracy with which LLMs imitate human viewpoints?
Results
- Using demographic information alone did not produce meaningful human–LLM alignment.
- When agents were given a single belief seed, alignment improved substantially for topics connected within the belief network.
- No improvement was observed for topics outside the belief network.
- The degree of alignment increased proportionally with the factor loadings within the belief network.
Findings
- Demographics Are Insufficient:
- Role-playing based solely on demographic cues failed to align LLM outputs with human beliefs in a meaningful way.
- Impact of a Belief Seed:
- LLM agents seeded with a single belief aligned more closely with human responses on related topics.
- Limited Alignment:
- On some topics, such as the death penalty, alignment remained at zero even when the correct opinion was given.
- Structural Dependence:
- Alignment varied with the strength of the connections in the belief network.
- Ethical Risks:
- Because false or harmful beliefs can also be simulated, the approach carries a risk of manipulation.
Scores
- LLM Models: 5
- Synthetic Data: 3
- Method: 4
- Speed: 1
- Ethics: 4
- Accuracy: 4
- Demographics: 3
If you would like to explore this research in more detail, click here to read the full paper.