Research Questions
- How can AI systems and digital products be tested with heterogeneous users when human evaluation is costly, slow, and hard to scale?
- How can a population-scale persona dataset be built so that it preserves real-world dependencies and distributions?
- Do persona agents actually adhere to their assigned attributes — does what they say and do reflect the assigned persona?
- Do simulated-user studies reveal consistent differences across systems, tasks, and user groups?
Results
- MatrAIx provides an end-to-end evaluation infrastructure built from three components: Persona 8B, 8.3 billion records under a shared 1,290-dimensional schema; the MatrAIx Playground, which runs four environments; and MatrAIx Applications, a library of 1,010 tasks.
- A coreset of approximately 1 million personas was released for research: 599,847 human-grounded and 400,000 synthetic records.
- 18,189 evaluation trials were run across eight representative tasks, with persona agents powered by Claude Opus 4.8, GPT 5.5, and Claude Haiku 4.5.
- In a 400-trial controlled behavioral study, the assigned behavior was expressed or correctly suppressed in 91.5% of trials (366/400).
- Extraction quality for human-grounded personas was supported by six human raters on a source-matched subset of 100 personas, with a mean quality score of 4.135 out of 5.
Findings
- Population Construction:
- 1,290 categorical dimensions are grouped into background (238), psychology (210), capability (331), behavior and interaction (124), and lifestyle (387). Synthetic records are sampled through a dependency-preserving directed acyclic graph (DAG), so relationships such as age–education or region–language are not broken.
- Human-Grounded Records:
- Wikipedia biographies, Amazon review histories, the Stack Overflow Developer Survey, the General Social Survey, PRISM alignment profiles, and the consent-based MatrAIx Persona Survey are all mapped into the same schema, and de-identified by removing direct identifiers such as names and contact details.
- Four Evaluation Environments:
- Survey (price sensitivity, concept testing), AI Chatbot (satisfaction, latency tolerance, willingness to continue after a failure), Web (browser automation and computer-using agents), and App (a Docker-based desktop plus a remote macOS/iOS simulator).
- Task Library:
- 1,010 tasks span more than 25 domains — 621 Survey, 371 AI Chatbot, 12 Web, and 6 App — anchored on Commerce, Software, Finance, and Healthcare.
- Adherence by Environment:
- In the controlled behavioral study, success rates were 96% for Survey, 92% for Chatbot, 95% for Web, and 83% for App; 33 of the 40 attribute-by-environment cells reached at least four of five successes in both arms.
- Model Dependence:
- In the OpenBB task, trust level separated subgroups in the same order under all three persona-agent models (Cramér’s V = 0.228–0.363, all q < 10⁻⁸). Because persona effects can still vary by model, the persona-agent model must be reported with every result.
- Limitations:
- In the meal-planning task, subgroup differences (e.g. empty nesters 66% versus career changers 46%) did not remain significant after Benjamini–Hochberg correction; the authors present such results as descriptive rather than confirmed persona effects. Human studies remain necessary before generalizing to real populations or consequential decisions.
Scores
- LLM Models: 3
- Synthetic Data: 5
- Method: 5
- Speed: 4
- Ethics: 4
- Accuracy: 4
- Demographics: 5
If you would like to explore this research in more detail, click here to read the full paper.