How AI Synthetic Testers Reach ~90% of Human Reliability

New peer-reviewed research shows how to get AI synthetic testers to match ~90% of human reliability — and it has nothing to do with which model you use.

"Just ask the AI to rate it 1 to 5" is the trap

If you've tested an AI synthetic panel yourself, you've probably noticed something odd: no matter what you show it, the model keeps answering "3."

That's not a fluke. It's a documented, measurable failure mode — and a new study finally explains why it happens and how to fix it.

The paper, LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings, was published in October 2025 by researchers at PyMC Labs working with Colgate-Palmolive. It's one of the most rigorous benchmarks of LLM-based synthetic respondents to date: 57 real consumer research surveys, 9,300 human participants, tested against synthetic panels built on GPT-4o and Gemini-2.0-flash.

At Uxia, we work with synthetic testers every day, so a benchmark this size — with a real corporation's real survey data as the ground truth — was worth digging into in detail.

What the researchers actually tested

The team compared three ways of getting a purchase-intent score out of an LLM:

1. Direct Likert Rating (DLR) — ask the model to reply with a single digit, 1 to 5. This is how most naive "AI panel" tools work today.

   
   

2. Follow-up Likert Rating (FLR) — let the model write a short free-text reaction first, then have it (or another instance of it) convert that into a number.

   
   

3. Semantic Similarity Rating (SSR) — the method the paper introduces. The model writes a free-text reaction, and that text is mapped onto a 1–5 distribution by comparing its embedding to a set of reference statements for each point on the scale.

   
   

The "everything is a 3" problem, quantified

When the researchers asked the model for a direct number, the ranking of products came out reasonably close to human data — about 80% correlation attainment (a test–retest-adjusted reliability score the authors define, where 100% would mean the synthetic panel is indistinguishable from a second human panel).

But the shape of the responses was way off. Real respondents clustered heavily at 4 and 5. The AI models almost never picked 1 or 5, defaulting instead to safe, central answers. Distributional similarity — how closely the synthetic response curve matched the real one — came out at just 0.26–0.39 on a 0–1 scale for direct ratings.

"Models typically replied with response '3', i.e. a 'safe' regression to the center of the scale." — Maier et al. (2025)

In other words, a naive AI panel can look directionally right while still describing a completely different, flatter emotional reality than your actual users.

The fix: let the model talk first

Once the researchers switched to text-first elicitation — SSR in particular — both problems improved sharply:

• Distributional similarity jumped to above 0.85 (versus 0.26–0.39 for direct ratings)

• Correlation attainment reached ~90% of human test–retest reliability

• No fine-tuning, no training data, no extra infrastructure — just a different way of asking the question

This is the core finding worth sitting with: the ceiling on synthetic-panel accuracy isn't set by the underlying model's intelligence. It's set by whether you're asking it to reason in language before you force it into a number.

Persona depth isn't a nice-to-have

The second finding is just as relevant to anyone building or buying synthetic testing tools. The researchers ran an experiment where they stripped all demographic detail out of the prompts — no age, no income, no location, nothing.

The result: the model defaulted to rating almost everything positively. Correlation attainment collapsed from 92% down to 50%.

A thin, generic "synthetic user" doesn't just underperform — it stops giving you a signal at all. It tells you everything is good, which is the one thing you can't act on.

Where the paper is honest about the limits

What we liked about this study is that the authors don't oversell it. They're explicit that:

• SSR's accuracy depends on how well the reference statements are designed

• Some demographic patterns (age, income) were reproduced well; others (gender, region, ethnicity) were not consistently replicated

• The method is bounded by how much the underlying LLM actually knows about a category — it works well where there's abundant public discussion of a product type, and will be shakier in niche domains

• This is framed as a way to screen and prioritize before human research, not a replacement for it

That last point is the one we'd underline for anyone evaluating this space: even the most carefully validated synthetic method in this study isn't positioned as a full substitute for human research on high-stakes decisions.

What this means if you're evaluating any AI research tool

A few takeaways worth applying directly, whether you're using Uxia or comparing options:

• Ask how the tool elicits responses, not just which model it runs. A platform that quietly asks an LLM for a raw numeric score is likely hitting the "everything is a 3" problem this paper documents — regardless of how capable the model is.

• Persona detail changes the output, not just the framing. Thin personas produce flat, uniformly positive feedback. Detailed ones produce a signal you can actually act on.

• Treat synthetic results as a fast, directional read — not a verdict. Use them to catch obvious friction early and decide what's worth a deeper human study.

• The qualitative reasoning matters as much as the score. A number without an explanation is easy to misread; the "why" behind it is usually where the real insight lives.

• Validate against real users periodically, especially for high-stakes decisions. No synthetic method, including this one, claims to replace that.

This is close to how we've approached synthetic testing at Uxia from day one: AI-driven testing for speed and early validation, human research reserved for the deeper trust and behavioral questions synthetic panels still can't fully answer. It's a useful, independent confirmation to see a peer-reviewed study — run with a company the size of Colgate-Palmolive — land on the same conclusion.

See it for yourself

If you want to see synthetic and human testers compared side by side on a real product, not just a research benchmark, our public reports are at uxia.app/reports — including a head-to-head validation study built the same way this paper validates its results: same test, same conditions, synthetic and human side by side.

Source: Benjamin F. Maier, Ulf Aslak, Luca Fiaschi, Nina Rismal, Kemble Fletcher, Christian C. Luhmann (PyMC Labs), Robbie Dow, Kli Pappas (Colgate-Palmolive), and Thomas V. Wiecki (PyMC Labs). "LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings." October 9, 2025. arXiv:2510.08338