Synthetic Users in UX Research: Help or Mislead?

On this page

• What are synthetic users (and what they’re not)?
• When synthetic users help: cheap hypotheses, fast
• When synthetic users mislead: six failure modes
• Synthetic vs real participants: the evidence gap
• A hybrid workflow: synthetic first, humans confirm
• In practice: de-risking a redesign study
• Decision checklist: use, validate, or avoid
• Frequently asked questions

Synthetic users promise research insight without the cost and delay of recruiting real people. When budgets are tight and timelines short, the pitch lands well. But a fluent, confident answer from an AI persona is not evidence — and teams keep treating it as if it were. Here’s where synthetic users earn a place in your process, the six ways they mislead, and a workflow that keeps them honest.

What are synthetic users (and what they’re not)?

Synthetic users are AI-generated participant profiles. You prompt a large language model (LLM) with demographic and behavioural data for a target segment, and it answers your interview or survey questions as if it were a person in that segment. Some tools layer on personality models — OCEAN-style trait profiles covering openness, conscientiousness, extraversion, agreeableness, and neuroticism — and hold context between questions so a “participant” stays consistent across a session. Multi-agent setups run several personas in parallel. The architecture is tidy. It does not change what the output is: predicted text, not observed behaviour.

Two distinctions matter. Synthetic users are AI participants you deploy on purpose, as subjects. They are not fraudulent AI respondents slipping into a real panel for the incentive — that is a data-quality problem, covered in detecting fake and AI-generated participants. Nor are they AI running sessions with real humans; see AI-moderated interviews with real people for that.

Set the expectation now. Treat synthetic outputs as cheap hypotheses that help you prioritise and de-risk a study. They are not a replacement for real participants.

When synthetic users help: cheap hypotheses, fast

Synthetic users earn their keep before you spend recruitment budget, when a rough, directional answer is enough.

Front-loading the problem space. Run a few synthetic interviews to absorb the vocabulary of a domain, map the pain points people are likely to raise, and spot gaps in your thinking. You walk into real fieldwork with sharper questions and fewer wasted warm-up minutes.

Concept and message triage. Say you have fifteen value propositions or onboarding flows and can only test four with real users. Synthetic runs help you cut the list. They screen out the obviously weak options — they don’t pick the winner.

Unfamiliar segments and quiet periods. Briefed on a segment you have never researched? Synthetic users give you a starting mental model to pressure-test. Between funded rounds, they can keep a lightweight insight loop running so momentum does not stall.

Pitch and proposal work. Early-stage strategy and pitch decks often need a defensible point of view before anyone funds a study. Synthetic input gives you that, as long as everyone knows validation comes later.

A word on accuracy. Published comparisons report roughly 85–92% thematic overlap between synthetic and human responses on logic-heavy tasks such as pricing structure or feature trade-offs. On emotional, open-ended, or identity-driven questions the gap is far wider. Treat any parity claim as task-dependent and unverified until your own data confirms it. For a sense of which questions suit which method, see usability testing vs user interviews.

When synthetic users mislead: six failure modes

1. Agreeable mode collapse. LLMs are tuned to be helpful and polite. Ask a synthetic user what it thinks of your concept and you usually get encouragement. We have watched synthetic runs praise ideas that real users later rejected outright — false confidence that feels like validation.

2. Missed surprises. Models predict the common path. They rarely volunteer the odd workaround, the misread label, or the sequence no one designed for — and those edge cases are usually where the real design flaw sits.

3. Training-data bias. An LLM knows most about the users who are best documented online: majority-language, higher-income, digitally fluent. Under-served groups are thin in the data and come back flattened or missing. If inclusion matters to your product, synthetic users quietly narrow your sample. See inclusive UX research methods for approaches built to close that gap.

4. No stakes. A synthetic user has never paid for anything, waited on hold, felt judged at a checkout, or used your app one-handed on a bus. Stated willingness to pay from an AI is fiction, and so is its account of how a task feels under pressure.

5. Fluent fabrication. Ask for specifics — a brand they use, a price they paid, a step in their routine — and a model invents plausible detail, then delivers it with the confidence of a real memory. You cannot separate the invented parts from the grounded ones.

6. The meta-failure. The most common mistake is treating synthetic output as evidence rather than input. Everything above is manageable while the output only feeds hypotheses. It turns dangerous the moment it lands in a decision log as a finding.

Synthetic vs real participants: the evidence gap

Here’s the hard line. Synthetic users produce plausible answers, but never behavioural data. They do not use the product. Nothing they say is grounded in a real attempt to complete a real task.

Real participants give you what synthetic ones structurally cannot:

• Friction under real conditions — the hesitation, the backtrack, the abandoned flow.
• Nonverbal cues — the frown, the pause, the tone that contradicts the words.
• Emotional depth — frustration and delight that come from something actually being at stake.
• Genuine task failure — the moment someone cannot do the thing, which no model reliably simulates.

Map your questions to the right method. Attitudinal and exploratory questions — how people think about a category, what language they use — tolerate synthetic input as a first pass. Behavioural and evaluative questions — can they complete this, where do they drop off, would they actually pay — need real humans doing real tasks. A method like first click testing only means something with real users clicking.

One rule with no exceptions: never send synthetic-only findings into a launch decision, an accessibility assessment, or any high-stakes stakeholder call.

A hybrid workflow: synthetic first, humans confirm

The point of synthetic users is to make your real research tighter and cheaper, not to replace it. A four-step loop keeps that boundary clear.

Step 1 — Generate. Use synthetic users to surface hypotheses, draft interview questions, and propose candidate segments. Run several personas, vary the traits, and collect the pain points and objections they raise. Output: a list of things that might be true.

Step 2 — Prioritise. Rank those hypotheses and concepts by potential impact and by how much nuance they need. Cut the shortlist hard. Recruitment budget should go where a real person’s reaction will differ most from a predicted one — emotional response, trust, edge cases, willingness to pay.

Step 3 — Validate. Run interviews or usability sessions with real, well-screened participants against the shortlist. Each hypothesis gets confirmed, adjusted, or overturned by someone attempting a real task. This is the step that produces evidence; guidance on sourcing the right people is in recruit real participants for user research.

Step 4 — Reconcile. Write down where synthetic and real findings agreed and where they diverged. The divergences are the useful part. They show where the model’s assumptions about this audience are wrong, and you can fold that back into sharper prompts next time. Tools like empathy mapping from interview data help you keep the two sources visibly separate.

Governance. Label every synthetic-derived claim as unvalidated until a human study confirms it — a tag in the research repository and a line in any deck: “hypothesis from synthetic research, not yet tested with users”. If a claim never reaches Step 3, it never gets presented as a finding.

In practice: de-risking a redesign study

On a concept-testing programme for a consumer fintech preparing to launch a crypto payment card, we ran synthetic users before any recruitment. The team had a long list of landing-page messages and feature ideas, and a fixed interview budget across three user segments.

Synthetic runs did two things well. They helped rank the segments — flagging active traders and freelancers as the strongest fit before we spoke to anyone — and they sharpened the discussion guide, surfacing fee transparency and payment reliability as themes worth probing in depth. We took a shortlist of roughly a dozen messages down to four.

They also got something wrong. The synthetic personas read an automated tax-reporting feature as a clear selling point. Real participants, across three rounds of interviews, read the same feature as a risk signal: handing tax data to a new provider made them more cautious, not less. A feature the synthetic runs favoured turned out, in practice, to be a barrier.

The payoff was a tighter guide that needed only minor edits after the first two sessions, fewer wasted questions, and a faster route to a recommendation we could defend. The redesign direction only shipped after real-user validation confirmed it. If you are weighing whether to run this in-house, see outsourcing research to an agency vs in-house.

Decision checklist: use, validate, or avoid

Green-light for synthetic-only work: early hypothesis generation, learning a domain’s vocabulary, drafting and sequencing interview questions, triaging a long concept list down to a testable shortlist.

Use, then validate with real people: segment exploration, message and positioning tests, roadmap and feature prioritisation. Synthetic input narrows the options; real sessions decide.

Avoid entirely: launch calls, accessibility evaluation, emotional-response testing, final pricing commitments, and anything shown to stakeholders as evidence. These need real users doing real tasks.

Litmus test: If a wrong answer would cost real money, harm users, or be presented to a decision-maker as proof, you need real participants — not synthetic ones.

Frequently asked questions

Can synthetic users replace real participants in UX research?

No. Synthetic users generate plausible answers, but they produce no behavioural data, no lived context, and no genuine willingness to pay. They have never used your product or felt the stakes of a real task. Use them to prioritise questions and de-risk a study, then validate with real, screened participants before anything reaches a decision.

How accurate are synthetic users?

Accuracy is task-dependent. Published comparisons report roughly 85–92% thematic overlap with human responses on logic-heavy tasks such as pricing structure and feature trade-offs. On emotional, open-ended, or identity-driven questions the gap is much wider. Treat any accuracy figure as unverified until your own real-user data confirms it for your audience and questions.

When should you not use synthetic users?

Avoid them for launch decisions, accessibility assessments, emotional-response testing, final pricing commitments, and anything presented to stakeholders as evidence. Those questions depend on real people doing real tasks under real conditions — exactly what a synthetic user cannot reproduce.

How do you validate synthetic user research with real people?

Treat every synthetic output as a hypothesis. Shortlist the ones worth testing, run interviews or usability sessions with well-screened participants, and let each hypothesis be confirmed, adjusted, or overturned. Document where synthetic and real findings diverged, and feed those corrections back into your prompts for the next study.

---

About Glasgow Research — Glasgow Research helps B2B SaaS teams turn customer and market research into product decisions. Work with us.