Synthetic Respondents and AI Interviews vs Surveys: When to Use Each

Contents
Synthetic Respondents and AI Interviews vs Surveys: When to Use Each

Synthetic respondents are personas simulated by a language model that answer research questions in place of people. AI-moderated interviews are the opposite: real people answer, and an AI asks the questions and the follow-ups. Both help early in a project, and neither can measure what a market thinks. This guide covers what the research says about each, a decision table by task and a hybrid workflow that ends with a survey of real people.

The three methods get confused because vendors sell all of them as "AI research". They answer different questions, though, and the difference decides whether a result can go into a pricing decision or only into a brainstorm.

Key points

  • Synthetic respondents are personas simulated by a language model that answer in place of people, while AI-moderated interviews keep real participants and automate the interviewer.
  • Studies show synthetic answers can approximate averages for well-studied groups, but they vary less than human answers, shift with prompt wording and represent some groups poorly.
  • Use synthetic respondents to generate ideas, AI-moderated interviews to understand people, and a survey with real people to measure them.
  • Label synthetic output as synthetic, never mix it with real responses in one dataset and never report synthetic percentages as market data.

Synthetic respondents and AI-moderated interviews: what each one is

Synthetic respondents (also called silicon samples, digital twins or AI personas) are answers generated by a language model that has been told to act as a certain kind of person: "You are a 34-year-old nurse in Chicago who shops online twice a month." You can ask a thousand of them a questionnaire in minutes. No human answers anything.

AI-moderated interviews are conversations with real participants in which an AI plays the interviewer. It reads from a discussion guide, asks follow-up questions based on the answer, and summarizes the transcripts afterwards. The people and their opinions are real; the moderator and the first pass of analysis are automated.

A survey with real people asks a defined sample the same fixed questions, so answers can be counted, compared between groups and, when the sample represents the population, projected onto it with a known margin of error.

Synthetic respondentsAI-moderated interviewsSurvey with real people
Who answersA language model playing a personaReal participantsReal participants
What you getPlausible answers in secondsDepth and reasons, from dozens of peopleComparable numbers, from hundreds or thousands
Main strengthSpeed and zero recruitingFollow-up questions at scaleMeasurement you can defend
Main riskAnswers reflect the model, not your marketSmall, self-selected samples; missed follow-upsBad questions or a biased sample

What synthetic respondents get right

The idea gained credibility from a 2023 paper in Political Analysis, "Out of One, Many: Using Language Models to Simulate Human Samples" by Argyle and colleagues. They conditioned GPT-3 on thousands of demographic backstories taken from real participants in large US surveys, creating what they called "silicon samples". The simulated answers reproduced many of the response patterns of the human subgroups, including how attitudes related to each other. The authors called this property "algorithmic fidelity" and argued that models with enough of it could be a useful tool for social research.

Later work showed that the data behind each persona matters a great deal. Park and colleagues built agents for 1,052 Americans from two-hour interviews with each of them (arXiv preprint, first posted 2024). On held-out questions from the General Social Survey, agents built from interviews reached 83% of the consistency that the same people showed when they answered again two weeks later. Agents given only demographics reached 74%.

So synthetic answers can approximate averages for well-studied populations on well-studied topics, especially when each persona is grounded in data from a real person. That is a narrower claim than "we can skip talking to customers".

Where synthetic respondents fail

The same body of research documents limits that matter for business decisions.

Answers vary less than real people's. Bisbee, Clinton, Dorff, Kenkel and Larson asked ChatGPT personas to rate social and political groups and compared the results with the American National Election Study (Political Analysis, 2024). The averages came close. The spread was narrower than in the human data, relationships between variables often differed from the survey's, small changes in prompt wording shifted the results, and the same prompt gave different answers three months apart. A narrow spread is exactly what hides the customers who love or hate a product.

Example distributions of ratings: real people spread from strong dislike to strong liking, while synthetic personas cluster tightly around a similar average. Bisbee and colleagues (Political Analysis, 2024) found this pattern, and a narrow spread hides the customers who love or hate a product.

Some groups are poorly represented. Santurkar and colleagues compared language model opinions with 1,498 questions from Pew Research's American Trends Panel across 60 US demographic groups ("Whose Opinions Do Language Models Reflect?", ICML 2023). They found substantial misalignment that persisted even when the model was told which group to represent. People aged 65 and over, widowed people and people who attend religious services often were among the groups no model represented well.

Personas can flatten and misportray identities. Wang, Morgenstern and Dickerson ran studies with 3,200 people across 16 demographic identities and four language models (accepted at Nature Machine Intelligence, 2025). Prompted personas tended to sound like an outsider's view of a group and to erase the differences within it. Minority segments, often the ones a product team most needs to understand, are where this hurts most.

New products are hard to simulate. Brand, Israeli and Ngwe tested language models on willingness-to-pay studies ("Using LLMs for Market Research", Harvard Business School working paper). Out-of-the-box models reproduced willingness to pay for existing features in a product category, but needed fine-tuning on earlier human results to simulate reactions to new features, and that fine-tuning may not carry over to unrelated categories. A brand-new product is precisely the case with no human data to lean on.

Taken together, these studies suggest that synthetic respondents reproduce what is typical in the text the model learned from. Your next customer segment, your new feature and your price are rarely typical.

What AI-moderated interviews are good for

AI-moderated interviews keep the people real and automate the moderator. Their appeal is depth at a scale human teams struggle with: fifty interviews in a week, in several languages, each with follow-up questions, and transcripts coded the same day.

Evidence on quality is still thin. In a small study published at the LaTeCH-CLfL 2025 workshop, Wuttke and colleagues randomly assigned students to an AI or a human interviewer using the same questionnaire. They report data quality comparable to human-led interviews, and participants gave longer answers to the AI. The AI interviewer also often failed to ask a follow-up after a surprising or unclear answer, which is the moment a good human moderator earns their fee. The study covered eleven interviews, so treat it as an early signal.

Three limits apply to any AI-moderated study. Participants are still a small, self-selected group, so the results describe reasons and patterns, not shares of a market. The quality of the discussion guide decides the quality of the follow-ups. And the AI's summary of the transcripts needs the same checks as any AI coding of open answers: read a sample yourself and confirm every quote. Our guide to analyzing survey data with AI has a verification table that works for interview transcripts too, and the thematic analysis guide covers the manual method.

Interviews also collect personal data: a voice, sometimes a face on video, and whatever people share about their lives. Before the interview starts, tell participants that an AI will ask the questions, that the conversation is recorded and transcribed, who processes the recordings (including the interview platform) and how long you keep them, then ask for their consent. Collect only what the research question needs, and check the privacy rules that apply where participants live, such as the GDPR in Europe or the CCPA in California.

AI interviews vs surveys vs synthetic respondents: which to use for which task

In short, use synthetic respondents to generate ideas, AI-moderated interviews to understand people, and a survey with real people to measure them. The table applies that to common research tasks.

Synthetic respondents, where a model answers, only generate hypotheses and wording to test and stay a draft. With real people, AI-moderated interviews explain reasons from dozens of people as supporting evidence, and a survey measures how common an opinion is across hundreds or thousands as the main evidence for a decision.

TaskSynthetic respondentsAI-moderated interviewsSurvey with real people
Brainstorm hypothesesYes, as a fast first draftYes, to find hypotheses you did not think ofLater, to test the shortlist
Pre-test a questionnaireYes, to catch confusing wording and missing optionsYes, as cognitive interviews with real peopleYes, as a small pilot before launch
Explore motives and contextOnly to draft the interview guideBest fitPartly, with open questions
Size a market or a segmentNoNo, samples are too smallYes, with a sample that represents the population
Measure satisfaction or NPSNo, the model has never used your productTo explain scores, not to produce themYes
Set a priceOnly to draft price points to testTo learn how buyers judge value and alternativesYes, with price questions to real buyers
Make a go/no-go decisionNoAs supporting evidenceYes, as the main evidence

A survey can only measure what a population thinks if the sample resembles that population. Our glossary entry on representativeness explains what that means in practice, and the sample size calculator shows how many answers you need for a given margin of error.

A hybrid workflow: personas, interviews, then a survey

The methods work best in sequence. Each step narrows what the next one has to answer, and only the last step produces numbers you report.

A five-step hybrid research workflow: AI drafts hypotheses and personas, synthetic personas stress-test the questionnaire, then 10 to 30 interviews with real people, a survey of real people and a final comparison where real data wins. Only the survey of real people produces numbers you report, and synthetic answers stay labelled and separate.

1. Draft hypotheses and personas with AI

Start by writing down the decision you need to make, then let AI help with the first draft of your thinking. Draft personas for the segments you expect, list hypotheses about what each one needs, and ask synthetic personas to react to your concept. Label everything from this step as "AI draft". Its job is to widen the list of possibilities and expose gaps in your assumptions.

2. Stress-test the questionnaire on synthetic personas

Before a single real person sees your survey, ask a few personas to answer it and explain what each question means to them. Language models are good at noticing double-barreled questions, missing answer options and jargon. They will not tell you how real people answer, but they will point at questions that need another look.

3. Explore with real people in interviews

Run 10 to 30 interviews with people from your target segments, moderated by a person or by an AI. Ask about recent, concrete behavior: the last time they solved the problem, what they used, what it cost them. Look for reasons and words you did not expect, and drop hypotheses nobody supports. This is primary research, and it is the first step that tells you something new about your market.

4. Measure with a survey of real people

Turn the surviving hypotheses into closed questions, keep one or two open questions for reasons, and send the survey to a sample sized for the precision you need. Our guide to quantitative research in surveys covers design and analysis. Before you trust the data, screen for automated and AI-written answers, which now reach online surveys too; the guide to bots and AI responses in surveys shows how.

5. Compare, then decide

If you ran synthetic personas in step one, compare their answers with what real people said. Where they diverge, the real data wins, and the gap tells you how far to trust synthetic answers on your next project. Make the decision on the survey results, with interview quotes explaining the numbers.

Rules for using synthetic answers honestly

  • Label synthetic output as synthetic in every document and slide.
  • Never mix synthetic answers with real responses in one dataset.
  • Never report synthetic percentages as market data.
  • Never present a synthetic answer as a quote or opinion of a real customer, in a report, a pitch or marketing material.
  • Record the model, the prompt and the date, because results can change between runs and over time.
  • Check synthetic results against at least one real benchmark before relying on them for a related question.
  • Be most skeptical about minority segments, new products and prices.

How to plan the research in SurveyNinja

The free AI tools on SurveyNinja cover the early, idea-generating steps. The research planner turns a business goal into a plan with objectives, audience, sample size, questions, channels, risks and a timeline. The hypothesis generator turns a problem or assumption into hypotheses, null hypotheses, survey test ideas and metrics. The persona generator drafts research personas with goals, pains, behavior and channels from a description of your product. Treat its personas the way this guide treats synthetic respondents: as a starting point to validate with research.

When it is time to measure, Ninja AI (Beta) in the survey builder drafts the survey from a description, a file such as a PDF or DOCX, or an image, on every plan including Free. It can also edit questions, logic and settings, and every change waits for you to click Apply. The survey then collects answers from real people, which is the step no simulation replaces.

Frequently asked questions

What are synthetic respondents?

Synthetic respondents are answers generated by a language model that has been prompted to act as a type of person, such as a young parent or an IT manager. They are fast and cheap, but no real person answers, so the results reflect patterns in the model's training data rather than the opinions of your market.

Can synthetic respondents replace surveys?

No. Studies show synthetic answers can approximate averages for well-studied groups, but they vary less than human answers, shift with prompt wording and represent some groups poorly. They are useful for brainstorming and for pre-testing a questionnaire. Decisions about markets, prices and satisfaction need answers from real people.

What is an AI-moderated interview?

It is an interview with a real participant in which an AI asks the questions, follows up on the answers and summarizes the transcript. It lets a team run dozens of interviews quickly. The participants are real, but samples are small, so the results explain reasons rather than measure how common they are. Participants should know an AI is asking the questions and agree to the recording.

Are AI interviews better than surveys?

They answer a different question. AI-moderated interviews are good for exploring motives, context and language with dozens of people. Surveys measure how common an opinion is across hundreds or thousands of people and allow comparisons between groups. Most projects need interviews first to find the right questions and a survey afterwards to measure the answers.

When is it fine to use synthetic data in market research?

Use it to brainstorm hypotheses, draft personas and interview guides, and find confusing wording in a questionnaire before a pilot. Keep it labeled, separate from real responses and out of reported numbers. Be most careful with minority segments, new products and prices, where research shows synthetic answers are least reliable.

How many interviews do you need before running a survey?

For exploratory interviews, 10 to 30 conversations within your target segments are a common starting point for finding the main reasons and the words people use. Stop when new interviews stop adding new themes. Then turn the findings into survey questions and size the survey sample with a sample size calculator for the precision you need.

0