Pew study finds AI survey replicas miss public opinion
Synthetic respondents overstated certainty and missed human answers
Pew Research Center’s September 30 report finds substantial gaps when AI systems stand in for human survey respondents.
The experiment reproduced three American Trends Panel surveys from early 2026. Its selected setup used Claude Opus 4.6 with demographic and attitude profiles, low reasoning and a model generated interpretation of each persona.
Across nearly 300 questions, synthetic estimates missed human results by about 12 percentage points on average. Almost half the questions had an option no synthetic respondent chose. Humans chose not sure around four times as often where offered.
What the result covers
The human comparison includes only panelists who had completed an earlier political typology survey. Pew says it has no plans to replace human respondents with AI.
The methods compare replicas answering sequentially with real panelists, rather than models predicting population averages. Most tested questions concerned politics. The experiment evaluates particular models and a specific approach to synthetic polling, not every use of AI in survey research.
Illustrative survey graphic by Tungilik under Creative Commons CC0. Placed on a white background and converted to WebP. It is unrelated to Pew’s survey interface.
