Bots in Surveys: How to Spot and Prevent Fraudulent and AI-Generated Responses

Contents
Bots in Surveys: How to Spot and Prevent Fraudulent and AI-Generated Responses

Bots in surveys are scripts and AI agents that fill in online questionnaires, usually to collect an incentive. Together with people who rush through or answer several times, they produce fraudulent survey responses that look like real data. No single check catches them all, so this guide gives a layered checklist: where the risk comes from, what to set up before fielding, which checks to put inside the survey and how to clean the data with rules you write in advance.

The approach is methodological and works with any survey tool. The term itself, and the main types of fraud, are covered in our glossary entry on fraud detection.

Key points

  • Fake responses gather where an open link, a reward worth collecting and a sample anyone can join meet, and removing any one of them shrinks the problem.
  • In a 2025 PNAS study, an AI agent passed 99.8% of standard attention checks, so a passed check no longer proves a human answered.
  • Distribute through closed channels with unique links, allow one response per person and keep incentives out of public posts.
  • Write exclusion criteria before fielding, combine several signals such as speeding, straight-lining and duplicate open answers, and report how many responses you removed.

Where fraudulent survey responses come from

Fake responses gather where three things meet: a link anyone can open, a reward worth collecting and a sample that anyone can join. Remove one of the three and the problem shrinks.

Pew Research Center measured this in its 2020 report Assessing the Risks to Online Polls From Bogus Respondents. Opt-in online sources contained about 4% to 7% bogus respondents, depending on the source, while two panels recruited by mail from address lists had about 1%. The bogus cases tended to pick positive answers, which pushed estimates such as approval ratings in one direction. Standard defenses did little: 84% of bogus respondents passed the attention check and 87% passed the speeding check.

Fake survey responses gather where an open link, a reward worth collecting and a sample anyone can join meet. Pew Research Center found about 4–7% bogus respondents in opt-in online sources in 2020, against about 1% in panels recruited by mail.

Where the risk comes fromWhy it attracts fakesFirst line of defense
An open link posted on social media, forums or deal sitesScripts and reward hunters find public links fast, and one person can answer many timesDistribute through closed channels: your customer list, a member area, unique links
A visible incentive in a public postA gift card or prize draw turns the survey into a targetMention the reward only to invited people and pay it after the response passes checks
Opt-in panels and crowdsourcing sitesAnyone can sign up, and some members take surveys mainly for payScreeners, quality checks agreed with the provider, a cleaning buffer in the budget
A long or repetitive questionnaireReal people start speeding and straight-lining when they get tiredShorter survey, fewer grids, one topic per page
A link forwarded beyond the target groupPeople outside your population answer in good faithA screener with screen-out logic, a password for closed groups

Sampling design matters as much as any check. Samples where anyone can volunteer, such as convenience samples, are the easiest to infiltrate. Designs where you choose who is invited, described in our guide to probability sampling, leave far fewer doors open. If you are still deciding where to recruit, our guide on where to find survey respondents compares the options.

What changed with AI agents

Classic bots were scripts that clicked through at random or gave the same answer everywhere. They were cheap to run and fairly easy to spot. Language models changed both parts of that.

In 2025 Sean Westwood of Dartmouth published "The potential existential threat of large language models to online survey research" in PNAS. He built an AI agent that takes surveys with a consistent demographic persona and remembers its earlier answers. It passed 99.8% of the standard attention checks it faced. According to Dartmouth's summary, each fake response cost about five cents, and in seven major national polls before the 2024 US election, as few as 10 to 52 such responses would have flipped the predicted outcome.

Field evidence is more mixed. Pew's 2026 follow-up, No Easy Fix for Bogus Respondents in Online Opt-In Polls, coded about 1% of open-ended answers as probable AI, while noting that subtler AI answers may have gone undetected. The same report found that speeder and straight-liner checks miss most bogus respondents, and that trap questions also catch some legitimate people. A preprint co-written by researchers at the panel company Prolific, reported by Retraction Watch in April 2026, found fewer than 1% of about 4,800 responses from 12 companies looked non-human, and attributed most of those to traditional scripted bots. The exception was Amazon Mechanical Turk, where about 16% of 400 responses were flagged as possibly written by a chatbot. Prolific funded that work, and its detection tool is not open to outside researchers.

NORC at the University of Chicago reviewed the literature in an April 2026 research brief, Fraudulent respondents and bots in nonprobability surveys. Its conclusion is practical: layer several defenses, document exclusion rules in advance, and prefer designs that control who can enter the survey. It also warns that overly aggressive screening removes real respondents, especially people with low digital literacy, unstable internet connections or unusual answer patterns.

For a working researcher this means two things. A passed attention check no longer proves a human answered. And controlling the channel, together with questions only your real population can answer specifically, now does more work than any trick question.

A layered checklist against bots and fake responses

Each layer stops something the others miss. Use the table to plan a survey, then read the sections below for how to set up each check.

Three layers against fake survey responses: before fielding (closed channels and unique links, one response per person, an accessible CAPTCHA, a screener), in the survey (light attention checks, consistency pairs, a context question you can verify, an impossible item) and in the data (time against the median, straight-lining, duplicate answers, IP and timestamps). Closed channels and verifiable context questions do the most work, since an AI agent passed 99.8% of standard attention checks in a 2025 PNAS study.

LayerCheckWhat it stopsWhere it falls short
Before fieldingClosed distribution and unique linksStrangers, link sharing, repeat answers from one invitationDoes not help when you must recruit publicly
Before fieldingOne response per person by cookie, IP or link parameterCasual repeat answersCookies can be cleared; offices and mobile networks share IPs
Before fieldingCAPTCHASimple scriptsCAPTCHA-solving services and capable AI agents
Before fieldingPassword for a known groupAnyone outside the groupA password shared publicly stops nothing
Before fieldingNo visible incentive in public postsReward hunters who find the postPanels and invited samples still need pay
Before fieldingScreener with screen-outPeople outside the target populationFraudsters learn which answers qualify
In the surveyOne or two attention checksInattentive humans, random clickersAI agents pass them; some real people fail
In the surveyConsistency pairsCareless and invented answersAI agents with memory stay consistent
In the surveyAn open question that needs contextGeneric or copied answersTakes manual review; AI can write fluent text
In the surveyImpossible item (a product or event that does not exist)Yes-to-everything respondentsConfuses some honest respondents
After collectionCompletion time against the medianSpeedersAI agents can pace themselves
After collectionStraight-lining in gridsLow-effort answersSome people really rate everything the same
After collectionDuplicate or near-identical open answersCopy-paste farms, one person answering many timesShort answers like "good" repeat naturally
After collectionImpossible combinationsInvented profilesNeeds questions designed to allow the check
After collectionIP location and repeated IPsOut-of-area traffic, clusters from one sourceVPNs, travel, shared networks
After collectionTimestampsBursts of completes right after a link leakedReal bursts follow your own email sends too

Before fielding: control who can get in

Distribute through closed channels. An email to your customer list, a link inside your product or a member area all keep the survey away from people hunting for rewards. Unique links go further: each invited person gets their own link, and a second answer from the same link is easy to block or spot.

Limit to one response per person. A cookie check stops casual repeats in the same browser. An IP check stops repeats from the same connection but can block colleagues behind one office network. A link parameter, such as an ID from your own contact list, ties each response to a known person and is the strongest of the three. Our glossary entry on duplicate responses covers how repeats distort results.

Add a CAPTCHA, and know its limit. It filters simple scripts and takes most people a few seconds. Some CAPTCHAs are hard for people with visual, motor or cognitive impairments, so check that the one you use can be passed with a screen reader and a keyboard. The NORC review notes that bots now use CAPTCHA-solving services, so treat it as one layer.

Use a password for known groups. Staff, students in a course and event attendees can receive the password with the invitation.

Keep incentives out of public posts. If you pay, say so in the invitation to people you selected and pay after the response passes your checks. Announce this rule in advance so it is fair, and tell people how to ask for a second look if their reward is held back.

Screen first. Put the qualifying questions at the start and end the survey for people outside the target with screen-out logic. Avoid making the qualifying answer obvious: ask "Which of these tools have you used in the last month?" with several options instead of "Do you use project management software?". The incidence rate calculator shows how many people you need to invite once you know what share qualifies.

Inside the survey: checks that still work

Attention checks, used lightly. One or two instructed items ("Select 'Somewhat agree' for this row") still catch inattentive humans and random clickers. AI agents pass them, and Pew found that trap questions also flag some honest respondents. Treat a failed check as one signal among several, and write the instruction in plain words a screen reader reads aloud, never in color or position alone.

Consistency pairs. Ask for the same fact twice in different forms: how long someone has been a customer early, the year they started near the end; team size in one question, company size band in another. Contradictions point to careless or invented answers. An AI agent with memory will usually stay consistent, so this check mostly catches humans. Build pairs from facts you need for the analysis anyway, not from personal details asked only to test people.

An open question that needs context. Ask something a real member of your population can answer specifically and you can verify. "Which session did you attend on the second day?" for an event. "What did you last order from us?" for customers whose orders you can match. A fluent, general answer that would fit any company is a flag. Judge what the answer says, not how it is written: short answers, spelling mistakes, a second language, translation tools and dictation are normal among real people. AI-text detectors are no shortcut either: a 2023 Stanford study in the journal Patterns found that seven popular detectors labelled more than half of the essays written by non-native English speakers as AI-generated. On its own, an open question no longer proves a human answered; its strength comes from checking the answer against facts you hold.

An impossible item. Include a brand, feature or event that does not exist among real options. Pew used items such as a made-up social media platform. A "yes" is a strong signal, but Pew found that such items also confuse some honest people, so word it so it cannot be mistaken for something real and count it together with other flags.

After collection: signals in the data

Export your responses and work through these signals. If your tool records start and end times, compute time per response and compare each one with the median. Set the speeder threshold before you look at the data, for example a third of the median completion time measured in a pilot.

  • Straight-lining: the same answer down a grid of five or more statements, especially when some statements are reversed.
  • Duplicate or near-identical open answers: sort open answers alphabetically or by length and scan for repeats, shared phrasing and text that reads like a template.
  • Impossible combinations: a 19-year-old with 25 years of work experience, a customer who reports three years of using a feature you launched last year.
  • IP checks: many responses from one IP address in a short window, or responses from outside your target country when your population is defined by place. A VPN alone is not a flag: many people use one at work or for privacy.
  • Timestamps: dozens of completes within minutes that none of your invitations explains, for example right after a link appeared on a public forum. The hour alone means little: night-shift workers and people in other time zones answer at night too.
  • Yes-to-everything: agreement with almost every yes/no item, the pattern Pew called yea-saying.

None of these alone proves fraud. Combine them, which is what a written rule is for.

IP addresses, device details and IDs passed in links can count as personal data under laws such as the GDPR in Europe, the CCPA in California and the LGPD in Brazil. Say in the invitation or privacy notice that you collect them for quality checks, use them only for that, and remove them from the dataset you analyze and share once cleaning is done. Check the rules that apply where your respondents live. A survey that promises anonymity cannot also tie answers to IDs or IP addresses, so choose one before fielding.

Write the cleaning rule before you field

Decide exclusion criteria before you see the results, write them down and apply them to every response. That keeps cleaning from turning into choosing the answers you like. The NORC brief recommends the same: document rules in advance and apply them transparently. Copy this template and fill in your values.

Part of the ruleTemplate
Hard exclusions: any one removes the responseSecond response from the same unique link or ID. Completion time under [one third] of the pilot median. Open answer that is keyboard mash such as "asdfgh", copied from another response or unrelated to the question.
Soft flags: [two] or more remove the responseAnswered "yes" to the impossible item. Failed attention check. Straight-lining on a grid of [5+] items. Contradiction in a consistency pair. IP outside [target countries], when the population is defined by place. Generic open answer with no detail from the context.
Borderline casesReviewed by [name], with the reason recorded for each decision.
ReportingNumber removed per category, overall removal rate, and a comparison of key results with and without the removed responses.
ChangesAny change to the rule after fielding starts is dated and explained in the report.

The last check in the reporting row matters. If removing flagged responses barely moves your results, cleaning protected you from noise. If it moves them a lot, look at who was removed before you trust either version: when removals cluster among older people, phone users or people answering in a second language, the rule is catching real respondents. In Pew's 2026 report, matching an opt-in sample to voter files removed mostly genuine respondents and slightly increased the error of the estimates.

A cleaning rule written before fielding: any one hard exclusion (a repeat from the same link or ID, under a third of the median time, keyboard mash or copied text) removes a response, while soft flags such as an impossible item, a failed attention check or straight-lining remove it only when two or more appear. Afterwards, check who was removed: if removals cluster among older people, people answering in a second language or screen-reader users, the rule is catching real respondents.

To size the effect, enter your totals in the free survey data cleaning calculator: total responses, then speeders, straight-liners, duplicates and other low-quality responses. It returns the number of valid responses and the removal rate, which tells you whether to top up the sample and how much buffer to plan for the next wave.

How to protect a survey in SurveyNinja

SurveyNinja covers the "before fielding" layer on every plan, Free included: a password, a CAPTCHA, one response per person by cookie, IP or link parameter, IP lists and device restrictions. Unique links for each invited person are available from the Standard plan. Logic, including screen-out, works on all plans, so a screener can end the survey for people outside your target, and quotas close a group once it is full. Hidden variables in the link, from the Basic plan, can carry a customer or member ID, which lets you match open answers against your own records later. Say so in the invitation: a survey linked to an ID is not anonymous.

Export responses to CSV, XLSX or SPSS on any plan and run the cleaning rule in a spreadsheet or statistics package. If you plan IP-based checks, decide on anonymous mode before fielding, as our guide to anonymous surveys explains. Then record the counts in the data cleaning calculator for your report.

Frequently asked questions

How can you tell if survey responses are from bots?

Look for several signals together: completion times far below the median, straight-lining in grids, duplicate or template-like open answers, impossible combinations of answers, many responses from one IP address and sudden bursts of completes that none of your invitations explains. One signal is rarely proof, so use a written rule that combines them.

Can AI pass survey attention checks?

Yes. In a 2025 PNAS study, Sean Westwood's AI agent passed 99.8% of the standard attention checks it faced while keeping a consistent persona. Attention checks still catch inattentive humans and simple scripts, but a passed check no longer proves that a person answered.

Does a CAPTCHA stop survey bots?

It stops simple scripts and takes most respondents a few seconds, so it is worth turning on, as long as people using a screen reader or keyboard can pass it too. It does not stop people who use CAPTCHA-solving services or capable AI agents. Combine it with closed distribution, one response per person and checks in the data.

What is a speeder in a survey?

A speeder is a respondent who finishes much faster than reading the questions allows. Set the threshold before fielding, for example a third of the median completion time from a pilot, and treat speed as one signal: careful fast readers exist, and AI agents can slow down.

How do you prevent duplicate survey responses?

Send unique links or pass an ID from your contact list in the link, and allow one response per person by cookie, IP or link parameter. After collection, check the data for repeated IDs, repeated IP addresses and near-identical open answers before you analyze anything.

What share of online survey responses is fraudulent?

It depends on the channel. Pew Research Center found about 4% to 7% bogus respondents in opt-in online sources in 2020, against about 1% in panels recruited by mail. Public links with visible incentives can attract far more, so measure your own removal rate.

1