Bots in Surveys: How to Spot and Prevent Fraudulent and AI-Generated Responses
Survey design
Mike Taylor Nov 16, 2026 Reading time ≈ 19 min
Contents
- Where fraudulent survey responses come from
- What changed with AI agents
- A layered checklist against bots and fake responses
- Before fielding: control who can get in
- Inside the survey: checks that still work
- After collection: signals in the data
- Write the cleaning rule before you field
- How to protect a survey in SurveyNinja
- Frequently asked questions
Bots in surveys are scripts and AI agents that fill in online questionnaires, usually to collect an incentive. Together with people who rush through or answer several times, they produce fraudulent survey responses that look like real data. No single check catches them all, so this guide gives a layered checklist: where the risk comes from, what to set up before fielding, which checks to put inside the survey and how to clean the data with rules you write in advance.
The approach is methodological and works with any survey tool. The term itself, and the main types of fraud, are covered in our glossary entry on fraud detection.
Key points
- Fake responses gather where an open link, a reward worth collecting and a sample anyone can join meet, and removing any one of them shrinks the problem.
- In a 2025 PNAS study, an AI agent passed 99.8% of standard attention checks, so a passed check no longer proves a human answered.
- Distribute through closed channels with unique links, allow one response per person and keep incentives out of public posts.
- Write exclusion criteria before fielding, combine several signals such as speeding, straight-lining and duplicate open answers, and report how many responses you removed.
Where fraudulent survey responses come from
Fake responses gather where three things meet: a link anyone can open, a reward worth collecting and a sample that anyone can join. Remove one of the three and the problem shrinks.
Pew Research Center measured this in its 2020 report Assessing the Risks to Online Polls From Bogus Respondents. Opt-in online sources contained about 4% to 7% bogus respondents, depending on the source, while two panels recruited by mail from address lists had about 1%. The bogus cases tended to pick positive answers, which pushed estimates such as approval ratings in one direction. Standard defenses did little: 84% of bogus respondents passed the attention check and 87% passed the speeding check.
| Where the risk comes from | Why it attracts fakes | First line of defense |
|---|---|---|
| An open link posted on social media, forums or deal sites | Scripts and reward hunters find public links fast, and one person can answer many times | Distribute through closed channels: your customer list, a member area, unique links |
| A visible incentive in a public post | A gift card or prize draw turns the survey into a target | Mention the reward only to invited people and pay it after the response passes checks |
| Opt-in panels and crowdsourcing sites | Anyone can sign up, and some members take surveys mainly for pay | Screeners, quality checks agreed with the provider, a cleaning buffer in the budget |
| A long or repetitive questionnaire | Real people start speeding and straight-lining when they get tired | Shorter survey, fewer grids, one topic per page |
| A link forwarded beyond the target group | People outside your population answer in good faith | A screener with screen-out logic, a password for closed groups |
Sampling design matters as much as any check. Samples where anyone can volunteer, such as convenience samples, are the easiest to infiltrate. Designs where you choose who is invited, described in our guide to probability sampling, leave far fewer doors open. If you are still deciding where to recruit, our guide on where to find survey respondents compares the options.
What changed with AI agents
Classic bots were scripts that clicked through at random or gave the same answer everywhere. They were cheap to run and fairly easy to spot. Language models changed both parts of that.
In 2025 Sean Westwood of Dartmouth published "The potential existential threat of large language models to online survey research" in PNAS. He built an AI agent that takes surveys with a consistent demographic persona and remembers its earlier answers. It passed 99.8% of the standard attention checks it faced. According to Dartmouth's summary, each fake response cost about five cents, and in seven major national polls before the 2024 US election, as few as 10 to 52 such responses would have flipped the predicted outcome.
Field evidence is more mixed. Pew's 2026 follow-up, No Easy Fix for Bogus Respondents in Online Opt-In Polls, coded about 1% of open-ended answers as probable AI, while noting that subtler AI answers may have gone undetected. The same report found that speeder and straight-liner checks miss most bogus respondents, and that trap questions also catch some legitimate people. A preprint co-written by researchers at the panel company Prolific, reported by Retraction Watch in April 2026, found fewer than 1% of about 4,800 responses from 12 companies looked non-human, and attributed most of those to traditional scripted bots. The exception was Amazon Mechanical Turk, where about 16% of 400 responses were flagged as possibly written by a chatbot. Prolific funded that work, and its detection tool is not open to outside researchers.
NORC at the University of Chicago reviewed the literature in an April 2026 research brief, Fraudulent respondents and bots in nonprobability surveys. Its conclusion is practical: layer several defenses, document exclusion rules in advance, and prefer designs that control who can enter the survey. It also warns that overly aggressive screening removes real respondents, especially people with low digital literacy, unstable internet connections or unusual answer patterns.
For a working researcher this means two things. A passed attention check no longer proves a human answered. And controlling the channel, together with questions only your real population can answer specifically, now does more work than any trick question.
A layered checklist against bots and fake responses
Each layer stops something the others miss. Use the table to plan a survey, then read the sections below for how to set up each check.
| Layer | Check | What it stops | Where it falls short |
|---|---|---|---|
| Before fielding | Closed distribution and unique links | Strangers, link sharing, repeat answers from one invitation | Does not help when you must recruit publicly |
| Before fielding | One response per person by cookie, IP or link parameter | Casual repeat answers | Cookies can be cleared; offices and mobile networks share IPs |
| Before fielding | CAPTCHA | Simple scripts | CAPTCHA-solving services and capable AI agents |
| Before fielding | Password for a known group | Anyone outside the group | A password shared publicly stops nothing |
| Before fielding | No visible incentive in public posts | Reward hunters who find the post | Panels and invited samples still need pay |
| Before fielding | Screener with screen-out | People outside the target population | Fraudsters learn which answers qualify |
| In the survey | One or two attention checks | Inattentive humans, random clickers | AI agents pass them; some real people fail |
| In the survey | Consistency pairs | Careless and invented answers | AI agents with memory stay consistent |
| In the survey | An open question that needs context | Generic or copied answers | Takes manual review; AI can write fluent text |
| In the survey | Impossible item (a product or event that does not exist) | Yes-to-everything respondents | Confuses some honest respondents |
| After collection | Completion time against the median | Speeders | AI agents can pace themselves |
| After collection | Straight-lining in grids | Low-effort answers | Some people really rate everything the same |
| After collection | Duplicate or near-identical open answers | Copy-paste farms, one person answering many times | Short answers like "good" repeat naturally |
| After collection | Impossible combinations | Invented profiles | Needs questions designed to allow the check |
| After collection | IP location and repeated IPs | Out-of-area traffic, clusters from one source | VPNs, travel, shared networks |
| After collection | Timestamps | Bursts of completes right after a link leaked | Real bursts follow your own email sends too |
Before fielding: control who can get in
Distribute through closed channels. An email to your customer list, a link inside your product or a member area all keep the survey away from people hunting for rewards. Unique links go further: each invited person gets their own link, and a second answer from the same link is easy to block or spot.
Limit to one response per person. A cookie check stops casual repeats in the same browser. An IP check stops repeats from the same connection but can block colleagues behind one office network. A link parameter, such as an ID from your own contact list, ties each response to a known person and is the strongest of the three. Our glossary entry on duplicate responses covers how repeats distort results.
Add a CAPTCHA, and know its limit. It filters simple scripts and takes most people a few seconds. Some CAPTCHAs are hard for people with visual, motor or cognitive impairments, so check that the one you use can be passed with a screen reader and a keyboard. The NORC review notes that bots now use CAPTCHA-solving services, so treat it as one layer.
Use a password for known groups. Staff, students in a course and event attendees can receive the password with the invitation.
Keep incentives out of public posts. If you pay, say so in the invitation to people you selected and pay after the response passes your checks. Announce this rule in advance so it is fair, and tell people how to ask for a second look if their reward is held back.
Screen first. Put the qualifying questions at the start and end the survey for people outside the target with screen-out logic. Avoid making the qualifying answer obvious: ask "Which of these tools have you used in the last month?" with several options instead of "Do you use project management software?". The incidence rate calculator shows how many people you need to invite once you know what share qualifies.
Inside the survey: checks that still work
Attention checks, used lightly. One or two instructed items ("Select 'Somewhat agree' for this row") still catch inattentive humans and random clickers. AI agents pass them, and Pew found that trap questions also flag some honest respondents. Treat a failed check as one signal among several, and write the instruction in plain words a screen reader reads aloud, never in color or position alone.
Consistency pairs. Ask for the same fact twice in different forms: how long someone has been a customer early, the year they started near the end; team size in one question, company size band in another. Contradictions point to careless or invented answers. An AI agent with memory will usually stay consistent, so this check mostly catches humans. Build pairs from facts you need for the analysis anyway, not from personal details asked only to test people.
An open question that needs context. Ask something a real member of your population can answer specifically and you can verify. "Which session did you attend on the second day?" for an event. "What did you last order from us?" for customers whose orders you can match. A fluent, general answer that would fit any company is a flag. Judge what the answer says, not how it is written: short answers, spelling mistakes, a second language, translation tools and dictation are normal among real people. AI-text detectors are no shortcut either: a 2023 Stanford study in the journal Patterns found that seven popular detectors labelled more than half of the essays written by non-native English speakers as AI-generated. On its own, an open question no longer proves a human answered; its strength comes from checking the answer against facts you hold.
An impossible item. Include a brand, feature or event that does not exist among real options. Pew used items such as a made-up social media platform. A "yes" is a strong signal, but Pew found that such items also confuse some honest people, so word it so it cannot be mistaken for something real and count it together with other flags.
After collection: signals in the data
Export your responses and work through these signals. If your tool records start and end times, compute time per response and compare each one with the median. Set the speeder threshold before you look at the data, for example a third of the median completion time measured in a pilot.
- Straight-lining: the same answer down a grid of five or more statements, especially when some statements are reversed.
- Duplicate or near-identical open answers: sort open answers alphabetically or by length and scan for repeats, shared phrasing and text that reads like a template.
- Impossible combinations: a 19-year-old with 25 years of work experience, a customer who reports three years of using a feature you launched last year.
- IP checks: many responses from one IP address in a short window, or responses from outside your target country when your population is defined by place. A VPN alone is not a flag: many people use one at work or for privacy.
- Timestamps: dozens of completes within minutes that none of your invitations explains, for example right after a link appeared on a public forum. The hour alone means little: night-shift workers and people in other time zones answer at night too.
- Yes-to-everything: agreement with almost every yes/no item, the pattern Pew called yea-saying.
None of these alone proves fraud. Combine them, which is what a written rule is for.
IP addresses, device details and IDs passed in links can count as personal data under laws such as the GDPR in Europe, the CCPA in California and the LGPD in Brazil. Say in the invitation or privacy notice that you collect them for quality checks, use them only for that, and remove them from the dataset you analyze and share once cleaning is done. Check the rules that apply where your respondents live. A survey that promises anonymity cannot also tie answers to IDs or IP addresses, so choose one before fielding.
Write the cleaning rule before you field
Decide exclusion criteria before you see the results, write them down and apply them to every response. That keeps cleaning from turning into choosing the answers you like. The NORC brief recommends the same: document rules in advance and apply them transparently. Copy this template and fill in your values.
| Part of the rule | Template |
|---|---|
| Hard exclusions: any one removes the response | Second response from the same unique link or ID. Completion time under [one third] of the pilot median. Open answer that is keyboard mash such as "asdfgh", copied from another response or unrelated to the question. |
| Soft flags: [two] or more remove the response | Answered "yes" to the impossible item. Failed attention check. Straight-lining on a grid of [5+] items. Contradiction in a consistency pair. IP outside [target countries], when the population is defined by place. Generic open answer with no detail from the context. |
| Borderline cases | Reviewed by [name], with the reason recorded for each decision. |
| Reporting | Number removed per category, overall removal rate, and a comparison of key results with and without the removed responses. |
| Changes | Any change to the rule after fielding starts is dated and explained in the report. |
The last check in the reporting row matters. If removing flagged responses barely moves your results, cleaning protected you from noise. If it moves them a lot, look at who was removed before you trust either version: when removals cluster among older people, phone users or people answering in a second language, the rule is catching real respondents. In Pew's 2026 report, matching an opt-in sample to voter files removed mostly genuine respondents and slightly increased the error of the estimates.
To size the effect, enter your totals in the free survey data cleaning calculator: total responses, then speeders, straight-liners, duplicates and other low-quality responses. It returns the number of valid responses and the removal rate, which tells you whether to top up the sample and how much buffer to plan for the next wave.
How to protect a survey in SurveyNinja
SurveyNinja covers the "before fielding" layer on every plan, Free included: a password, a CAPTCHA, one response per person by cookie, IP or link parameter, IP lists and device restrictions. Unique links for each invited person are available from the Standard plan. Logic, including screen-out, works on all plans, so a screener can end the survey for people outside your target, and quotas close a group once it is full. Hidden variables in the link, from the Basic plan, can carry a customer or member ID, which lets you match open answers against your own records later. Say so in the invitation: a survey linked to an ID is not anonymous.
Export responses to CSV, XLSX or SPSS on any plan and run the cleaning rule in a spreadsheet or statistics package. If you plan IP-based checks, decide on anonymous mode before fielding, as our guide to anonymous surveys explains. Then record the counts in the data cleaning calculator for your report.
Frequently asked questions
How can you tell if survey responses are from bots?
Look for several signals together: completion times far below the median, straight-lining in grids, duplicate or template-like open answers, impossible combinations of answers, many responses from one IP address and sudden bursts of completes that none of your invitations explains. One signal is rarely proof, so use a written rule that combines them.
Can AI pass survey attention checks?
Yes. In a 2025 PNAS study, Sean Westwood's AI agent passed 99.8% of the standard attention checks it faced while keeping a consistent persona. Attention checks still catch inattentive humans and simple scripts, but a passed check no longer proves that a person answered.
Does a CAPTCHA stop survey bots?
It stops simple scripts and takes most respondents a few seconds, so it is worth turning on, as long as people using a screen reader or keyboard can pass it too. It does not stop people who use CAPTCHA-solving services or capable AI agents. Combine it with closed distribution, one response per person and checks in the data.
What is a speeder in a survey?
A speeder is a respondent who finishes much faster than reading the questions allows. Set the threshold before fielding, for example a third of the median completion time from a pilot, and treat speed as one signal: careful fast readers exist, and AI agents can slow down.
How do you prevent duplicate survey responses?
Send unique links or pass an ID from your contact list in the link, and allow one response per person by cookie, IP or link parameter. After collection, check the data for repeated IDs, repeated IP addresses and near-identical open answers before you analyze anything.
What share of online survey responses is fraudulent?
It depends on the channel. Pew Research Center found about 4% to 7% bogus respondents in opt-in online sources in 2020, against about 1% in panels recruited by mail. Public links with visible incentives can attract far more, so measure your own removal rate.
Published: Nov 16, 2026
Google Forms Limits: Responses, File Uploads, Timers and Logic, With Workarounds 

