Fraud detection

Say you launched a survey through a panel and collected 600 responses in three days. You look at the data - NPS 72, everything looks great. But if you check the completion time, it turns out that 90 people filled in a 20-question survey in 40 seconds. Another 60 picked the same option in every question

These are not respondents - they are noise that skews your results. Fraud detection is a set of methods that lets you find such responses before they reach your analysis.

Definition

Fraud detection (identifying low-quality responses) - a set of methods and procedures aimed at detecting and excluding poor-quality data in surveys. Such data includes responses from bots, inattentive or dishonest respondents, duplicate participations, and attempts to manipulate the results. Fraud detection includes technical protection measures, behavioral analysis, and statistical validation of the data.

Types of problematic responses

Low-quality data in surveys falls into several categories, each with its own signs.

Bots and automated responses. Programs that fill in surveys without human involvement. Signs: completion time of a few seconds, atypical patterns in open-ended fields (meaningless text, random characters), identical answers from different IPs. This is especially relevant for surveys with rewards through online panels.

Speeders. Real people who fill in the survey too fast - without reading the questions. They are identified through completion time: if a person spends less than a third of the median time across the whole sample, their answers are most likely random. With 20 questions, the median time is 8-10 minutes. Completing it in 2 minutes is a warning sign.

Straight-liners. Respondents who pick the same option across all questions in a matrix or a series of scale questions. "Strongly agree" to all 12 statements in a row is a pattern typical of inattention or minimal effort. It partially overlaps with acquiescence bias, but in fraud detection the focus is precisely on mechanical uniformity.

Duplicate participations. One person takes the survey several times from different browsers or after clearing cookies. This is especially common in prize draws and surveys with rewards. It is detected through matching IPs, browser fingerprints, or device identifiers.

Random clickers. Answers are chosen at random, without logic. They are hard to tell from honest answers by a single metric - you need a combination of signals: anomalous time + inconsistent answers to logically related questions.

Attention checks and trap questions

The most direct method of spotting inattentive respondents is trap questions built into the survey. They work in several formats:

Direct instruction. The question reads like a regular one ("Rate the following statements"), but one of them says: "Choose the 'Strongly disagree' option - this is a verification question". Whoever read it will pick the right one. Whoever filled in mechanically will pick something else.

Impossible items. Statements that cannot be true at the same time: "I have never used the internet" in an online survey. Or: "I use our product daily" and 5 questions later "I have never tried our product". A contradiction in the answers is a sign of inattention or random selection.

Red herring questions. Questions about non-existent products, brands, or features. "How satisfied are you with our MultiSync Pro feature?" - if there is no such feature, but a person rates it 4/5, then they did not read carefully or are giving socially desirable answers. The method overlaps with the validity scale.

The recommended number is 1-2 attention check questions per survey. More than that irritates conscientious respondents and lowers the completion rate.

Behavioral analysis: time and patterns

Completion time is one of the most informative quality indicators. The standard procedure:

• Calculate the median time across all responses
• Flag as suspicious those who finished in less than 1/3 of the median (speeders)
• Flag those who took too long - more than 3-4 standard deviations above the median (possible distractions, a tab left open)

Time on its own is not a criterion for exclusion - it works together with other signs. A person may have gone through quickly because they know the topic well, not because they clicked at random.

Answer patterns in matrix questions are analyzed through a dispersion index: if a person gave the same rating across all 10 rows, that is suspicious. A normal respondent varies the answers. Straight-lining in a matrix of 8+ rows is almost always a signal to check.

Technical protection methods

CAPTCHA. Basic protection against bots at the survey entrance. It reduces automated submissions but does not protect against inattentive people. More complex CAPTCHAs lower the response rate - it is worth using only when there are clear risks of automated attacks.

IP deduplication. A limit on taking the survey from a single IP address. Effective against simple duplicates, but it also blocks families and corporate networks where several people may share an IP. It is supplemented with browser fingerprinting - unique device characteristics.

Cookies and identifiers. On the first pass, a marker is written to the device. On a repeat attempt the system recognizes it and blocks repeat participation. It is bypassed by clearing cookies, but it covers most accidental repeats.

Limits based on screening questions. A mismatch in demographic data between different parts of the survey - age, region, job title - may indicate random answers or deliberate falsification.

Example: spotting problematic responses in an NPS survey

A company collected 800 responses to an NPS survey through an online panel. After a basic check it found the following:

• Speeders (time < 90 sec with a median of 7 min): 54 surveys - 6.8%
• Straight-liners in a matrix block of 8 questions: 38 surveys - 4.8%
• Failed the attention check: 47 surveys - 5.9%
• Duplicate IPs: 22 surveys - 2.8%

Overlaps between the groups yielded 120 unique surveys to exclude - 15% of the sample. After exclusion, NPS changed from 72 to 61. The 11-point difference is the result of systematic inflation of scores by dishonest respondents. Without fraud detection, the company would have made a decision based on inflated data.

What to do with detected responses

Suspicious responses are not deleted automatically - first an assessment is carried out. Three approaches:

Hard exclusion. All surveys that failed a threshold number of checks (for example, 2 out of 3) are excluded from the analysis. Suitable for quantitative research where data cleanliness matters.

Weighting. Doubtful responses are assigned a lower weight during aggregation. A more cautious approach - no data is lost, but the influence of unreliable answers is reduced.

Manual review. Borderline cases (only one sign) are reviewed manually, especially if the sample is small and every response matters. Automatic rejection at n < 100 can significantly distort the results in the other direction.

After exclusion, it is important to recalculate the baseline metrics and make sure the representativeness of the sample has not been broken. If many responses from a single demographic segment have been removed - that is already a selection bias problem.

Common mistakes in fraud detection

Using only one criterion. Time alone or an attention check alone is not enough. Every individual sign produces false positives. A reliable system is built on 2-3 independent indicators.

Excluding too aggressively. A hard time threshold (< 3 minutes) will weed out real experts who know the topic well. An inflated straight-lining threshold will weed out people with genuinely uniform opinions.

Not checking the data before the final analysis. Fraud detection is not a one-off measure. A pilot launch helps test how the traps work and calibrate the thresholds before full-scale collection.

Ignoring the problem in short surveys. Microsurveys of 3-5 questions seem safe - but even in them straight-lining and speeders occur, especially when working with panels. A minimal set of checks (time + IP) is always needed.

Fraud detection in SurveyNinja

SurveyNinja provides built-in data protection tools. Limiting one response per device or IP is configured in the limit settings. To build attention check questions, standard question types are used with logic jumps - on a wrong answer the survey can be ended early or flagged.

Completion time is recorded for every response and available when viewing and downloading data - this lets you spot speeders right in the results table. For panel studies with a high risk of dishonest responses, it is recommended to build in a 10-15% buffer when calculating the needed panel size.

Fraud detection is not paranoia, it is data hygiene. Bots, speeders, and straight-liners make up, on average, 5-20% of responses in online panels. Multi-layered validation - time, attention checks, IP deduplication, answer patterns - lets you clean the sample before analysis rather than discover the problem after decisions have been made.

Frequently asked questions

How many attention check questions should I add to a survey?

One to two per survey is optimal. One trap question in the middle and one closer to the end is standard practice. More than two start to annoy attentive respondents and lower the completion rate. For short surveys of up to 10 questions, one is enough.

What is a speeder and how do you detect one?

A speeder is a respondent who completed the survey significantly faster than normal. The threshold: less than 1/3 of the median time across the whole sample. If the median is 8 minutes, a speeder is someone who finished in 2-3 minutes. Completion time is recorded in most survey systems and available in the data export.

Is fraud detection needed for internal corporate surveys?

For HR surveys and employee surveys the relevance is lower - there is no financial incentive to falsify. But straight-lining occurs here too: employees who do not trust the anonymity may give mechanically neutral answers. A basic check of patterns and timing is useful for samples of 100+ people.

How do you tell a speeder-expert from a dishonest responder?

By time alone, you cannot. You need a combination of signals: fast completion + a failed attention check, or fast completion + straight-lining in a matrix. An expert who knows the topic well will fill out the survey quickly, but will vary the answers and will not fail the trap. This is exactly why fraud detection is built on a combination of indicators, not a single one.

What should you do if there is not enough sample left after removing bad responses?

Collect more data, building in a buffer for a repeat collection. Standard practice is to build in 15-20% over the target volume when working with panels. If it turns out after collection - you can launch an additional wave with the same screening criteria. Reducing the sample below the calculated minimum is not acceptable - it lowers the statistical significance of the results.