Response bias

Response bias is a systematic difference between the answers people give in a survey and their true opinions, behavior or characteristics. It comes from how questions are worded and ordered, from the survey setting and from respondents' own tendencies, such as wanting to look good or agreeing with whatever is asked. Because it pushes results in one direction, a bigger sample does not fix it, so it has to be prevented through survey design.

Response bias is one kind of survey bias. It concerns the answers of people who did respond. Problems with who responds belong to selection bias and non-response bias.

Response bias vs random error

Random error goes up and down and averages out as the sample grows. Response bias does not. If employees overstate their satisfaction because they fear being identified, surveying 5,000 of them instead of 500 gives you a more precise wrong answer.

Main types of response bias

The term satisficing comes from Jon Krosnick's 1991 work on survey response: when a task feels demanding, some respondents give an answer that is good enough instead of their best one.

Example: how wording creates bias

A company asks half of its customers "Do you agree that our support team resolved your issue quickly?" and the other half "How quickly or slowly did our support team resolve your issue?" with a balanced 5-point scale. The agree/disagree version invites acquiescence and states the expected answer, so it usually produces a higher positive share. The difference between the two halves shows how much of the first result was created by the wording. Running both versions on random halves of the sample is a simple way to test any question you are unsure about.

How to spot response bias in your data

• Straightlining: the same answer down a whole matrix.
• Contradictions: agreeing with a statement and with its opposite.
• Unusual pile-ups: very high shares on the midpoint, on "agree", or on the first option in a list.
• Very short completion times compared with the median.
• Gaps with known facts: self-reported behavior that does not match records, such as purchase or usage data.

How to reduce response bias

1. Write neutral questions. Describe both directions ("satisfied or dissatisfied"), avoid loaded adjectives and ask one thing per question. See how to write survey questions.
2. Use balanced scales with as many positive as negative options and labeled points.
3. Prefer item-specific questions ("How easy was…") over agree/disagree statements.
4. Randomize the order of answer options in lists and of statements in matrices, where the order has no logic of its own. See randomization.
5. Collect answers anonymously on sensitive topics, and say so in the introduction.
6. Keep the survey short and use skip logic so people see only relevant questions.
7. Pretest. A pilot study or cognitive interviews reveal questions people read differently from how you meant them.

In SurveyNinja, anonymous mode, random order of questions and answer options, and skip logic are available on all plans.

Common mistakes

• Assuming a large sample removes bias. It reduces random error only.
• Assuming anonymity solves everything. It helps with social desirability, not with acquiescence, order effects or fatigue.
• Comparing waves with different wording and reading the difference as a real change.
• Ignoring it in the report. State which steps you took and which limitations remain, for example: "Questions on sensitive topics were anonymous; answer options were randomized; self-reported frequencies may be affected by recall bias."