A/B testing

Say you've launched a customer satisfaction survey. The response rate is 8%. Of those who start, half drop off on the third question. You change the email subject line from "Take our survey" to "Three minutes and we'll get better for you" — and the response rate jumps to 14%.

Coincidence? Or a pattern? To answer that question for sure rather than guess, there's A/B testing — a method that turns intuition into verifiable facts.

What A/B testing is

A/B testing (A/B Testing, Split Testing) is a method of comparing two (or more) variants of a single element, in which the audience is split into groups at random, each of which sees its own version. The difference in results between the groups lets you determine which variant works better — based on data, not opinions.

In marketing, A/B tests are associated first and foremost with landing pages and "Buy" buttons. But in the world of surveys this method is no less valuable — and yet it's used far less often. And that's a shame: the wording of a question, the length of a questionnaire, the tone of the invitation, the distribution channel — all of this affects the quality and quantity of the data you collect. And all of it can be tested.

Why test surveys

It might seem that a survey is text that either works or it doesn't. In practice, between "works" and "doesn't work" lies a spectrum of options, and the difference between a good and a great survey can be huge — several times over on response rate, by tens of percentage points on the share of completed questionnaires.

Wording determines the answer. A classic example from sociology: the question "Should public speeches against democracy be forbidden?" and the question "Should public speeches against democracy be allowed?" yield statistically different results — even though logically it's the same question, just inverted. The word "forbid" and the word "allow" activate different mental frames. Without an A/B test, you'll never know how much your particular wording shifts the answers.

Small details have a disproportionate effect. The subject line of the invitation email, the presence or absence of a progress bar, the order of answer options, the number of questions per page — each of these elements seems insignificant, but together they determine whether the respondent reaches the end. A single test can yield a 5–10% gain in completed questionnaires — and that's hundreds of additional full responses with large samples.

Subjectivity is removed. On any team there'll be a marketer who "feels" that a long headline is better, and an analyst who's "sure" about a short one. An A/B test replaces the argument with a number: variant A gave a 12% conversion, variant B — 17%. Discussion over.

What you can test in surveys

The list of elements that lend themselves to A/B testing is far broader than most questionnaire authors think. Here are the main ones — from the most obvious to the least.

Question wording

The main territory for tests. The same idea can be expressed in a dozen ways, and each gives a slightly different distribution of answers.

Example. Version A: "How likely are you to recommend us to friends or colleagues?" (the classic NPS). Version B: "Would you recommend us to someone you know?" (less formal). The test will show whether the tone of the wording affects the average score and the distribution between promoters, passives, and detractors.

Questionnaire length

How many questions are optimal? 8? 12? 20? The only way to find out for your particular audience is a test. Group A receives the full questionnaire of 15 questions, group B — a shortened one of 8. You compare response rate, abandonment rate, and data completeness. Often it turns out that the short questionnaire collects more useful information — simply because people finish it.

Question order

Start with an easy general question or get straight to the point? Put CSAT at the beginning or the end? Order affects both the answers (Context Effect) and the completion share. An A/B test of two sequences will show which structure works better for your task.

Visual design

Button color, the presence of the company logo, the progress bar style, the number of questions on a single page — all of these are factors that influence respondent behavior. One of the less obvious but effective tests: a page with one question vs. a page with three. The first option looks simpler but lengthens the path; the second is more compact but may be perceived as a "wall of text."

Invitation subject and copy

If the survey is distributed via email, the subject line is the first (and often only) element a potential respondent sees. Testing email subject lines is one of the fastest-paying A/B experiments: the difference between a good and a poor subject line can be a factor of 2–3 in open rate.

Example. Subject A: "Customer satisfaction survey — 2026." Subject B: "We have one question for you (takes 2 minutes)." Subject C: "Dmitry, tell us — what should we improve?" You test all three on equal shares of the base, and measure opens and link clicks.

Incentive to participate

Offer a discount for completing it? A gift? Entry into a prize draw? Or nothing at all — just a polite request? An A/B test lets you understand how much an incentive increases response and — no less important — whether it lowers the quality of answers. Respondents who came for a prize sometimes click through the questionnaire without looking, just to get the reward.

How to run an A/B test of a survey: step by step

Step 1. Choose one variable

The main rule of A/B testing: one change per experiment. If you simultaneously change the wording of a question, the order of blocks, and the button design, it's impossible to understand what exactly affected the result. The "one variable at a time" discipline requires patience, but it's also what guarantees the cleanliness of your conclusions.

The exception is multivariate testing (MVT), when several variables are checked simultaneously in different combinations. But this requires a significantly larger sample, and for most applied tasks the classic A/B is sufficient.

Step 2. Define the success metric

Before launching the test, formulate what exactly you consider the "better" result. Typical metrics:

• Response Rate — the share of people who started the survey, out of the total invited
• Completion Rate — the share of those who reached the end, out of those who started
• Average completion time — an indirect indicator of engagement
• Share of "junk" answers — straight-lining, identical ratings on all questions, meaningless text in open fields
• Answer distribution — if one of the wording variants shifts the scale, this may indicate a leading nature of the question

Step 3. Split the audience at random

Randomness is the foundation of the experiment. If group A is the morning respondents and group B the evening ones, the difference in answers may be caused by the time of day, not by the change being tested. Random Assignment guarantees that the groups are roughly identical across all parameters — age, loyalty, mood — and the only difference between them is the element being tested.

Step 4. Wait for statistical significance

This is the most common mistake: the test author sees that version B is leading after 50 answers and declares it the winner. But with a small sample the difference may be random. For the result to be reliable, you need a sufficient volume of data.

A guideline: to detect a difference of 5 percentage points (for example, Completion Rate 60% vs. 65%) at a confidence interval of 95%, you need about 1,500 answers in each group. For a difference of 10 p.p. — about 400. The smaller the expected difference, the larger the sample needed.

If your audience is small, focus on testing elements with a potentially large effect (email subject line, questionnaire length) rather than on the nuances of wording — there the difference will be noticeable even with modest volumes.

Step 5. Interpret the result and roll it out

If the difference is statistically significant — roll out the winning version. If not, that's a result too: it means the element being tested doesn't affect the metric, and you can choose by other criteria (convenience, style, brand guidelines). A negative A/B test result is time saved on future debates.

Common mistakes

Testing everything at once. You changed the wording, rearranged the questions, and changed the background color — and then you try to understand what worked. That's not an A/B test, it's roulette. One experiment — one variable.

Stopping the test too early. "Oh, version B is leading after 30 answers — let's launch it!" No. With small numbers, random fluctuations are enormous. Determine the required sample size before launching and stick to that figure.

Ignoring answer quality. The version with the short survey showed a response rate 20% higher? Great. But if at the same time 40% of the answers are click-through without reading, the final amount of useful data may end up lower than for the long version with a smaller response. Always look at the totality of metrics, not at one.

Testing on an irrelevant audience. If you're optimizing a survey for B2B clients but run the test on random site visitors — the results won't carry over. The test audience must match the target audience as closely as possible.

Not documenting tests. Six months later you'll forget what you tested, what the hypotheses were, and why you chose variant B. Keep a table: date, hypothesis, element tested, metric, sample size, result. This is the basis for accumulating knowledge about your audience.

A/B testing in SurveyNinja

In the SurveyNinja builder, A/B testing of surveys is implemented through a combination of several built-in tools.

Hidden variables for tagging groups. Through hidden variables in the URL you can pass a variant identifier (for example, ?variant=A or ?variant=B). The respondent doesn't see the parameter, but it's saved in the results — and you can filter answers by group, comparing metrics.

Logic jumps for different versions. Logic jumps let you route respondents to different versions of a question or block within a single questionnaire. For example, based on the value of a hidden variable, group A sees the wording "How likely is it that you'd recommend us?", while group B sees "Would you recommend us to people you know?"

Built-in analytics with filtering. In the analytics section you can filter answers by any parameter, including hidden variables. This lets you compare the results of group A and group B right in the interface — without exporting to Excel.

Duplicating the survey. For tests where it's not a single question that differs but the entire questionnaire (for example, a short vs. a long version), you can create a copy of the survey and distribute the two links in parallel. The results are compared in the analytics of each survey.

A/B testing of surveys isn't a luxury for large research departments but a working tool available to anyone willing to spend a little time checking hypotheses. A single test of an email subject line can double the response rate — and pay back all the effort of setting it up.