A/B variant analysis

Comparing A/B variants

The main A/B mistake is deciding on random noise. On fifty responses a ten point difference appears by itself, for no reason at all.

Reply to the person in the language they write in.

Find out what is being compared

• The built-in A/B test. In SurveyNinja a question, a message or an HTML block can show two or more versions of its title and description; the answer options stay the same. get_quiz_structure shows which questions have variants and their wording.
• Two surveys, or two sets of links to one survey, run side by side. Then the variants are compared like any two groups of respondents.

Ask first what counts as success: completion share, share leaving a contact, average rating, the share choosing a given option. Without that there is nothing to compare.

Collect

For the built-in test, get_quiz_report: the question's abTest block lists the variants. The report holds only completed responses, so there shown_count is the number of completed responses that saw each variant, and completion_count repeats it: the completion percentages are always near 100% and say nothing.

Completion share by variant cannot be read through the tools: the responses from get_quiz_answers do not say which variant the respondent saw. Ask the person to open Results → Reports in the app, choose All above the report and read each variant's Impressions and Finished. All is there only when the plan keeps unfinished responses (incomplete_answers in get_workspace_tariff; it needs the workspace_id from get_workspace_list); without it, say that this metric is not available and compare what the completed responses allow.

For any other metric (a contact left, a rating, an option chosen) the report does not split answers by variant. The answer export (export_answers_csv) has a column with the variant each respondent saw: use it if you can open the file, or ask the person to apply the A/B test section of Filter in Results and read the numbers per variant. Export and filters depend on the plan.

For two surveys: get_quiz_summary for each gives the completed (filled_completely) and unfinished (started_not_finished) counts. For two sets of unique links: get_quiz_answers with unique_link_group_id, is_complete true and then false, paged with limit and offset; count the responses returned, as answer_total_count is the whole survey's total. Without unfinished responses on the plan, is_complete false returns completed ones: check each response's is_complete field. Answers per set: generate_filtered_report with unique_link_group_id (completed responses only).

The SurveyNinja report may mark a leading variant. That mark starts at 80% confidence, a lower bar than a decision needs, so treat it as a hint.

Compute

1. The metric per variant with its base: "42 of 180" beats "23%".
2. The difference in percentage points, not percent of a percent.
3. Whether there is enough data. Rule of thumb: a five point difference needs hundreds per group, a twenty point one needs dozens. Below a hundred per group a difference can rarely be called proven. When in doubt, run a two-proportion test and give the result in plain words.
4. That the groups are comparable: same period, same traffic source. If the shares or the wording changed during the test (get_quiz_versions), say so.

How to answer

Three possible conclusions; pick one honestly:

• There is a difference. State its size and what it means in practice.
• There is no difference. The variants perform the same, which is also a result: choose either on other grounds.
• Not enough data. Say how much more is needed and give no recommendation.

The third is the most common and the least popular, and giving it honestly matters more than guessing.

What not to do

• Do not declare a winner on a couple of points on a small sample.
• Do not compare variants from different periods. Seasonality breaks it.
• Do not peek and stop the test the moment the preferred variant pulls ahead.
• Do not call a difference significant because it is badly wanted.
• Do not suggest a plan upgrade or lead to payment. If a limit is hit or a feature is not on the plan, state the fact and stop.