Statistical significance

Statistical significance means that an observed difference or relationship in survey or experiment data would be unlikely if there were no real effect. It is used to tell real changes in metrics from random fluctuations caused by sampling.

For example, customer satisfaction was 78% last month and 81% this month. Before reporting "we grew by 3 percentage points", you check whether such a gap could easily appear by chance with your sample size. Without an understanding of significance it is easy to mistake statistical noise for success.

What is statistical significance in plain language

Statistical significance is an indication that an observed difference or relationship in the data is unlikely to have arisen by chance under the given conditions of an experiment or survey. Formally it is assessed through the p-value: the smaller it is, the less compatible the data are with the assumption that there is no effect.

Intuitively you can think of it this way: if there were really no effect, a result as extreme as yours would rarely appear from sampling chance alone. Significance does not tell you that you are "definitely right", but it shows how convincing your hypothesis is from a mathematical standpoint.

What p-value is and how to read it

In reports on A/B tests, marketing research and analytical articles you often see notation like "p < 0.05". Behind it lies the idea: if there really were no difference between the variants, we would obtain such a (or a more extreme) result no more than 5% of the time.

Low p-value (for example, 0.01). The observed effect is poorly compatible with the hypothesis that "there is no difference at all". If there were no difference, a result like this would appear rarely, in about 1% of samples or less.

High p-value (for example, 0.3). The data fit the hypothesis that "nothing has changed" quite well, and there are no statistical grounds to claim a significant difference. This does not mean the effect is definitely absent — only that it did not show up convincingly in the current sample.

Significance only addresses random error. Systematic distortions in how the sample was drawn or how people answered are a separate problem, covered in the article on bias.

Significance and effect size are not the same thing

It is important to distinguish "is it significant" from "how large is the difference". With a very large sample size, even a tiny difference (for example, 0.5 of a percentage point) can turn out to be statistically significant — simply because you have a lot of data. But from a practical standpoint such a shift may be inconsequential.

Conversely, in small samples you can observe noticeable differences (for example, a 10-point gap in satisfaction) that do not pass the significance test — there is not enough data to confidently distinguish the effect from random fluctuation.

That is why, when working with survey results, it is useful to look at both the p-value and the effect size: how large the difference is in percentage terms, how important it is for the business, and how it relates to the margin of error and the goals of the study. How to measure the scale of a difference is covered in the article "Effect size".

How to test significance in surveys

Comparing proportions. For questions with answer options ("yes/no", "satisfied/not satisfied") you usually apply tests for comparing proportions — the z-test or chi-square. The SurveyNinja glossary has the term Z-Test, which describes one of these approaches in detail.

Comparing means. For scale questions (a rating from 1 to 10, a satisfaction index) you use tests for means. In practice this means you do not simply compare an average score of 7.2 against 7.8, but check how compatible such a difference is with the idea that the metrics are "equal".

Multifactor comparisons. When you account for several factors at once (region, customer segment, product type), more sophisticated methods come to the rescue: regression analysis, factor analysis, multivariate models. An introduction to such approaches can be found in the articles "Quantitative research" and "Cluster analysis", which show how to work with multivariate data and group respondents.

Example: significance of the difference between two pricing plans

Suppose you are comparing customer satisfaction across two pricing plans: basic and premium. The survey was taken by 400 customers in each group. In the basic plan 76% of respondents are satisfied, in the premium plan — 82%. On a chart this looks like a tangible difference, and you are tempted to declare a winner.

To understand how reliable this difference is, you compute confidence intervals for each proportion and run a test for comparing proportions (for example, the Z-Test). If the calculations show that the p-value is below 0.05, you can say that the probability of seeing such a difference "purely by chance" is low and the effect is statistically significant. If, however, the p-value is high, it is wiser to treat the result with caution and view it as a hypothesis for further testing rather than as a final conclusion.

Where significance matters most

Comparing periods in tracking studies. In regular research (for example, a quarterly NPS or eNPS) it is easy to see small swings in the metrics and mistake them for a trend. Testing significance helps you separate real changes from random ones, especially when you are looking at many segments and metrics at once.

A/B testing of surveys and communications. When comparing different question wordings, invitations to participate or survey scenarios, it is important to distinguish "this variant looks slightly better on the chart" from "this variant statistically and consistently wins". Without a significance test you may choose a strategy that is not actually better than the alternatives.

Customer and HR decisions with a high cost of error. When survey results influence changes to a product, a service or a company policy, testing significance adds confidence: you are relying not only on individual cases and intuition, but also on a formal assessment of how reliable the effect is.

How it looks in reports and in SurveyNinja

Even if you do not run full statistical tests, it is useful to build into your reports elements that help readers avoid overrating small differences, such as sample sizes next to each percentage and a note on the margin of error.

In SurveyNinja, summary reports and filters let you quickly see the differences between groups, and, if necessary, export the data for subsequent testing in statistical packages. The help section on working with reports ("Viewing the survey report") describes how to read the report and compare answers.

Common mistakes when working with significance

Hunting for a "pretty" p-value. If you repeatedly cycle through segments, wordings and slices until you find a "significant" difference, you run a high risk of stumbling onto a false positive. The more tests you run, the higher the probability of accidentally getting p < 0.05 simply because of the number of comparisons.

Substituting correlation for causation. A statistically significant relationship between two metrics does not yet mean that one causes the other. For example, a rise in satisfaction may coincide with the launch of a new feature and with a change in external conditions — and the test by itself will not tell you which of the causes matters more.

Ignoring the sizes of subsamples. In small segments even a significant difference can be unstable. If there are only a few dozen respondents inside a group, one or two atypical answers can shift the result substantially — and a formal significance test does not always reflect the real reliability of the conclusion here.

Practical recommendations

Do not declare every shift an "improvement". Before celebrating a rise in a metric, check how large it is relative to the margin of error and how stable it is over time. Sometimes it is better to wait for another wave of data than to make decisions based on a random spike.

Look at the whole picture. When assessing significance, take into account not only the p-value but also the effect size, the context and the practical importance. A small but stable shift in a business-critical metric can matter more than a large but one-off spike in a secondary indicator.

Be honest in your wording. Instead of categorical "this variant is better", use more careful phrasing: "we observe a statistically significant improvement" or "the differences are within the statistical margin of error". This helps the team develop a more mature attitude to data.

Statistical significance is a way to assess whether an observed effect could easily be explained by random fluctuation. The better you understand its limitations and combine it with practical common sense, the more reliable the decisions you make based on surveys and analytics.