Statistical power and power analysis

Statistical power is the probability that a test finds an effect of a given size when that effect really exists. Power analysis is the calculation, done before a study, of how many respondents you need to reach that probability for your chosen significance level and the smallest effect that matters to you.

Without it, a study may be too small to detect a real difference, and a "not significant" result then tells you almost nothing. With it, you know in advance what the study can and cannot show.

The four quantities

Power analysis links four numbers. Fix any three and the fourth follows:

• Significance level (α): the risk of a false positive, finding a difference that is not there. Usually 0.05.
• Power (1 − β): the chance of detecting a real effect. β is the risk of a false negative.
• Effect size: how large a difference you want to be able to detect, in raw units or standardized (Cohen's d for means).
• Sample size (n): respondents per group.

Where 0.8 and 0.05 come from

Jacob Cohen proposed the conventions in Statistical Power Analysis for the Behavioral Sciences (2nd edition, 1988): when there is no reason to choose otherwise, aim for power of 0.80 with α = 0.05. That pairs a 20% risk of missing a real effect with a 5% risk of a false alarm, treating the false alarm as about four times more serious. Cohen also offered rough benchmarks for d: 0.2 small, 0.5 medium, 0.8 large. He meant them as a fallback when you have no better idea of what difference matters in your field.

These are conventions, not laws. If missing an effect is costly, for example a safety issue, plan for 0.90 or higher.

How to calculate sample size for two groups

For comparing two means with a two-sided test, the normal approximation is:

n per group = 2 × (z(1−α/2) + z(1−β))² × σ² / Δ²

where σ is the standard deviation and Δ the difference you want to detect. For α = 0.05, z(1−α/2) = 1.96; for power 0.80, z(1−β) = 0.84; for power 0.90, it is 1.28.

Worked example: means

You want to compare satisfaction on a 0–20 index between two customer segments. A difference of 2 points matters, and past data shows a standard deviation of about 5, so d = 2 / 5 = 0.4.

• Power 0.80: n = 2 × (1.96 + 0.84)² × 25 / 4 ≈ 98.1, so about 99 per group.
• Power 0.90: about 132 per group.
• For a medium effect, d = 0.5, the same formula gives about 63 per group; Cohen's tables, based on the t-test, give 64.

Worked example: proportions

You expect 50% of one group and 60% of another to choose an option. At α = 0.05 and power 0.80, a two-proportion test needs about 388 per group. Halving the difference to 5 points roughly quadruples the sample, because n grows with 1/Δ².

The two-group sample size calculator runs both versions: enter two expected proportions or means, α and power. If you have pilot or past data, the effect size calculator turns means and standard deviations into Cohen's d and Hedges' g.

How to read the result

The sample size from a power analysis is the minimum number of valid responses per group. To get the number of invitations, divide by the expected response rate, the incidence of qualified people, the completion rate and the share you keep after cleaning.

The reverse calculation is just as useful. With a fixed sample, power analysis gives the smallest effect you can reliably detect, the minimum detectable effect. If that effect is larger than anything realistic, the study is not worth running in that form.

Common mistakes

• Calculating "observed power" after the test. Power computed from the effect you just observed adds nothing to the p-value. Hoenig and Heisey explain why in "The abuse of power" (The American Statistician, 2001). Report a confidence interval instead.
• Using an optimistic effect size to keep the sample small. Base it on the smallest difference that would change a decision, or on prior data.
• Planning for the total sample but analyzing subgroups. Each comparison you plan to report needs its own power.
• Ignoring several comparisons. Testing many outcomes at α = 0.05 raises the false-positive risk; a stricter α needs a larger sample.
• Reading a non-significant result from an underpowered study as "no difference". It means the study could not tell. See statistical significance.

Planning a powered survey

Write the hypothesis and the minimum effect first, then run the calculator and set group targets. In SurveyNinja, quotas on every plan, including Free, stop each group at its target. If you compare two wordings of a question with SurveyNinja's A/B test, which is also available on every plan, plan the views per variant with the same calculator. For the test itself, see t-test and hypothesis testing.