Standard error

The average NPS for a sample of 50 customers is 42. For a sample of 500 it is also 42. The same numbers, but you should trust them to different degrees.

In the first case the average could have "jumped" by 5-7 points had you surveyed a different 50 people. In the second the swing would have been three times smaller. The standard error is exactly the number that shows how unstable your estimate is because of a limited sample size.

Definition

Standard error (SE) is a measure of the precision of a sample estimate of a population parameter. It shows how much the sample mean fluctuates from sample to sample at a fixed sample size. It is calculated as the standard deviation divided by the square root of the sample size: SE = SD / √n. The larger the sample and the smaller the spread of the data, the smaller the standard error and the more precise the estimate.

Standard error vs standard deviation

These are two different things that are often confused - even in academic papers.

Standard deviation (SD) describes the spread of the data within a sample: how much individual responses differ from the mean. It is a characteristic of the data, not of the precision of the estimate. As the sample grows, SD does not decrease - it describes the real variability in the population.

Standard error (SE) describes the precision of the estimate of the mean: how close the sample mean is to the true mean of the population. As the sample grows, SE decreases - proportionally to the square root of n. Doubling the precision = increasing the sample fourfold.

An analogy: SD is the spread of people's height in a city. SE is the precision with which a sample of 100 people estimates the average height of the whole city. The spread of height will not change if we survey more people. But our estimate of the mean will become more precise.

Formula and calculation

For the mean:

SE = SD / √n

Where SD is the sample standard deviation and n is the sample size.

Example: a satisfaction survey, 100 respondents, mean 7.2, SD = 2.0.

SE = 2.0 / √100 = 2.0 / 10 = 0.2

This means: if we repeated the survey on another random sample of 100 people, the mean would most likely fall in the range 7.2 ± 0.2 (in roughly two thirds of cases). Increase the sample to 400:

SE = 2.0 / √400 = 2.0 / 20 = 0.1

Twice as precise. At the cost of a fourfold increase in the sample.

Standard error and the confidence interval

SE is the building block for the confidence interval. The 95% confidence interval for the mean is built as:

CI = M ± 1.96 × SE

Where 1.96 is the z-value for the 95% confidence level. For small samples (n < 30) a t-value from the t-distribution table is used instead of 1.96.

For the example above (M = 7.2, SE = 0.2):

95% CI = 7.2 ± 1.96 × 0.2 = [6.81; 7.59]

Interpretation: with 95% probability the true mean in the population lies between 6.81 and 7.59. This interval width - 0.78 points - shows how imprecise our estimate is. With a sample of 400 the interval narrows to 0.39 points.

Standard error for proportions

In surveys you often need the SE not for a mean but for a proportion: "32% of customers rated the service 9-10" - how precise is that figure? For a proportion the formula is different:

SE_p = √(p × (1-p) / n)

Where p is the observed proportion and n is the sample size.

Example: p = 0.32, n = 200:

SE_p = √(0.32 × 0.68 / 200) = √(0.001088) ≈ 0.033

95% CI for the proportion: 0.32 ± 1.96 × 0.033 = [0.255; 0.385], that is 25.5% - 38.5%.

This is important information: "32%" sounds precise, but the real range is from 26% to 38%. When making decisions you need to keep this interval in mind, not the point estimate.

Example: comparing the precision of estimates across different surveys

A company collects satisfaction data through three channels:

Channel	n	Mean	SD	SE	95% CI
Email campaign	400	7.4	2.1	0.105	[7.19; 7.61]
Website widget	80	7.1	2.3	0.257	[6.60; 7.60]
QR offline	25	6.8	1.9	0.380	[6.05; 7.55]

The means differ: 7.4, 7.1, 6.8. But look at the confidence intervals: all three overlap. The confidence intervals of email and QR overlap almost completely. The conclusion "customers are less satisfied offline" is not statistically justified - the difference falls within the measurement error. To draw a confident conclusion you need more data for the offline channel.

How sample size affects SE

The dependence of SE on n is nonlinear - this is important to understand when planning a study:

• n = 25 → SE = SD / 5
• n = 100 → SE = SD / 10 (twice as precise as at n=25)
• n = 400 → SE = SD / 20 (four times as precise as at n=25)
• n = 1600 → SE = SD / 40 (eight times as precise)

Every doubling of precision requires a fourfold increase in the sample. This is the law of diminishing returns in data collection: going from n=25 to n=100 gives a big gain in precision, while going from n=900 to n=1600 gives only a small one. This is one of the arguments for calculating the minimum required sample size in advance - so as not to waste resources on excessive data and not to get imprecise estimates from insufficient data.

Common mistakes

Confusing SE and SD in reports. "Mean 7.2 ± 0.2" - what is this about? The spread of the data (SD) or the precision of the estimate (SE)? In academic publications this is always specified. In business reports it often is not. Always state explicitly what the ± stands for.

Not calculating SE for small subgroups. A total sample of 500 people is good. But if you analyze a subgroup of 18 people (for example, customers from a specific region), the SE for that group will be large and the conclusions unreliable. Small subgroups require special caution in interpretation.

Ignoring SE when comparing two estimates. "This quarter NPS rose from 38 to 42." It looks like progress. But if the SE of both estimates is ~4 points, then the intervals overlap and the increase is not statistically significant. Without SE you cannot claim the change is real and not measurement noise.

SE in survey data analysis

The standard error is a fundamental metric when interpreting any sample estimates from surveys. Means, proportions, indices (NPS, CSI, eNPS) - they all have an SE, which determines the width of the confidence interval and the minimum detectable change.

When you export data from SurveyNinja via data export you get raw responses, on the basis of which you can calculate SE in Excel, R or Python. For a quick estimate of the precision of a proportion - the SE_p formula in any spreadsheet. For an estimate of the precision of a mean - SE = SD / √n, where SD and n are in the basic statistics of any report.

The standard error is the margin of your measurement caused by a finite sample size. A mean without an SE is incomplete information. Two estimates cannot be compared without taking their SE into account: the difference may be real, or it may just be noise. SE = SD / √n: more data means less error, but not linearly.

Frequently asked questions

How does standard error differ from margin of error?

The margin of error is half the width of the confidence interval, usually for the 95% confidence level: MOE = 1.96 × SE. The standard error is the base quantity; the margin of error is its practical expression in the form of "±X%" used in public reports and the media.

Why does SE decrease as the sample grows, but SD does not?

SD describes the real variability in the data - it does not depend on the sample size because it reflects a property of the population. As the sample grows, SD does not decrease. SE describes the precision of the estimate of the mean - the larger the sample, the more precisely we estimate the true mean. This is a mathematical consequence of the law of large numbers.

What size of SE is considered acceptable?

It depends on the task. A universal guideline: the SE should be substantially smaller than the differences you want to detect. If it is important for you to capture a 5-point change in NPS, the SE of the estimate should be no more than 2-3 points. If SE = 6, then a 5-point change is statistically indistinguishable from noise.

How is SE related to the power of a statistical test?

The power of a test (the probability of detecting a real effect) depends directly on SE: the smaller the SE, the higher the power. This is precisely why increasing the sample raises the power - it lowers SE and makes the test more sensitive to real differences. Calculating the required sample size through MDE effectively sets a target level of SE.

Can SE from different studies be compared?

Directly - only if the scales are the same. SE = 0.3 on a 1-10 scale and SE = 0.3 on a 1-100 scale mean different things. For comparison, the relative error is used: SE is divided by the mean or by the range of the scale. This gives a dimensionless measure that is comparable across studies.