Sample size

Picture this: a marketer is planning a customer survey. There are 50,000 people in the database. Their manager asks, "How many responses do we need?", the marketer says "the more, the better" and budgets for 5,000 respondents. The survey runs, the money is spent — and suddenly it turns out that 400-500 responses would have been plenty for the same conclusions.

The opposite extreme: a team launches a quick-and-dirty study, collects 40 responses and draws far-reaching conclusions about the entire audience. A couple of months later it becomes clear that decisions were made on noise rather than data. In both cases the problem is the same — no one thought ahead about the sample size and how it relates to the accuracy and reliability of the survey.

What sample size means in plain language

Sample size is the number of respondents whose answers you use to analyze and draw conclusions about an entire target group. More simply: how many people you need to survey so that the result reflects the opinion not only of those people, but of the whole population with an acceptable margin of error.

It's important to distinguish three figures: how many invitations you sent, how many people started the survey, and how many completed it. Statistically, the sample size is precisely the number of completed, valid responses. Metrics such as Response Rate, Abandonment Rate and Yield Rate help you work out how many invitations you need to send to obtain the required number of such responses.

What determines how many people you need to survey

You sometimes hear rules of thumb like "100 responses are enough for any study." They're convenient, but wrong. The correct sample size depends on several parameters.

1. The size of the population. How many people in total are in the group you want to draw conclusions about: 2,000 customers, 50,000 subscribers, a million city residents? Intuitively it seems that a million would require dozens of times more responses than two thousand. In practice, starting at around 100,000, the influence of population size on the calculation weakens: for similar accuracy settings you'll still need on the order of a few hundred respondents.

2. The acceptable margin of error. How much imprecision are you willing to tolerate? In consumer research a confidence interval of ±5% is often used. This means that if 60% answer "yes," the true value in the population very likely lies between 55% and 65%. The narrower the interval (±3%, ±2%), the larger the sample you need.

3. The confidence level in your results. The standard in applied research is a 95% confidence level. For critical management decisions 99% is sometimes used; for quick screening surveys, 90%. Intuitively: the more demanding you are about reliability, the more people you need to survey.

4. The expected distribution of answers. If you expect opinions to split roughly in half (50/50), you need the maximum sample size. But if almost everyone will surely choose one option (for example, 90% of customers use only one plan), the required sample is smaller — because variability is lower. When nothing is known in advance, calculations usually assume the "worst" scenario — 50%.

The rigorous math behind these calculations is described in articles on the confidence interval and statistical margins of error. For practice it's more important to remember the logic: you adjust the "sliders" of accuracy and confidence — and the required sample size is selected to match them.

Intuitive benchmarks without formulas

A full calculation is usually done with a statistical calculator, but it's useful to keep a few orders of magnitude in mind.

Around 100 responses. Suitable for exploratory surveys, hypothesis checks, and wording tests. It lets you see rough trends, but doesn't provide reliable quantitative estimates.

Around 300-400 respondents. The classic benchmark for marketing research with a margin of error of about ±5% and a 95% confidence level. This is the order of magnitude most often used when people talk about a "representative survey of city residents" or customers.

800-1,200 and more. Needed when it's important to analyze subgroups: men and women, regions, age categories. If you want to reliably compare segments within a sample, each segment must also contain enough observations.

The key takeaway: beyond a certain threshold, adding another few hundred respondents yields less and less gain in accuracy. Doubling the sample doesn't double reliability. So there's no point in endlessly "padding" a survey with responses — it's more important to understand in advance what volume is justified for your task.

How sample size and budget are linked

In real projects the sample size is rarely limited by statistics alone. Most often there's a financial and time ceiling: the study has a deadline and a budget for collecting responses.

The cost of a single response. If you use a respondent panel or paid acquisition channels, every completed survey costs money. Then increasing the sample by 200 people is not only a gain in accuracy, but also an extra line in the budget.

The timeline of the field stage. Even with a free acquisition channel (your own customer base, social media), a large number of responses takes time. The faster you want to finish collecting, the more you'll depend on the response rate and the size of the invited audience.

In SurveyNinja you can control the volume of data technically: using settings for response collection limits (a cap on the number of respondents) and your choice of distribution channel. If you work with a respondent panel through a paid panel provider, you specify the required number of completed questionnaires and the audience parameters in advance — and the provider collects the necessary volume for you.

Example: how to calculate the sample size for an online store

Imagine a company that wants to measure satisfaction with a purchase at an online store. Around 20,000 customers place orders per month. Management wants to understand how accurate the figures in the report will be, while not overpaying for unnecessary responses.

The team formulates its requirements: "We're comfortable with a margin of error of about ±5% and a standard confidence level of 95%. We're interested in the share of satisfied customers, and we don't know in advance how it will be distributed — let's take the conservative assumption of 50%."

Plugging these parameters into any statistical calculator, they get a benchmark on the order of 380-400 valid responses. Next, the marketer looks at the history of past surveys and sees that usually 10% of those invited respond. That means they need to send the invitation to about 4,000 customers, and in the survey settings cap collection at 450-500 responses to leave a small reserve for data cleaning.

But if the task changes — for example, the company wants to separately compare buyers from the capital and the regions — they'll have to calculate not only the total volume, but also the minimum number of questionnaires within each segment. Then the final plan may grow to 700-800 responses, but this is now a deliberate decision rather than a "just in case."

Quantitative and qualitative research: when to count people, and when depth

It's important to remember that the discussion of sample size applies primarily to quantitative research, where you work with percentages and confidence intervals. Read more about such approaches in the article "Everything about quantitative research".

In qualitative methods (in-depth interviews, focus groups, ethnographic research) the math is different. There it's not the number of participants per se that matters, but saturation: the moment when new respondents stop bringing fundamentally new information. The term Qualitative Research in the glossary covers qualitative approaches in detail.

A typical mistake is to "drag" quantitative thinking into the qualitative field: to assume that 20 interviews are "few" and 100 are "reliable." In reality, for many tasks studying motivation and behavior scenarios, 10-20 well-conducted interviews are enough, whereas estimating shares and the percentage of satisfied customers already requires a full numerical sample.

Common mistakes related to sample size

Too small a sample. With 20-30 responses it's easy to see a "nice-looking chart," but hard to tell a pattern from random noise. One or two active respondents can sharply shift the average, and the conclusions will turn out to be exaggerated.

Ignoring the structure of the audience. You can survey 500 people but draw almost all of them from a single subgroup, and get a distorted picture. Sample size and its structure must be considered together: sometimes it's better to survey 300 people but ensure a balanced representation of segments.

Overspending the budget for a "nice number." The desire to see a "round" figure in the report (1,000, 2,000 respondents) sometimes makes the study unjustifiably expensive, while the quality of the conclusions barely changes compared with a carefully calculated smaller sample.

No calculation at the planning stage. People often start thinking about sample size only after launching the survey: "Whatever we collect is what we'll have." This approach makes results harder to interpret and doesn't let you assess in advance how reliable the conclusions will be.

Practical recommendations

Start with the goal and the acceptable margin of error. Spell out which decisions will be made based on the survey results, and determine what level of imprecision is acceptable. For strategic decisions it makes sense to set stricter parameters and, consequently, a larger sample size.

Use online calculators or built-in tools. You don't have to derive the formula on paper. It's enough to enter the audience size, the desired margin of error and the confidence level — and get a benchmark for the number of responses. The main thing is to set the input parameters deliberately.

Plan not only the number of responses, but the path to them. Estimate what percentage of those invited usually answer your surveys, and calculate how many people you need to invite to reach the target volume. This helps avoid disappointment when, a week later, you see only dozens of responses instead of hundreds.

Build in a reserve for data cleaning. There are always responses you'll have to delete: duplicates, "click-throughs," and obviously frivolous questionnaires. So it's better to set the planned sample size slightly higher than to end up with fewer valid data points than you need.

Sample size isn't a magic number, but a deliberate compromise among accuracy, budget and timeline. The sooner you ask yourself "what margin of error are we prepared to accept?" and "how many decisions depend on this survey?", the easier it will be to choose a reasonable sample size and avoid overpaying for unnecessary responses.