Meta-analysis

You're looking for the answer to "do short surveys work better than long ones." You google it — and find 20 studies. Some say "yes, short ones get a higher response rate," others say "no, the difference is negligible," and still others say "long ones produce higher-quality data."

Each has its own sample, its own methodology, its own context. Which one do you trust? Meta-analysis is a method that combines all these studies into a single statistical result with a reliability estimate, turning a contradictory mosaic into a coherent picture.

Definition

Meta-analysis is a statistical method for combining the results of several independent studies on the same topic to obtain a more precise and generalized estimate of the effect under study. It combines quantitative results (effect sizes) rather than mere descriptions, which is what distinguishes a meta-analysis from an ordinary literature review. It is widely used in medicine, psychology, marketing, and social research to synthesize knowledge and make decisions based on the totality of the data.

Why you need a meta-analysis

Increased statistical power. A single study of 100 people may fail to detect an effect because the sample is too small. Twenty such studies together make up 2,000 participants, and the probability of detecting a real effect is much higher.

Reduced influence of chance. An individual study may produce an atypical result because of the peculiarities of its sample or sheer chance. Aggregating across several studies smooths out these fluctuations and reveals a stable effect.

Resolving contradictions. When different studies reach different conclusions, a meta-analysis systematically analyzes how they differ: different samples, methodologies, contexts. This helps you understand under which conditions the effect is present and under which it is not.

Assessing generalizability. A single result may be specific to a particular audience. A meta-analysis that combines studies on different populations lets you assess how universal the effect is.

The meta-analysis procedure

1. Formulating the research question. A clear, specific question: "How does the length of an online survey affect completion rate among a B2C audience?" Vague wording produces vague results.

2. Systematic search for studies. Defining databases, keywords, and inclusion and exclusion criteria. The goal is to find all relevant studies, not just those that confirm the expected conclusion (publication bias).

3. Selecting studies. Applying predefined criteria: methodology (quantitative), population (target audience), metrics (comparable indicators), quality (minimum standards). The criteria are fixed before the results are reviewed.

4. Extracting the data. From each study: sample size, effect size (Cohen's d, correlation r, odds ratio), confidence intervals, sample and methodology characteristics.

5. Calculating the combined effect. A weighted average of the effect sizes, where each study's weight is proportional to its precision (inversely proportional to its variance). Large, precise studies receive greater weight.

6. Assessing heterogeneity. How much do the results of different studies diverge from one another? If the spread is large, you need to look for its source: moderators (conditions under which the effect differs), methodological errors, differences in samples.

7. Testing for publication bias. Formal methods (funnel plot, Egger's test) check whether the results are biased because "unsuccessful" studies are published less often.

Fixed-effect vs random-effects models

There are two main approaches to calculating the combined effect:

Fixed-effect model. It assumes that all studies measure the same "true" effect and that the differences between them are due only to random sampling error. It is appropriate when the studies are very similar in methodology and population.

Random-effects model. It assumes that the true effect may differ between studies (for example, different populations yield different effects), and the meta-analysis estimates the average of these true effects. This is a more realistic model for most cases — studies are rarely identical.

In practice, the random-effects model is used more often, especially when heterogeneity is present. It produces more conservative (wider) confidence intervals and better reflects the real uncertainty.

Example: a meta-analysis of the effect of a thank-you screen on response rate

Question: does the presence of a personalized thank-you screen affect the response rate in repeat surveys of the same respondent?

We found 8 studies with measurable data (the effect size is the difference in response rate between the group with personalization and the group without):

• Study 1 (n=400): +3.2%
• Study 2 (n=1,200): +4.5%
• Study 3 (n=250): +7.1%
• Study 4 (n=800): +2.8%
• Study 5 (n=150): +1.2%
• Study 6 (n=600): +4.9%
• Study 7 (n=350): +3.6%
• Study 8 (n=950): +2.1%

Weighted combined value (accounting for sample sizes): +3.4%, 95% CI [2.1%; 4.7%]. Conclusion: a personalized thank-you screen yields an increase in response rate of roughly 2-5 percentage points in subsequent surveys. This is a small but stable and statistically significant effect. A single isolated study could have shown anything from +1% to +7% — the meta-analysis provides a reliable estimate.

Common problems with meta-analysis

Publication bias. "Successful" studies are published more readily than unsuccessful ones. A meta-analysis based only on published work may systematically overstate the effect. Countermeasure: search the grey literature (dissertations, reports, preprints) and run tests for bias.

Heterogeneity of studies. When studies differ greatly in methodology, population, and context, combining them is dangerous. It mixes apples with oranges. You need to test for heterogeneity statistically (the Q-test, the I² index) and, when values are high, either narrow the inclusion criteria or run a subgroup analysis.

Garbage in, garbage out. If the included studies are of low quality (small samples, weak methodology), combining them will not improve the picture. Worse, it will lend false confidence to unreliable conclusions. A rigorous quality assessment of each study before inclusion is essential.

Double counting. The same data may appear in several publications. Including it twice artificially inflates the sample and distorts the result. You need to check for overlaps in authors, collection periods, and methodologies.

Meta-analysis vs systematic review

They are often used together, but they are different things:

• A systematic review is a rigorous, methodologically documented collection and qualitative analysis of all studies on a topic. It may not include a quantitative combination
• A meta-analysis is a statistical combination of the quantitative results of studies. It is usually conducted as part of a systematic review

A good meta-analysis is always based on a systematic review. The reverse is not true: a systematic review can be useful even without a meta-analysis, if the studies are too heterogeneous to combine.

Meta-analysis and triangulation

Meta-analysis is one way of triangulating knowledge: combining different sources for a more reliable picture. Its difference from classic triangulation is that meta-analysis works with quantitative results, whereas triangulation can include qualitative methods (qualitative analysis, interviews, observations). In complex research programs, both are often used: meta-analysis for quantitative data, triangulation for combining it with qualitative data.

Application in applied work

For most teams, a full-fledged meta-analysis is an overly complex tool. But its principles are always useful:

• Don't rely on a single study — look for confirmation across several sources
• Take into account the effect size and the sample size, not just significance
• Critically assess the methodology of the studies you rely on
• Understand that contradictory results are often explained by different contexts

For product and marketing teams, the "internal meta-analysis" format means gathering all your previous A/B tests on one topic and analyzing the aggregate effects. This gives more reliable insights than relying on a single most recent test.

A meta-analysis is not just a "literature review with numbers." It is a formal method of synthesizing knowledge that turns contradictory individual results into a well-grounded combined conclusion with a reliability estimate. The key elements are: systematic search, selection criteria, heterogeneity assessment, weighted combination of effects. The principles of meta-analysis are also useful in everyday work with data: look for confirmation, take effect size into account, and critically assess your sources.

Frequently asked questions

What is the minimum number of studies needed for a meta-analysis?

Technically, two. In practice, for reliable conclusions, a minimum of 5-10 studies is desirable. A smaller number does not allow you to properly assess heterogeneity and run tests for publication bias. With 2-3 studies, a systematic review without a formal meta-analysis is more appropriate.

How does a meta-analysis differ from simply averaging the results?

Fundamentally, in two ways: weighting (large, precise studies receive greater weight than small ones) and heterogeneity assessment (checking how appropriate it is to combine the studies). Simple averaging gives equal weight to all and ignores the spread — this can lead to incorrect conclusions.

Can a meta-analysis be conducted based on survey research?

Yes, and it is common practice in the marketing and social sciences. It is important that the studies measure comparable variables (for example, the effect of variable X on NPS) using comparable methodologies. Different scales require standardization before combining.

What is I² and how do you interpret it?

I² is a measure of heterogeneity in a meta-analysis; it shows the percentage of variability between studies explained by real differences (rather than chance). I² < 25% — low heterogeneity, 25-50% — moderate, 50-75% — substantial, > 75% — high. With high heterogeneity, you need to look for moderators or doubt whether the combination is appropriate.

Can a meta-analysis prove the absence of an effect?

Yes, but with a caveat. If a meta-analysis on a large aggregate sample shows an effect close to zero with a narrow confidence interval, this is strong evidence of the absence of a practically significant effect. But this is not the same as "the effect does not exist at all" — it may be very small or appear only under specific conditions.