Chi-square test (χ²)

The chi-square test (χ²) checks whether the counts you observed in categories differ from the counts you would expect by chance. In surveys it answers two questions: are two categorical answers related (the test of independence on a cross-tabulation), and does one answer follow an expected distribution (the goodness-of-fit test).

Karl Pearson introduced the statistic in 1900, which is why you will also see it called Pearson's chi-square test. It works with counts of people, so it fits single-choice questions such as plan type, region, age group or "yes / no / not sure".

Two versions of the test

How to calculate chi-square

Both versions use the same formula, where O is the observed count in a cell and E is the expected count:

χ² = Σ (O − E)² / E

For a test of independence, the expected count of each cell is E = row total × column total / grand total, and the degrees of freedom are df = (rows − 1) × (columns − 1). For goodness of fit, E is the sample size times the expected share, and df = number of categories − 1.

Worked example: test of independence

200 customers answered "Would you recommend us?" You split them by plan:

Expected counts are 78 and 42 for the free plan, 52 and 28 for the paid plan (for example, 120 × 130 / 200 = 78). The four terms are 64/78 + 64/42 + 64/52 + 64/28 = 0.82 + 1.52 + 1.23 + 2.29, so χ² ≈ 5.86 with df = 1. The critical value at α = 0.05 is 3.84, and the p-value is about 0.015. Recommendation and plan are related: paid users say "yes" more often (75% against 58%).

Worked example: goodness of fit

You expect your 300 respondents to split 50/30/20 across three regions, so E = 150, 90 and 60. You got 165, 80 and 55. χ² = 225/150 + 100/90 + 25/60 ≈ 3.03 with df = 2. The critical value is 5.99 and p ≈ 0.22, so the sample does not differ significantly from the expected split.

You can check your own table with the chi-square calculator: paste counts from a 2×2 up to a 5×5 table and get χ², the degrees of freedom, the p-value and Cramér's V.

How to read the result

• p < 0.05 (or your chosen α): the variables are associated, or the distribution differs from the expected one. See statistical significance for what this threshold does and does not mean.
• The p-value says nothing about strength. For that, use Cramér's V = √(χ² / (n × (k − 1))), where k is the smaller of the number of rows and columns. In the example, V = √(5.86 / 200) ≈ 0.17, a weak association.
• Chi-square does not say where the difference is. Compare observed and expected counts cell by cell, or look at the standardized residuals, to see which cells drive the result.
• Association is not causation. Paid users may recommend more because they got more value, or because happier users chose to pay.

Assumptions and common mistakes

• Use counts, not percentages or averages. Running the test on percentages makes χ² depend on what base you picked.
• Each respondent belongs to one cell. Multiple-choice questions where people tick several options break this rule; test each option as its own yes/no variable instead.
• Expected counts must not be too small. Cochran's rule (1954): no expected count below 1, and no more than 20% of cells below 5. If your table fails, merge categories or use Fisher's exact test for a 2×2 table.
• Yates' correction. Some statistics packages apply a continuity correction to 2×2 tables by default, which gives a slightly smaller χ². Say which version you report.
• Many tables, many false alarms. If you test 20 crosstabs at α = 0.05, about one can come out significant by chance. Decide in advance which comparisons matter.
• Large samples flag tiny differences. With thousands of respondents a 2-point gap can be significant, so always read the effect size next to the p-value.
• Ordered categories lose their order. Chi-square treats "rarely, sometimes, often" as unordered labels. For ordinal answers a rank test or a test for trend can be more sensitive. See hypothesis testing for choosing a test.

Running a chi-square test on survey data

1. Write the hypothesis before fieldwork, for example "the share of promoters differs by plan".
2. Ask both variables as closed-ended questions with single choice, and keep the categories mutually exclusive.
3. Plan the sample size so that each cell gets enough people.
4. Count responses per combination of answers in a spreadsheet or statistics package. SurveyNinja exports responses to CSV, XLSX and SPSS on every plan, including Free.
5. Paste the table into the calculator and report χ², df, p and Cramér's V together.

For a longer walkthrough of building crosstabs from survey answers, see the guide to cross-tabulation in surveys.