Statistical Inference

You measured 40 patients, but you care about the millions you will never measure. You polled 1,000 voters, but you want to know what 50 million will do. Statistical inference is the disciplined art of reasoning from the part you can see to the whole you cannot — and, crucially, of stating honestly how uncertain that reasoning is. It is what turns a pile of data into a defensible claim about the world.

This is arguably the most consequential topic in all of statistics, because it is where mathematics meets decision-making. Drug approvals, election forecasts, factory quality control, and scientific discoveries all hinge on it. It is also the most misused topic in statistics, so learning it well means learning both the machinery and its many traps.

Learning Objectives

By the end of this page, you should be able to:

• Distinguish a population from a sample, and a parameter from a statistic.
• Explain why random sampling matters and identify common sources of bias.
• Interpret a confidence interval and compute a margin of error for a mean or proportion.
• Explain, intuitively and carefully, what a p-value does and does not tell you.
• Carry out a simple hypothesis test and state its conclusion correctly.
• Explain why correlation does not imply causation, and what would.

Quick Answer

Statistical inference uses a random sample to draw conclusions about a larger population. Because different samples give different results, every estimate carries uncertainty, which we quantify. A confidence interval gives a range of plausible values for an unknown quantity (like a population mean) together with a confidence level such as 95%. A hypothesis test asks whether the data are surprising under some default assumption (the null hypothesis); the p-value measures that surprise — the probability of data at least this extreme if the null were true. A small p-value casts doubt on the null but never proves the alternative, and it is not the probability the null is true. Finally, association between two variables (correlation) does not by itself establish that one causes the other.

Where It Came From

For most of history, "data analysis" meant describing what you had: averages, totals, tables. The leap to inference — making rigorous, quantified statements about the unseen — was forced by two very practical needs: agriculture and medicine.

In the early 1900s, Ronald A. Fisher worked at Rothamsted Experimental Station in England, wrestling with decades of messy crop data. Which fertilizer actually worked, when weather, soil, and drainage varied wildly from plot to plot? Fisher's genius was to realize that randomization — deliberately assigning treatments to plots at random — was not a nuisance to be minimized but the very thing that licensed inference. Randomness introduced on purpose gave a mathematical basis for saying "this difference is bigger than chance would produce." Fisher formalized the significance test and popularized the p-value (and, somewhat casually, the 0.05 threshold) in the 1920s and 1930s.

Slightly later, Jerzy Neyman and Egon Pearson built a more decision-oriented framework: rather than just measuring evidence against a null, they framed testing as choosing between two hypotheses while controlling two kinds of error (false positives and false negatives). Neyman also developed the confidence interval in 1937, giving a rigorous meaning to "margin of error." The two schools — Fisher's evidential significance testing and Neyman-Pearson's decision-theoretic testing — were fiercely debated (Fisher and Neyman genuinely disliked each other's approaches), and the hybrid taught today is a somewhat uneasy blend of both.

The stakes rose enormously with clinical trials. The 1948 UK Medical Research Council trial of streptomycin for tuberculosis is often cited as the first modern randomized controlled trial, and it made inference literally a matter of life and death. In recent decades a replication crisis — many published findings failing to reproduce — has sharpened awareness that inference is easy to abuse, especially by fishing for small p-values. The mathematics was never the problem; the interpretation was.

Populations, Samples, and the Idea of a Sampling Distribution

The population is the entire collection you care about (all voters, all widgets from a factory, all people with a disease). A parameter is a fixed but unknown number describing it — for example the population mean $\mu$ or population proportion $p$. A sample is the subset you actually observe, and a statistic (like the sample mean $\bar{x}$ or sample proportion $\hat{p}$) is computed from it. Inference is the bridge: use the statistic to say something about the parameter.

The key insight is that a statistic is itself random — draw a different sample and you get a different value. If you imagine repeating the sampling many times, the statistic has its own distribution, called the sampling distribution. The Central Limit Theorem tells us that for a large enough sample, the sample mean is approximately normally distributed with mean $\mu$ and standard deviation $\sigma/\sqrt{n}$. That quantity $\sigma/\sqrt{n}$, the standard error, is the engine behind every margin of error: it shrinks as $\sqrt{n}$ grows, which is why bigger samples give sharper estimates — but with diminishing returns.

Sampling and Bias

Good inference requires the sample to be representative. The gold standard is a simple random sample, where every member of the population is equally likely to be chosen. Bias creeps in when the sampling method systematically favors some members:

• Selection bias: e.g. polling only landline phones misses younger voters.
• Non-response bias: those who answer differ from those who don't.
• Voluntary response bias: online polls attract people with strong opinions.

Worked example. The classic cautionary tale is the 1936 Literary Digest poll. It mailed 10 million surveys and got 2.4 million back — a huge sample — and confidently predicted Alf Landon would beat Franklin Roosevelt. Roosevelt won in a landslide. The sample was drawn from telephone and automobile registries, which in the Depression skewed wealthy and Republican. A smaller but properly random sample (George Gallup used a few thousand) got it right. Lesson: size cannot fix bias. A biased estimate stays biased no matter how much data you pile on; you just become more confident in the wrong answer.

Confidence Intervals and Margin of Error

A point estimate like $\hat{p} = 0.52$ is almost never exactly right. A confidence interval (CI) attaches an honest range. For a proportion, an approximate 95% CI is

$\hat{p} \pm z^{*}\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$

where the second term is the margin of error and $z^{*} = 1.96$ for 95% confidence (from the normal distribution).

Worked example. In a poll of $n = 1000$ voters, 520 favor candidate A, so $\hat{p} = 0.52$.

$\text{SE} = \sqrt{\frac{0.52 \times 0.48}{1000}} = \sqrt{\frac{0.2496}{1000}} = \sqrt{0.0002496} \approx 0.0158.$

Margin of error $= 1.96 \times 0.0158 \approx 0.031$, or about 3.1 percentage points. So the 95% CI is $0.52 \pm 0.031$, i.e. roughly $[0.489, 0.551]$. Because the interval dips below 0.50, the poll does not establish that A is ahead — the race is a statistical tie.

What "95% confidence" actually means. It does not mean "there is a 95% probability the true $p$ is in this specific interval" — $p$ is a fixed number, either in or out. It means the procedure works 95% of the time: if you repeated the whole sampling-and-interval process many times, about 95% of the intervals produced would contain the true value. Confidence is a property of the method, not of any single interval.

Notice the margin of error scales as $1/\sqrt{n}$. To halve it you must quadruple the sample. Quadrupling 1,000 to 4,000 only shrinks 3.1 points to about 1.55 points — expensive precision.

Hypothesis Testing and p-values

A hypothesis test starts from a null hypothesis $H_0$ — a default, "nothing interesting is happening" claim (the coin is fair, the drug has no effect) — and an alternative $H_1$. We ask: if $H_0$ were true, how surprising is our data? The p-value answers exactly this: it is the probability of getting a result at least as extreme as the observed one, assuming $H_0$ is true. Small p-value = the data would be surprising under $H_0$ = evidence against $H_0$.

Worked example. A coin is flipped 100 times and lands heads 63 times. Is it fair? Under $H_0: p = 0.5$, the number of heads has mean $np = 50$ and standard deviation $\sqrt{np(1-p)} = \sqrt{100 \times 0.5 \times 0.5} = \sqrt{25} = 5$. The observed count is

$z = \frac{63 - 50}{5} = \frac{13}{5} = 2.6.$

A $z$ of 2.6 has a two-sided p-value of about $0.0093$ (roughly 0.9%). Since this is below the conventional threshold $\alpha = 0.05$, we reject $H_0$: the data provide significant evidence the coin is biased toward heads. If instead we had seen 55 heads, $z = 1.0$, p-value $\approx 0.317$ - quite ordinary under a fair coin, so we would fail to reject $H_0$.

Crucial cautions. The p-value is slippery, so pin these down:

• It is not the probability that $H_0$ is true. It assumes $H_0$ and asks about the data, not the reverse.
• "Fail to reject" is not "prove $H_0$ true." Absence of evidence is not evidence of absence.
• Statistical significance is not practical significance. With a huge $n$, a trivially small, useless effect can be "significant."
• The 0.05 threshold is a convention, not a law of nature. It draws an arbitrary line through a continuous scale of evidence.
• p-hacking: if you test 20 useless hypotheses, on average one will hit $p < 0.05$ by chance alone. Running many tests and reporting only the winners manufactures false discoveries — a central driver of the replication crisis.

Two Kinds of Error

Because we decide under uncertainty, we can be wrong two ways. A Type I error is rejecting a true $H_0$ (a false positive); its probability is $\alpha$. A Type II error is failing to reject a false $H_0$ (a false negative); its probability is $\beta$, and $1 - \beta$ is the power of the test. Lowering $\alpha$ (demanding stronger evidence) raises $\beta$ unless you increase the sample size. Choosing $\alpha$ is really a judgment about which error is worse — convicting the innocent or freeing the guilty.

Correlation vs Causation

Two variables are correlated when they move together. But correlation alone cannot tell you why. Ice-cream sales and drowning deaths are strongly correlated — because a lurking variable, hot weather, drives both. Ice cream does not cause drowning.

Association can arise from (1) $X$ causing $Y$, (2) $Y$ causing $X$ (reverse causation), (3) a common cause / confounder $Z$, or (4) pure coincidence. To move from correlation to a credible causal claim you generally need a randomized controlled experiment: by assigning the treatment at random, you break any link between the treatment and lurking variables, so a difference in outcomes can be attributed to the treatment itself. This is precisely why Fisher's randomization was revolutionary, and why drug trials randomize patients. When experiments are impossible (you cannot randomly assign people to smoke), causation must be argued through careful design, multiple lines of evidence, and controlling for confounders — a much harder road.

Real-World Applications

• Medicine: randomized controlled trials use hypothesis tests to decide whether a drug beats placebo; CIs report the size of the benefit and its uncertainty.
• Politics and media: election polls report a candidate's support with a margin of error; understanding it prevents over-reading a within-margin "lead."
• Manufacturing: quality control samples items from a production line and tests whether the defect rate has drifted from specification.
• A/B testing in tech: websites randomly show version A or B to users and test which yields more clicks or sales — a live experiment run millions of times a day.
• Economics and public policy: estimating unemployment, inflation, or the effect of a minimum-wage change from survey samples, always with reported uncertainty.
• Science generally: from particle physics (the "5-sigma" discovery standard for the Higgs boson) to psychology, inference is the grammar of empirical claims.

Common Mistakes

1. "The p-value is the probability the null hypothesis is true." Wrong: the p-value is computed assuming the null is true; it is $P(\text{data} \mid H_0)$, not $P(H_0 \mid \text{data})$. Correction: treat a small p-value as evidence against $H_0$, nothing more precise.

2. "There's a 95% chance the true value lies in my confidence interval." Wrong: the parameter is fixed; a given interval either contains it or not. Correction: 95% describes the long-run success rate of the procedure that generated the interval, not the single interval you got.

3. "A bigger sample removes bias." Wrong: bias is systematic, not random. A larger biased sample just gives a tighter, more confidently wrong estimate (see the Literary Digest). Correction: fix bias through how you sample (randomization), not how much.

4. "Statistically significant means important." Wrong: with enough data, a negligible effect can pass $p < 0.05$. Correction: always report and judge the effect size and its practical meaning, not just significance.

5. "They're correlated, so one causes the other." Wrong: a confounder or reverse causation can produce correlation with no direct causal link. Correction: seek randomized experiments or careful control of confounders before claiming causation.

Comparison and Connections

Concept	Question it answers	Output
Confidence interval	What values of the parameter are plausible?	A range plus a confidence level
Hypothesis test	Is the data surprising under a default claim?	A decision (reject / fail to reject) and a p-value
Point estimate	What's my single best guess?	One number (no uncertainty shown)
Correlation	Do two variables move together?	A number in $[-1, 1]$ (association only)

A CI and a two-sided test at level $\alpha$ are two views of the same information: a 95% CI contains exactly those null values that a 5% test would not reject. If the null value falls outside the CI, the test rejects it. Descriptive statistics (means, spreads) summarize your sample; inference generalizes beyond it. Probability runs the machinery in reverse: probability reasons from a known population to possible samples, while inference reasons from an observed sample back to the unknown population.

Practice Questions

Recall

State the difference between a parameter and a statistic, and give one example of each.

Answer: A parameter is a fixed unknown number describing the whole population (e.g. the true proportion $p$ of voters supporting a candidate). A statistic is computed from a sample (e.g. the sample proportion $\hat{p}$ from a poll) and is used to estimate the parameter.

Understanding

A study reports a 95% confidence interval for a mean of $[12.1, 15.9]$. A student says "there's a 95% probability the true mean is between 12.1 and 15.9." What's wrong, and what's the correct interpretation?

Answer: The true mean is a fixed number, so it is either in that interval or not — no probability attaches to this specific interval. Correctly: the method that produced this interval captures the true mean 95% of the time in repeated sampling.

Application

In a sample of 400 items, 60 are defective. Give a 95% confidence interval for the defect rate.

Answer: $\hat{p} = 60/400 = 0.15$. $\text{SE} = \sqrt{0.15 \times 0.85 / 400} = \sqrt{0.1275/400} = \sqrt{0.00031875} \approx 0.01785$. Margin $= 1.96 \times 0.01785 \approx 0.035$. CI $= 0.15 \pm 0.035 = [0.115, 0.185]$, i.e. roughly 11.5% to 18.5%.

Analysis

A researcher runs 20 independent tests of useless hypotheses, each at $\alpha = 0.05$, and excitedly reports the one that gave $p = 0.04$. Why is this misleading, and what's the probability of at least one false positive?

Answer: Each test has a 5% chance of a false positive under a true null. With 20 independent tests, the chance of at least one is $1 - (0.95)^{20} \approx 1 - 0.358 = 0.642$, about 64%. So a "significant" result is expected by chance alone; reporting only the winner (p-hacking) grossly overstates the evidence. Multiple-comparison corrections or pre-registration are needed.

FAQ

Why 95% and not some other level? It's a convention balancing confidence against interval width. Higher confidence (99%) gives wider, less useful intervals; lower confidence (90%) gives narrower but riskier ones. Physics often demands far stricter standards (5-sigma). Choose the level before seeing the data.

Is a p-value of 0.049 really different from 0.051? Practically, no — the evidence is nearly identical. The sharp cliff at 0.05 is an artifact of thresholding a continuous quantity. Modern guidance is to report the actual p-value and effect size rather than a bare "significant/not significant."

What sample size do I need? It depends on the precision you want. Since margin of error scales as $1/\sqrt{n}$, to reach a target margin $m$ for a proportion you need roughly $n \approx (z^{*})^2 \hat{p}(1-\hat{p})/m^2$. For a 3% margin at 95% with $\hat{p} = 0.5$, that's about $n \approx (1.96)^2(0.25)/(0.03)^2 \approx 1068$.

If my result isn't significant, does that prove no effect? No. "Fail to reject" can also mean your sample was too small to detect a real effect (low power). A non-significant result with a wide confidence interval is simply inconclusive.

How do randomized experiments justify causal claims when observational studies can't? Randomly assigning the treatment makes the treatment group and control group similar on average in every respect — known and unknown — except the treatment. So any systematic outcome difference can be pinned on the treatment. In observational data, unmeasured confounders can always masquerade as an effect.

What is the replication crisis, and how does it relate to this page? Across several fields, many published "significant" findings failed to reproduce in fresh data. Major causes are p-hacking, running many analyses and reporting the best, small samples with low power, and treating $p < 0.05$ as truth. It's a warning that inference is a tool for weighing evidence, not a machine for manufacturing certainty.

Quick Revision

• Population/parameter = the whole and its fixed unknown number; sample/statistic = the part and your estimate.
• Standard error $= \sigma/\sqrt{n}$; precision improves only as $\sqrt{n}$ (quadruple $n$ to halve the margin).
• 95% CI for a proportion: $\hat{p} \pm 1.96\sqrt{\hat{p}(1-\hat{p})/n}$.
• Confidence is a property of the procedure, not of one interval.
• p-value $= P(\text{data at least this extreme} \mid H_0)$. Small p = evidence against $H_0$. It is not $P(H_0 \text{ true})$.
• Type I = false positive (prob $\alpha$); Type II = false negative (prob $\beta$); power $= 1 - \beta$.
• Bias is systematic and cannot be fixed by more data; randomization is the cure.
• Correlation $\ne$ causation: beware confounders and reverse causation; randomized experiments justify causal claims.

Related Topics

Prerequisites

• Descriptive Statistics — means, standard deviation, and summaries you must know first.
• Probability Basics — the language of chance underlying every p-value.
• Distributions — the normal distribution and Central Limit Theorem that power inference.

Related Topics

• Conditional Probability and Bayes — the Bayesian alternative view of $P(H_0 \mid \text{data})$.
• Statistics and Probability overview — how these pieces fit together.

Next Topics

• Economics — where inference drives empirical estimation of policy effects.