Sampling and Experimental Design

Every headline that begins "Studies show..." or "A poll finds..." rests on a hidden foundation: how the data was collected. You can run the most sophisticated statistical test in the world, but if the numbers came from the wrong people, or from a study that quietly confused a cause with a coincidence, the analysis is worthless. Sampling and experimental design is the branch of statistics that decides whether the data deserves to be trusted at all — and it does its work before a single average is computed.

This page teaches you to think like a designer of studies, not just a consumer of them. You will learn how to pull a trustworthy sample out of a huge population, how bias sneaks in even when everyone is trying to be honest, and why a randomized controlled experiment is the single most powerful tool humanity has for separating real causes from convincing illusions.

Learning Objectives

• Distinguish a population from a sample, and a parameter from a statistic.
• Describe and compare simple random, stratified, and cluster sampling, and choose the right one for a situation.
• Identify major sources of bias — selection, nonresponse, and response bias — and explain how each distorts conclusions.
• Explain the difference between an observational study and a designed experiment, and why only the latter can establish causation.
• Apply the three pillars of good experiments: control, randomization, and replication, plus blinding.
• Interpret famous failures (the 1936 and 1948 polls) and the origins of experimental design.

Quick Answer

A population is the entire group you want to understand; a sample is the subset you actually measure. Because measuring everyone is usually impossible, we sample — but the sample must be chosen so it fairly represents the population. Random sampling (every member has a known, nonzero chance of selection) is what makes a sample trustworthy and lets us quantify uncertainty. Bias is a systematic error that pushes estimates consistently away from the truth; it comes from how you sample, not from bad luck. To learn about causes rather than mere associations, you need a designed experiment with a control group, random assignment of treatments, and ideally blinding. Observational studies can reveal associations but cannot rule out confounding variables.

Where It Came From

The need was ancient but the mathematics was late. Governments have counted people for taxation and armies for millennia — the word census is Roman — but a full census is expensive and slow, and for many questions it is overkill. In the late 1800s the Norwegian statistician Anders Kiaer argued for a "representative method": measure a carefully chosen part instead of the whole. The idea was controversial. Could a part really speak for the whole?

The decisive breakthrough came in 1934, when the Polish-born statistician Jerzy Neyman published a paper proving that probability sampling — selecting members by a known random mechanism — is what makes inference valid, and that purposive (hand-picked "representative") sampling does not. Randomness, counterintuitively, is the source of reliability: it is precisely because selection is left to chance that no hidden preference of the researcher can creep in, and that the laws of probability let us attach a margin of error.

Meanwhile, a parallel revolution was happening in agriculture. Ronald A. Fisher, working at the Rothamsted Experimental Station in England in the 1920s and 1930s, faced a maddening problem: crop yields varied so much from plot to plot — because of soil, drainage, and sunlight — that it was nearly impossible to tell whether a new fertilizer actually helped. Fisher's answer, laid out in his 1935 book The Design of Experiments, was to build randomness into the experiment on purpose. By randomly assigning treatments to plots, replicating each treatment, and grouping similar plots into blocks, he could mathematically separate the effect of the fertilizer from the noise of the land. Fisher also gave us the famous "lady tasting tea" thought experiment, which crystallized how randomization creates a valid test of a claim.

The cost of ignoring these ideas was made brutally public by two polling disasters. In 1936, Literary Digest magazine mailed 10 million straw-poll ballots and, from 2.4 million replies, confidently predicted Alf Landon would beat Franklin Roosevelt in a landslide. Roosevelt won 46 states. The magazine's sample — drawn from telephone and automobile-registration lists during the Great Depression — skewed wealthy and Republican, and only the enthusiastic mailed back. A young pollster named George Gallup, using a much smaller but more representative sample, called it correctly. Then in 1948, the pollsters themselves were humbled: relying on outdated quota sampling, they declared "Dewey Defeats Truman" — and were wrong, pushing the industry toward the probability methods Neyman had championed. These episodes are the reason modern polling obsesses over sampling frames.

Populations, Samples, and the Language of Inference

The population is every individual or item of interest — every registered voter in a country, every 500 g bag off a production line, every patient with a given disease. A number that describes the whole population is a parameter (for example, the true proportion $p$ of voters supporting a candidate). We almost never know a parameter directly.

Instead we take a sample and compute a statistic from it — for instance the sample proportion $\hat{p}$. The statistic is our estimate of the parameter. The central promise of statistics is that if the sample was chosen well, $\hat{p}$ will be close to $p$, and we can say how close.

Worked example. A university has 20,000 students (the population). We want the mean number of hours studied per week — the parameter $\mu$. We survey 400 students and find a sample mean of $\bar{x} = 14.2$ hours with sample standard deviation $s = 6$ hours. A rough 95% margin of error for the mean is

$2 \times \frac{s}{\sqrt{n}} = 2 \times \frac{6}{\sqrt{400}} = 2 \times \frac{6}{20} = 0.6 \text{ hours}.$

So we estimate $\mu$ is about $14.2 \pm 0.6$ hours. Notice the crucial detail: this margin is only meaningful if the 400 were randomly chosen. If we had surveyed 400 students leaving the library, the arithmetic would be identical but the answer would be badly biased — library-goers study more than average.

Random, Stratified, and Cluster Sampling

Not all sampling is equal. Three probability-based designs dominate practice.

Simple random sampling (SRS). Every possible sample of size $n$ is equally likely; equivalently, every member has the same chance of selection. Imagine numbering all 20,000 students and using a random-number generator to pick 400. SRS is the gold-standard baseline, but it can be impractical (you need a complete list) and can, by chance, under-represent small subgroups.

Stratified sampling. Divide the population into strata — non-overlapping groups that are internally similar — then take a random sample from each stratum. If our university is 60% undergraduate and 40% graduate, we sample 240 undergraduates and 160 graduates. Stratification guarantees each group is represented and usually reduces the margin of error, because it removes between-group variation from the estimate.

Worked example (stratified estimate). Suppose undergraduates average 12 hours (from a sample of 240) and graduates average 17 hours (from a sample of 160). The stratified estimate of the overall mean weights each stratum by its true population share:

$\bar{x}_{\text{strat}} = 0.60 \times 12 + 0.40 \times 17 = 7.2 + 6.8 = 14.0 \text{ hours}.$

Because each group's proportion is fixed to match the population, a random over-sampling of graduates cannot inflate the total the way it might under plain SRS.

Cluster sampling. Divide the population into clusters — groups that ideally each resemble the whole population — then randomly pick a few whole clusters and measure everyone in them. To survey students you might randomly choose 10 of the university's 200 dorm floors and interview every resident. Clustering is cheap (you travel to few locations) but less precise if people within a cluster are too similar to each other.

The key contrast: stratified sampling takes a bit from every group and is efficient; cluster sampling takes everything from a few groups and is cheap. Strata should be internally homogeneous and different from one another; clusters should each be internally heterogeneous and resemble the whole.

Bias: The Error That Does Not Shrink

Bias is a systematic tendency for an estimate to miss the truth in the same direction, no matter how large the sample. This is what makes it dangerous: increasing $n$ shrinks random error but does nothing to bias. A precisely wrong answer is still wrong.

• Selection (undercoverage) bias: the sampling method systematically excludes part of the population. The Literary Digest used car-and-phone owners; poor voters had no chance of selection.
• Nonresponse bias: chosen people who do not respond differ from those who do. If only strongly opinionated people return a survey, results skew toward extremes.
• Response bias: the answers themselves are distorted — by badly worded or leading questions, by sensitive topics ("Have you ever cheated?"), or by interviewer influence.

Worked example. A radio host asks listeners to call in: "Should taxes be cut?" Of 5,000 callers, 4,200 say yes — an 84% "yes" rate. This voluntary-response sample is worthless as a population estimate: the listeners already share the host's leanings (selection), and people who feel strongly are far likelier to bother calling (nonresponse). The 84% describes motivated callers, not the public. A random sample of 600 citizens might well show a near-even split.

Observational Studies vs Experiments

In an observational study you measure individuals without intervening — you record who happens to smoke and who develops cancer. In an experiment you actively impose a treatment — you assign subjects to take a drug or a placebo — and compare outcomes.

The difference is everything, because of confounding variables: a lurking third factor associated with both the supposed cause and the effect. Observational data famously showed that coffee drinkers had higher lung-cancer rates. Coffee causes cancer? No — coffee drinkers were more likely to smoke. Smoking was the confounder. Observation alone cannot untangle this; you see an association, not a cause.

An experiment breaks confounding by randomly assigning the treatment. If subjects are split into groups by a coin flip, then on average the groups are balanced on every variable — age, diet, genetics, even ones you never thought to measure. Any later difference in outcome can then be attributed to the treatment. This is why the phrase "correlation does not imply causation" is really a warning that most published associations come from observational data.

Control, Randomization, and Blinding

Fisher's framework rests on three principles, plus a fourth that protects against human psychology.

1. Control — include a control group that does not receive the treatment (or receives a placebo), so you have a baseline for comparison. Without a control, you cannot tell whether improvement came from the drug or from time, hope, or the natural course of illness.
2. Randomization — assign subjects to groups by chance. This balances confounders and validates the probability calculations behind your test.
3. Replication — apply each treatment to many subjects (and, ideally, repeat the whole study). One patient recovering proves nothing; a reliable effect must show up across many.

Blinding guards against expectations. In a single-blind study, subjects don't know which group they're in, so the placebo effect — real improvement driven purely by belief — hits both groups equally and cancels out. In a double-blind study, the researchers measuring outcomes also don't know, preventing them from unconsciously scoring the treatment group more favorably.

Worked example. Testing a new painkiller, we enroll 200 patients. We randomly assign 100 to the drug and 100 to an identical-looking placebo (control + randomization). Neither patients nor the nurses recording pain scores know who got what (double-blind). After a week, mean pain (0–10 scale) is 3.1 in the drug group and 4.9 in the placebo group, a difference of 1.8 points. Because assignment was random and both groups experienced the placebo effect equally, that 1.8-point gap is credibly caused by the drug — a conclusion no observational study could support.

Real-World Applications

• Clinical trials: Drug approval by agencies like the FDA hinges on double-blind randomized controlled trials — the direct descendants of Fisher's plots.
• Political polling: Modern pollsters use stratified and weighted random samples, and report margins of error, precisely to avoid a repeat of 1936 and 1948.
• Quality control: Factories randomly sample items off the line to estimate defect rates without inspecting every unit.
• A/B testing: Tech companies randomly show version A or version B of a webpage to users — a live experiment — to measure which drives more clicks, causally.
• Public health & agriculture: Randomized field trials still test crop varieties, vaccination strategies, and education programs.

Common Mistakes

1. "A bigger sample is always better." Why wrong: Size fixes random error, not bias. The Literary Digest had 2.4 million responses and was catastrophically wrong. Correction: A small random sample beats a huge biased one. Fix the method first, then worry about size.

2. "Correlation shows causation." Why wrong: Observational associations can be entirely driven by a confounding variable. Correction: Only a randomized experiment (or very careful causal analysis) supports a causal claim; otherwise report an association.

3. "Voluntary responses represent everyone." Why wrong: People who opt in — online polls, call-ins, product reviews — differ systematically from those who don't (selection and nonresponse bias). Correction: Use a probability sample where you choose respondents by chance, not one where respondents choose you.

Comparison and Connections

Stratified and cluster sampling are the two most frequently confused designs. Observational versus experimental is the other axis students blur.

Feature	Stratified sampling	Cluster sampling
How groups are formed	Homogeneous strata (similar within)	Heterogeneous clusters (each mirrors whole)
What is sampled	A random sample within every group	All members of a few random groups
Main goal	Precision — reduce error	Cost — fewer locations
Best when	Groups differ a lot from each other	Population is spread out; a full list is hard

Feature	Observational study	Experiment
Researcher's role	Observes, does not intervene	Imposes a treatment
Assignment	Self-selected / natural	Random assignment
Can show causation?	No — only association	Yes, if well designed
Main threat	Confounding variables	Poor blinding, small $n$

These ideas feed directly into confidence intervals and hypothesis testing, which quantify the uncertainty that good sampling makes calculable, and rest on the probability framework of random selection.

Practice Questions

Recall

Q: Define parameter and statistic, and state which one we usually know. A: A parameter is a number describing the whole population (e.g. true mean $\mu$); a statistic is a number computed from a sample (e.g. $\bar{x}$). We usually know only the statistic and use it to estimate the parameter.

Understanding

Q: A town has three neighborhoods of very different income levels. You want to estimate average household spending. Would you use stratified or cluster sampling, and why? A: Stratified. Because the neighborhoods differ strongly from one another (heterogeneous between groups, homogeneous within), making each a natural stratum. Sampling within each guarantees all income levels are represented and lowers the margin of error. Cluster sampling could accidentally miss a whole neighborhood.

Application

Q: A population is 70% women and 30% men. Using proportional stratified sampling for $n = 200$, how many of each should you sample, and if women average 40 and men average 46 on a score, what is the stratified estimate of the overall mean? A: Sample $0.70 \times 200 = 140$ women and $0.30 \times 200 = 60$ men. Estimated mean $= 0.70 \times 40 + 0.30 \times 46 = 28 + 13.8 = 41.8$.

Analysis

Q: A study finds people who take vitamin supplements live longer. A journalist writes "Vitamins extend life." Critique this, naming the study type and at least one confounder. A: It is an observational study, so it shows association, not causation. Supplement takers likely differ systematically — they tend to be wealthier, exercise more, eat better, and see doctors more often. Any of these (healthy-user effect) could be the real cause of longer life. Only a randomized controlled trial could isolate the vitamins' effect.

FAQ

Is a sample of 1,000 really enough to represent millions of people? Yes, if it is random. The margin of error depends on the sample size, not the population size (once the population is large). A random 1,000 gives roughly a $\pm 3\%$ margin whether the country has 1 million or 300 million people — this surprises almost everyone.

What exactly is the "sampling frame," and why does it matter? The sampling frame is the actual list from which you draw your sample. If the frame omits part of the population — as phone lists omitted the poor in 1936 — you get undercoverage bias no matter how randomly you sample from the flawed list.

Can you ever prove causation without an experiment? Rarely and only with great care. When experiments are unethical (you can't force people to smoke), scientists build causal cases from many observational studies, dose-response patterns, and mechanisms — the way smoking-and-cancer was eventually established. But a single observational study cannot prove cause.

What is the placebo effect and why does it require a control group? It's real improvement caused by a patient's belief that they're being treated. Because it affects anyone who thinks they're treated, you need a placebo control group that experiences the same belief, so the treatment group's extra improvement can be attributed to the actual treatment.

Why randomize instead of carefully matching the groups myself? You can only match on variables you know and measure. Randomization balances the groups on everything, including confounders you never imagined — and it is what makes the probability behind your test valid.

Quick Revision

• Population (parameter) vs sample (statistic); we estimate the former with the latter.
• SRS: everyone equally likely. Stratified: sample within homogeneous groups (precision). Cluster: all members of a few groups (cheap).
• Bias = systematic error; a bigger $n$ does not fix it. Types: selection, nonresponse, response.
• Margin of error (mean) $\approx 2 s / \sqrt{n}$ — depends on $n$, not population size.
• Observational study → association only. Experiment → causation, via random assignment.
• Three pillars: control, randomization, replication; add blinding (single/double) against the placebo effect.
• History: Neyman (1934, probability sampling), Fisher (1935, experimental design), and the 1936/1948 polling failures.

Related Topics

Prerequisites

• Statistics and Probability overview

Related Topics

• Confidence Intervals and margin of error
• Hypothesis Testing
• Probability and random variables

Next Topics

• Return to the Statistics and Probability branch
• Explore the Mathematics subject index