Hypothesis Testing in Psychology

Learning Objectives

• State a null and alternative hypothesis correctly for a given research question.
• Explain the roles of significance level (α), test statistic, and p-value in a hypothesis test.
• Walk through the steps of hypothesis testing from formulating hypotheses to interpreting results.
• Distinguish Type I and Type II errors and explain the trade-off between them.
• Select the correct hypothesis test (one-sample t, two-sample t, paired t, ANOVA, chi-square) for a research scenario.

Quick Answer

Hypothesis testing is the formal decision-making procedure psychologists use to determine whether a pattern observed in sample data is likely to reflect a real effect in the population, or is more likely just random sampling variation. The process always starts with two competing statements — a null hypothesis (H0, "no effect" or "no difference") and an alternative hypothesis (H1, "there is an effect") — and ends with a decision to either reject H0 (evidence favors a real effect) or fail to reject H0 (not enough evidence). This structured, probability-based procedure is what separates a scientific claim ("this therapy works, p < .05") from an anecdotal one ("this therapy seemed to help my client").

The Logic: Innocent Until Proven Guilty

Hypothesis testing borrows the logic of a courtroom trial. The null hypothesis is treated like a defendant presumed innocent (no effect exists) until the sample data provide strong enough evidence to reject that assumption. You never "prove H0 true" — you either find enough evidence against it, or you don't.

Null hypothesis (H0): states there is no effect, no difference, or no relationship (e.g., μ1 = μ2). Alternative hypothesis (H1 or Ha): states there is an effect, difference, or relationship (e.g., μ1 ≠ μ2).

Steps in Hypothesis Testing

1. Formulate H0 and H1 based on the research question.
2. Choose a significance level (α) — conventionally 0.05, meaning you'll accept a 5% risk of wrongly rejecting a true H0.
3. Collect sample data.
4. Calculate a test statistic (t, F, χ², z) that measures how far the sample result is from what H0 predicts.
5. Determine the critical value or p-value for that test statistic.
6. Make a decision: if p < α, reject H0; if p ≥ α, fail to reject H0.
7. Interpret the result in plain language, tied back to the original research question.

Worked Example: One-Sample t-test

A researcher believes a new study technique changes average exam scores from the known population mean of 70. A sample of 16 students using the technique scores a mean of 74, with a sample SD of 8.

H0: μ = 70 (no change) H1: μ ≠ 70 (a change exists)

Standard error = SD / √n = 8 / √16 = 8/4 = 2

t = (sample mean − population mean) / SE = (74 − 70) / 2 = 2.0

With df = 15 and α = .05 (two-tailed), the critical t-value is approximately 2.13. Since our calculated t (2.0) is less than 2.13, we fail to reject H0 — the observed 4-point increase isn't large enough, relative to the variability in the sample, to conclude the study technique reliably changes scores at the 5% significance level. This is a good example of why "the numbers moved in the right direction" isn't the same as "statistically significant."

Types of Hypothesis Tests

Test	Used when	Example
One-sample t-test	Comparing one sample mean to a known population mean	Comparing a sample's average IQ to the population mean of 100
Independent two-sample t-test	Comparing means of two separate, unrelated groups	Reaction time: caffeine group vs. no-caffeine group
Paired t-test	Comparing two measurements from the same subjects	Anxiety scores before vs. after therapy in the same clients
ANOVA	Comparing means across three or more groups	Exam scores across three teaching methods
Chi-square test	Testing association between categorical variables	Gender and preference for a therapy type

Errors in Decision-Making: Type I and Type II

No decision procedure is perfect. Hypothesis testing can go wrong in two distinct ways:

	H0 is actually true	H0 is actually false
Reject H0	Type I error (false positive), probability = α	Correct decision
Fail to reject H0	Correct decision	Type II error (false negative), probability = β

• Type I error: concluding an effect exists when it really doesn't (e.g., claiming a useless drug works). Controlled by setting α (usually 0.05).
• Type II error: missing a real effect (e.g., concluding a genuinely effective therapy "doesn't work"). Reduced by increasing sample size or statistical power.

There's an inherent trade-off: making α stricter (say, 0.01) reduces Type I errors but increases the risk of Type II errors, unless sample size also increases.

Real-World Application

Educational researchers testing whether a new reading curriculum improves comprehension scores compared to the standard curriculum will formulate H0 (no difference between curricula) and H1 (curricula differ), collect scores from two randomly assigned classrooms, run an independent-samples t-test, and report a p-value. If p < .05, the school district gets statistical grounds to justify a costly curriculum change — the exact reasoning process outlined above, just applied to a real policy decision.

Key Terms

Term	Definition
Null hypothesis (H0)	The hypothesis of no effect, no difference, or no relationship
Alternative hypothesis (H1)	The hypothesis that an effect, difference, or relationship exists
Significance level (α)	The probability threshold (commonly 0.05) for rejecting H0
p-value	The probability of observing data this extreme (or more) if H0 were true
Test statistic	A calculated value (t, F, χ², z) summarizing how far sample data deviate from H0
Type I error	Rejecting a true H0 (false positive)
Type II error	Failing to reject a false H0 (false negative)
Statistical power	The probability of correctly rejecting a false H0

Common Mistakes

1. Misconception: "Failing to reject H0 means H0 is proven true." Why it's wrong: Failing to reject H0 simply means the evidence wasn't strong enough to rule out "no effect" — it doesn't confirm no effect exists; the study might just lack statistical power. Correct: Say "we failed to find sufficient evidence against H0," never "we proved H0."

2. Misconception: "A p-value tells you the probability that the null hypothesis is true." Why it's wrong: The p-value is calculated assuming H0 is true — it tells you the probability of the data (or more extreme data), not the probability of the hypothesis itself. Correct: p-value = P(data this extreme | H0 is true), not P(H0 is true | data).

3. Misconception: A statistically significant result is automatically a practically important one. Why it's wrong: With a large enough sample, even a tiny, trivial difference can become statistically significant. Correct: Always examine effect size alongside significance to judge whether a result matters in real-world terms.

Comparison and Connections

Concept	Type I Error	Type II Error
What happens	Reject a true H0	Fail to reject a false H0
Also called	False positive	False negative
Controlled by	Significance level (α)	Sample size / statistical power
Example consequence	Concluding an ineffective drug works	Concluding an effective drug doesn't work

Test	Groups compared	Data type
One-sample t-test	Sample vs. known population value	Interval/ratio
Independent t-test	Two separate groups	Interval/ratio
Paired t-test	Same subjects, two conditions	Interval/ratio
ANOVA	Three or more groups	Interval/ratio
Chi-square	Categorical group membership	Nominal

Practice Questions

Recall

1. State the null and alternative hypotheses in general symbolic form for a two-group comparison of means. Answer guidance: H0: μ1 = μ2; H1: μ1 ≠ μ2.
2. Define Type I and Type II errors. Answer guidance: Type I = rejecting a true H0 (false positive); Type II = failing to reject a false H0 (false negative).

Understanding 3. Explain why a p-value is not the probability that the null hypothesis is true. Answer guidance: The p-value is computed under the assumption that H0 is true; it reflects how likely the observed data (or more extreme) would be in that scenario, not the probability of H0 itself being true. 4. Why does lowering α from 0.05 to 0.01 reduce Type I errors but potentially increase Type II errors? Answer guidance: A stricter α makes it harder to reject H0, reducing false positives, but this also makes it easier to miss a real effect (false negative) unless statistical power is compensated for, e.g., with a larger sample.

Application 5. A clinical researcher compares depression scores before and after 8 weeks of the same clients' therapy. Which test is appropriate, and why? Answer guidance: Paired t-test, because the same subjects are measured twice (before/after), making the two sets of scores related rather than independent. 6. A study finds p = .03 when testing whether a new memory technique improves recall. Using α = .05, what decision is made, and how would you phrase the conclusion? Answer guidance: Since p (.03) < α (.05), reject H0; conclude there is statistically significant evidence that the memory technique affects recall, while noting this doesn't establish the size or practical importance of the effect.

Analysis 7. A study with a very large sample (n = 10,000) finds a statistically significant difference in reaction time between two groups, but the actual difference is only 2 milliseconds. Evaluate the practical significance of this finding. Answer guidance: Despite reaching statistical significance due to the large sample's power, a 2ms difference is likely trivial in real-world terms; the researcher should report effect size and consider whether the difference matters practically, not just statistically. 8. Compare a Type I error and Type II error in the context of a courtroom trial for a new medication. Which error would regulators typically consider more serious, and why? Answer guidance: Type I (approving/claiming a useless or harmful drug "works") could expose patients to an ineffective or harmful treatment; Type II (rejecting a genuinely effective drug) delays a beneficial treatment. Regulators often weigh Type I errors as more serious for safety, but the "worse" error depends on context (e.g., for a fatal illness with no other treatment, missing a real effect via Type II error might be more costly).

FAQ

Why do we test the null hypothesis instead of directly testing what we believe (H1)? Statistical theory lets us calculate exact probabilities under a precise "no effect" assumption; we then see how unlikely our data would be under that assumption, which indirectly builds the case for H1.

What's the difference between a one-tailed and two-tailed test? A one-tailed test predicts a direction of effect (e.g., "scores will increase") and places the entire rejection region on one side; a two-tailed test allows for an effect in either direction, splitting the rejection region between both tails. Two-tailed tests are the more conservative, common default.

Why is α conventionally set at 0.05? It's a historical convention (popularized by Ronald Fisher) balancing the risk of false positives against practicality; some fields now push for stricter thresholds (e.g., 0.01 or 0.005) for stronger claims.

Does a non-significant result mean the study "failed"? No — a well-designed, adequately powered study that fails to reject H0 still provides useful information (e.g., ruling out a meaningful effect), especially if the sample size was large enough to detect one had it existed.

How does statistical power relate to sample size? Larger samples generally increase statistical power (the ability to detect a real effect), because they reduce standard error and make the test statistic more precise — this is why power analysis is done before data collection to determine adequate sample size.

Quick Revision

• H0 = no effect/no difference; H1 = an effect/difference exists.
• α (significance level) is the risk of Type I error you're willing to accept, conventionally 0.05.
• p-value = probability of the observed data (or more extreme) assuming H0 is true — not the probability H0 is true.
• Decision rule: if p < α, reject H0; if p ≥ α, fail to reject H0.
• Type I error = false positive (rejecting a true H0); Type II error = false negative (failing to reject a false H0).
• Statistical power = probability of correctly detecting a real effect; increases with larger sample size.
• One-sample t-test compares a sample to a known value; independent t-test compares two separate groups; paired t-test compares the same subjects twice; ANOVA compares 3+ groups; chi-square tests categorical associations.
• Statistical significance ≠ practical/real-world importance — always check effect size.
• "Fail to reject H0" is not the same as "H0 is proven true."

Related Topics

Prerequisites: Inferential Statistics, Descriptive Statistics.

Related: Correlation and Regression, Data Interpretation.

Next: Correlation and Regression, which extends hypothesis testing logic to relationships between variables.