Correlation and Regression

Whenever two things seem to move together — hours studied and exam scores, height and weight, advertising spend and sales — we naturally want to know two things: how strongly are they linked, and can I use one to predict the other? Correlation answers the first question with a single number between $-1$ and $1$. Regression answers the second by fitting the "best" straight line through the cloud of data. Together they are among the most-used tools in all of science, economics, and machine learning — and among the most-abused, because a strong relationship is endlessly mistaken for a causal one.

This page teaches you to read a scatterplot, compute and interpret the correlation coefficient $r$, build the least-squares regression line by hand, use it to predict, and — crucially — to know when your prediction is trustworthy and when the whole thing is a mirage.

Learning Objectives

• Read and describe the pattern in a scatterplot (direction, form, strength, outliers).
• Compute the Pearson correlation coefficient $r$ and interpret its sign and magnitude.
• Derive and calculate the least-squares regression line $\hat{y} = a + bx$.
• Use a regression line to make predictions and understand interpolation vs. extrapolation.
• Interpret the slope, intercept, and coefficient of determination $r^2$ in context.
• Explain clearly why correlation does not imply causation, with lurking variables.

Quick Answer

Correlation measures the strength and direction of a linear relationship between two quantitative variables using $r$, which ranges from $-1$ (perfect negative) through $0$ (no linear relationship) to $1$ (perfect positive). The least-squares regression line is the unique straight line that minimizes the sum of squared vertical distances from the points; its slope is $b = r \cdot (s_y / s_x)$ and it always passes through the mean point $(\bar{x}, \bar{y})$. The square of the correlation, $r^2$, tells you the fraction of variation in $y$ explained by $x$. You can predict $y$ from $x$ within the range of your data, but a strong $r$ never proves that $x$ causes $y$ — a hidden third variable or reverse causation can produce the same pattern.

Where It Came From

Regression was born from an obsession with heredity. In the 1870s and 1880s the Victorian polymath Francis Galton — cousin of Charles Darwin — wanted to understand how traits like height passed from parents to children. He collected the heights of hundreds of parents and their grown children and plotted them. He noticed something strange: tall parents did tend to have tall children, but the children were on average closer to the population mean than their parents were. Exceptionally tall parents had children shorter than themselves; exceptionally short parents had children taller than themselves. Galton called this pull toward the average "regression toward mediocrity" (today: regression to the mean). The line he drew through his data to summarize the parent–child relationship became the first "regression line," and the name stuck to the entire method — an accident of biological history.

Galton had the picture but lacked the mathematics to measure the strength of such relationships precisely. That was supplied by his younger colleague Karl Pearson around 1895–1896. Building on earlier work by the French physicist Auguste Bravais, Pearson formalized the product-moment correlation coefficient $r$ — the standardized measure we still use — giving science a single, dimensionless number to quantify association. The deep need these tools answered was universal: nature is noisy. Two quantities rarely line up perfectly, so researchers needed a rigorous way to say "these vary together, and here is exactly how much" rather than relying on gut impression. That need — measuring imperfect relationships in messy data — is why correlation and regression became the backbone of modern statistics.

Scatterplots: Seeing the Relationship First

Before any calculation, plot the data. A scatterplot puts one variable on the horizontal axis ($x$, the explanatory or independent variable) and the other on the vertical axis ($y$, the response or dependent variable). Each observation is one dot. When you look at a scatterplot, describe four things:

• Direction — positive (up to the right), negative (down to the right), or none.
• Form — linear, curved, or clustered.
• Strength — how tightly points hug an imagined line.
• Outliers — points far from the overall pattern.

This step is not optional decoration. The correlation coefficient only measures linear strength, so a perfect U-shaped parabola can have $r$ near $0$ despite a flawless relationship. The famous Anscombe's quartet consists of four datasets with identical $r$, identical regression lines, and identical means — yet radically different scatterplots (one is curved, one is dominated by a single outlier). The lesson: numbers summarize, but the picture reveals.

The Correlation Coefficient r

Pearson's $r$ standardizes both variables and averages the products. The most transparent formula is

$r = \frac{1}{n-1} \sum \left( \frac{x_i - \bar{x}}{s_x} \right)\left( \frac{y_i - \bar{y}}{s_y} \right)$

where $s_x$ and $s_y$ are the sample standard deviations. Each factor is a z-score. When a point is above average in both $x$ and $y$, the product is positive; when it is above in one and below in the other, the product is negative. Add these up and $r$ captures whether points tend to be "on the same side" of both means.

Key properties:

• $-1 \le r \le 1$ always.
• $r$ has no units and is unchanged if you rescale (e.g., cm to inches).
• $r$ is symmetric: the correlation of $x$ with $y$ equals that of $y$ with $x$.
• $r$ only measures linear association and is sensitive to outliers.

Worked Example: Computing r

Suppose five students report hours studied ($x$) and their test score ($y$):

Student	$x$ (hours)	$y$ (score)
A	1	50
B	2	60
C	3	70
D	4	65
E	5	80

Means: $\bar{x} = 15/5 = 3$ and $\bar{y} = 325/5 = 65$.

Build a table of deviations. Let $dx = x - \bar{x}$ and $dy = y - \bar{y}$:

$dx$	$dy$	$dx \cdot dy$	$dx^2$	$dy^2$
$-2$	$-15$	$30$	$4$	$225$
$-1$	$-5$	$5$	$1$	$25$
$0$	$5$	$0$	$0$	$25$
$1$	$0$	$0$	$1$	$0$
$2$	$15$	$30$	$4$	$225$

Sums: $\sum dx\,dy = 65$, $\sum dx^2 = 10$, $\sum dy^2 = 500$.

$r = \frac{\sum dx\,dy}{\sqrt{\sum dx^2 \cdot \sum dy^2}} = \frac{65}{\sqrt{10 \times 500}} = \frac{65}{\sqrt{5000}} \approx \frac{65}{70.71} \approx 0.919$

An $r$ of about $0.92$ signals a strong positive linear relationship: more study hours go with higher scores.

Least-Squares Regression Lines

Correlation says how strong; regression gives the actual line so we can predict. We write the fitted line as

$\hat{y} = a + bx$

where $\hat{y}$ (read "y-hat") is the predicted value. For any real point, the residual is $e_i = y_i - \hat{y}_i$, the vertical miss. The least-squares line is the one choice of $a$ and $b$ that makes $\sum e_i^2$ as small as possible. Squaring keeps positive and negative misses from cancelling and punishes big misses harder. Solving that minimization (via calculus) gives clean formulas:

$b = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2} = r \cdot \frac{s_y}{s_x}, \qquad a = \bar{y} - b\bar{x}$

The second form of the slope is beautiful: it links regression directly to correlation. Because $a = \bar{y} - b\bar{x}$, the line is forced to pass through the mean point $(\bar{x}, \bar{y})$ — a fact worth remembering.

Worked Example: Fitting the Line

Using the same study data, we already have $\sum dx\,dy = 65$ and $\sum dx^2 = 10$. So

$b = \frac{65}{10} = 6.5$

$a = \bar{y} - b\bar{x} = 65 - 6.5 \times 3 = 65 - 19.5 = 45.5$

The regression line is

$\hat{y} = 45.5 + 6.5x$

Interpreting the slope: each additional hour of study is associated with about $6.5$ more points on the test. Interpreting the intercept: a student who studies $0$ hours is predicted to score $45.5$ — plausible here, but often the intercept is just a mathematical anchor with no real meaning.

Prediction, and Its Limits

To predict the score for a student who studies $3.5$ hours:

$\hat{y} = 45.5 + 6.5 \times 3.5 = 45.5 + 22.75 = 68.25$

This is interpolation — predicting inside the observed range ($1$ to $5$ hours) — and is reasonable. Predicting for $20$ hours ($\hat{y} = 45.5 + 130 = 175.5$, an impossible score above $100$) is extrapolation: assuming the linear pattern continues far beyond the data. It rarely does, and it is one of the most common ways predictions go badly wrong.

r-squared: How Much Does the Line Explain?

The coefficient of determination $r^2$ is the fraction of the variation in $y$ that is explained by the linear relationship with $x$. Here $r^2 = 0.919^2 \approx 0.845$, so about 85% of the variation in test scores is accounted for by study hours; the remaining $15\%$ is other factors (sleep, prior knowledge, luck). Values of $r^2$ range from $0$ to $1$, and it is often more meaningful to report than $r$ itself because it has a concrete "proportion of variance" interpretation.

Correlation Versus Causation

This is the single most important idea on the page. A strong $r$ tells you two variables move together; it says nothing about why. There are always three possible explanations for an observed correlation between $x$ and $y$:

1. $x$ causes $y$ (what people assume).
2. $y$ causes $x$ (reverse causation).
3. A lurking (confounding) variable $z$ causes both.

Classic illustration: across a town, ice cream sales correlate strongly with drowning deaths. Ice cream does not cause drowning. The lurking variable is hot weather — heat drives both more ice cream buying and more swimming. Another: children with bigger shoe sizes read better. Shoes do not cause literacy; age confounds both. Only a well-designed randomized controlled experiment — where the researcher randomly assigns the value of $x$ — can establish causation, because randomization breaks the link to lurking variables. Observational correlation, no matter how strong, cannot.

Real-World Applications

• Medicine: regression relates drug dosage to blood pressure response; but establishing that a drug works requires randomized trials, not just correlation.
• Economics and finance: the "beta" of a stock is a regression slope measuring how the stock moves relative to the market. Analysts regress consumer spending on income.
• Machine learning: linear regression is the foundational predictive model, and least squares underlies much of modern data science.
• Sports analytics: teams regress player statistics (on-base percentage, expected goals) to predict future performance and set salaries.
• Public health: the smoking–lung cancer link began as a correlation; it took decades of converging evidence and mechanism studies to firmly establish causation.

Common Mistakes

Mistake 1: Concluding causation from correlation. A student sees $r = 0.9$ between two variables and writes "$x$ causes $y$." Why it's wrong: correlation is symmetric and blind to mechanism; a lurking variable or reverse causation can produce identical numbers. Correction: say the variables are "associated" or "related," and reserve causal language for experimental evidence.

Mistake 2: Trusting $r$ without plotting the data. A dataset with a clear curve or one dominant outlier can yield a misleading $r$. Why it's wrong: $r$ only measures linear strength; a parabola can have $r \approx 0$ despite a perfect relationship. Correction: always inspect the scatterplot before and after computing $r$ (remember Anscombe's quartet).

Mistake 3: Extrapolating beyond the data. Using $\hat{y} = 45.5 + 6.5x$ to predict a score for $20$ study hours gives an impossible $175.5$. Why it's wrong: the linear pattern was only observed from $1$ to $5$ hours; nothing guarantees it holds outside that range. Correction: restrict predictions to the observed range of $x$, and flag any extrapolation as speculative.

Comparison and Connections

Correlation and regression are two views of the same relationship, but they are not interchangeable.

Feature	Correlation ($r$)	Regression ($\hat{y}=a+bx$)
Question answered	How strong/what direction?	Predict $y$ from $x$?
Output	One number, $-1$ to $1$	A line (slope and intercept)
Units	None (dimensionless)	Slope has units of $y$ per $x$
Symmetry	Symmetric in $x$ and $y$	Not symmetric — $y$ on $x$ differs from $x$ on $y$
Requires which is "response"?	No	Yes

They connect through $b = r \cdot (s_y/s_x)$ and $r^2$. Note the asymmetry of regression: the line predicting scores from hours is not the same as the line predicting hours from scores, even though $r$ is identical both ways. Correlation is also related to covariance — $r$ is just covariance standardized by both standard deviations, which strips out the units.

Practice Questions

Recall

What is the range of possible values for the correlation coefficient $r$, and what do the endpoints mean? Answer: $r$ ranges from $-1$ to $1$. A value of $1$ is a perfect positive linear relationship, $-1$ a perfect negative one, and $0$ no linear relationship.

Understanding

Explain why the least-squares line uses squared residuals rather than just the sum of residuals. Answer: The plain sum of residuals is always zero for the best line, so it cannot distinguish good fits from bad. Squaring makes all terms positive (preventing cancellation) and penalizes large misses more heavily, yielding a unique, well-behaved minimum.

Application

For a dataset, $\bar{x}=10$, $\bar{y}=50$, $s_x=2$, $s_y=8$, and $r=0.75$. Find the regression line. Answer: $b = r \cdot s_y/s_x = 0.75 \times 8/2 = 3$. Then $a = \bar{y} - b\bar{x} = 50 - 3\times 10 = 20$. Line: $\hat{y} = 20 + 3x$.

Analysis

A study finds that cities with more police officers have more crime ($r = 0.8$). Does hiring police cause crime? Explain. Answer: No. This is almost certainly a lurking-variable effect: larger cities have both more crime and more police. Population confounds both variables. Reverse causation is also plausible (high-crime cities hire more police). Correlation cannot settle causation; only a controlled study could.

FAQ

Is $r = 0.5$ "strong"? It depends on the field. In physics, $0.5$ might be weak; in social science, where human behavior is noisy, it can be meaningful. As a rough guide: $|r|$ near $0.1$ is weak, $0.3$ moderate, $0.5$ or above fairly strong — but always judge in context and via $r^2$.

What's the difference between $r$ and $r^2$? $r$ carries the sign (direction) and measures linear strength; $r^2$ (always $0$ to $1$) is the proportion of variance in $y$ explained by $x$. Because squaring shrinks values below $1$, $r=0.7$ gives $r^2 = 0.49$ — only about half the variation explained, which surprises many students.

Why does the regression line always go through $(\bar{x}, \bar{y})$? Because the intercept is defined as $a = \bar{y} - b\bar{x}$. Substituting $x = \bar{x}$ gives $\hat{y} = \bar{y} - b\bar{x} + b\bar{x} = \bar{y}$. The mean point is the balance point of the line.

Can I have a strong relationship with $r = 0$? Yes, if the relationship is nonlinear. A perfect symmetric U-shape (like $y = x^2$ centered on zero) has $r \approx 0$ because $r$ only detects linear trends. Always look at the scatterplot.

Which variable goes on the x-axis? The explanatory (independent) variable — the one you think explains or predicts the other. The response (dependent) variable goes on the y-axis. In study-hours vs. score, hours is explanatory. Getting this backwards changes the regression line.

Quick Revision

• Scatterplot first: describe direction, form, strength, outliers.
• Correlation: $r = \frac{\sum dx\,dy}{\sqrt{\sum dx^2 \sum dy^2}}$, ranges $-1$ to $1$, unitless, measures linear strength only.
• Regression line: $\hat{y} = a + bx$ with $b = r\,s_y/s_x$ and $a = \bar{y} - b\bar{x}$; always passes through $(\bar{x}, \bar{y})$.
• Least squares minimizes $\sum (y_i - \hat{y}_i)^2$.
• $r^2$ = proportion of variation in $y$ explained by $x$.
• Interpolate (inside data range) safely; never extrapolate blindly.
• Correlation is not causation — beware lurking variables and reverse causation.
• History: Galton (regression to the mean, 1880s), Pearson (the coefficient $r$, 1890s).

Related Topics

Prerequisites

• Statistics and Probability overview
• Descriptive statistics: mean and standard deviation
• Reading and building graphs and coordinate plots

Related Topics

• Covariance and standardization (z-scores)
• Sampling and experimental design (why randomization enables causal claims)

Next Topics

• Multiple regression and the basics of predictive modeling
• Hypothesis testing for the significance of a correlation