Machine Learning Fundamentals

Learning Objectives

• Define machine learning and explain what problem it solves that traditional programming cannot.
• Trace the historical milestones that shaped modern ML.
• Distinguish supervised, unsupervised, and reinforcement learning by their data and goals.
• Explain how linear regression works, including its equation and cost function.
• Implement and interpret a simple linear regression model in Python.
• Identify the limitations of linear regression and when to use other algorithms instead.

Quick Answer

Machine learning is a subset of AI where a system learns a mapping from inputs to outputs by analyzing data, rather than following rules a programmer wrote by hand. It matters because many real-world problems — predicting prices, recognizing images, detecting fraud — are too complex or too variable to describe with explicit rules, but they do have consistent statistical patterns that a model can learn from examples. ML splits broadly into three types: supervised learning (learning from labeled examples), unsupervised learning (finding structure in unlabeled data), and reinforcement learning (learning through trial-and-error rewards). Linear regression, one of the simplest supervised algorithms, illustrates the core ML workflow: define a model, measure its error, and adjust it to reduce that error.

Overview

Traditional programming is: input + rules → output. You write the rules. Machine learning inverts this: input + output examples → the algorithm figures out the rules (the model). This matters because for many tasks — like predicting whether a loan applicant will default — nobody can write a complete, accurate rule; but with enough historical examples, an algorithm can discover the statistical relationship on its own.

ML's practical rise took decades. Early theoretical ideas from the 1950s (Turing's "learning machines," Rosenblatt's perceptron) showed promise but were limited by weak computers and small datasets. The 1980s brought backpropagation, a practical way to train multi-layer networks, but it wasn't until the 1990s–2000s, when computing power and data availability exploded, that ML became genuinely useful at scale. Today's ML boom rests on that same idea — learn from data — combined with vastly more data and computing power.

Core Concepts

The Three Types of Machine Learning

Definition: Supervised learning trains models on labeled data (input-output pairs); unsupervised learning finds structure in unlabeled data; reinforcement learning trains agents to choose actions that maximize a reward.

Explanation: The type of learning depends on what data you have and what question you're asking. If you have historical examples with known correct answers (house prices, past loan outcomes), supervised learning fits. If you only have raw data with no known "correct" answer (customer purchase records with no predefined groups), unsupervised learning finds hidden structure like clusters. If you're building a system that must make sequential decisions and learn from the outcomes of those decisions (game-playing, robotics), reinforcement learning applies.

Example: Given emails labeled "spam" or "not spam," a supervised algorithm learns the boundary between the two. Given emails with no labels at all, an unsupervised algorithm might still group similar emails together based on shared vocabulary, without ever calling any group "spam."

Real-World Example: Netflix's recommendation engine uses a mix: it uses supervised techniques to predict how much you'll like a movie based on past ratings, and unsupervised clustering to group viewers with similar tastes.

Why It Matters: Choosing the wrong learning type wastes effort — you can't apply supervised learning if you have no labels, and you shouldn't ignore available labels by using an unsupervised method when a more accurate supervised one is possible.

Common Misunderstanding: Students often think unsupervised learning is "worse" because it lacks labels. It isn't worse — it solves a different problem (discovering structure) that supervised learning cannot solve because supervised learning requires labels that frequently don't exist or are expensive to obtain.

Linear Regression

Definition: Linear regression is a supervised learning algorithm that models the relationship between one or more input features and a continuous output as a straight line (or hyperplane in higher dimensions).

Explanation: The model assumes the output y can be approximated as y = mx + b for a single feature (or y = w₁x₁ + w₂x₂ + ... + b for multiple features). Training means finding the values of m (slope/weights) and b (intercept) that minimize the total error between the model's predictions and the actual values in the training data — typically measured using mean squared error. This search is usually performed with an algorithm like gradient descent, which nudges the parameters in the direction that reduces error, or with a closed-form solution for simple cases.

Example: With data points (1,1), (2,2), (3,1.3), (4,3.75), (5,2.25), linear regression finds the single straight line that minimizes the sum of squared vertical distances from each point to the line — even though no line passes through all the points exactly.

Real-World Example: Real estate platforms use linear regression (or its more advanced relatives) to estimate a home's value from features like square footage, number of bedrooms, and location, based on patterns learned from past sales.

Why It Matters: Linear regression is the simplest possible ML model, and understanding it — the idea of a loss function and minimizing error — is the conceptual foundation for almost every other ML algorithm, including neural networks.

Common Misunderstanding: Students often think linear regression can model any relationship if you just add enough features. In reality, plain linear regression can only capture linear relationships between features and the output; genuinely curved or interacting relationships require polynomial features, other algorithms, or neural networks.

Visual Learning

This diagram shows how the presence or absence of labeled data determines whether a problem is supervised or unsupervised, while reinforcement learning follows a separate feedback loop driven by rewards rather than static labels.

Real-World Applications

• E-commerce: Regression models predict expected demand for products, helping companies manage inventory.
• Insurance: Regression-based risk models estimate the likely cost of insuring a customer based on historical claims data.
• Marketing: Businesses use ML to predict customer lifetime value and target advertising more efficiently.
• Scientific research: Researchers use regression to quantify relationships between variables, such as how a drug dosage affects patient recovery time.

Professionals rely on ML fundamentals like regression because they provide interpretable, fast-to-train baselines: before reaching for a complex deep learning model, most practitioners try a simple algorithm like linear regression first to establish a benchmark.

Common Mistakes

1. Misconception: Machine learning is a recent invention from the 2010s. Why it's wrong: The theoretical foundations date back to the 1950s (Turing, Rosenblatt's perceptron); what changed recently is the availability of data and computing power that made these old ideas practical. Correct explanation: ML is decades old conceptually; the "AI boom" reflects an explosion in data and compute, not the invention of new fundamental theory.

2. Misconception: Linear regression is outdated and rarely used compared to neural networks. Why it's wrong: For problems where the relationship between features and output really is roughly linear, linear regression is faster to train, easier to interpret, and often just as accurate as a much more complex model. Correct explanation: Algorithm choice should match the data and problem — simple algorithms remain the right choice whenever they perform adequately, and complexity should only be added when it demonstrably improves results.

3. Misconception: A model's job is done once it fits the training data well. Why it's wrong: A model that fits the training data perfectly may have simply memorized noise rather than learned the actual underlying relationship, leading to poor performance on new data. Correct explanation: ML practice always requires evaluating a model on separate test data to check that its learned pattern generalizes.

Comparison and Connections

Type	Data Required	Goal	Example Algorithms
Supervised Learning	Labeled input-output pairs	Predict outputs for new inputs	Linear regression, decision trees, SVM
Unsupervised Learning	Unlabeled data	Discover hidden structure	K-means, PCA, GMM
Reinforcement Learning	Reward signal from environment	Learn a policy that maximizes reward	Q-learning, policy gradients

Practice Questions

Recall

1. What are the three main types of machine learning? Answer guidance: Supervised, unsupervised, and reinforcement learning.
2. What equation does linear regression use to model a relationship between one input feature and an output? Answer guidance: y = mx + b, where m is the slope and b is the intercept.

Understanding

3. Explain why traditional programming struggles with tasks like predicting house prices, while ML handles it well. Answer guidance: House prices depend on many interacting factors with no single fixed rule; ML can learn the statistical relationship between features and price directly from historical sales data instead of requiring an explicit formula.
4. Why does linear regression need a loss function like mean squared error? Answer guidance: The loss function quantifies how far off the model's predictions are from the actual values, giving the training algorithm a target to minimize when adjusting the model's parameters.

Application

5. A company has historical data on customer churn (whether a customer left) along with usage statistics, but no clear rule for who churns. Which ML type should they use, and why? Answer guidance: Supervised learning (classification), since churn is a labeled outcome (churned/not churned) they want to predict from feature data.
6. Given the code in this page, describe step by step what happens when model.fit(X, y) is called on a LinearRegression object. Answer guidance: The model computes the slope and intercept that minimize the mean squared error between the predicted line and the actual y values in the training data, typically via a closed-form or iterative optimization method.

Analysis

7. Compare using linear regression versus a decision tree to predict house prices. What factors would influence your choice? Answer guidance: Linear regression assumes a straight-line relationship and is more interpretable and less prone to overfitting on small data, while decision trees can capture non-linear relationships and interactions but risk overfitting without proper tuning; choice depends on data size, whether relationships are linear, and interpretability needs.
8. A student trains a linear regression model and reports it fits all five training points almost perfectly, then claims the model will work well on new house price data. What is missing from this evaluation? Answer guidance: The claim ignores generalization — the model needs to be tested on a separate holdout/test set not used during training, since fitting training data well doesn't guarantee similarly low error on unseen data.

FAQ

How is machine learning different from statistics? They overlap heavily — both study patterns in data — but ML tends to emphasize predictive accuracy on new data and often uses more complex, less interpretable models, while classical statistics emphasizes interpretability and formal inference about relationships.

Why do we split data into training and testing sets? To estimate how well the model will perform on data it hasn't seen. Training accuracy alone can be misleading because a model can memorize training examples without learning a pattern that generalizes.

Is linear regression only used for a single input feature? No. Linear regression easily extends to multiple input features (multiple linear regression), where the model learns a separate weight for each feature.

What's the difference between the "slope" in high school math and the weights in ML? They're conceptually the same thing — a weight tells you how much the output changes per unit change in an input feature — but ML models often have many weights (one per feature) rather than just one.

Do I need to memorize the math behind gradient descent to use ML? Not to get started — libraries like scikit-learn handle the optimization for you. But understanding what gradient descent is doing (iteratively reducing error) helps you debug models that fail to train properly.

Quick Revision

• ML: systems that learn patterns from data instead of following explicit rules.
• Three types: supervised (labeled data), unsupervised (unlabeled data), reinforcement (reward-driven).
• Key milestones: 1950s theory (Turing, Rosenblatt), 1980s backpropagation, 1990s-2000s scale-up with data and compute.
• Linear regression models y = mx + b, minimizing prediction error (commonly mean squared error).
• Dependent variable: what you're predicting; independent variable(s): the inputs used to predict it.
• Training a model means adjusting its parameters to reduce a loss function.
• Linear regression can only capture linear relationships — non-linear patterns need other models.
• Always evaluate models on a separate test set, not just training data.
• Simple models like linear regression remain useful as fast, interpretable baselines.
• More complex algorithms (decision trees, SVMs, neural networks) build on the same core idea: minimize error on data.

Related Topics

Prerequisites: Introduction to AI, basic algebra and statistics, basic Python.

Related Topics: Supervised Learning, Unsupervised Learning.

Next Topics: Supervised Learning, Reinforcement Learning.