Machine Learning for Data Science

Learning Objectives

• Distinguish supervised, unsupervised, and deep learning, and identify which fits a given problem.
• Explain regression versus classification and pick the right one for a task.
• Recognize overfitting and underfitting, and describe how to detect and fix each.
• Interpret model evaluation metrics: accuracy, precision, recall, and F1-score.
• Implement and compare Linear Regression, Decision Trees, and Support Vector Machines in Python using scikit-learn.
• Explain feature selection and engineering and why they affect model performance.

Quick Answer

Machine learning is the branch of data science where a computer improves its performance on a task by learning patterns from data, rather than following rules a programmer wrote by hand. It matters because most real-world prediction problems — will this customer churn, is this transaction fraudulent, what will tomorrow's demand be — have patterns too complex or too numerous for a human to encode as explicit rules. Instead, an algorithm is given examples (labeled or unlabeled) and finds the pattern itself. The result is a model: a mathematical object that takes new input and produces a prediction, whose quality is judged with metrics like accuracy, precision, and recall rather than by reading its code.

Overview

Machine learning sits at the modeling stage of the data science pipeline: after data has been collected, cleaned, and explored, machine learning turns that prepared data into a system that predicts or classifies. This chapter covers the vocabulary you need before touching any algorithm, then walks through three foundational algorithms — Linear Regression, Decision Trees, and Support Vector Machines — with working Python code for each.

Core Concepts

Supervised vs. Unsupervised Learning

Definition. Supervised learning trains a model on data where each example already has the correct answer attached (a "label"). Unsupervised learning trains on data with no labels, so the algorithm must find structure on its own.

Explanation. In supervised learning, you show the algorithm thousands of past examples of input → correct output pairs (house features → actual sale price), and it learns a function that maps one to the other. In unsupervised learning, there is no "correct output" to learn from — the algorithm instead groups similar data points together (clustering) or reduces the data to its most important dimensions (like PCA).

Example. Supervised: given emails labeled "spam" or "not spam," train a classifier. Unsupervised: given emails with no labels at all, group them into clusters of similar content and let a human decide what each cluster represents.

Real-world example. Netflix uses supervised learning to predict whether you'll finish a show you started (labeled by historical viewing completion), and unsupervised learning to discover natural genre-like clusters of shows that don't map to any official genre label.

Why it matters. The type of learning available depends entirely on whether labels exist — this is usually the very first design decision in any machine learning project, before any algorithm is chosen.

Common misunderstanding. Students think unsupervised learning is "worse" because it has no correct-answer to check against. In reality, unsupervised learning solves a different problem (discovering unknown structure) that supervised learning cannot solve even with labels, because there's nothing to label in the first place.

Regression vs. Classification

Definition. Regression predicts a continuous numeric value. Classification predicts a discrete category or class label.

Explanation. Both are supervised learning tasks; the difference is the type of output. "How much will this house sell for?" is regression (any dollar value is possible). "Will this email be marked as spam?" is classification (only two possible outputs: spam or not spam).

Example. Predicting a student's exact exam score (0–100) is regression. Predicting whether a student passes or fails (pass/fail) is classification.

Real-world example. A bank uses regression to estimate how much credit line to approve for an applicant, and classification to decide whether to approve the application at all.

Why it matters. Choosing the wrong task type breaks the entire pipeline — using a classifier's algorithm to predict a continuous price, or a regression algorithm to predict a category, requires different loss functions, evaluation metrics, and often different algorithms entirely.

Common misunderstanding. The name "logistic regression" leads students to assume it produces continuous output like linear regression. It is actually a classification algorithm — it estimates a probability, which is then thresholded (usually at 0.5) into a discrete class.

Overfitting and Underfitting

Definition. Overfitting occurs when a model learns the training data too precisely, including its noise, and performs poorly on new data. Underfitting occurs when a model is too simple to capture the real pattern, performing poorly on both training and new data.

Explanation. Think of fitting a curve to a scatter of points. An overly flexible model (a high-degree polynomial) can wiggle through every single point, but that wiggle is chasing noise, not signal — hand it a new point and it predicts wildly. An overly rigid model (a straight line through clearly curved data) misses the pattern even in the data it was trained on.

Example. A decision tree with unlimited depth that has a unique leaf for every training example has memorized the training set (overfitting). A depth-1 decision tree, which only asks one question, usually underfits.

Real-world example. A spam filter trained too specifically on last month's spam emails (overfitting) will fail on new spam-writing tactics next month; a spam filter that just checks for the word "free" (underfitting) will miss spam that doesn't use that word and flag legitimate promotional emails that do.

Why it matters. The gap between training performance and test performance is the single most important diagnostic a data scientist checks before trusting a model — a model that looks perfect on training data but is never tested on unseen data is a common cause of real-world failures.

Common misunderstanding. Students think "more accurate on the training set" always means "better model." A model that is 100% accurate on training data but only 60% accurate on a held-out test set is worse than one that is 90% accurate on both — the second model generalizes; the first has memorized.

Model Evaluation Metrics

Definition. Model evaluation metrics are quantitative measures — accuracy, precision, recall, and F1-score — used to judge how well a classification model performs.

Explanation.

• Accuracy = (correct predictions) / (total predictions). Simple, but misleading on imbalanced data.
• Precision = (true positives) / (true positives + false positives). "Of everything I flagged as positive, how much was actually positive?"
• Recall = (true positives) / (true positives + false negatives). "Of everything that was actually positive, how much did I catch?"
• F1-score = harmonic mean of precision and recall — a single number balancing both.

Example. In a fraud-detection dataset with 1,000 transactions where only 10 are fraud, a model that predicts "not fraud" every single time scores 99% accuracy but 0% recall — it catches zero actual fraud cases.

from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

y_true = [0, 0, 1, 1, 0, 1, 0, 0, 1, 0]
y_pred = [0, 0, 1, 0, 0, 1, 0, 0, 1, 0]

print("Accuracy:", accuracy_score(y_true, y_pred))
print("Precision:", precision_score(y_true, y_pred))
print("Recall:", recall_score(y_true, y_pred))
print("F1-score:", f1_score(y_true, y_pred))

Tracing this: there are 4 actual positives (y_true has four 1s) and the model correctly predicts 3 of them (one, at index 3, is missed). Accuracy counts all 9 correct predictions out of 10 (0.9). Precision looks only at the 3 predicted positives, all of which are correct (1.0). Recall looks at the 4 actual positives and finds only 3 were caught (0.75). F1 balances precision and recall into roughly 0.857.

Real-world example. A hospital screening test for a rare disease is tuned to maximize recall (catch every real case) even at the cost of precision (some false alarms), because missing a real case is far more costly than a follow-up test that turns out negative.

Why it matters. Choosing the wrong metric to optimize can produce a model that looks great on paper but fails at the actual goal — a fraud detector optimized purely for accuracy will miss most fraud, since fraud is rare.

Common misunderstanding. Students assume you can maximize precision and recall simultaneously without a trade-off. In practice, tightening a classification threshold to raise precision (flag fewer positives, be more sure of each) typically lowers recall (miss more true positives), and vice versa.

Feature Selection and Engineering

Definition. Feature selection is choosing which input variables to keep for a model; feature engineering is transforming raw data into new, more useful input variables.

Explanation. Not every column in a dataset helps a model, and some raw columns are more useful reshaped. Feature engineering examples: turning a "date of birth" column into "age," combining "length" and "width" into "area," or applying one-hot encoding to a categorical column.

Example. Given a raw "timestamp" column, engineering "hour_of_day" and "day_of_week" as separate features often reveals patterns (like traffic peaking at 5 PM) that the raw timestamp alone cannot express to a linear model.

Real-world example. Ride-sharing pricing models engineer features like "distance to nearest event venue" and "minutes until predicted rain" from raw GPS and weather data — none of these exist directly in the raw logs, but they are strong predictors of demand.

Why it matters. A well-engineered feature can improve model performance more than switching to a fancier algorithm — many winning solutions in machine learning competitions succeed primarily through better features, not more complex models.

Common misunderstanding. Students think adding more features always helps. Irrelevant or redundant features can add noise, increase overfitting risk, and slow training — feature selection (removing unhelpful columns) is often as important as engineering new ones.

Fundamental Algorithms

1. Linear Regression

Definition. Linear regression models the relationship between a continuous target variable and one or more input features as a weighted sum plus an error term.

Explanation. The model is represented as:

y = β₀ + β₁x₁ + β₂x₂ + ... + βₖxₖ + ε

where y is the predicted value, β₀ is the intercept, β₁...βₖ are learned coefficients (weights) for each input feature x₁...xₖ, and ε is irreducible error. Training means finding the coefficients that minimize the total squared difference between predicted and actual y values (least squares).

Example.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

# Sample data: single feature, linear relationship with noise
rng = np.random.default_rng(42)
X = np.arange(1, 51).reshape(-1, 1)
y = 3 * X.flatten() + 7 + rng.normal(0, 5, size=50)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = LinearRegression()
model.fit(X_train, y_train)

y_pred = model.predict(X_test)
print("Coefficient:", model.coef_[0], "Intercept:", model.intercept_)
print("R-squared:", r2_score(y_test, y_pred))
print("MSE:", mean_squared_error(y_test, y_pred))

plt.scatter(X, y, color='blue', label='Data points')
plt.plot(X_test, y_pred, color='red', label='Regression line')
plt.xlabel('Input Feature')
plt.ylabel('Target Variable')
plt.title('Linear Regression Example')
plt.legend()
plt.show()

Tracing this: model.fit() finds the coefficient and intercept that minimize squared error on the 40 training points. model.coef_[0] should come out close to 3 and model.intercept_ close to 7, matching the true relationship used to generate the data (noise keeps it from being exact). r2_score reports what fraction of the variance in y_test is explained by the model — closer to 1 is better.

Real-world example. Real estate platforms use linear regression as a fast, interpretable baseline to estimate home value from square footage, bedroom count, and location — the coefficients directly tell you "each additional bedroom adds about $X to the estimated price."

Why it matters. Linear regression is simple, fast to train, and directly interpretable — each coefficient has a clear meaning — which makes it the standard first model to try before reaching for anything more complex.

Common misunderstanding. Students think linear regression can only fit a straight line, so it's "useless" for curved relationships. Polynomial features (x², x³, etc.) can be added as new input columns, letting the same linear-in-parameters model fit curves.

2. Decision Trees

Definition. A decision tree is a non-parametric supervised learning method that predicts a target by repeatedly splitting the data based on feature-value questions, forming a tree of decisions.

Explanation. Each internal node asks a question like "is petal length < 2.5 cm?" and routes the example left or right based on the answer. The tree keeps splitting (choosing the question that best separates the classes at each step) until it reaches a stopping condition, and the leaf a data point lands in gives its predicted class or value.

Example.

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn import tree
import matplotlib.pyplot as plt

# Load dataset
iris = load_iris()
X = iris.data
y = iris.target

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# max_depth limits tree size to reduce overfitting
dt_model = DecisionTreeClassifier(max_depth=3, random_state=42)
dt_model.fit(X_train, y_train)

y_pred = dt_model.predict(X_test)
print("Test accuracy:", accuracy_score(y_test, y_pred))

plt.figure(figsize=(10, 8))
tree.plot_tree(dt_model, feature_names=iris.feature_names,
                class_names=iris.target_names, filled=True)
plt.title('Decision Tree Visualization')
plt.show()

Tracing this: max_depth=3 caps the tree at 3 levels of questions, which is a direct defense against overfitting — an unlimited-depth tree on this small dataset could create a leaf for nearly every training example. tree.plot_tree visualizes exactly which feature and threshold each node split on, which is why decision trees are called "easy to interpret" — you can read the entire decision logic off the diagram.

Real-world example. Loan approval systems often use decision trees (or their ensemble form, Random Forest) specifically because regulators require an explainable reason for rejecting an application — "income below $X and credit history under Y years" is a rule a human can audit, unlike a neural network's internal weights.

Why it matters. Decision trees handle non-linear relationships and mixed data types (numeric and categorical) without requiring feature scaling, and their splits are directly human-readable — a rare combination of flexibility and interpretability.

Common misunderstanding. Students believe a single decision tree is a top-performing algorithm because it's commonly taught first. In practice, single decision trees overfit easily and are usually outperformed by ensembles of many trees (Random Forest, Gradient Boosting), which average out each tree's individual mistakes.

3. Support Vector Machines (SVM)

Definition. A Support Vector Machine is a classification algorithm that finds the hyperplane (decision boundary) that maximizes the margin — the distance — between the closest points of each class.

Explanation. Rather than just finding any line that separates two classes, SVM finds the one with the widest possible "street" between the classes, which tends to generalize better to new data. The points closest to this boundary (the ones that "support" it) are called support vectors. For data that isn't linearly separable, SVM can use a "kernel trick" to implicitly map data into a higher-dimensional space where a linear separator does exist.

Example.

from sklearn import datasets
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
import matplotlib.pyplot as plt

# Load dataset
iris = datasets.load_iris()
X = iris.data[:, :2]  # first two features, for 2D visualization
y = iris.target

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

svm_model = SVC(kernel='linear')
svm_model.fit(X_train, y_train)

y_pred = svm_model.predict(X_test)
print("Test accuracy:", accuracy_score(y_test, y_pred))

plt.scatter(X_train[:, 0], X_train[:, 1], c=y_train, s=30, cmap='autumn', label='Train')
plt.scatter(X_test[:, 0], X_test[:, 1], c='blue', s=30, marker='x', label='Test')
plt.title('SVM Decision Boundary Visualization')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.legend()
plt.show()

Tracing this: kernel='linear' tells SVM to find a straight-line (or flat hyperplane) boundary rather than a curved one. Using only the first two of the iris dataset's four features (X[:, :2]) sacrifices some accuracy but makes the decision boundary plottable in 2D — with all four features, the same idea applies in four-dimensional space, which cannot be drawn directly.

Real-world example. Early handwriting and digit-recognition systems (like postal code readers) relied heavily on SVMs with non-linear kernels because they performed well on high-dimensional pixel data with relatively small training sets, before deep learning became practical for that task.

Why it matters. SVMs remain effective even when the number of features is larger than the number of training samples, a situation where many other algorithms struggle — making them useful in domains like genomics, where you might have thousands of gene features but only hundreds of patient samples.

Common misunderstanding. Students assume SVM only works for linearly separable data because the simplest examples use kernel='linear'. The kernel trick (e.g., kernel='rbf') lets SVM handle complex, non-linear boundaries by implicitly operating in a transformed feature space.

Visual Learning

Key Terms

Term	Definition
Supervised Learning	Learning from labeled input-output pairs to predict outputs for new inputs
Unsupervised Learning	Learning patterns or groupings from unlabeled data
Overfitting	A model matches training data (including its noise) too closely and fails to generalize to new data
Underfitting	A model is too simple to capture the real pattern in the data, performing poorly everywhere
Precision	Of all instances predicted positive, the fraction that are actually positive
Recall	Of all instances that are actually positive, the fraction the model correctly identified
F1-score	The harmonic mean of precision and recall, balancing both into one number
Hyperplane	The decision boundary an SVM finds to separate classes with maximum margin
Feature Engineering	Creating new, more informative input variables from raw data

Common Mistakes

1. Misconception: "A model with high training accuracy is a good model." Why it's wrong: High training accuracy can simply mean the model has memorized the training data (overfitting), which tells you nothing about how it will perform on data it hasn't seen. Correct explanation: Always evaluate on a separate held-out test set (or via cross-validation); a good model performs comparably well on both training and test data.

2. Misconception: "Logistic regression is a type of regression that predicts continuous numbers, just like linear regression." Why it's wrong: Despite the shared name, logistic regression outputs a probability that is thresholded into a discrete class — it is a classification algorithm, not a regression algorithm in the predictive sense. Correct explanation: Use linear regression for continuous targets (price, temperature) and logistic regression for binary classification tasks (spam/not spam, fraud/not fraud).

3. Misconception: "Accuracy is always the right metric to judge a classifier." Why it's wrong: On imbalanced datasets (e.g., 1% fraud rate), a model that always predicts the majority class scores very high accuracy while being completely useless at the actual task. Correct explanation: Use precision, recall, and F1-score (or a confusion matrix) alongside accuracy, especially whenever one class is much rarer than the other.

Comparison and Connections

Algorithm	Task Type	Interpretability	Handles Non-Linear Data	Typical Use Case
Linear Regression	Regression	High (coefficients are directly meaningful)	No (unless features engineered)	Price prediction, trend estimation
Decision Tree	Classification/Regression	Very high (visual, rule-based)	Yes	Loan approval, medical diagnosis rules
SVM	Classification (mainly)	Low to medium	Yes, with kernel trick	High-dimensional data, image/text classification
K-Means (unsupervised)	Clustering	Medium	Depends on distance metric	Customer segmentation

Practice Questions

Recall

1. What is the key difference between supervised and unsupervised learning? Answer guidance: Supervised learning uses labeled data (input-output pairs); unsupervised learning uses unlabeled data and finds structure on its own.
2. List the four model evaluation metrics discussed for classification and what each one measures. Answer guidance: Accuracy (overall correctness), precision (correctness of positive predictions), recall (coverage of actual positives), F1-score (balance of precision and recall).

Understanding

3. Explain why a decision tree with unlimited depth is prone to overfitting. Answer guidance: Should explain that an unrestricted tree keeps splitting until leaves may contain a single training example, effectively memorizing the training set including its noise, rather than learning a generalizable pattern.
4. Why can a model have 99% accuracy on a fraud dataset and still be a poor model? Answer guidance: Should identify class imbalance — if fraud is rare (e.g., 1% of transactions), predicting "not fraud" always achieves 99% accuracy while catching zero actual fraud, so recall on the minority class must be checked.

Application

5. You need to predict whether a bank loan applicant will default (yes/no) using income, credit score, and loan amount. Which of Linear Regression, Decision Tree, or SVM would you start with, and how would you evaluate it? Answer guidance: Should recognize this as a classification task (not regression, since output is yes/no), suggest a Decision Tree for interpretability (regulatory need) or SVM/logistic regression as alternatives, and propose evaluating with precision/recall/F1 given likely class imbalance (most applicants don't default).
6. Write the scikit-learn code to train a Decision Tree classifier with a maximum depth of 4 on a dataset X, y, and explain why you set that limit. Answer guidance: Should produce code using DecisionTreeClassifier(max_depth=4).fit(X, y) and explain that limiting depth is a regularization technique to reduce overfitting risk.

Analysis

7. Compare Decision Trees and SVMs for a dataset with 10,000 features but only 200 training samples (e.g., gene expression data). Which is more appropriate and why? Answer guidance: Should favor SVM, noting it remains effective when features outnumber samples, while decision trees tend to overfit badly with so many features relative to so few examples.
8. A student trains a linear regression model on housing data and gets a very high R-squared on the training set but a much lower R-squared on the test set. Diagnose the likely problem and propose two fixes. Answer guidance: Should diagnose overfitting (possibly from too many engineered/polynomial features relative to data size) and propose fixes such as regularization (Ridge/Lasso), reducing feature count, or gathering more training data.

FAQ

Is machine learning the same thing as artificial intelligence? No. Artificial intelligence is the broader goal of building systems that perform tasks requiring intelligence. Machine learning is one approach to achieving that goal — specifically, learning patterns from data rather than hand-coding rules. Deep learning is, in turn, a subset of machine learning using neural networks.

Why does scikit-learn split data into training and test sets? Because evaluating a model on the same data it was trained on tells you how well it memorized the data, not how well it will perform on new, unseen data. The test set simulates "new data" the model has never encountered, giving a more honest performance estimate.

Do I need to scale my features before using a Decision Tree? No. Decision trees split based on threshold comparisons on one feature at a time, so the scale of features doesn't affect the splits. Feature scaling matters for algorithms like SVM, K-Means, or KNN that rely on distance calculations across multiple features at once.

Which is better: Decision Trees or SVM? Neither is universally better — it depends on the data and priorities. Decision trees are more interpretable and handle mixed data types well; SVMs often perform better on high-dimensional data but are harder to interpret and require careful kernel/parameter tuning. Many practitioners try both and compare validation performance.

How do I know if my model is overfitting? Compare performance on the training set versus a held-out test set (or use k-fold cross-validation). A large gap — for example, 98% training accuracy but 70% test accuracy — is a clear sign of overfitting.

Quick Revision

• Supervised learning uses labeled data; unsupervised learning finds structure in unlabeled data.
• Regression predicts continuous values; classification predicts discrete categories (logistic regression is classification despite its name).
• Overfitting: too well-fit to training noise, poor generalization. Underfitting: too simple, poor performance everywhere.
• Accuracy, precision, recall, and F1-score each answer a different question — check all of them, especially on imbalanced data.
• Feature engineering (creating better inputs) often improves performance more than switching algorithms.
• Linear Regression: y = β₀ + β₁x₁ + ... + βₖxₖ + ε, minimizes squared error, highly interpretable.
• Decision Trees split data via feature-threshold questions; limiting max_depth reduces overfitting.
• SVM finds the maximum-margin hyperplane; the kernel trick lets it handle non-linear boundaries.
• SVMs remain effective even when features outnumber samples; decision trees can overfit in that scenario.
• Always evaluate models on a held-out test set, never solely on training data.
• Precision and recall usually trade off against each other as a classification threshold changes.
• Decision trees don't require feature scaling; SVM, K-Means, and KNN do.

Related Topics

Prerequisites: Data Analytics using Python and R (data cleaning, EDA, and the analytics pipeline), basic statistics (mean, variance, probability), Python programming fundamentals.

Related Topics: Linear Algebra and Probability for Computer Science (mathematical foundation for model training), Data Structures and Algorithms (efficiency of training and prediction), Statistical Modeling (regression and inference concepts shared with ML).

Next Topics: Deep Learning and Neural Networks, Ensemble Methods (Random Forest, Gradient Boosting), Model Deployment and MLOps for putting trained models into production.