Reinforcement Learning

Learning Objectives

• Define reinforcement learning and explain how it differs from supervised and unsupervised learning.
• Identify the core components of an RL system: agent, environment, state, action, reward, and policy.
• Trace how the exploration-exploitation trade-off shapes agent behavior.
• Compare episodic, continuous, and partially observable RL problems.
• Explain how Q-learning updates its value estimates using a worked numeric example.
• Identify real-world domains where RL is used and why RL fits those problems better than supervised learning.

Quick Answer

Reinforcement learning (RL) is a branch of machine learning where an agent learns to make decisions by interacting with an environment and receiving reward or penalty feedback, rather than learning from labeled examples. The agent tries actions, observes the resulting state and reward, and gradually updates a policy (a strategy for choosing actions) to maximize its cumulative reward over time. RL matters because many real problems, such as playing a game, controlling a robot, or optimizing an ad campaign, have no "correct answer" dataset to learn from — only a way to measure whether an outcome was good or bad. RL powers systems like AlphaGo, robotic control, and recommendation engines that improve through trial and error rather than static training sets.

Overview

Supervised learning needs a labeled dataset of correct answers. Reinforcement learning needs something different: an environment the agent can act in, and a reward signal that tells it how well it's doing. There's no teacher showing the "right" move at every step — the agent discovers good behavior by trying actions and seeing what pays off, sometimes only much later.

This is closer to how humans and animals actually learn many skills. A child learns to ride a bicycle not by studying labeled examples of "correct handlebar angle" but by trying, wobbling, falling, and adjusting based on the outcome (staying upright or falling over). RL formalizes this trial-and-error process mathematically: an agent observes a state, picks an action, receives a reward and a new state, and repeats — gradually learning which actions lead to the best long-term outcomes.

The central tension in RL is exploration vs. exploitation: should the agent try a new, untested action that might turn out better (explore), or stick with the action it already knows works well (exploit)? Get this balance wrong and the agent either never discovers the best strategy or wastes too much time experimenting.

Core Concepts

The Agent-Environment Loop

Definition: RL is built around a loop where an agent observes the current state of an environment, takes an action, and receives a reward plus a new state in return.

Explanation: At each time step t, the agent is in state S_t. It selects an action A_t according to its policy. The environment responds with a reward R_{t+1} and a new state S_{t+1}. This repeats until the task ends (an episode finishes) or indefinitely (continuous tasks). The agent's only goal is to choose actions that maximize the sum of rewards it collects over time, not just the immediate reward.

Example: In a maze-solving agent, the state is the agent's current cell, the action is a direction to move, and the reward might be -1 per step (encouraging speed) and +100 for reaching the exit.

Real-World Example: A thermostat-optimizing RL system observes room temperature and occupancy (state), decides whether to heat or cool (action), and receives a reward that balances comfort against energy cost.

Why It Matters: This loop is the same regardless of whether the agent is playing chess, trading stocks, or controlling a robot arm — recognizing this common structure lets you apply the same algorithms across very different problems.

Common Misunderstanding: Students often think the agent is rewarded for every individual action being "correct." In reality, RL rewards are often delayed and sparse — an action might only pay off many steps later, which is why the "credit assignment" problem (deciding which past action deserves credit for a later reward) is a core challenge.

Policy, Value Function, and Reward

Definition: The policy is the agent's strategy — a mapping from states to actions. The value function estimates how much cumulative reward the agent can expect from a state (or state-action pair) if it follows its policy from there onward. The reward is the immediate numerical feedback received after an action.

Explanation: Rewards are short-term signals; value functions look further ahead. A state can have a low immediate reward but a high value if it leads to great rewards later (like sacrificing a chess piece to win the game). Learning algorithms use estimated values to improve the policy: the agent gradually shifts toward actions that lead to higher-value states.

Example: In Q-learning, the value function is represented as a table Q(s, a) — the expected cumulative reward of taking action a in state s and then acting optimally afterward.

Real-World Example: In a stock-trading RL agent, a single "hold" action might have zero immediate reward but a high estimated value if the model predicts the stock will rise significantly.

Why It Matters: Separating short-term reward from long-term value is what allows RL agents to make sacrifices now for bigger payoffs later — something a purely greedy, reward-maximizing-per-step agent could never do.

Common Misunderstanding: Students often assume a higher reward at every step is always better. An agent that always takes the action with the highest immediate reward can get stuck in a local optimum and completely miss a better long-term strategy — this is exactly why value functions, not raw rewards, drive good policies.

Exploration vs. Exploitation

Definition: Exploration means trying new or uncertain actions to gather more information about the environment; exploitation means choosing the action currently believed to be best.

Explanation: Early in training, an agent knows little about the environment, so pure exploitation would lock it into a possibly mediocre strategy discovered by chance. Pure exploration, on the other hand, never uses what's been learned and wastes reward on random behavior. Most algorithms balance the two, commonly with an epsilon-greedy strategy: choose a random action with probability ε, otherwise choose the current best-known action. ε typically starts high and decays as the agent gains experience.

Example: An epsilon-greedy Q-learning agent with ε = 0.2 picks a random action 20% of the time and its best-known action 80% of the time, gradually lowering ε as training progresses.

Real-World Example: A news recommendation system must occasionally show articles it isn't confident about (exploration) instead of only ever showing the currently most-clicked article (exploitation), or it will never discover new content users would actually prefer.

Why It Matters: Getting this trade-off right determines whether an agent converges to a genuinely good strategy or settles prematurely on a mediocre one.

Common Misunderstanding: Students often think exploration is simply "wasted" effort compared to exploitation. Without enough exploration, an agent can converge confidently on a suboptimal policy simply because it never tried the better action enough times to find out it was better.

Visual Learning

This diagram shows the core RL feedback loop: the agent chooses an action based on its current policy, the environment returns a reward and a new state, and the agent uses that feedback to refine its policy before choosing the next action. This cycle repeats thousands or millions of times during training.

Worked Example: Q-Learning Update

Q-learning maintains a table Q(s, a) estimating the value of taking action a in state s. After each step, it updates that estimate using the Bellman equation:

Q(s, a) ← Q(s, a) + α [ r + γ · max_a' Q(s', a') − Q(s, a) ]

Where α is the learning rate, γ is the discount factor (how much future rewards matter), r is the reward just received, and s' is the new state.

Concrete trace: Suppose Q(s, a) = 2.0, the agent takes action a, receives reward r = 1, moves to state s', where the best next action has Q(s', a') = 5.0. With α = 0.1 and γ = 0.9:

Q(s, a) ← 2.0 + 0.1 × [1 + 0.9 × 5.0 − 2.0]
        ← 2.0 + 0.1 × [1 + 4.5 − 2.0]
        ← 2.0 + 0.1 × 3.5
        ← 2.35

The value estimate for taking action a in state s nudges upward from 2.0 to 2.35, because the actual outcome (reward plus the best future value) was better than what the table currently believed. Repeating this update across many episodes gradually makes Q(s, a) converge to accurate estimates of long-term value.

import numpy as np
import random

class GridWorld:
    """A 5x5 grid where the agent must navigate from (0,0) to (4,4)."""
    def __init__(self, size=(5, 5), start=(0, 0), goal=(4, 4)):
        self.size, self.start, self.goal = size, start, goal
        self.state = start

    def reset(self):
        self.state = self.start
        return self.state

    def step(self, action):
        r, c = self.state
        moves = {0: (r - 1, c), 1: (r + 1, c), 2: (r, c - 1), 3: (r, c + 1)}
        nr, nc = moves[action]
        nr, nc = max(0, min(nr, self.size[0] - 1)), max(0, min(nc, self.size[1] - 1))
        self.state = (nr, nc)
        reward = 1.0 if self.state == self.goal else -0.01
        done = self.state == self.goal
        return self.state, reward, done

def q_learning(grid, episodes=500, alpha=0.1, gamma=0.95):
    q_table = np.zeros((*grid.size, 4))
    epsilon, min_epsilon, decay = 1.0, 0.1, 0.99
    for _ in range(episodes):
        state, done = grid.reset(), False
        while not done:
            action = random.randint(0, 3) if random.random() < epsilon else np.argmax(q_table[state])
            next_state, reward, done = grid.step(action)
            best_next = np.max(q_table[next_state])
            q_table[state][action] += alpha * (reward + gamma * best_next - q_table[state][action])
            state = next_state
        epsilon = max(min_epsilon, epsilon * decay)
    return q_table

This is the same Bellman update from the worked trace above, applied automatically at every step of every episode until the Q-table converges on a good policy for reaching the goal.

Types of Reinforcement Learning Problems

• Episodic tasks: Each run has a clear start and terminal state, such as one game of Pac-Man or Mario Kart. Learning happens across many independent episodes.
• Continuous (continuing) tasks: The agent acts indefinitely with no natural end, such as a robotic arm running on a factory line or a thermostat controller that never "finishes."
• Partially Observable MDPs (POMDPs): The agent cannot see the full state of the environment, only partial observations, such as a robot that only sees what's directly in front of it rather than the entire building layout.

Common Algorithms

Algorithm	Type	Key Idea
Q-learning	Value-based, off-policy	Learns Q(s,a) using the Bellman equation regardless of the policy being followed
SARSA	Value-based, on-policy	Updates Q(s,a) using the action actually taken next, not the best possible one
Deep Q-Networks (DQN)	Value-based, deep RL	Uses a neural network to approximate Q(s,a) for large/continuous state spaces
Policy Gradient Methods	Policy-based	Directly optimizes the policy's parameters to maximize expected reward
Actor-Critic	Hybrid	Combines a policy ("actor") with a value function ("critic") that evaluates the actor's choices

Real-World Applications

• Robotics: Robotic arms and legged robots learn manipulation and locomotion skills through repeated trial and error in simulation before deploying on real hardware.
• Game playing: DeepMind's AlphaGo and AlphaZero used RL (combined with deep learning) to reach superhuman performance in Go, chess, and shogi by playing millions of self-play games.
• Finance: Portfolio management systems use RL to decide how to allocate assets, treating market returns as the reward signal.
• Recommendation systems: Streaming and e-commerce platforms increasingly use RL to balance showing users content they'll likely enjoy now against exploring new content that might improve long-term engagement.

RL is chosen over supervised learning in these domains because there's no fixed "correct" dataset — the right action depends on a changing environment and long-term consequences that only reveal themselves through interaction.

Common Mistakes

1. Misconception: Reinforcement learning requires a labeled dataset like supervised learning does. Why it's wrong: RL agents learn from reward signals generated by interacting with an environment, not from a fixed set of labeled input-output pairs prepared in advance. Correct explanation: The "training data" in RL is generated on the fly through the agent's own experience — sequences of (state, action, reward, next state) tuples collected during exploration.

2. Misconception: An RL agent should always choose the action with the highest immediate reward. Why it's wrong: Greedy, reward-maximizing-per-step behavior ignores long-term consequences and can trap the agent in a poor overall strategy. Correct explanation: Good RL agents optimize cumulative (often discounted) future reward, using value functions to weigh actions that may have low immediate payoff but lead to much better outcomes later.

3. Misconception: More exploration is always better because it helps the agent learn more. Why it's wrong: Excessive exploration wastes time and reward on actions already known to be poor, slowing convergence to a good policy. Correct explanation: Effective RL balances exploration and exploitation, typically exploring heavily early in training and shifting toward exploitation as the agent's value estimates become more reliable (e.g., decaying epsilon in epsilon-greedy).

Comparison and Connections

Approach	Learns From	Feedback Signal	Typical Use Case
Supervised learning	Labeled input-output pairs	Correct answer for each example	Classification, regression on fixed datasets
Unsupervised learning	Unlabeled data	None (finds structure)	Clustering, dimensionality reduction
Reinforcement learning	Interaction with an environment	Delayed, scalar reward	Sequential decision-making, control, games
Q-learning vs. SARSA	Same value-based framework	Q-learning uses the best possible next action; SARSA uses the action actually taken	Off-policy vs. on-policy control problems

Practice Questions

Recall

1. Name the five core components of the RL agent-environment loop. Answer guidance: Agent, environment, state, action, reward (policy and value function are also acceptable additions).
2. What does the discount factor γ control in the Bellman equation? Answer guidance: How much the agent values future rewards relative to immediate ones; γ close to 1 means future rewards matter almost as much as immediate ones, while γ close to 0 makes the agent short-sighted.

Understanding

3. Explain why RL doesn't need labeled training data the way supervised learning does. Answer guidance: The agent generates its own experience by acting in the environment and receiving reward signals, rather than learning from pre-labeled correct answers.
4. Why is the exploration-exploitation trade-off necessary, and what happens if an agent explores too little? Answer guidance: Too little exploration means the agent may never discover a better action than the first mediocre one it found, converging on a suboptimal policy; the trade-off balances gathering new information against using what's already known.

Application

5. You're designing an RL agent to manage a warehouse robot that must find the fastest path to pick items without a map given in advance. Which RL problem type applies (episodic, continuous, or POMDP), and why? Answer guidance: Likely a POMDP if the robot can't see the full warehouse layout, and episodic if each pick-and-return trip is treated as a separate episode; both can apply simultaneously.
6. A game-playing agent has Q(s, a) = 3.0, receives a reward of 2, moves to a state where the best next Q-value is 4.0, with α = 0.5 and γ = 0.9. Calculate the updated Q(s, a). Answer guidance: Q(s,a) ← 3.0 + 0.5 × [2 + 0.9×4.0 − 3.0] = 3.0 + 0.5 × [2 + 3.6 − 3.0] = 3.0 + 0.5×2.6 = 4.3.

Analysis

7. Compare Q-learning and SARSA in terms of how they would behave differently near a dangerous state (e.g., a cliff edge with a large penalty). Answer guidance: Q-learning (off-policy) assumes the agent will act optimally afterward, so it learns the value of the shortest, riskiest path; SARSA (on-policy) accounts for the actual exploration policy, including occasional random risky moves, so it tends to learn a safer, more cautious path near danger.
8. A student claims their RL agent isn't learning because its reward per episode isn't increasing after 10 training episodes. Evaluate this claim. Answer guidance: Ten episodes is usually far too few for value estimates to converge, especially with a high initial exploration rate; the claim is premature — reward trends should be evaluated over hundreds or thousands of episodes, along with checking that the exploration rate is decaying and hyperparameters (α, γ) are reasonable.

FAQ

Is reinforcement learning the same as deep learning? No. RL is a learning paradigm defined by learning through interaction and reward; deep learning is a technique for building models with many-layered neural networks. They combine in "deep RL" (like DQN) when neural networks are used to approximate value functions or policies for large or continuous state spaces.

Why does RL sometimes take so long to train? Because the agent must discover good behavior largely through trial and error, and rewards can be sparse or delayed, it may need to try millions of actions before learning which sequences lead to good outcomes — this is the "sample complexity" problem.

What's the difference between a policy and a value function? A policy tells the agent what action to take in a given state; a value function estimates how good a state (or state-action pair) is in terms of expected future reward. Some algorithms learn a value function and derive a policy from it (value-based methods); others learn the policy directly (policy-based methods).

Can RL be used without a simulator? It's much easier with one, since RL typically needs many trial-and-error interactions, which can be costly, slow, or unsafe in the real world. Many RL systems (robotics, especially) are trained extensively in simulation before being fine-tuned on real hardware.

Why do RL agents sometimes find "cheating" strategies? Because the agent only optimizes the exact reward signal it's given, not the designer's true intent. If the reward function has a loophole (e.g., a reward for "reducing distance to goal" that can be gamed by circling near it), the agent will often find and exploit that loophole rather than solving the task as intended — a phenomenon called reward hacking.

Quick Revision

• RL: an agent learns a policy by interacting with an environment and maximizing cumulative reward, without labeled data.
• Core components: agent, environment, state, action, reward, policy, value function.
• Exploration vs. exploitation: trying new actions vs. using known-good ones; epsilon-greedy is a common balancing strategy.
• Q-learning update rule: Q(s,a) ← Q(s,a) + α[r + γ·max Q(s',a') − Q(s,a)].
• Q-learning is off-policy (learns from the best possible next action); SARSA is on-policy (learns from the action actually taken).
• Episodic tasks have clear start/end points; continuous tasks run indefinitely; POMDPs involve partial state observability.
• DQN uses a neural network to approximate Q-values for large state spaces.
• Policy gradient methods optimize the policy directly instead of learning a value function first.
• Actor-critic methods combine a policy (actor) and a value function (critic).
• Credit assignment problem: figuring out which past actions deserve credit for a later reward.
• Reward hacking: an agent exploiting loopholes in the reward function instead of solving the intended task.
• RL is widely used in robotics, game-playing AI, finance, and recommendation systems.

Related Topics

Prerequisites: Introduction to AI, basic probability, Machine Learning Fundamentals (supervised vs. unsupervised learning).

Related Topics: Neural Networks and Deep Learning (for deep RL), Markov Decision Processes, Dynamic Programming.

Next Topics: Neural Networks and Deep Learning, AI Ethics and Applications.