Skip to main content

8. Debugging Embedded Systems

Learning Objectives​

  • Distinguish hardware debugging from software debugging and identify which tool fits which class of bug
  • Explain how JTAG/SWD in-circuit debugging works and why it's more powerful than print statements
  • Read a logic analyzer/oscilloscope capture to diagnose a communication protocol timing issue
  • Identify and fix three classic embedded bugs: missing volatile, stack overflow, and floating pin
  • Use watchpoints and breakpoints effectively during a debugging session
  • Analyze a described symptom (intermittent crash, wrong sensor reading) and propose a debugging strategy

Quick Answer​

Debugging embedded systems means finding and fixing faults that can originate in hardware, software, or the interaction between them — which is what makes it harder than debugging a desktop application. A print statement can tell you a variable's value, but it can't tell you why a UART line is glitching at the electrical level, or why a CPU occasionally resets under load. Embedded debugging combines software techniques (breakpoints, watchpoints, print/serial logging) with hardware tools (oscilloscopes, logic analyzers, JTAG/SWD in-circuit debuggers) precisely because the bug's root cause could be a logic error, a timing violation, a wiring mistake, or an electrical noise issue — and the right tool depends entirely on which category the symptom points to.

Hardware Debugging vs. Software Debugging​

Hardware debugging targets faults in the circuit itself: a floating input pin, a signal that doesn't reach the expected voltage, a communication bus with incorrect timing, or a short circuit. Tools: multimeter, oscilloscope, logic analyzer.

Software debugging targets faults in the code's logic: an off-by-one error, a race condition, incorrect register configuration, or a memory corruption bug. Tools: breakpoints, watchpoints, print/serial logging, JTAG/SWD debuggers.

In practice, most non-trivial embedded bugs live at the boundary between the two — a "software" symptom (sensor reads garbage) frequently has a hardware root cause (missing pull-up resistor, bad ground connection), and a "hardware-looking" symptom (signal never appears on the scope) can have a software root cause (peripheral never actually initialized). This is why experienced embedded engineers check both categories, not just the one that feels more familiar.

JTAG/SWD: In-Circuit Debugging​

JTAG (Joint Test Action Group) and SWD (Serial Wire Debug) are hardware debugging protocols that let a debugger probe (e.g., an ST-LINK/V2) connect directly to the microcontroller's debug port. Unlike print statements, which only show you what you explicitly logged, JTAG/SWD lets you:

  • Set breakpoints — halt execution the instant the CPU reaches a specific instruction address
  • Set watchpoints — halt execution the instant a specific memory address changes value, regardless of which line of code changed it
  • Inspect and modify registers and memory live, while the CPU is halted
  • Single-step through code one instruction at a time

This is fundamentally more powerful than Serial.println() debugging because it requires no code changes (no recompiling to add/remove print statements), works even before main() runs, and can catch bugs that print statements would never reveal — like memory corruption from a stray pointer write, caught the instant it happens via a watchpoint on that address.

Classic Bug 1: Missing volatile​

// BUGGY — compiler may cache 'flag' in a register, never re-reading it
uint8_t flag = 0;

void TIMER_IRQHandler(void) {
    flag = 1;
}

int main(void) {
    while (!flag) {
        // Compiler may optimize this to an infinite loop if it assumes
        // 'flag' never changes within this function — it has no idea
        // an ISR modifies it
    }
    handle_timer_event();
}

Symptom: The code appears to hang forever in the while loop even though the ISR clearly runs (verified by toggling an LED inside it). Root cause: without volatile, the compiler's optimizer may load flag into a CPU register once and never re-check main memory, because nothing in the visible code path changes it. Fix: declare volatile uint8_t flag = 0; — this tells the compiler the value can change outside normal program flow, forcing a fresh memory read on every check.

Classic Bug 2: Stack Overflow​

// BUGGY — deep recursion or a large local array can exceed a small task's stack
void process_reading(int depth) {
    uint8_t large_buffer[512];   // 512 bytes on the stack, every call
    if (depth < 20) {
        process_reading(depth + 1);   // recursion multiplies stack usage
    }
}

Symptom: The system randomly resets or corrupts unrelated variables, often only under specific input conditions — a notoriously hard bug to reproduce because it depends on exact call depth and buffer sizes at the time of the overflow. Root cause: each recursive call allocates another 512-byte buffer on a stack that might only be 1–2KB total on a small MCU or RTOS task; the stack grows past its allocated region and starts overwriting adjacent memory (other variables, or the interrupt vector table). Fix: eliminate unnecessary recursion, reduce local buffer sizes, or increase the task's stack size — and use the RTOS's built-in stack high-water-mark check (uxTaskGetStackHighWaterMark() in FreeRTOS) to verify actual stack usage during testing rather than guessing.

Classic Bug 3: Floating Pin Misread as Real Input​

// BUGGY — no pull-up/pull-down configured
pinMode(BUTTON_PIN, INPUT);       // floating if button isn't pressed!

void loop() {
    if (digitalRead(BUTTON_PIN) == HIGH) {
        trigger_action();          // fires randomly due to noise pickup
    }
}

Symptom: trigger_action() fires intermittently even when the button is definitely not pressed. Root cause: with no pull-up or pull-down resistor, the pin "floats" between logic states, picking up electrical noise from nearby wires and radio interference, and randomly reads HIGH or LOW. Fix: pinMode(BUTTON_PIN, INPUT_PULLUP); and invert the logic (== LOW means pressed), or add an external pull-down resistor. This exact symptom — a signal that looks fine on the code review but misbehaves unpredictably on real hardware — is the classic signature of a floating input, and it's one of the fastest things to check with a multimeter or oscilloscope before assuming a software bug.

Reading a Logic Analyzer Capture​

When a UART, SPI, or I2C communication isn't working, a logic analyzer (e.g., Saleae Logic) captures the actual electrical signal transitions and decodes the protocol, showing you exactly what bits were transmitted and at what timing — invaluable when the two devices "should" agree on a protocol but clearly aren't communicating.

A typical diagnostic sequence:

  1. Connect logic analyzer probes to the relevant signal lines (e.g., SDA/SCL for I2C).
  2. Trigger a capture around the moment the failing transaction occurs.
  3. Use the analyzer's protocol decoder to overlay the actual bit values on the waveform.
  4. Compare against the expected protocol: Is the clock frequency correct? Is there an ACK bit where expected? Is a start/stop condition malformed?

A common finding: a device fails to respond on I2C because the actual bus speed configured in software (e.g., 400kHz) exceeds what the specific sensor supports (100kHz standard mode only) — a mismatch invisible in code review but immediately obvious once you see the waveform and observed non-response.

Why It Matters​

An embedded engineer who only knows software debugging techniques will spend hours chasing a "logic bug" that's actually a missing pull-up resistor, and one who only knows hardware tools will miss a race condition that never shows up on a scope trace. Real debugging efficiency comes from correctly classifying the symptom first — is this a timing/electrical issue, or a logic issue? — and reaching for the matching tool, rather than defaulting to whichever technique is most familiar.

Key Terms​

TermDefinitionRelated Concept
JTAG/SWDHardware debug interfaces allowing breakpoints, watchpoints, and register inspection on real siliconBreakpoint, watchpoint
BreakpointA debugger-set point that halts execution when a specific instruction address is reachedJTAG, single-stepping
WatchpointA debugger-set trigger that halts execution when a specific memory address's value changesJTAG, memory corruption
Logic analyzerA tool that captures and decodes multiple digital signal lines over timeProtocol decoding, UART/SPI/I2C
OscilloscopeA tool that displays analog voltage waveforms over time, showing signal shape and timingSignal integrity
Stack overflow (embedded)Stack memory usage exceeding its allocated region, corrupting adjacent memoryRecursion, RTOS task stack
Stack high-water markThe measured worst-case stack usage of a task, used to verify stack size is adequateStack overflow

Common Mistakes​

Misconception: Print/serial statement debugging is always sufficient for embedded systems, just like it often is for desktop scripts. Why it's wrong: Print debugging cannot reveal electrical/timing issues (a glitching signal, incorrect bus speed), requires recompiling and can itself alter timing enough to mask race conditions ("Heisenbugs"), and doesn't work before serial communication is even initialized. Correct understanding: Print debugging is one tool among several; timing-sensitive and hardware-rooted bugs require an oscilloscope, logic analyzer, or JTAG/SWD debugger that doesn't require modifying the code under test.

Misconception: An intermittent, hard-to-reproduce bug is probably caused by a rare and complex software edge case. Why it's wrong: In embedded systems, intermittent symptoms are frequently caused by much more mundane hardware issues — a floating pin, marginal power supply, or a stack overflow that only manifests under specific call-depth conditions — that are simpler to find with the right hardware tool than by staring at code. Correct understanding: Before assuming a subtle software bug, check the simple hardware suspects first: pull-up/pull-down resistors, power supply stability, and stack/memory usage.

Misconception: Adding a print statement to check a variable's value never affects the bug you're chasing. Why it's wrong: Print/serial statements take real time to execute and can shift the timing of a race condition or interrupt interaction enough to make the bug disappear while debugging and reappear once removed — a classic "Heisenbug." Correct understanding: For timing-sensitive bugs, prefer non-intrusive tools (JTAG watchpoints, logic analyzer captures) that don't alter the timing of the system under test.

Comparison and Connections​

ToolBest forRequires code changes?Shows electrical signal?
Print/serial loggingSimple logic tracing, low-cost setupsYesNo
Breakpoints/watchpoints (JTAG/SWD)Precise logic and memory-state bugsNoNo
OscilloscopeSignal voltage, timing, noise, glitchesNoYes (analog waveform)
Logic analyzerMulti-signal digital protocol decoding (UART/SPI/I2C)NoYes (digital, decoded)

Practice Questions​

Recall

  1. What is the difference between a breakpoint and a watchpoint? Answer guidance: A breakpoint halts execution at a specific instruction address; a watchpoint halts execution when a specific memory address's value changes, regardless of which instruction caused the change.

  2. Name two hardware debugging tools and what each is best suited to observe. Answer guidance: Oscilloscope — analog voltage waveforms, signal integrity, timing; logic analyzer — multiple digital signals decoded into protocol-level data (UART, SPI, I2C).

Understanding

  1. Explain why a missing volatile keyword can cause a while(!flag) loop to hang forever even though an ISR is correctly setting flag = 1. Answer guidance: The compiler's optimizer may load flag into a register once, assuming it cannot change since nothing in the visible code path (from the compiler's view) modifies it — it doesn't know the ISR runs asynchronously and updates the actual memory location, so the cached register value never reflects the change.

  2. Why can adding a print statement sometimes make a race-condition bug temporarily disappear? Answer guidance: The print statement takes measurable execution time, shifting the relative timing between the main code and an interrupt or other task — sometimes enough to avoid the exact interleaving that triggers the race condition, without actually fixing the underlying bug.

Application

  1. A student's I2C sensor never responds. Using a logic analyzer, they see the software is toggling SCL at 400kHz, but the sensor's datasheet specifies standard mode only (100kHz max). Diagnose the fix. Answer guidance: Reconfigure the I2C peripheral's clock speed setting in software to 100kHz (standard mode) to match the sensor's supported speed; the 400kHz "fast mode" clock exceeds what the sensor can decode, causing it to never respond.

  2. A digital input reads HIGH randomly even when nothing is connected to trigger it. Propose the fastest way to confirm the diagnosis and the fix. Answer guidance: Use a multimeter or oscilloscope on the pin to observe an unstable, noisy voltage rather than a clean HIGH or LOW — confirming a floating input. Fix: configure INPUT_PULLUP (or add an external pull-down resistor) so the pin has a defined rest state.

Analysis

  1. A system resets unpredictably only when a specific function is called with deep recursion and large local buffers. Analyze the likely root cause and how to confirm it without guessing. Answer guidance: Likely a stack overflow from recursive calls each allocating a large local buffer, exceeding available stack space and corrupting adjacent memory (possibly the interrupt vector table, explaining the reset). Confirm using the RTOS's stack high-water-mark function or a debugger-based stack usage check, rather than assuming based on symptoms alone.

  2. Compare the debugging strategy you'd use for a bug that reproduces every single time versus one that only occurs "once every few days" in the field. What tools and techniques differ? Answer guidance: A consistently reproducible bug is well-suited to breakpoints/single-stepping since you can pause and inspect state reliably. A rare field bug requires non-intrusive tools that don't alter timing (watchpoints, hardware logging/black-box recording) and often benefits from adding persistent, low-overhead logging (e.g., to flash) that survives a reset, since you can't sit at a debugger waiting days for the event.

FAQ​

Why does my code work fine with the debugger attached but fail when running standalone? This is a classic sign of a race condition or timing-sensitive bug — the debugger itself (breakpoints, single-stepping, or even just the slight overhead of being attached) changes timing enough to avoid the exact interleaving that triggers the bug. Try non-intrusive tools like watchpoints without single-stepping, or hardware-based signal capture instead.

When should I reach for a logic analyzer instead of an oscilloscope? Use a logic analyzer when you need to see the decoded meaning of multiple digital signals together (e.g., what data actually transferred over I2C). Use an oscilloscope when you need to see the analog voltage shape of a single signal — rise/fall time, noise, ringing, or whether a "digital" signal is actually reaching valid logic levels.

How do I know if a bug is hardware or software in origin when the symptom looks identical either way? Start with the cheapest check: measure the actual voltage at the suspected pin with a multimeter or scope. If the electrical signal is correct and stable but the software still misbehaves, the issue is in code logic. If the signal itself is wrong, noisy, or missing, the root cause is electrical/wiring, no matter how the software appears to behave.

Why do JTAG/SWD debuggers require a physical hardware connection instead of just working over serial? JTAG/SWD accesses the microcontroller's debug port directly at the silicon level, letting it halt the CPU, inspect registers, and set watchpoints independent of whatever software is (or isn't) running — capabilities that require dedicated hardware pins and protocol support built into the chip, not something achievable purely over a general-purpose serial connection.

What's the single most valuable habit for efficient embedded debugging? Forming a specific, falsifiable hypothesis before reaching for a tool — "I believe the sensor's data line is floating when disconnected" is testable in seconds with a multimeter, while randomly adding print statements or poking at code without a hypothesis wastes far more time than it saves.

Quick Revision​

  • Hardware debugging (oscilloscope, logic analyzer, multimeter) targets electrical/timing faults; software debugging (breakpoints, watchpoints, logging) targets logic faults
  • JTAG/SWD gives non-intrusive breakpoints, watchpoints, and live memory inspection without recompiling
  • Missing volatile on ISR-shared variables can cause infinite loops due to compiler register caching
  • Stack overflow from recursion/large local buffers causes intermittent resets and corrupted unrelated variables
  • Floating input pins (no pull-up/pull-down) cause random, noise-driven logic level reads
  • Print/serial debugging can itself alter timing enough to hide race conditions ("Heisenbugs")
  • Logic analyzers decode multi-signal digital protocols (UART/SPI/I2C); oscilloscopes show single-signal analog waveforms
  • Always check simple hardware suspects (pull-ups, power stability, stack usage) before assuming a complex software bug
  • Use stack high-water-mark checks (e.g., FreeRTOS uxTaskGetStackHighWaterMark) to verify stack sizing rather than guessing
  • Match debugging tool to symptom type: electrical symptom → scope/analyzer; logic symptom → breakpoints/watchpoints

Prerequisites: Embedded System Programming, Embedded System Architecture

Related Topics: Real-Time Operating Systems, Embedded System Interfaces

Next Topics: Embedded System Interfaces (protocols debugged with logic analyzers), Future Trends in Embedded Systems