Speech Processing

Learning Objectives

• Describe the defining acoustic characteristics of a speech signal, including voiced and unvoiced sounds
• Explain why speech is treated as non-stationary and why this shapes the analysis techniques used
• Describe the standard speech-processing pipeline: acquisition, preprocessing, feature extraction, recognition, synthesis
• Explain spectral gating as a noise reduction technique and how the STFT enables it
• Identify why the sampling rate chosen for speech (e.g., 8 kHz for telephony) relates to the Nyquist theorem
• Apply a basic noise reduction technique to a speech signal using STFT-based spectral gating

Quick Answer

Speech processing analyzes, cleans, and synthesizes the acoustic signal produced by the human vocal tract. What sets speech apart from a generic audio signal is that it is non-stationary — its frequency content changes continuously as different sounds are produced — and it mixes voiced segments (periodic, driven by vocal cord vibration) with unvoiced segments (noise-like, from turbulent airflow, as in "s" or "f" sounds). Because speech changes character on a timescale of tens of milliseconds, it's processed in short overlapping frames rather than as one long stationary signal, using tools like the Short-Time Fourier Transform. This underlies applications from noise reduction and telephony to speech recognition and synthetic voice generation.

What Makes Speech a Distinctive Signal

A speech signal is a time-varying acoustic waveform, typically carrying most of its energy below about 4 kHz, though the full range of human speech sounds can extend up to roughly 8-10 kHz for sibilants like "s". Two properties define it:

• Non-stationarity — the statistical properties (frequency content, energy, pitch) of speech change continuously as a speaker moves between sounds. A single, full-length Fourier transform of a sentence would average away exactly the information that distinguishes one phoneme from the next.
• Voiced vs. unvoiced structure — voiced sounds (vowels, "m", "n") are produced by periodic vibration of the vocal cords, giving them a clear pitch (fundamental frequency) and harmonic structure. Unvoiced sounds ("s", "f", "sh") are produced by turbulent airflow with no vocal cord vibration, giving them a noise-like, non-periodic spectrum. Recognizing which category a sound belongs to is one of the earliest steps in most speech-analysis pipelines.

Because speech changes so quickly, it's standard practice to analyze it in short frames — typically 20-30 ms — over which the signal is assumed to be approximately stationary. This "quasi-stationary" assumption is what makes applying a Fourier transform meaningful at all for a signal that is globally non-stationary.

The Speech-Processing Pipeline

1. Signal acquisition — capturing speech via a microphone or reading from a recorded file, converting the acoustic wave to a digital signal via sampling. Telephony traditionally used 8 kHz sampling, matching the ~4 kHz "voice band" it needed to carry: the Nyquist rate for a 4 kHz bandlimited signal is exactly 8 kHz.
2. Preprocessing — cleaning the raw signal, most commonly removing background noise, before further processing. This step determines how usable the signal is for every later stage.
3. Feature extraction — deriving compact numerical descriptors from the waveform, such as pitch, formant frequencies, or Mel-Frequency Cepstral Coefficients (MFCCs), the standard feature set used in most speech recognition systems.
4. Pattern recognition — matching extracted features against known patterns, whether to recognize words (speech recognition) or identify a speaker (speaker verification).
5. Synthesis — generating artificial speech from text or extracted parameters, the reverse direction of the pipeline.

Noise Reduction via Spectral Gating

One of the most common preprocessing steps is removing background noise while preserving the speech itself. Spectral gating does this by working in the frequency domain, frame by frame:

import numpy as np
import librosa

signal, sample_rate = librosa.load('speech.wav', sr=None)

def noise_reduction(signal, noise_factor=0.5):
    stft = librosa.stft(signal)              # frame-by-frame Fourier transform (STFT)
    magnitude, phase = librosa.magphase(stft)
    noise_mean = np.mean(magnitude)
    # Keep only frequency components stronger than a noise-floor threshold
    magnitude_denoised = np.where(magnitude > noise_mean * noise_factor, magnitude, 0)
    stft_denoised = magnitude_denoised * phase
    return librosa.istft(stft_denoised)       # back to the time domain

denoised = noise_reduction(signal)

The logic is exactly frequency-domain filtering applied to a non-stationary signal one short frame at a time: compute the STFT, zero out magnitude components below an estimated noise floor (keeping the phase to allow accurate reconstruction), and invert back to the time domain. This is the same "transform, mask, inverse-transform" pattern used for simple noise removal on any signal, just applied frame-by-frame via the STFT instead of a single whole-signal FFT, because speech's frequency content shifts too quickly for a single global spectrum to work well.

Why It Matters

Speech processing underlies telephony (bandwidth-limited transmission), voice assistants and dictation software (recognition), hearing aids (noise suppression and amplification tuned to speech frequencies), and text-to-speech systems (synthesis). Getting the non-stationary nature of speech right — analyzing it frame by frame rather than as one long stationary block — is the single idea that separates competent speech processing from a naive application of generic signal-processing tools.

Real-World Applications

• Telecommunications — voice compression codecs (like those used in mobile networks) exploit speech-specific statistical models to achieve far higher compression than a generic audio codec.
• Voice assistants and dictation — feature extraction (MFCCs) followed by pattern recognition (often via neural networks) converts speech into text or commands.
• Hearing aids — real-time noise reduction and frequency-selective amplification tuned specifically to speech frequency ranges.
• Speaker verification and biometrics — pitch and formant characteristics act as a form of vocal "fingerprint" for identity verification.

Key Terms

Term	Definition	Related Concept
Voiced sound	Speech sound produced by periodic vocal cord vibration, with clear pitch	Fundamental frequency, harmonics
Unvoiced sound	Speech sound produced by turbulent airflow, noise-like and non-periodic	Fricatives (s, f, sh)
Non-stationarity	Property where a signal's statistical characteristics change over time	Short-Time Fourier Transform
Quasi-stationary frame	A short segment (~20-30 ms) over which speech is approximately stationary	STFT windowing
Feature extraction	Deriving compact numerical descriptors (e.g., MFCCs) from a raw speech signal	Speech recognition
MFCC	Mel-Frequency Cepstral Coefficients — standard feature set for speech recognition	Feature extraction
Spectral gating	Noise reduction technique zeroing frequency components below a noise-floor threshold	STFT, noise reduction
Voice band	The ~300-3400 Hz frequency range historically used for telephone speech transmission	Nyquist rate, 8 kHz sampling

Common Mistakes

Misconception: A single Fourier transform of an entire spoken sentence gives a useful picture of its speech content. Why it's wrong: A full-length FFT averages frequency content across the whole sentence, discarding exactly the information that distinguishes one phoneme from the next as the speaker's mouth and vocal tract shape change over time. Correct understanding: Speech must be analyzed frame by frame (via the STFT) because its frequency content is non-stationary — meaningfully changing on a timescale of tens of milliseconds.

---

Misconception: All speech sounds have a clear, identifiable pitch. Why it's wrong: Only voiced sounds, produced by periodic vocal cord vibration, have a well-defined pitch (fundamental frequency). Unvoiced sounds like "s" or "f" are generated by turbulent, noise-like airflow with no periodic structure and therefore no pitch. Correct understanding: Speech is a mixture of voiced (periodic, pitched) and unvoiced (aperiodic, noise-like) segments, and processing techniques often need to treat the two differently.

---

Misconception: Telephone-quality 8 kHz sampling is simply an outdated, low-quality choice compared to modern audio sampling rates. Why it's wrong: 8 kHz sampling was chosen deliberately to match the Nyquist rate for the ~4 kHz "voice band" carrying the bulk of speech intelligibility — it was an engineering decision optimized for bandwidth-limited telephone lines, not an arbitrary quality shortcut. Correct understanding: The choice reflects a genuine trade-off between bandwidth and intelligibility appropriate to telephony; modern applications requiring higher fidelity (like music or high-quality voice recording) use higher sampling rates because they need to preserve frequency content well beyond the 4 kHz voice band.

Comparison and Connections

Aspect	Voiced Sounds	Unvoiced Sounds
Source	Periodic vocal cord vibration	Turbulent airflow
Spectral character	Harmonic, clear fundamental frequency	Noise-like, broadband
Example sounds	Vowels, m, n, l	s, f, sh, th
Pitch present?	Yes	No

Speech Processing Concept	General Signal Processing Analogue
STFT-based frame analysis	Handling non-stationary signals generally
MFCC feature extraction	Dimensionality reduction / feature engineering
8 kHz telephony sampling	Nyquist rate for a bandlimited (4 kHz) signal
Spectral gating noise reduction	Frequency-domain filtering (masking a spectrum)

Practice Questions

Recall

1. What distinguishes a voiced speech sound from an unvoiced speech sound? Answer guidance: Voiced sounds are produced by periodic vibration of the vocal cords and have a clear pitch/fundamental frequency; unvoiced sounds are produced by turbulent airflow with no vocal cord vibration and are noise-like, with no defined pitch.

2. List the five stages of the standard speech-processing pipeline. Answer guidance: Signal acquisition, preprocessing, feature extraction, pattern recognition, and synthesis.

Understanding

3. Explain why speech is described as "non-stationary" and why this property requires frame-based (STFT) analysis rather than a single full-length Fourier transform. Answer guidance: Non-stationary means the signal's statistical/frequency properties change continuously over time, as the speaker forms different sounds. A single full-length Fourier transform would average this changing content into one static spectrum, losing the information about which sound occurred when. Breaking the signal into short frames, over which the signal is approximately stationary, and analyzing each with its own Fourier transform (the STFT) preserves this time-varying detail.

4. Why was 8 kHz historically chosen as the sampling rate for telephone speech? Answer guidance: Speech intelligibility is carried mostly within the ~300-3400 Hz "voice band," roughly bounded by 4 kHz. By the Nyquist-Shannon theorem, a signal bandlimited to 4 kHz requires a sampling rate greater than 8 kHz to avoid aliasing, so 8 kHz was chosen as an efficient rate that preserves intelligibility while minimizing bandwidth/storage requirements.

Application

5. You are building a noise reduction system for a voice recording made in a noisy café. Describe, in terms of the pipeline covered, how spectral gating would remove the background noise. Answer guidance: Compute the STFT of the recording to get magnitude and phase per frame; estimate a noise floor from the average magnitude; zero out (gate) any frequency component in each frame whose magnitude falls below a threshold based on that noise floor, while keeping the phase for reconstruction; then apply the inverse STFT to reconstruct a denoised time-domain signal.

6. A speech recognition system needs compact numerical features rather than raw waveform samples to feed into a recognition model. What feature extraction technique is standard for this, and why is a compact representation preferred? Answer guidance: Mel-Frequency Cepstral Coefficients (MFCCs) are the standard feature set. A compact representation reduces the amount of data the recognition model must process per frame, focuses on perceptually and phonetically relevant information, and generally improves recognition accuracy and computational efficiency compared to using raw samples directly.

Analysis

7. Compare processing a sustained vowel sound versus processing a spoken sentence in terms of whether a single Fourier transform is adequate. Answer guidance: A sustained, unchanging vowel is approximately stationary over its duration, so a single Fourier transform of that segment gives a reasonably accurate and stable picture of its harmonic structure. A full spoken sentence, by contrast, is strongly non-stationary — its frequency content changes continuously as different phonemes are produced — so a single Fourier transform over the whole sentence would be inadequate, and frame-based STFT analysis is required.

8. A student argues that since spectral gating in speech noise reduction is "just zeroing weak frequencies," it must always improve audio quality by fully removing noise. Evaluate this claim. Answer guidance: Overstated. Spectral gating can also remove or distort quiet parts of the actual speech signal if they fall below the estimated noise threshold, introducing a musical-noise or watery artifact, and it depends heavily on an accurate noise-floor estimate. Overly aggressive gating trades noise removal for speech distortion, echoing the general trade-off in any thresholding/filtering approach.

FAQ

Why can't a general-purpose audio codec (like a generic lossless compressor) match the compression a speech-specific codec achieves? Speech-specific codecs exploit statistical models of how the human vocal tract produces sound (e.g., linear predictive coding models the vocal tract as a simple resonant filter), which captures speech-specific redundancy that a generic, content-agnostic audio compressor has no way to exploit.

What is a formant, and why does it matter for speech processing? A formant is a resonant frequency of the vocal tract that shapes the spectral envelope of a voiced sound — the pattern of formant frequencies is largely what distinguishes one vowel sound from another, and formant tracking is used in both speech recognition and speech synthesis.

Does spectral gating work equally well for all kinds of background noise? No — it works best for relatively steady, broadband noise whose spectral profile can be reasonably estimated. Highly non-stationary or structured noise (like a competing talker in the background) is much harder to separate from the target speech using simple thresholding alone, and typically needs more sophisticated source-separation techniques.

Why is the STFT preferred over a plain FFT for speech, when both are based on the same Fourier mathematics? The plain FFT applied to the whole signal produces one spectrum, averaged over the full duration, discarding when different frequencies occurred. The STFT applies the FFT to short overlapping windows, preserving an approximate timeline of how the spectrum evolves — essential for a genuinely non-stationary signal like speech.

How is speech synthesis (text-to-speech) related to the same signal-processing concepts covered here? Modern speech synthesis reverses the feature-extraction pipeline: it generates parameters (pitch, formants, or learned neural representations) frame by frame and then reconstructs a time-domain waveform from them, effectively running the acquisition/analysis pipeline in reverse, frame by frame, just like reconstruction reverses sampling in general signal processing.

Quick Revision

• Speech is a non-stationary acoustic signal, mixing voiced (periodic, pitched) and unvoiced (noise-like) sounds
• Most speech energy lies below ~4 kHz, though sibilants extend higher
• Because speech is non-stationary, it's analyzed in short quasi-stationary frames (20-30 ms) via the STFT, not one global FFT
• The speech-processing pipeline: acquisition, preprocessing, feature extraction, pattern recognition, synthesis
• MFCCs are the standard compact feature set for speech recognition
• 8 kHz telephony sampling matches the Nyquist rate for the ~4 kHz voice band
• Spectral gating removes noise by zeroing frequency components below an estimated noise floor, frame by frame
• Aggressive spectral gating can distort quiet speech and introduce musical-noise artifacts
• Formants — vocal tract resonances — distinguish vowel sounds and are tracked in recognition and synthesis
• Speech synthesis reverses the analysis pipeline, generating a waveform from extracted or predicted parameters

Related Topics

Prerequisites: Fourier Transform, Signal Sampling and Reconstruction, Digital Filters

Related Topics: Image Processing, Signal Compression, Frequency-Domain Analysis

Next Topics: Applications of Signal Processing