Speech Recognition Engineer Interview Questions: Complete Prep Guide 2026

Answer: CTC loss allows training sequence-to-sequence models without requiring frame-level alignments between audio and text. It works by:

• Introducing a "blank" token that represents no output
• Allowing multiple paths through the output sequence that collapse to the same final text
• Summing probabilities of all valid paths that produce the target sequence
• Using dynamic programming to compute this efficiently

Why it matters: CTC was revolutionary for ASR because you don't need phoneme-level timestamps—just the audio and final transcription.

Follow-up they might ask: "What are the limitations of CTC?" (Answer: Can't model output dependencies, blank token overhead, assumes conditional independence)

Answer:

WER: Measures errors at word level. Formula: (Substitutions + Deletions + Insertions) / Total Words

CER: Measures errors at character level. Same formula but applied to characters.

When to use:

• WER: English and other space-separated languages, end-user facing metrics
• CER: Languages without clear word boundaries (Chinese, Japanese), when word tokenization is unclear

Pro tip: Always report which metric you're using—a 5% WER sounds great until you realize the baseline was 2%.

Answer: End-to-end ASR models typically have three components:

1. Encoder: Converts audio features (mel-spectrograms) into high-level representations. Usually a stack of CNNs + RNNs or Transformers. Takes variable-length audio input.
2. Attention Mechanism: Learns to focus on relevant parts of the encoded audio when predicting each output token. Allows the model to "attend" to different parts of the audio at different times.
3. Decoder: Generates output sequence (characters or subwords) autoregressively. Uses previous predictions and attended encoder outputs.

Key advantage over traditional pipeline: Single neural network trained end-to-end, no need for separate acoustic model, pronunciation dictionary, and language model.

Answer: Beam search is a decoding algorithm that maintains the top K most likely partial hypotheses at each step:

1. Start with K=beam_width empty hypotheses
2. For each hypothesis, generate all possible next tokens
3. Score each extension (usually log probability)
4. Keep only the top K scored complete hypotheses
5. Repeat until end-of-sequence or max length

Tradeoffs:

• Larger beam (K=10-20): Better accuracy, slower inference, more memory
• Smaller beam (K=1-5): Faster inference, less memory, might miss optimal path
• K=1 (greedy): Fastest but often suboptimal

Production insight: Most systems use K=5-8 as a sweet spot. Beyond K=10, gains plateau.

Answer: Mel-spectrograms are time-frequency representations of audio that use the mel scale, which better matches human perception of sound.

How they're created:

1. Take raw audio waveform
2. Apply Short-Time Fourier Transform (STFT) to get spectrogram
3. Convert frequency axis to mel scale (logarithmic)
4. Often apply logarithm to amplitudes

Why mel scale? Humans perceive pitch logarithmically—doubling from 100Hz to 200Hz sounds like the same "distance" as 1000Hz to 2000Hz. Mel scale captures this.

Why use them? Better than raw audio (too high-dimensional) or linear spectrograms (don't match human perception).

Non-streaming (offline): Process entire audio file at once, can look forward and backward, higher accuracy.

Streaming (online): Process audio in real-time as it arrives, can only look backward (and limited lookahead), must maintain low latency.

Technical challenges of streaming:

• Latency: Must emit results within ~200-500ms for real-time feel
• Chunking: How to split audio while maintaining context?
• Look-ahead limitations: Can't use future context that works well offline
• Stability: Results shouldn't change after being emitted (no "flickering")
• State management: Need to maintain decoder state between chunks

Common solutions: RNN-Transducer architecture, limited lookahead windows, causal attention mechanisms.

Answer: Wav2Vec 2.0 is a self-supervised learning approach for speech that learns representations from raw audio without transcriptions.

How it works:

1. Masking: Randomly mask portions of the audio input (like BERT does for text)
2. Quantization: Discretize the audio into a finite set of representations
3. Contrastive Learning: Train the model to predict the correct quantized representation from a set of "distractors"
4. Fine-tuning: After pre-training, add a small CTC head and fine-tune on labeled data

Why it's important: Achieves strong results with as little as 10 minutes of labeled data for low-resource languages, vs. thousands of hours needed for traditional approaches.

Related work: HuBERT, WavLM, Data2Vec (similar ideas with variations)

def compute_wer(reference, hypothesis):
    """
    Compute Word Error Rate using Levenshtein distance.
    
    Args:
        reference: Ground truth string
        hypothesis: Predicted string
    
    Returns:
        WER as a float (0.0 to 1.0+)
    """
    ref_words = reference.split()
    hyp_words = hypothesis.split()
    
    # Build edit distance matrix
    d = [[0] * (len(hyp_words) + 1) for _ in range(len(ref_words) + 1)]
    
    # Initialize first row and column
    for i in range(len(ref_words) + 1):
        d[i][0] = i
    for j in range(len(hyp_words) + 1):
        d[0][j] = j
    
    # Fill matrix
    for i in range(1, len(ref_words) + 1):
        for j in range(1, len(hyp_words) + 1):
            if ref_words[i-1] == hyp_words[j-1]:
                d[i][j] = d[i-1][j-1]  # No error
            else:
                substitution = d[i-1][j-1] + 1
                insertion = d[i][j-1] + 1
                deletion = d[i-1][j] + 1
                d[i][j] = min(substitution, insertion, deletion)
    
    # WER = edit distance / reference length
    return d[len(ref_words)][len(hyp_words)] / len(ref_words) if ref_words else 0.0

# Test
ref = "the quick brown fox"
hyp = "the qwick brown fox"
print(f"WER: {compute_wer(ref, hyp):.2f}")  # 0.25 (1 error out of 4 words)

Follow-up questions:

• "How would you optimize this for very long sequences?" (Answer: Use NumPy, only keep two rows)
• "What if you need to return the specific errors?" (Answer: Backtrack through the matrix)

def greedy_ctc_decode(logits, blank_id=0):
    """
    Greedy CTC decoding: take argmax at each timestep, collapse repeats and blanks.
    
    Args:
        logits: (T, vocab_size) model outputs before softmax
        blank_id: ID of blank token (usually 0)
    
    Returns:
        List of predicted token IDs
    """
    import numpy as np
    
    # Get argmax at each timestep
    predictions = np.argmax(logits, axis=1)
    
    # Collapse: remove consecutive duplicates and blanks
    output = []
    previous = None
    
    for pred in predictions:
        # Skip if same as previous (collapse repeats)
        if pred == previous:
            continue
        # Skip blank tokens
        if pred == blank_id:
            previous = pred
            continue
        # Add to output
        output.append(pred)
        previous = pred
    
    return output

# Example usage
# logits shape: (50, 29) for 50 timesteps, 29 tokens (26 letters + 3 special)
# After greedy decode: might get [8, 5, 12, 12, 15] -> "hello"

Extension: "Now implement beam search CTC decoding" (significantly harder, usually just discuss approach)

import librosa
import numpy as np

def extract_mel_spectrogram(audio_path, sr=16000, n_mels=80, 
                           n_fft=400, hop_length=160):
    """
    Extract mel-spectrogram features from audio file.
    
    Args:
        audio_path: Path to audio file
        sr: Sample rate
        n_mels: Number of mel bands
        n_fft: FFT window size
        hop_length: Hop length for STFT
    
    Returns:
        Mel-spectrogram (n_mels, time)
    """
    # Load audio
    audio, _ = librosa.load(audio_path, sr=sr)
    
    # Compute mel-spectrogram
    mel_spec = librosa.feature.melspectrogram(
        y=audio,
        sr=sr,
        n_fft=n_fft,
        hop_length=hop_length,
        n_mels=n_mels,
        fmin=0,
        fmax=sr/2
    )
    
    # Convert to log scale (dB)
    mel_spec_db = librosa.power_to_db(mel_spec, ref=np.max)
    
    return mel_spec_db

# Usage
features = extract_mel_spectrogram('speech.wav')
print(f"Feature shape: {features.shape}")  # (80, T)

Discussion points:

• "Why these specific hyperparameters?" (16kHz is standard for speech, 80 mels is common, hop of 10ms)
• "What's the time resolution?" (hop_length/sr = 10ms per frame)

Key components to discuss:

1. Wake Word Detection (on-device)

• Tiny neural network (1-5MB) running on device
• Always listening, very low power
• High recall (catch all wake words), lower precision OK
• Sends audio to cloud only after detection

2. ASR Service (cloud)

• Streaming ASR (RNN-T or similar)
• Auto-scaling based on load
• Regional deployment (latency)
• GPU inference servers
• Target latency: <200ms for first word

3. NLU (Intent Classification)

• Extract intent and entities from transcription
• Route to appropriate service (music, weather, etc.)
• Fast inference (CPU or small GPU)

4. Response Generation

• TTS for voice response
• Caching common responses
• Multiple voice options

Scale considerations:

• 10M concurrent: Need thousands of inference servers
• Load balancing: Geographic routing, queue management
• Cost optimization: Batch where possible, cache aggressively
• Monitoring: Latency p50/p95/p99, WER, uptime

Tradeoffs to discuss:

• On-device vs cloud processing (privacy vs accuracy)
• Model size vs accuracy (smaller = faster but less accurate)
• Streaming vs batch (latency vs throughput)

Architecture components:

1. Audio Ingestion

• WebSocket or WebRTC from client
• Audio chunking (1-5 second segments)
• Queue system (Kafka/RabbitMQ)

2. ASR Pipeline

• Streaming ASR (Whisper or similar)
• Speaker diarization (who spoke when)
• Punctuation restoration
• Word-level timestamps

3. Post-Processing

• Filler word removal (um, uh, like)
• Paragraph segmentation
• Named entity recognition
• Action item extraction (ML model)

4. Storage & Retrieval

• Audio in S3/cloud storage
• Transcripts in database (PostgreSQL)
• Full-text search (Elasticsearch)

Scale math:

• 1000 concurrent meetings, 1 hour avg = 1000 audio-hours/hour
• Real-time factor (RTF) = 0.2-0.5 (process 1 hour in 12-30 min)
• Need ~50-200 GPU instances for ASR
• Storage: 1000 meetings/day * 30 days * 100MB audio = 3TB/month

Answer: They sit at opposite ends of the training-paradigm spectrum.

• Whisper: an encoder-decoder Transformer trained with ordinary supervision on ~680k hours of weakly-labeled multilingual audio scraped from the web. The decoder is autoregressive and behaves much like a language model, steered by special tokens for task (transcribe vs translate), language, and timestamps.
• Wav2Vec 2.0: self-supervised pretraining first, with a contrastive objective over masked latent speech representations against quantized targets, then a CTC head fine-tuned on a comparatively small labeled set.

What follows from that: Whisper is robust out of the box and multilingual, but it inherits a strong language-model prior, is locked to a 30-second input window, and hallucinates on silence or noise. Wav2Vec 2.0 is smaller and faster, but needs fine-tuning plus an external LM to reach competitive WER.

Follow-up they might ask: "Why does Whisper hallucinate?" (Answer: an autoregressive decoder with a strong LM prior will happily produce fluent text when the acoustic evidence is absent. It is generating, not just recognizing.)

Answer: The RNN Transducer has three parts:

1. Encoder: processes the audio, same role as in any other architecture.
2. Prediction network: consumes the label history. Functionally an internal language model.
3. Joint network: combines encoder and prediction outputs to emit the next token or a blank.

Why it wins for streaming: unlike CTC it models dependencies between output tokens, and unlike attention-based seq2seq it never needs the full utterance. Alignment is monotonic and decoding is frame-synchronous, so you can emit words while the user is still talking. This is why it runs on-device at Google and Apple.

Follow-up: "What makes it expensive to train?" (Answer: the joint network materializes a T×U lattice, which is memory-hungry. Hence function-merging and pruned transducer losses, like the pruned RNN-T in k2/icefall.)

Answer: A Conformer is a convolution-augmented Transformer block, usually arranged macaron-style: feed-forward, multi-head self-attention, convolution module, feed-forward, layer norm.

The intuition: speech needs two kinds of context at once. Self-attention captures long-range global structure such as sentence-level context and speaker characteristics. Convolution captures local structure such as formant transitions and phone boundaries. Pure Transformers modeled the first well and the second poorly.

Why it matters practically: Conformer became the default encoder across NeMo, ESPnet and icefall, so "which encoder are you using" usually has the answer "a Conformer variant" in 2026.

Answer: Go up the cost ladder only as far as you need to:

1. Contextual biasing at decode time. Shallow fusion with a small LM or a biasing FST built from the term list. No retraining, applied per request, so it can be personalized per user or per call.
2. Hotword boosting. Whisper's initial_prompt, or lattice rescoring in a hybrid system.
3. Lexicon additions. In hybrid systems, add grapheme-to-phoneme pronunciations for the new words.
4. Fine-tuning on in-domain audio. Most expensive, and only worth it once you have real labeled audio rather than just a word list.

The tradeoff they want you to name: biasing lists degrade general accuracy as they grow, because you are up-weighting rare words everywhere. Production systems typically cap the active list at a few thousand entries and scope it per request.

Answer: This is the most common real-world ASR question, and they want a method, not a guess.

Check the trivial cause first: sample-rate mismatch. Telephony audio is 8 kHz; a model trained at 16 kHz fed upsampled audio degrades badly. This is the single most frequent cause of exactly this symptom.

Then stratify the errors by speaker, by SNR, by utterance length, and look at the error-type balance:

• High deletions: VAD or endpointing is clipping audio, or the model is failing on fast speech.
• High insertions: hallucination on noise or silence.
• High substitutions: vocabulary or accent mismatch.

Then name the domain-shift axes: channel and codec, noise and reverb, spontaneous versus read speech, accents, overlapping speech, and domain-specific vocabulary. LibriSpeech is read audiobooks by design, so almost every axis differs.

Fixes, cheapest first: match the front-end sample rate, augment (SpecAugment, noise and RIR mixing, codec simulation), fine-tune on in-domain audio, and only then reconsider architecture.

Answer: Data augmentation applied directly to the mel-spectrogram rather than the waveform, with three operations: time warping, frequency masking, and time masking.

Why it works: it is a regularizer. Masking frequency bands stops the model leaning on a narrow part of the spectrum; masking time regions forces it to recover words from partial evidence, which is what real noisy audio looks like. Because it operates on features, it costs almost nothing compared to re-synthesizing audio.

Follow-up: "What else would you augment with?" (Answer: speed perturbation at 0.9/1.0/1.1, room impulse response convolution for reverb, MUSAN noise mixing, and codec simulation if you are targeting telephony.)

Answer: Training from scratch is not on the table. The plan is transfer plus text.

• Start from a multilingual self-supervised model such as XLS-R, MMS or Whisper, and fine-tune. Cross-lingual representations transfer well even to unseen languages.
• Use character or BPE outputs so you do not need a pronunciation lexicon you cannot build.
• Add an n-gram LM from text-only data. Text is almost always far more available than transcribed audio, and shallow fusion gives disproportionate gains exactly in the low-resource regime.
• Pseudo-label unlabeled in-language audio with the fine-tuned model, filter by confidence, and retrain. Iterate.

Evaluation nuance worth raising: report CER alongside WER. Morphologically rich languages inflate WER because a single wrong affix marks the whole word wrong.

The classical pipeline: voice activity detection → segmentation into speaker-homogeneous chunks → speaker embedding extraction (x-vector or ECAPA-TDNN) → clustering (agglomerative or spectral) → optional resegmentation.

Where it breaks:

• Overlapping speech. Clustering assigns one speaker per frame by construction, so simultaneous talkers are unrecoverable. In real meetings and calls this is 10-20% of speech.
• Unknown speaker count. You are usually estimating it from the clustering threshold, which is brittle.
• Short turns. Under about a second there is not enough audio for a reliable embedding, so backchannels ("mm-hm", "right") get misassigned.
• Domain mismatch. Embeddings trained on clean speech degrade on telephony.

The modern answer: end-to-end neural diarization (EEND) with permutation-invariant training handles overlap natively, and target-speaker VAD works well when you know who to look for.

The metric: DER, the sum of missed speech, false alarm and speaker confusion, usually scored with a forgiveness collar around boundaries.

Answer: Three different problems people routinely conflate.

• Diarization: "who spoke when." No enrolled identities at all. You are partitioning audio into anonymous speaker clusters.
• Identification: closed-set. Which of N enrolled speakers is this? Scored by accuracy across the N classes.
• Verification: one-to-one. Is this the speaker they claim to be? An accept/reject decision against a threshold, scored by equal error rate.

Why interviewers like it: the follow-up is usually "which one does a voice authentication product need?" (Verification, with an EER target and a spoofing/anti-replay story attached.)

Answer: Evaluate the two stages separately, then together.

• ASR stage: WER, but this is the trap. Overall WER is a weak predictor of tagging quality. What matters is error rate on the keywords that trigger the tag. Report keyword recall as its own number.
• Tagging stage: precision and recall per tag. Compliance events are rare, so accuracy is meaningless. A model that never fires scores 99% accurate.

Setting the threshold: this is a business decision, not a modeling one. A missed compliance breach costs far more than a false alarm a reviewer dismisses in five seconds, so you bias toward recall and staff the review queue accordingly.

Ground truth: rotating human-labeled samples, because the label distribution drifts as agents change scripts.

The modeling part is the easy part: prosodic features (F0 contour, energy, speaking rate) plus representations from a self-supervised model. Worth knowing: mid layers of WavLM and HuBERT carry more paralinguistic information than the top layers, which have specialized toward phonetic content.

Why it is genuinely hard:

• The labels are subjective. Inter-annotator agreement on emotion is frequently around 0.6 kappa. You are fitting a noisy target and your ceiling is human disagreement.
• Acted corpora do not transfer. Models trained on IEMOCAP or RAVDESS collapse on spontaneous speech, because acted anger is not real anger.
• Speaker identity confounds emotion. Without speaker-independent splits you will measure speaker recognition and call it emotion recognition.
• Emotional expression is culturally variable, so a model tuned on one population misreads another.

Evaluate with unweighted average recall rather than accuracy, because the classes are badly imbalanced.

VAD is voice activity detection: classifying audio as speech or non-speech. Endpointing is the downstream decision that the speaker has finished a turn, which is what actually triggers a response.

Why it is high-stakes:

• Too aggressive and you cut users off mid-sentence. This shows up as deletions in your WER and as fury in your user feedback.
• Too permissive and the agent feels slow, because every response waits out the silence timeout.

The point to make: in a voice agent, endpointing latency usually dominates perceived latency far more than model inference does. Teams spend months optimizing inference by 200ms while a 700ms silence threshold sits untouched.

Answer: Stack the cheap wins before touching the model.

• Do not run the model on silence. VAD-gating alone often removes a large fraction of audio in call and meeting workloads. Cheapest win available.
• Change the runtime, not the weights. CTranslate2/faster-whisper, ONNX Runtime or TensorRT give large speedups on identical weights.
• Quantize to INT8 or FP16. Usually a fraction of a WER point for a multiple in throughput.
• Batch aggressively if the workload is offline. GPU utilization on unbatched ASR is typically dismal.
• Right-size per traffic tier. Route clean, short audio to a small or distilled model and reserve the large one for hard audio, using a confidence or audio-quality signal to decide.

The senior framing: state the WER budget first. "10x cheaper" is only meaningful alongside how much accuracy you are permitted to spend.

The constraints change completely: fixed memory, limited or absent floating-point throughput, thermal and battery budgets, and no dynamic batching because requests arrive one at a time.

What that forces:

• Streaming architecture. RNN-T rather than attention-based seq2seq, since you cannot buffer the whole utterance.
• Quantization-aware training rather than post-training quantization, because the accuracy headroom is thinner.
• Pruning and a lighter feature front-end, sometimes fixed-point arithmetic end to end.

ARM-specific detail worth knowing: on embedded Linux the bottleneck is usually memory bandwidth rather than raw FLOPs, so INT8 NEON kernels and cache-friendly layouts buy more than shaving parameters.

Why bother: privacy and data residency, offline operation, latency, and eliminating per-request cost.

Answer: You cannot compute WER, so you triangulate.

• Model-internal signals: average confidence, posterior entropy, empty-output rate, and repetition-loop detection. That last one matters specifically because degenerate repetition is a classic Whisper production failure.
• Input-side signals: SNR, clipping rate, sample rate, audio duration distribution. Most quality regressions are upstream audio changes, not model changes.
• Downstream behavioral signals: user retries, manual corrections, abandonment, escalation to a human agent. These are your best proxy for real quality because they measure whether the output was usable.
• Sampled human transcription on a rotating basis to estimate true WER with a confidence interval. Expensive, so you sample rather than measure.

The design principle: alert on distribution shift, not on an absolute WER number you cannot compute anyway.

What they are testing: judgment, not loyalty to self-hosting. A candidate who reflexively argues for building is failing this question.

Cases where the stakeholder is right: low or spiky volume, no domain vocabulary, no hard latency requirement, no data-residency constraint. Buy, do not build.

Cases where they are not:

• Unit economics at scale. There is a crossover volume where self-hosting wins. Name it as a number.
• Compliance. HIPAA, GDPR and data residency frequently rule out sending audio to a third party at all.
• Real-time. A batch transcription API is not a streaming system, and retrofitting one into the other goes badly.
• Control. You cannot pin a model version, adapt to domain vocabulary, or fix hallucination on silence in someone else's API.

The strong answer ends with a cost, latency and compliance comparison plus a break-even volume, so the stakeholder can make the call themselves. If you are weighing this for a real team, our guide on hiring a Whisper engineer covers the build-versus-buy decision from the hiring side.

What they're really asking: Can you advocate for your ideas while staying collaborative?

Good answer structure (STAR method):

• Situation: "We were deciding between RNN-T and Transformer for streaming ASR..."
• Task: "I believed RNN-T was better for our latency requirements..."
• Action: "I prepared a doc with benchmark data, presented to the team, and we decided to prototype both..."
• Result: "RNN-T won, but the process helped us align on latency goals..."

Red flags to avoid: Being stubborn, not listening to others, making it personal

Why they ask: Speech tech moves fast. Can you adapt?

Good example topics:

• Learning Whisper when it came out and applying it to your use case
• Picking up Kaldi despite its steep learning curve
• Diving into self-supervised learning papers and implementing Wav2Vec

Key points to emphasize:

• How you approached learning (papers, code, experiments)
• Timeline (weeks not months)
• Concrete outcome (shipped feature, improved metric)

Bad answer: "It's a hot field" or "Good salary"

Good answer shows genuine interest:

• "I'm fascinated by how much context ASR requires—acoustic + linguistic + sometimes visual..."
• "Voice is the most natural interface, but we're still far from solving it..."
• "I built a project using Whisper and realized how challenging low-resource languages are..."
• "The combination of signal processing, deep learning, and linguistics is unique..."

Connect to company: "Your work on [specific product] aligns with my interest in [area]..."