Speculators: Engineering the Unified Speculative Decoding Library for Production LLM Inference

Speculative decoding has moved from a research curiosity to a production necessity. The math is compelling: verify N candidate tokens in a single forward pass instead of generating one token at a time, and you get 2–5x latency reduction without any quality degradation. But the engineering gap between a paper’s algorithm description and a production-ready serving stack has been substantial — every team rolling out speculative decoding needed to reimplement training pipelines, convert checkpoints, and wire them into the inference engine.

The Speculators library (GitHub, docs) closes that gap. Built by the vLLM team in partnership with Red Hat’s AI Model Optimization group, Speculators provides a unified framework for training, evaluating, and serving speculative decoding draft models that deploy directly into vLLM. As of v0.6.0 (July 2026), it supports four algorithm families, online and offline training, and a HuggingFace-compatible model format that enables one-command deployment. This is not a research prototype — it is the production substrate for speculative decoding at scale.

Architecture: From Hidden States to Draft Tokens

The Speculators pipeline breaks into three phases: data generation, training, and serving.

Data generation extracts the verifier model’s internal hidden states alongside token IDs and loss masks. The library uses a custom vLLM worker extension that patches the model’s forward pass during the prefill phase to capture intermediate activations. For Eagle3, it saves per-sample .pt files containing input_ids, hidden_states (tensors from three captured layers), and loss_mask — the binary mask that restricts training to assistant-response spans, excluding user prompts. Asynchronous I/O via ThreadPoolExecutor overlaps disk writes with ongoing generation to maximize throughput.

A critical detail is the vocabulary mapping. The draft model operates on a reduced vocabulary (10k–32k tokens versus the verifier’s full vocabulary of 128k+). Speculators builds t2d (target-to-draft) and d2t (draft-to-target) tensors from token frequency statistics, which it collects during preprocessing. This mapping is what enables the draft model to project its predictions back into the verifier’s output space for verification.

Algorithm Support: Four Families, One Interface

Eagle3 (v0.3.0)

Eagle3 remains the most widely deployed speculative decoding algorithm. It fuses hidden states from three intermediate verifier layers as input to a lightweight autoregressive draft model. Speculators implements Eagle3 training with train-time-testing — a technique that simulates the multi-step draft sampling process during training rather than treating each position independently. This is where FlexAttention (He et al., 2024) becomes essential:

# Conceptual: Eagle3 train-time-testing attention masking
# FlexAttention splits the attention mask into blocks and computes
# only non-empty regions, making sparse multi-step masking tractable.
from speculators.eagle3 import Eagle3DraftModel, Eagle3Config

config = Eagle3Config(
    verifier_name_or_path="meta-llama/Llama-3.1-8B-Instruct",
    draft_vocab_size=16384,
    speculative_tokens=5,
    use_flex_attention=True,
)
model = Eagle3DraftModel(config)
# train.py handles batching via sequence concatenation + block-sparse masks

Without FlexAttention, the sparse attention masks from multi-step drafting would inflate activation memory by 4–6x. Speculators batches by concatenating sequences along the sequence dimension and configuring attention masks to treat them independently — integrating FlexAttention with torch.compile for the backward pass.

DFlash (v0.5.0)

DFlash (arXiv:2602.06036, ICML 2026) replaces autoregressive drafting with block diffusion. The draft model generates all K candidate tokens in a single forward pass using a non-causal attention pattern where queries within a block attend to all other tokens in the same block. This is fundamentally different from Eagle3’s autoregressive approach:

Eagle3:   token_1 → token_2 → token_3 → ... → token_K  (K serial passes)
DFlash:   [token_1, token_2, ..., token_K]              (1 parallel pass)

Training DFlash requires anchoring — randomly selecting positions in the sequence where predicted blocks are attached, keeping the number of predicted blocks fixed regardless of sequence length. v0.6.0 adds sliding window attention for DFlash, reducing KV cache allocation versus full attention and enabling efficient long-context handling.

P-EAGLE (v0.6.0)

P-EAGLE (contributed by AWS) extends Eagle3 with parallel drafting: instead of generating draft tokens autoregressively, the draft model produces all K candidate tokens in one forward pass, then constructs verification trees. On coding benchmarks, P-EAGLE reaches 4–5x speedup over standard decoding and 20–30% improvement over Eagle3 alone.

MTP (v0.6.0)

Multi-Token Prediction training finetunes the verifier’s own embedding layer to predict multiple future tokens simultaneously. The MTP head is small (~100M–400M parameters), so training runs on pre-extracted hidden states without loading the full verifier. After training, stitch_mtp.py merges the finetuned weights back into the verifier checkpoint for deployment with vLLM’s native MTP support.

Production Deployment

The serving contract is a speculators_config embedded in the model’s config.json on HuggingFace. This config specifies the algorithm, verifier model path, speculative token count, acceptance tolerance, and vocabulary mapping paths. Deployment becomes a single command:

# Serve an Eagle3 speculator for Llama 3.1 8B
vllm serve RedHatAI/Llama-3.1-8B-Instruct-speculator.eagle3

# Override verifier and tune speculative tokens
vllm serve RedHatAI/Llama-3.1-8B-Instruct-speculator.eagle3 \
  --speculative-config '{"verifier": "meta-llama/Llama-3.1-8B-Instruct-FP8", "speculative_tokens": 7}'

The key production knob is --speculative-disable-by-batch-size. At high concurrency (>32 concurrent requests), speculative decoding can reduce throughput by 30–40% because the GPU is already saturated with batched inference — there’s no idle memory bandwidth to exploit. Setting this threshold ensures automatic fallback to standard decoding under load. Red Hat’s published benchmarks for Gemma 4 31B with a DFlash speculator show measurable latency gains across batch sizes when this knob is properly configured (v0.5.0 blog).

Engineering Takeaways

The Speculators library is a case study in how to bridge research-to-production for ML techniques:

1. Standardized config format — The speculators_config embeds all deployment parameters in the model artifact itself, eliminating configuration drift between training and serving.

2. Algorithm diversity under one interface — Four fundamentally different draft strategies (autoregressive Eagle3, diffusion DFlash, parallel P-EAGLE, head-based MTP) share the same training/serving API, making algorithm selection a configuration change rather than a rewrite.

3. vLLM as both inference engine and data generator — By leveraging vLLM’s plug-in system for hidden states extraction, Speculators avoids maintaining a separate data pipeline. The same binary that serves production traffic generates training data.

4. Production guardrails — speculative-disable-by-batch-size, acceptance rate monitoring, and automatic fallback are not afterthoughts; they’re first-class features required for any production speculative decoding deployment.

For teams running LLM inference at scale, Speculators is rapidly becoming the standard way to deploy speculative decoding. The library is installable via pip install speculators and integrates with vLLM ≥ 0.18.0. The Red Hat AI Hub on HuggingFace hosts pre-trained speculator models for Llama 3.1, Qwen3, and Gemma 4 families, ready for vllm serve.

📖 Related Reads

• ToolBrain — tool reviews, LLM comparisons, and AI workflow guides

Cross-links automatically generated from CodeIntel Log.