
Configure Speculative Decoding in vLLM for Local Coding Agents
A coding agent can spend more time waiting for the next generated token than running the tool it just requested. Draft-model speculative decoding is worth testing when generation latency is the bottleneck. But there is no universal 3x switch.
Run a Draft Model with vLLM
The draft proposes several tokens; the target model verifies them. vLLM’s developer-preview draft-model guide configures this through --speculative-config, not the older standalone --speculative-model flag.
The table below summarizes an upstream Qwen3 example from developer-preview documentation, but is deliberately not a copy-paste launch command. vLLM’s official documentation is the source of the exact CLI for your installed version. Validate model availability, GPU architecture, memory headroom, tokenizer compatibility and the current CLI before creating an actual server.
| Configuration item | Example from the official guide | What to check |
|---|---|---|
| Target model | Qwen/Qwen3-4B-Thinking-2507 | Exact supported checkpoint and engine version |
| Draft model | Qwen/Qwen3-0.6B | Method and tokenizer compatibility |
| Speculation | method: draft_model; num_speculative_tokens: 5 | Supported speculative-config schema |
| Constrained upstream context example | max-model-len 2048 | Demonstration only; not a recommendation for a long-context coding agent |
| GPU memory allocation | No universal budget is established by the cited example | Size from installed release, real GPU headroom, draft weights and KV-cache measurements |
| Local-only HTTP API listener | Use 127.0.0.1 as a safety modification, not a value copied from the upstream example | Upstream example uses a wildcard listener; keep both API and internal worker ports isolated |
Security review (November 12, 2026): Explicitly bind the API listener to loopback for the local-agent scenario. The vLLM security guide warns that an API key does not protect every HTTP route and that internal worker ports need their own network boundary. Do not expose these services to untrusted networks. The example’s model pair is from the official guide. Loopback binding is a separate editorial safety change, and all runtime settings remain unexecuted here. Use the table as a checklist, not a working command.
The official draft-model example informs the configuration fields. Review loopback API binding, actual worker-network exposure and available GPU memory independently; a realistic coding-agent prompt, tool history and output may require a much longer context than the upstream 2048-token demonstration.
The verification step can preserve the target model’s output distribution under an exact speculative sampling implementation. That does not promise identical generated text across two sampling runs, and it does not guarantee a latency gain. For model-specific or experimental verification modes, check the vLLM method documentation.
When Speculation Helps, and When It Hurts
The interesting variable is the accepted draft length relative to the extra draft and verification work. Syntactically repetitive code may help a well-matched draft, but an acceptance percentage cannot be inferred from the fact that the prompt asks for Python.
| Condition | What to check | Possible outcome |
|---|---|---|
| Low-concurrency, memory-bound decoding | Inter-token latency at equal output lengths | A useful latency reduction |
| Busy, high-throughput serving | Output tokens per second across the same QPS | Little benefit or a regression |
| Weak draft-target agreement | Mean accepted tokens per proposal | Draft overhead outweighs saved work |
| Long contexts or tight GPU headroom | KV cache utilization and preemption | OOM, shorter context, or slower requests |
| Different tokenizers or incompatible model variants | Startup logs and vLLM compatibility checks | Configuration rejected or incorrect assumptions |
Measure your own agent’s QPS, prompt sizes, response lengths, and concurrency before deciding whether to use the draft model. The vendor example demonstrates the configuration method, not a speedup for any particular workload. A synthetic one-line code completion is not a proxy for a 40-tool-call repair session.
Compare Target-Only and Speculative Serving
For a target-only baseline, use the same target checkpoint, context budget, output limits, GPU, server version and approved loopback/network configuration, but leave the draft-model speculative configuration disabled. Derive and verify the actual launch syntax from the installed release’s official CLI reference, rather than assuming the above table is a validated command.
Repeat the test with the draft-model configuration above, using the same hardware, dataset, sampling parameters, output-token budget, and concurrency. Capture time to first token, inter-token latency, output tokens per second, and failure rate. Monitor draft acceptance separately; it describes one part of the mechanism, not end-to-end performance.
Before testing, confirm that the selected engine release accepts the shown speculative configuration and can load the target/draft pair. The upstream draft-model example documents the Qwen3 configuration pattern but provides no performance results for this site. Verify version, memory headroom and actual startup rather than reporting a speedup for an untested setup.
Keep the comparison reproducible. The official draft-model example is a starting point for configuring the target and draft models; collect your own latency and throughput measurements instead of treating the configuration as benchmark results.
Memory and Configuration Traps
Measure the draft-model resource overhead for your engine release. Before enabling speculation, check real GPU memory allocation for the target-only baseline and again with the selected draft model. We do not claim a universal extra VRAM amount, placement policy or performance cost; they depend on engine, model, context and concurrency.
Tokenizer and method compatibility must be checked. A draft from an unrelated model family is not a safe default. Consult the installed release documentation; do not assume any small model can act as a draft.
Sampling matters. Avoid fixed assertions such as “temperature 0.8 always causes less than 40% acceptance.” Actual acceptance depends on the token distributions, draft model, workload, and sampling algorithm.
Latency is not quality. Exact speculative verification is designed to preserve the target distribution, not to eliminate hallucinations or improve factual correctness. Judge coding-agent accuracy with the same tests you use for the unmodified target.
Connecting Coding Agents
Once the server is healthy, point your editor or agent’s OpenAI-compatible client at http://127.0.0.1:8000/v1. Use the served model ID actually exposed by your local server; it may be an alias configured for the target model rather than the raw checkpoint name. Do not publish an expected completion-time reduction until you have an actual before-and-after measurement.
The useful engineering rule is simple: measure with and without speculation on the workload you intend to run. If the draft adds cost without lowering latency, turn it off.