
Run Qwen3.8-27B Locally with vLLM on Linux: Model IDs, GPU Memory, and Tool Calling
A Qwen3.8-27B server may fail before serving its first response because of insufficient GPU memory, incompatible checkpoint/kernel versions, or mismatched parser settings. A model-family recipe and a quantized checkpoint card can specify different tool-call parsers; neither parser should be silently substituted without an end-to-end test.
The practical deployment order is select an accessible checkpoint, verify the GPU, install a compatible vLLM build, start with a bounded context window, and test an actual request. This guide uses published model IDs and commands instead of presenting a theoretical weight-size estimate as a successful hardware benchmark.
Scope: Linux with supported NVIDIA GPUs and vLLM; text-only chat and tool-call smoke tests. The linked vLLM tool-call parser page documents the latest/developer-preview behavior; it may differ from the installed release. Verify the exact versioned documentation for your runtime. Qwen3.8-27B also has a vision tower, but image/video throughput and memory use are not tested here. The examples below are sourced from project documentation; they are not hardware runs performed by this site. Version-specific commands may change after publication.
1. Check the actual model architecture and memory budget
The Qwen3.8-27B model card and vLLM model recipe describe a 27B-class model with a hybrid-attention backbone, a vision encoder and a built-in multi-token-prediction (MTP) head. Of 64 language layers, 16 use full attention and 48 use linear attention. Its model-configured native context limit is 262,144 tokens, not a guarantee that any GPU can serve that length.
For rough weight-only math, 27 billion parameters at 2 bytes each are approximately 54 billion decimal bytes (50.3 GiB); at 0.5 bytes each the lower bound is 13.5 billion decimal bytes (12.6 GiB). Those figures exclude quantization metadata, non-quantized modules, vision components, MTP, CUDA workspaces, execution graphs, KV cache and batching. Consequently, do not add an assumed fixed 6.8 GB KV-cache allowance or claim that every 24 GB card can handle 32K context.
| Checkpoint / format | Public checkpoint | Hardware guidance | What is verified |
|---|---|---|---|
| BF16 base | Qwen/Qwen3.8-27B | Use an authorized large-memory GPU matching a versioned upstream recipe, not a presumed single-H200 guarantee | Base model and vLLM H200 recipe published |
| INT4, W4A16 | RedHatAI/Qwen3.8-27B-INT4 | Lower weight memory; assess real free VRAM before trying 24 GB GPUs | Checkpoint and serving recipe published; 24 GB success not verified here |
| FP8 | Qwen/Qwen3.8-27B-FP8 | Multi-GPU setup such as the documented 2 × RTX 5090 configuration | FP8 checkpoint and TP2 recipe documented |
| NVFP4 | Inferact/Qwen3.8-27B-NVFP4 | Documented 1 × RTX 5090 case needs eager execution | Vendor/model-specific recipe documents the workaround |
The first thing to check is GPU availability, before downloading tens of gigabytes of model weights:
nvidia-smi --query-gpu=name,memory.total,memory.free,driver_version --format=csv,noheader
python3 --version
If the driver is present but PyTorch cannot see the GPU, check your actual driver/PyTorch CUDA availability on the host before changing quantization formats.
2. Create a compatible, isolated vLLM environment
The currently retrieved Qwen3.8-27B vLLM recipe labels its instructions vLLM 0.17.0+ and discusses transformers >= 5.8.0 for Qwen processor classes. That is a recipe label, not an independently tested compatibility floor for every quantized checkpoint, Python runtime, GPU, or extension. For an actual installation, select a specific tested vLLM/PyTorch/CUDA combination from the official vLLM GPU installation instructions and record the exact build versions.
# Inspection after installing a versioned, supported stack using official
# instructions for YOUR Python, CUDA, PyTorch and GPU configuration.
python3 --version
python3 -m pip show vllm transformers torch
vllm serve --help
This deliberately does not provide a universal unpinned pip install --upgrade or promise that Python 3.11 works with every release. Install only inside a reviewed, isolated environment with the hardware/vendor-specific version matrix and rollback plan.
Check the installed package versions and vllm serve --help on the target machine. Do not carry over the old vllm >= 0.8.0 compatibility claim or add --enable-reasoning merely because an older tutorial used it. The current vLLM reasoning documentation uses --reasoning-parser qwen3.
3. Choose a versioned upstream serving configuration
The four checkpoints below differ in quantization format, GPU requirements, chat-template parser and actual evidence. This publication does not offer a copy-paste vllm serve command that mixes those settings, because no signed startup log or completed request was produced for such a combination here. Open the linked, current upstream model recipe or checkpoint card, record its tested release, and use the exact instructions only on appropriately authorized hardware.
A. BF16 base model
The Qwen3.8-27B vLLM recipe lists a BF16 path for Qwen/Qwen3.8-27B and an eight-H200 node description that also shows tensor-parallel-size=1. Those details alone do not prove our claimed single-card deployment or its available memory; inspect the recipe’s exact selected hardware and command in context. Its examples use qwen3_coder for automatic tool-call parsing and qwen3 for reasoning. The large 262,144-token configured context is not a promise that your GPU has enough free KV cache. Select a shorter context only after checking your installed vLLM version’s supported CLI semantics; treat that as a new untested configuration until it starts and answers a bounded request.
B. Public INT4 W4A16 checkpoint
The RedHatAI/Qwen3.8-27B-INT4 card is the checkpoint-specific serving reference. Unlike the base recipe, the model-card example uses qwen3_xml as the tool-call parser. This difference matters: copying qwen3_coder from the base recipe into this model’s configuration was not validated. Follow the exact model-card recipe with a release-compatible runtime, then independently verify parsed tool calls. A lower weight precision alone is not proof that a 24 GB accelerator can serve the model, its vision components, a chosen context and multiple requests simultaneously.
C. FP8 checkpoint
The Qwen3.8 vLLM recipe discusses the Qwen/Qwen3.8-27B-FP8 checkpoint, multiple GPUs and versioned Blackwell tests. Do not substitute a two-card recipe for a different GPU topology or change its tensor-parallel size and chat parser while calling the resulting command verified. Capture your own version, device names, startup memory log and response output.
D. NVFP4 checkpoint
The vLLM Qwen3.8 recipe also describes Inferact/Qwen3.8-27B-NVFP4 and an RTX 5090 case whose specific build required eager execution to avoid a CUDA-graph issue. That workaround is not universal across all drivers, vLLM versions or NVFP4 variants. Check its latest upstream recipe before issuing a new server command.
Safe procedure before attempting any of the upstream commands
- Verify the exact model repository, access permissions, quantization scheme and revision rather than guessing a compatible-looking repository name.
- Record the chosen GPU, driver, tested vLLM release, PyTorch/CUDA build and checkpoint revision. The upstream recipe’s version label is not a complete compatibility matrix.
- Compare required VRAM with actual free GPU memory; include non-quantized modules, cache, runtime workspaces, batching and simultaneous processes.
- Review and apply only the upstream command for the selected model and environment; do not combine flags from another checkpoint or change its documented parser without a controlled test.
- Run on an isolated, authorized local service, capture startup logs, and perform a real small chat/tool-call test. Report failure instead of claiming a successful deployment from an import or config file alone.
4. Verify the running server, not just the Python import
An imported package or a successful model download is not proof of a usable inference endpoint. Once vLLM finishes loading and reports its KV-cache capacity, query the locally bound API:
curl -fsS http://127.0.0.1:8000/v1/models | python -m json.tool
Then send a small chat request using the exact model identifier reported by the server.
from openai import OpenAI
client = OpenAI(
base_url="http://127.0.0.1:8000/v1",
api_key="EMPTY", # Only for a local development server with no exposed network access.
timeout=120,
)
model_id = client.models.list().data[0].id
reply = client.chat.completions.create(
model=model_id,
messages=[{"role": "user", "content": "In two sentences, explain what CUDA OOM means."}],
max_tokens=256,
)
print(reply.choices[0].message.content)
Qwen3.8 models think by default. For a no-thinking request, pass extra_body={"chat_template_kwargs": {"enable_thinking": False}}, as described on the model card. The exact field used to expose reasoning in the API may vary with vLLM version; check the reasoning documentation rather than assuming an older reasoning_content attribute.
Illustrative, unexecuted tool-call inspection
Start a server with a working --enable-auto-tool-choice and --tool-call-parser pair, then test whether the response contains a structured tool call.
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
model_id = client.models.list().data[0].id
tools = [{
"type": "function",
"function": {
"name": "get_gpu_status",
"description": "Return GPU free memory information from a trusted local monitor.",
"parameters": {
"type": "object",
"properties": {},
"additionalProperties": False,
},
},
}]
reply = client.chat.completions.create(
model=model_id,
messages=[{"role": "user", "content": "Use get_gpu_status before advising me about model memory."}],
tools=tools,
tool_choice="auto",
max_tokens=300,
)
message = reply.choices[0].message
for call in message.tool_calls or []:
print("Requested tool:", call.function.name, call.function.arguments)
# Inspect only. Do not execute model-generated commands without an allowlist.
An automatic tool-choice response may legitimately contain no tool calls; that alone neither proves nor disproves parser correctness. If a tool_calls payload is returned, it demonstrates a small part of the parsing path, not the correctness of the tool or the safety of an autonomous agent. These client snippets were not run with a live server by this site. Inspect the actual installed OpenAI-compatible client and vLLM release before using this example. For real agent integration, separately validate the actual client, MCP tool permissions and model tool-call schema.
5. Handle startup errors with measured limits
CUDA out of memory during checkpoint load: confirm the model ID, precision and free memory. On a small GPU, try a published INT4 variant only if the runtime supports it. Lowering --max-model-len affects KV requirements, not the core weight footprint. For 1 × RTX 5090 NVFP4, read the recipe’s --enforce-eager caveat before changing settings.
The model's max sequence length is larger than the KV cache capacity: check the startup estimate and lower --max-model-len. A 262K context declaration is not 262K usable memory on your device. Distinguish the model-serving engine from the workload and actual KV-cache allocation; do not infer one engine always fits better.
Unknown parser or a tool call returned as plain text: compare vllm serve --help and the official tool-calling documentation with the specific checkpoint’s chat template. Do not assume qwen3_xml and qwen3_coder are aliases.
Connection refused: ensure vLLM finished startup before sending a request. The client examples only connect to 127.0.0.1; that does not bind or protect the vLLM server listener. Before launching any upstream recipe, check that your installed vllm serve --help supports the desired loopback-only --host 127.0.0.1 setting, apply it to the actual server command and verify the live listener with ss -ltnp in the intended network namespace. If the server binds to 0.0.0.0, other interfaces may be reachable unless independently restricted. Never expose an unauthenticated model API to untrusted networks.
CUDA driver / wheel mismatch: inspect nvidia-smi, the installed PyTorch CUDA build and the vLLM GPU installation guide. Changing an unrelated parser flag will not fix incompatible CUDA components.
6. What these examples do, and do not, establish
This article checks publicly documented checkpoint identities, model-card parser differences, and stated prerequisites as they appeared during the November 13, 2026 review. The exact upstream launch commands live on the cited official sites and were not executed by this publisher. It does not claim that every command was executed on the site’s own GPU, nor does it claim a throughput result, exact VRAM total, tool-use reliability score, or universal 24 GB fit. Those require a recorded GPU model, driver, vLLM version, checkpoint revision, context length and startup logs.
To choose a deployment, first record the GPU and model ID. Next run one bounded configuration and save the startup KV-cache capacity. Only then tune context, concurrency, multimodal handling, prefix caching and speculative decoding. For deep serving internals, see the vLLM official documentation and the Qwen3.8-27B model recipe.
Source notes: Qwen official model card · vLLM Qwen3.8 recipe · vLLM tool-call parser guide.
Primary-source check reviewed November 13, 2026 (KST). The listed commands were not run on this site’s GPU. Verify current upstream sources and exact build before production deployment.