
How to Fix 'RuntimeError: Expected all tensors to be on the same device' in PyTorch and vLLM on Linux
Expected all tensors to be on the same device is a placement error, not a signal to allocate more GPUs or force every tensor onto cuda:0. First find the exact operation and the devices of its operands. A CPU tensor and a CUDA tensor, or tensors on distinct CUDA devices, can be incompatible for that operation.
Updated November 12, 2026: Earlier examples conflated per-node LOCAL_RANK with global process rank, reassigned Parameter.data inside forward(), assumed four GPUs and a large model, and recommended unrelated vLLM executor settings. This revision separates ordinary PyTorch placement, distributed training, and vLLM-managed serving. No GPU workload was executed to prepare this correction.
1. Inspect the actual mismatched operands
Capture the full exception and the operation where it occurred. Before a matrix multiplication or custom adapter, inspect device metadata without printing sensitive input values:
# Diagnostic fragment using variables from YOUR application.
print("activation:", activation.device, activation.dtype)
print("weight:", layer.weight.device, layer.weight.dtype)
assert activation.device == layer.weight.device, "check placement before matmul"
These variables are placeholders, not a complete runnable script. A model may be intentionally sharded across devices; do not automatically move the entire model to whichever GPU held the last input.
For a genuinely single-device model, move the model once during setup and place the input batch on that device:
# Fragment: assumes torch, an existing nn.Module, and input tensors.
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model = model.to(device)
inputs = inputs.to(device)
output = model(inputs)
Do not blindly cast attention_mask to the same floating-point dtype as the hidden states. Depending on the model, a mask may need a boolean, integer or additive floating-point representation. Move it to the correct device while preserving its expected type.
The PyTorch CUDA notes explain CUDA device selection and transfer semantics.
2. Avoid silently replacing trainable weights in forward()
A previous example reassigned self.lora_A.data and self.lora_B.data to a new device and dtype during each forward pass. This bypasses normal model/optimizer setup and is outside ordinary optimizer and autograd setup and is difficult to reason about safely.
Instead, construct the adapter on its intended device and dtype, move the module at model initialization, and configure the optimizer after final parameter placement. Treat a change to an active distributed training model as an explicit, controlled migration with optimizer state handling.
The PyTorch autograd documentation distinguishes recorded operations, no-grad updates and in-place changes. Direct .data replacement is not a universal solution for cross-device errors.
3. Local rank selects a local GPU; RANK is global
Under torchrun, LOCAL_RANK is a rank within one node and RANK is global. They can differ on multi-node jobs. WORLD_SIZE, MASTER_ADDR and MASTER_PORT describe the job rendezvous.
The PyTorch 2.15 torchrun reference explains that the launcher exports these values. Initialize a process group without overriding global rank with local rank:
# Fragment for an authorized NVIDIA GPU torchrun worker.
import os
import torch
import torch.distributed as dist
local_rank = int(os.environ["LOCAL_RANK"])
torch.cuda.set_device(local_rank)
dist.init_process_group(backend="nccl")
try:
device = torch.device(f"cuda:{local_rank}")
print("device:", device, "global rank:", dist.get_rank())
# Move the actual model and inputs before creating DDP.
finally:
dist.destroy_process_group()
This is an initialization pattern, not a tested training script. A CPU-only environment, unsupported NCCL build or an occupied GPU can fail. A documented command shape for one authorized machine is:
# Adjust worker count to your own hardware and workload.
torchrun --standalone --nnodes=1 --nproc-per-node=2 train.py
For multi-node training, configure a shared rendezvous host, unique job ID and the appropriate distributed backend; --standalone is for single-node jobs. See PyTorch distributed process groups.
4. Diagnose vLLM serving separately from custom PyTorch code
If a custom PyTorch layer fails, fix its operand placement. If an error arises within vLLM, record the specific engine version, model checkpoint, worker traceback, GPU topology and logs. Do not mix raw Hugging Face device-map assumptions into a vLLM-managed worker. Switching mp and Ray is not a demonstrated solution to a parameter-device error.
Check the vLLM CLI reference for version-supported arguments. Avoid universal 70B-model, 4/8-GPU, 16K-context or 90% GPU allocation recipes. The following is an unexecuted API-listener example, not a validated compatible checkpoint or memory budget:
vllm serve Qwen/Qwen2.5-0.5B-Instruct --host 127.0.0.1 --port 8000
The vLLM security guide warns that --api-key does not protect every HTTP path. Even with an API listener on loopback, internal distributed communication ports need separate firewall/network isolation.
For speculative decoding, consult the official --speculative-config schema rather than using deprecated --speculative-model flags.
Verification boundary
- Record the exact device pair and failing operation.
- Determine whether placement belongs to a single device, DDP, manually sharded PyTorch model or the vLLM engine.
- Change only the correct owning component while preserving input dtypes and optimizer state.
- Compare an authorized repeat of the exact workload, not just startup.
- Preserve logs and roll back changes that affect unrelated workers.
No CUDA training, Docker change, model loading or GPU benchmark was performed for this article.
Primary sources
- PyTorch CUDA semantics
- PyTorch Autograd mechanics
- PyTorch torchrun rank semantics
- PyTorch distributed process groups
- vLLM serving CLI
- vLLM speculative decoding
- vLLM security
Reviewed November 12, 2026. Examples are version-dependent diagnostic guidance, not a benchmark or guaranteed repair.