How to Fix HTTP 524 Timeouts and SSE Buffering for LLM Inference on Linux
LLM Serving(Updated )

How to Fix HTTP 524 Timeouts and SSE Buffering for LLM Inference on Linux

An LLM server may stream correctly on localhost yet deliver output in bursts, disconnect, or return an error through a reverse proxy. These are different failure modes. A delayed token is not automatically a Cloudflare 524, and a 524 does not prove that Nginx buffering caused the delay.

Updated November 12, 2026: The current Cloudflare Error 524 guide gives a 125-second default Proxy Read Timeout. This revision separates timeout and buffering layers and avoids universal heartbeat or kernel-tuning promises.

Identify the failing layer before changing a setting

Symptom First inspection Avoid assuming
SSE output arrives in batches Application yield times, Nginx buffering, intermediate proxy A 4 KB buffer must fill before each write
Nginx 504 while the upstream is quiet Upstream error logs and Nginx read-gap timeout Raising the timeout fixes every application stall
Cloudflare 524 Origin processing/response timing and Cloudflare logs One initial SSE comment makes the connection immune
Long idle connection drops Which network hop closed the connection Linux kernel keepalive is necessarily responsible

Nginx’s proxy module documentation explains that proxy_buffering off forwards upstream data as it arrives. Its proxy_read_timeout defaults to 60 seconds and measures the interval between successive upstream reads, not a maximum total request duration. Changing it cannot override an unrelated edge timeout.

Cloudflare’s 524 guide states 125 seconds for the default Proxy Read Timeout and 30 seconds for the separate Proxy Write Timeout. Its connection limits reference lists Enterprise-specific read-timeout adjustment. Sending SSE data can help with some inactivity problems, but it is not a universal bypass for Cloudflare, clients or upstream components.

Step 1: Compare direct and proxied responses

Capture the error code, request/first-byte timestamps, time of every output chunk, endpoint and proxy chain. Use an authorized local staging endpoint, not a production API with missing authentication.

curl -N --no-buffer --include http://127.0.0.1:8000/stream-test

The -N flag disables curl’s own output buffering. Use the same request via the staging proxy and compare server-side logs too. A successful curl output does not prove every proxy hop is healthy.

Step 2: Scope Nginx buffering changes to the affected route

This is a location fragment for review inside an existing authenticated, TLS-enabled server block, not a complete production configuration:

location /stream-test {
    proxy_pass http://127.0.0.1:8000;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_buffering off;
    proxy_cache off;
    # Inactivity gap between upstream reads, not a total deadline.
    proxy_read_timeout 180s;
}

The value 180s is an example only. Set it against measured upstream gaps and the entire proxy chain. Preserve existing authentication, rate limits and request-size controls. Run nginx -t in the administrator’s change procedure before any approved reload and keep a rollback configuration.

Step 3: Verify actual SSE generation

FastAPI documents StreamingResponse for async generators. This local demonstration generates three small events rather than a real inference workload.

import asyncio
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()

async def demo_events():
    yield ": stream opened\n\n"  # SSE comment, not an edge timeout promise
    for i in range(3):
        await asyncio.sleep(1)
        yield f"data: chunk-{i}\n\n"
    yield "data: [DONE]\n\n"

@app.get("/stream-test")
async def stream_test():
    return StreamingResponse(
        demo_events(),
        media_type="text/event-stream",
        headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
    )

The X-Accel-Buffering header helps only when an intermediary supports and honors it. A blocking inference function may prevent a timely yield; a staging SSE demo passing does not prove production inference can stream.

Step 4: Handle long processing gaps explicitly

When a legitimate inference job can stay silent longer than a permitted read interval, prefer job submission plus authenticated status polling, or a separately managed results channel. Cloudflare recommends polling for long-running HTTP operations.

An early :\n\n SSE comment is not a guarantee against 524. Moving an API hostname to DNS-only removes Cloudflare proxy protections; do not make that change without direct-origin TLS, authentication, firewall rules and rate limiting. Do not use host-wide sysctl --system or change /etc/sysctl.d to treat an HTTP issue with an unverified cause. TCP keepalive, response streaming, proxy idle intervals and client timeouts are different mechanisms.

Verification and limits

  1. Compare the same authorized staging request at every proxy hop, including per-chunk timestamps.
  2. Identify the component closing the connection using origin, Nginx, edge and client logs.
  3. Determine whether the failure is a read-gap timeout, total operation duration, blocked event loop or client disconnect.
  4. Revert ineffective timeout changes, and document what was actually measured.

No Cloudflare-plan experiment, GPU benchmark or empirical token-latency measurement was performed for this revision.

Official references

Reviewed November 12, 2026. These are diagnostic steps, not a guaranteed fix for every deployment.

BC

Written by B. Cho

Systems & AI Infrastructure Engineer

Specializing in GPU inference engines (vLLM, SGLang, Ollama), CUDA memory allocation, and Model Context Protocol (MCP) infrastructure on Linux workstations.