Cloudflare disclosed on October 26 that its GLM-5.3 model on Workers AI returned 155 responses containing data from other customers’ requests between August 27 and September 9, after a configuration mismatch collided with a bug in the SGLang inference engine. Security researcher Gadi Evron of Knostic reported the flaw on September 8, Cloudflare reproduced it within 25 minutes the next day, took the model offline, and restored it the same day after fixing the configuration and patching the engine. Cloudflare says it found no evidence of deliberate exploitation and that no other Workers AI model was affected. Operators running SGLang in split prefill-decode mode should upgrade to version 0.5.22 or later.
A Pool Too Small For The Context
Cloudflare deployed GLM 5.3, a text generation model from Z.ai built for agentic coding workflows, with a 1,048,576-token context window. The KV cache pool on its prefill GPUs, however, held only 681,280 tokens.
Cloudflare says it knew about the mismatch before launch. It expected the engine to return an error for any prompt it could not process, and its long-context tests returned successful responses, so the team did not believe the gap was an issue.
That testing had a gap Cloudflare now admits: the team validated HTTP response codes and engine performance at long contexts, without validating model output and coherence. It now checks KV pool size against context length directly from the running engine before any disaggregated deployment change ships.
How Stale Pages Reached Other Users
When a prompt longer than the pool arrived, SGLang’s scheduler rejected it and swapped it for a one-token placeholder so it could drain through the batch machinery. On a single node that worked: the client received an error.
In disaggregated mode, the admission queues on both prefill and decode checked whether a request was finished, not whether it was rejected. The rejected request reported itself as live, so prefill ran the placeholder, sent one page of KV cache to decode, and signalled transfer complete. Decode had reserved eight pages for the request, so it generated a response while reading stale pages that still held earlier requests’ data.
The client received HTTP 200 instead of HTTP 400, with a coherent reply that arrived faster than a full prefill should take. Every request over the threshold behaved this way. The bug had been reported to SGLang by another user as issue 30233 on July 6, 2026, Cloudflare noticed while investigating.
Fixed The Same Day, Disclosed Weeks Later
Evron’s report arrived via HackerOne on September 8 at 23:11 UTC. Cloudflare began verifying it the next morning, declared a security incident at 13:22, and set the model to private at 14:00. All prefill nodes moved to the corrected configuration by 14:52, and the model was restored to public at 19:40 after a rebuilt deployment with the engine patch, validated against the researcher’s reproduction script.
The configuration fix enlarged the prefill KV pool to about 2.04 million tokens, roughly twice the context length, so a valid prompt always fits. The engine fix retires a request carrying a pending rejection at the disaggregation admission queue, with its original error, before any bootstrap, allocation or transfer happens.
Upstream, the fixes landed in SGLang v0.5.20, v0.5.21 and v0.5.22, released October 19. Cloudflare published its report on October 26 as a coordinated disclosure with the researcher and the SGLang team at RadixArk.
Why It Matters
This leak matters because the failure sat in the inference plumbing, not in the model itself. GPU memory pages are not zeroed between requests; the next request simply overwrites them. That design is common and fast, but it makes the engine responsible for tracking exactly who owns each page.
When that tracking fails, the result is silent: a normal-looking answer, a success code, and another customer’s data inside it. The incident joins a run of hosted-platform failures, including the OpenCode console breach, in which provider keys were exposed through a platform flaw rather than a model flaw.
Cloudflare’s advice to operators is concrete: measure the KV pool against the context length from the engine’s startup log after every change, upgrade to SGLang v0.5.22 or later, run deferred KV release on both roles, and turn on corruption detection where the model supports it. Longer term, it wants page-level tags checked at read time, so the engine fails one request rather than returning someone else’s data.
Background
Workers AI runs GLM 5.3 with SGLang in a disaggregated setup: prefill GPUs process the prompt and hand its KV cache over remote direct memory access to decode GPUs that generate the response, on Cloudflare’s Workers AI developer platform. Cloudflare says it has been working on KV cache memory-safety bugs for several months, and this incident sped up work it had already started with the SGLang team.
Source: Cloudflare Blog: Cloudflare GLM leak disclosure
0 Comments