# SGLang 披露并修复 PD 分离服务中的跨请求 KVCache 竞态问题
> 原标题:SGLang security update: improving request isolation
- 来源:LMSYS Blog
- 发布时间:2026-10-25T16:00:00.000Z
- AIHOT:https://aihot.news/items/bb39zcs7eu91fi0ze136tayg2
- 原文:https://www.lmsys.org/blog/2026-10-26-cf-security
## 摘要
SGLang 团队与 Cloudflare 协调披露 2026 年 8 月 27 日至 9 月 9 日影响 Workers AI 上 @cf/zai-org/glm-5.3 的跨请求 KVCache 暴露问题。
## 推荐理由
SGLang 团队复盘了异步 KVCache 传输中的跨请求泄漏机制、复现方法和修复方案,对运维长上下文 PD 部署有直接指导意义。
## 正文 · 中文译文
2026 年 9 月 9 日,Cloudflare 通知我们,其 Workers AI 上的 GLM-5.3 部署存在跨请求 KV 缓存暴露问题。我们与 Cloudflare 合作,处理了涉及的请求拒绝路径,并加强了分离式服务中相关的缓存生命周期处理。
我们与 Cloudflare 以及通过 Cloudflare HackerOne 项目报告该问题的研究员协调发布此披露。Cloudflare 的报告涵盖其生产事故和客户影响,研究员的报告(即将发布)将描述发现过程和外部验证。本文重点介绍底层机制、我们所做的更改以及对 SGLang 运维人员的指导。
[在此阅读 Cloudflare 的报告。](https://blog.cloudflare.com/kv-cache-exposure-remediation/)
## 影响与受影响的部署
Cloudflare 报告称,该事故在 2026 年 8 月 27 日至 9 月 9 日期间影响了 Workers AI 上的 `@cf/zai-org/glm-5.3`。其调查未发现其他 Workers AI 模型受到影响。
Cloudflare 的调查发现,超过预填充 KV 缓存容量的请求可能返回另一用户会话的内容。有关事故范围和客户影响的详细信息,请参阅 Cloudflare 的事故报告。
该问题发生在分离式预填充/解码服务中,即由不同的 worker 处理提示并生成响应。单节点拒绝路径则会返回错误。
## 发生了什么
Cloudflare 的部署配置的上下文长度为 1,048,576 个 token,但其预填充 KV 缓存池只能容纳 681,280 个 token。因此,一个请求可能符合配置的上下文窗口,却超出可用于处理它的内存。
在 SGLang 的分离式预填充/解码路径中,被标记为拒绝的请求仍可能继续通过交接流程。随后解码可能读取并未为该请求填充的缓存页。配置不匹配使拒绝路径变得可达。下文描述的准入修复确保被拒绝的请求在进入分离式交接之前返回错误。它补充了比较池容量与配置上下文长度的部署检查。
Cloudflare 于 9 月 9 日禁用了对该模型的公开访问。它扩大了预填充池以容纳配置的上下文窗口,并部署了 Cloudflare 编写的引擎补丁,对拒绝的请求返回错误。Cloudflare 报告称,在当晚恢复模型之前,它已根据研究员的复现及其自身的金丝雀标记测试验证了这些更改。
上游修复 [#36988](https://github.com/sgl-project/sglang/pull/36988) 在处理中止请求的最终预填充结果时将其退役。后续更改 [#38935](https://github.com/sgl-project/sglang/pull/38935) 更早地停止被准入拒绝的请求,使其在进入分离式交接之前就被拦截,从而避免不必要的工作和延迟。
除了准入修复之外,我们还与 Cloudflare 合作处理 SGLang 在请求中断时如何管理 KV 缓存传输的问题。
在预填充-解码(PD)分离式服务中,已中止的请求仍可能有正在进行的 RDMA 写入。如果解码在这些写入完成之前释放并重新分配目标槽位,迟到的写入可能覆盖另一请求的 KVCache 并破坏其输出。
Cloudflare 的报告还描述了我们在相关 KV 缓存生命周期问题上的联合工作。以下各节解释了延迟释放机制、额外的传输保障措施和完整性检查,以及我们的验证结果和对运维人员的指导。
## 背景
在 PD 分离式服务中,prefill 阶段和 decode 阶段运行在独立的引擎上,每个请求的 KVCache 必须从 prefill 引擎迁移到 decode 引擎的内存池中。为了性能,这一传输是异步的。Mooncake、NIXL 或 MORI 等传输引擎将其作为单边 RDMA WRITE 操作发出,在 prefill 计算仍在进行时,将 KVCache 分块流式写入预先保留的目标槽位。
由于传输是异步的,一个请求可能在其部分写入仍在进行中时终止。此时 prefill 可以丢弃仍在其软件队列中等待的分块,但无法停止已经发往目标的写入,例如当时正在传输中的分块。在早期的实现中,当请求在传输中途终止(客户端断开连接、超时、传输故障)时,decode 会在终止时将其目标槽位归还给分配器,而不等待此类写入完成。如果分配器在该写入落地之前重新分配了这些槽位,已终止请求的 KVCache 就会落入另一个请求的 KVCache 中。在典型工作负载下,这种损坏很少发生,尽管长上下文会使其可能性大大增加(见下文事实 4)。每次发生都是静默的。传输报告成功,没有任何组件抛出异常或记录错误。单个被污染的槽位就会毁掉整个生成过程,因为注意力机制会读取每一个位置。
结果,一个要求模型从 1 数到 15 的请求,返回的却是一串不连贯的无关 token。随后我们着手让这一竞态条件可按需复现,并端到端地修复它,从传输引擎层一直到服务引擎。
## 问题所在:KVCache 槽位究竟何时才算真正空闲?
该故障分为四步:
1. 客户端断开连接、准入拒绝或传输故障导致请求 A 在传输中途中止,此时至少有一个已发出的写入尚未落地。
2. Decode 在中止时释放 A 的 KVCache 槽位。分配器将其标记为空闲,而远程写入器仍在向其写入。
3. 分配器将这些槽位交给受害请求 B。
4. A 已发出的写入落入 B 的 KVCache 中,B 使用外来的键和值进行解码,且没有任何错误。
这些步骤适用于任何使用异步 DMA(推送或拉取)在引擎之间迁移 KVCache 的 PD 实现,因为每个实现都必须决定槽位何时空闲。我们在 SGLang 中诊断并修复了这一类问题。

_**图 1:**竞态条件。中止操作释放了 A 的槽位,而 A 已发出的写入仍在落地;B 继承了这些槽位,迟到的写入落入其 KVCache 中。_
关于该系统的四个事实使每一步都成为可能。
**1\. 分配器可以在写入仍在进行中时将槽位标记为空闲。**在 PD 分离中,decode 引擎为每个传入请求预先保留目标 KVCache 槽位,其分配器将它们跟踪为已分配或空闲。但“空闲”还有第二个隐藏条件:只有当_没有远程写入器仍能触及该槽位_时,该槽位才可以安全复用。这一类 bug 正源于这两个定义之间的差距。每一个都是分布式 use-after-free,其中悬空指针是位于另一台机器 NIC 队列中的写请求。修复方案使第二个条件变得显式、可观测且有界。
**2\. 已启动的 RDMA 工作无法取消,push 和 pull 都是如此。** SGLang 的 KVCache 传输是基于 push 的 \[3\]。Prefill 将 KVCache _写入_ decode 的内存,因为 prefill 知道它产生每个 chunk 的时刻,并且可以逐 chunk 地将传输与正在进行的计算重叠。在 NIC 及以下层面,若不损害可用性,取消很难实现。RDMA API 没有可以取消已提交到 NIC 队列对的单个工作请求的调用。Python 无法在调用中途中断传输引擎的阻塞式 C++ 调用(例如 Mooncake 的 `batch_transfer_sync`)。如果该调用超时,其工作请求会保持已提交状态且没有完成记账,因此这些写入可能在未知的稍后时间落地。单边 RDMA WRITE 绕过接收方 CPU,因此 decode 不会收到每次写入的信号以供其等待。NIC 还按数据包放置数据:一次在传输中途被中断的写入仍会在目标端留下部分写入的前缀,因此即使看起来像取消的路径也可能改变目标内存。
基于 pull 的设计也无法摆脱这一点。如果 decode 改为发起 RDMA READ,那么在 pull 中途到达的中止会让 decode _自己的_已提交读取落入它刚刚释放的槽位,因为已提交的 READ 和已提交的 WRITE 一样无法收回。在 pull 下,危险还会反向发生:如果 prefill 在中止时释放其源页面,而分配器在 READ 仍在进行时重新分配这些页面,那么该读取会把另一个请求的 KVCache 取到 decode 中。释放后又被重新分配的槽位会在任一端破坏传输。

_**图 2:**Python 调度器之下的传输栈。中止可以停止未来的 chunk,但所有已经交给 C++ 引擎或 NIC 的内容都无法收回,而接收方 CPU 根本看不到这些写入。_
**3\. 完整的存活处理始于路由器,但不能止于路由器。** 只有当 P/D 对的_两半_都得知中止时,中止才是安全的。设计良好的 PD 路由器会成对取消,因此任何一方都不会为一个永远不会到来的对端等待 300 秒超时。在 `sgl-router` 中,[#41612](https://github.com/sgl-project/sglang/pull/41612) 在 [#39461](https://github.com/sgl-project/sglang/pull/39461) 的断连即中止机制之上构建了 PD 配对。该对共享一个由路由器生成的请求 id。客户端断连会中止 decode,prefill 失败会丢弃 decode 请求,从而使引擎中止它,而 prefill 通过 decode 引擎侧的 `ABORT` 通知得知中止。`sgl-model-gateway` 将同样两个事件配对([#19524](https://github.com/sgl-project/sglang/pull/19524) 用于断连,[#29017](https://github.com/sgl-project/sglang/pull/29017) 用于 prefill 失败)。在该引擎侧通知出现之前,只中止一侧的路由器必然导致竞态:decode 释放其槽位,而从未被告知的 prefill 继续推送。即使完美的路由器也只能缩小窗口。中止会异步到达 prefill,经过一些调度和网络延迟,而传输引擎已经启动的任何内容(进行中的 C++ 调用、已提交的工作请求)都会继续。路由器决定谁死亡(中止编排),引擎决定内存何时安全(排空正确性)。
**4\. 竞态窗口随 prefill 上下文长度而扩大。** SGLang 会分块传输并与 prefill 计算重叠,因此暴露窗口覆盖了 prefill 本身的大部分时间。短提示词的传输只持续几毫秒。而 128K–1M token 的上下文,根据模型不同,每个 token 的 KVCache 为数十到数百 KB,写入会在数秒到数分钟内持续进行,跨越许多分块和许多已提交的工作请求。中止_之后_的尾部也会随之增长。长上下文意味着更多分块在途、更多工作请求在中止发生时仍在排空,因此已启动的写入可能在分配器回收空闲槽位之后仍持续落地。在内存压力下,中止一个长请求会使“复用后写入”成为预期结果。长上下文工作负载也正是客户端超时并在请求中途断开连接的那类场景。因此,长上下文同时扩大了窗口、延长了中止后的尾部并提高了中止率,所以随着部署转向长上下文服务,这类 bug 就会浮现。
### 影响与暴露面
此处描述的中止传输竞态需要三个条件:
1. 启用了 PD 分离,并配有 KVCache 传输后端(Mooncake、NIXL、MORI 或自定义 `CommonKVManager` 后端)。没有这条跨引擎传输路径的单实例部署不受此特定竞态影响。
2. 请求在传输中途终止(客户端断开、路由器取消、超时、传输故障、prefill 失败或 hicache 恢复失败),同时其至少一个写入已启动但尚未落地。
3. 分配器在已启动的写入落地之前将空闲槽位重新分配给另一个请求,在内存压力下这会在几毫秒内发生。
去掉三者中的任意一个,损坏就不会发生。在 PD 部署中,默认配置从 v0.4.x 系列(2025 年初)PD 分离随 RDMA 后端发布时起,直到 v0.5.21(MORI 为 v0.5.22)均受影响。另一个具有相同特征的变体需要解码侧分层缓存卸载,该功能默认未开启(已由 [#24226](https://github.com/sgl-project/sglang/pull/24226) 和 [#30899](https://github.com/sgl-project/sglang/pull/30899) 修复)。
**确定性还是概率性?** 在生产环境中该竞态是概率性的,因此运维人员会看到偶发的损坏输出,却无法复现。一旦窗口被保持打开,它就能按需复现(下一节我们的测试装置,以及社区复现器 [#32564](https://github.com/sgl-project/sglang/pull/32564))。客户端可以通过发送长上下文请求并在传输中途断开来故意提高概率,这会最大化竞态窗口和中止率。客户端无法选择受害者或落地槽位,且数据流是单向的。被中止请求的 KVCache 会落到另一个请求的 KVCache _之中_,而不会有任何数据流回中止方。因此现实影响是单向的跨请求泄漏,被中止请求的上下文片段可能出现在另一个客户端的输出中,正如我们的复现中所见。中止方不会获知任何信息,但在多租户服务中,这既是完整性问题,也是数据泄漏问题。
**部署者能否判断暴露是否已经发生?** 部署者没有可靠的方法来判断,因为损坏不会留下错误或日志行。可选的 PD 校验和([#39500](https://github.com/sgl-project/sglang/pull/39500))可以捕获部分迟到的写入(见 PD 校验和)。升级后,使用 [#41128](https://github.com/sgl-project/sglang/pull/41128) 指标确认保护已生效(见运维人员建议操作)。
## 确定性复现
在声称修复之前,我们需要先复现这个竞态,因此我们构建了一个测试装置,可在 1P1D 对上按需触发它。prefill 的 `send_kvcache` 内部有一个由环境变量控制的延迟,让已启动的写操作保持在进行中,`--max-total-tokens 96` 强制立即复用槽位,而插桩会记录每个请求的槽位范围以及每次释放或保留的决策。
在窗口保持打开的情况下,重叠每次都会出现:Decode 在其写操作仍在进行时释放请求 A 的槽位 `[14..44]`,请求 B 分配 `[1..32]`,两个范围在 `[14..32]` 上发生冲突。软件延迟无法让写操作本身在单个 P/D 对上延迟落地,因为传输队列会将写操作串行化到同一个 decode 引擎。为了展示_错误输出_,而不仅仅是槽位号重叠,我们在受害者的提交点将已中止请求的已释放页复制到重叠区域上,这正是延迟写入所做的事。没有修复时,受害者的输出是乱码。有了修复后,decode 保留了捐赠者的槽位,受害者获得了新的槽位,输出与基线完全一致。
一旦两个 prefill 引擎共享一个 decode,真实的延迟落地也能在真实硬件上端到端复现,因此没有任何东西会串行化它们的写操作。我们在一个 2×H200 对(基于 RDMA 的 Mooncake)上运行了它,并禁用了保留机制。一次单边中止释放了捐赠者的 430 个槽位,受害者在其中预分配了 `[1..105]`,而捐赠者已启动的写操作在 decode 中途落地。受害者一直正确计数到 80,然后其输出变成了捐赠者的提示文本:另一个请求的 KVCache 从其生成过程中被读回。在启用保留机制且时序相同的情况下,decode 记录了保留、排空确认和释放,受害者等待释放,然后产生了与基线逐字节完全相同的输出。完整配方(测试装置补丁、启动脚本、驱动程序和观察到的输出)在[公开 gist](https://gist.github.com/ShangmingCai/bdcfb3131b000a17810b4dca256d40cf) 中。
## 修复:延迟 KVCache 释放
Decode 无法取消 prefill 已启动的写操作,因此它保留已中止请求的槽位的时间要长于这些写操作可能继续落地的时间:
```
T_hold > t_abort_propagation + t_remaining_chunks + t_in_flight_call
```
只有当这个不等式在每条中止路径上都成立时,修复才是安全的。设计的其余部分让每条中止路径都保留槽位(左侧),并约束右侧的每一项(详见[#44186](https://github.com/sgl-project/sglang/issues/44186))。等待完成也符合硬件特性:可靠连接 RDMA 按顺序执行工作请求,因此“所有写操作已排空”在发送方是单次完成队列观察。
### 延迟释放
我们分三步构建了这个机制:
- **中止时通知 prefill**([#27372](https://github.com/sgl-project/sglang/pull/27372))。在 decode 侧中止时,decode 向每个 prefill rank 发送 `ABORT`,每个 rank 丢弃该房间软件队列中仍在等待的每个块。这将 `t_remaining_chunks` 降至接近零。如果没有通知,传输的每个剩余块都会启动并落地,因此该项覆盖整个 prefill 的剩余部分。`t_in_flight_call` 是一个单独的项,用于中止生效时已在执行的调用。在 Mooncake 的按房间串行队列上,通知不会留下该调用之外的任何内容,而像 NIXL 这样异步提交块的 backend 仍可能有多个已启动的块未完成。无论哪种方式,已启动的工作都会继续写入;通知只停止尚未启动的内容。
- **延迟解码侧 KVCache 释放**([#35049](https://github.com/sgl-project/sglang/pull/35049),NIXL 移植 [#35360](https://github.com/sgl-project/sglang/pull/35360))。中止时,解码端_保留_请求所拥有的一切,而不是释放它们:其 KVCache 槽位、请求池中的行以及元数据缓冲区条目。这使得写入方的存活状态变得明确。每个 prefill rank 按 room 跟踪其未完成的工作,并在出队时统计 chunk 数量,因此计数为零即可证明已出队的内容都不再处于传输中。仍在队列中等待的 chunk 之所以安全是出于另一个原因:Prefill 在设置 ack 之前先将该 room 标记为失败,因此 worker 在出队时会丢弃它们。只有当该计数降为零时,该 rank 才回复 `ABORT_ACK`。解码端只有在所有被通知的 rank 都已 ack(正向排空信号)或超时触发(有界兜底)时才释放。超时是实用的兜底手段,而非证明。其时长([#35049](https://github.com/sgl-project/sglang/pull/35049) 中默认 30 秒)与传输引擎的单次调用预算相匹配,该预算在中止之前就已开始,而在健康路径上,每次发起的写入都远在其内完成。在降级路径上超时的调用可能留下引擎仍在重试的写入,因此 prefill 绝不会将失败的调用确认为已排空([#43719](https://github.com/sgl-project/sglang/pull/43719)),释放操作会等待完整的超时时间。解码端绝不会将以这种方式释放的槽位插入基数树;它会隔离失败 room 的 KVCache,并在不读取的情况下将其丢弃。
- **能力门控与默认开启**([#37454](https://github.com/sgl-project/sglang/pull/37454),[#41023](https://github.com/sgl-project/sglang/pull/41023))。该保留机制仅在支持 ack 的后端上有帮助。它最初作为带按后端门控的选项发布,并在 v0.5.21 中成为默认行为,此时 Mooncake 和 NIXL 都已支持排空 ack。
在 [#35049](https://github.com/sgl-project/sglang/pull/35049) 中,一个 room 的所有 chunk 都进入同一个传输队列,由单个 worker 线程处理。该 worker 是该 room 唯一的写入者,并且由它自己发送 `ABORT_ACK`,而不是留给 bootstrap 线程。解码端也会统计来自 dummy rank 的确认;这些 rank 会立即确认。一旦所有预期的确认都到达,任何 prefill rank 都不可能再写入这些槽位。
Room ID 会在不同请求间复用,因此解码端为每次中止创建新的 ack 跟踪器,并忽略其未跟踪的 room 的 ack。[#37077](https://github.com/sgl-project/sglang/pull/37077) 后来在消息本身中加入了代际标签。
我们在 2×H200 上进行了测试,通过保留路径在传输中途中止了 132 个请求。没有发生崩溃,每个被保留的槽位在其保留解除后都被归还。

_**图 3:**修复方案。中止时,解码端保留 A 的槽位而不是释放它们。每个 prefill rank 排空其已发起的写入并回复 `ABORT_ACK`,槽位在最后一个 ack 时释放,耗时以毫秒计。超时是有界兜底,只会延长保留时间,绝不会缩短它。_
Decode 会保留这些槽位,直到所有必需的 drain 确认到达或释放超时到期。缺少确认会让 decode 等待超时。基于超时的释放只有在届时未完成的写入已经停止时才是安全的;仅凭超时到期并不能确认这些写入已经排空。该机制只在 abort 路径上保留槽位(参见性能影响)。即使在 abort 路径上,除非池接近耗尽且传入请求无法分配、直到这些槽位归还,否则保留槽位不会带来任何代价;此时它会表现为短暂的准入延迟。一项正在开发中的功能将进一步降低这一代价:Decode 会先将 KVCache 接收到主机内存,仅在传输完成且请求即将开始解码时才将其复制到 VRAM。届时保留将占用主机内存,而不是 GPU KVCache 容量。
#### 失败的 Mooncake 发送
[#43719](https://github.com/sgl-project/sglang/pull/43719) 于 2026 年 10 月 12 日合并,关闭了 Mooncake drain ack 中的一条提前释放路径。在此之前,如果提交的 Mooncake 发送返回错误,prefill 会将房间标记为失败,但一旦 decode 的 `ABORT` 到达,仍会将其确认为已排空。来自 `batch_transfer_sync` 的错误并不意味着写入已停止:超时后,切片会继续重试;批次失败后,其他任务仍可能在进行中。Decode 在该确认后释放了槽位,而残留的写入可能落入另一个请求的 KVCache。#43719 对任何 KVCache、状态或 aux 发送失败的房间暂不发送 drain ack,因此 decode 会保留槽位直到其释放超时。现在每次发送失败都会带来这一保留代价,直到 Mooncake 能够报告失败的批次是否已排空。#43719 已在 main 上,将在 v0.5.22 中发布。
### 准入修复
准入控制从路由器开始。PD 路由器将每个请求同时发送给一个 prefill worker 和一个 decode worker,而 decode 在没有 prefill 的 KVCache 时无法生成 token。如果 prefill 拒绝或使请求失败,decode 这一半过去会在持有 decode 槽位的同时等待其 300 秒超时,因此一波 prefill 失败可能同时占住许多 decode 槽位。`sgl-model-gateway` 中的 [#29017](https://github.com/sgl-project/sglang/pull/29017) 和 `sgl-router` 中的 [#41612](https://github.com/sgl-project/sglang/pull/41612) 现在会在 prefill 返回错误时立即丢弃 decode 请求,decode 引擎会在数秒内中止它。
在引擎内部,[#38935](https://github.com/sgl-project/sglang/pull/38935) 于 2026 年 9 月 16 日合并,当 worker 在入口处拒绝的请求进入任一准入队列(`PrefillBootstrapQueue.add` 或 `DecodePreallocQueue.add`)时,会在任何交接状态存在之前将其退役。在此之前,被拒绝的请求仍会报告自己存活,因此它会完成 bootstrap 握手、预留元数据缓冲区,并在单 token 存根上运行一次前向传递;[#36988](https://github.com/sgl-project/sglang/pull/36988) 仅在该传递之后才将其退役。如果两个 worker 都拒绝请求,decode 现在会立即返回 400,而不是在 300 秒后超时。这使被拒绝的请求不进入分离式交接。
### Mooncake 传输批次大小调优
有两个设置守护 Mooncake 传输调用。第一个是 Mooncake 自身的 `MC_TRANSFER_TIMEOUT`(默认 30 秒),任何耗时超过该值的 `batch_transfer_sync` 调用都会失败。以这种方式失败的调用仍可能留下未完成的写入(事实 2),这就是 prefill 在发送失败后暂不发送 drain ack 的原因。
第二个是`SGLANG_MOONCAKE_MAX_TRANSFER_BATCH_INDICES`,在[#32758](https://github.com/sgl-project/sglang/pull/32758)中加入(2026年9月8日合并),它限制每次调用的数据量,从而限制每次调用的持续时间。在默认值0下,SGLang在一次调用中发送一个chunk的全部KVCache,这是高效路径。正值会将传输拆分为有序的多次调用,每次调用最多包含该数量的索引,因此挂起的调用最多只会滞留一批已提交的写入,而第一次失败的调用会在剩余批次提交之前结束循环。较小的批次会牺牲吞吐量和TTFT,因此只有在极端上下文长度下才应降低该限制作为额外保障。在#32758的测试中,一个800K token请求的单次调用携带了30 GB。
`SGLANG_DISAGGREGATION_DEFERRED_DECODE_KV_RELEASE_TIMEOUT`然后设置`T_hold`。60到120秒的值给保持留出宽裕的余量。更大的值增加的安全性有限,并会在中止风暴期间成倍增加被保持的KVCache。如果释放落到超时,被保持的容量约为中止率 × `T_hold` × 每请求页数,因此应按该乘积来规划decode池的大小。
### PD校验和
[#39500](https://github.com/sgl-project/sglang/pull/39500)于2026年9月16日合并,为传输的KVCache增加了可选的完整性检查,默认关闭。Prefill在GPU上对连续区域以及带索引的KVCache和状态页计算Adler-32校验和,并随最后一个chunk发送。Decode在提交传输时记录它,并在请求加入运行批次之前用其检查接收到的KVCache,因此能捕获在该时间点之前到达的延迟写入;保持机制会阻止后续写入进入被复用的槽位。Decode在撤回期间保留校验和。不匹配会中止请求并递增传输失败计数器;在CI中,不匹配会抛出错误。健康检查请求跳过该检查。
要启用它,请向prefill和decode引擎都传入`--disaggregation-enable-kv-checksum`;它们的KVCache布局必须匹配。该标志会增加一个可选的元数据缓冲区。它默认保持关闭,因为开启后每次传输都要付出额外的GPU读取KVCache的开销和额外的同步点。
我们还评估了在提交时检查传输长度,并否决了它:发布的长度源自同样用于判定Success的prefill核算,因此双方只可能一致。
## 性能影响
在2×H100上的21次运行中(1P1D Mooncake,并发256下1024个请求,每组7次重复),补丁前以及启用或禁用该特性时的吞吐量在统计上无法区分。所有差异均在±2%以内,且每个95%置信区间都跨越零。在默认值切换前五周,另一项在2×H200上的16次运行扫描测得默认启用该特性的成本为−0.38%(CI −2.12至+1.36,不显著)。
在中止路径本身上,成本取决于ack快速路径。若其存活,被保持的请求在排空后几毫秒内释放;若其失效,每次中止都要付出完整的释放超时。
## 对运维人员的建议
对于PD部署,我们建议如下:
- **升级。**v0.5.21 默认保护 Mooncake 和 NIXL 部署,覆盖所有故障类型。我们未将此修复线回溯移植到更早的发布分支。MORI 部署以及带 generation 标记的 ack 卫生,需要构建版本达到或超过 [#37077](https://github.com/sgl-project/sglang/pull/37077) 的合并点(2026-10-03,包含在 v0.5.22 中)。实践中,请将构建版本切到 [#41128](https://github.com/sgl-project/sglang/pull/41128) 之后,该 PR 于同日十二小时后合并,因此也包含下文所述的发布指标。当 prefill 和 decode 使用不同的确认格式时,drain 握手可能失败,释放会回退到配置的超时时间。新的 decode 会丢弃来自旧 prefill 的无 generation 确认,而旧的 decode 收不到来自新 prefill 的确认。在这些情况下,安全释放取决于未完成的写入能否在保持期内停止。
- **运行一个将 abort 成对处理的路由器。**引擎侧的保持只有在 abort 到达引擎之后才能保护部署。一个能取消成对两半的路由器(如 `sgl-router` 的 PD 模式所做的那样)是第一道防线。
- **在 v0.5.19 或 v0.5.20 上**,在 prefill 和 decode 服务器上都设置 `SGLANG_DISAGGREGATION_DEFERRED_DECODE_KV_RELEASE=1`。这会为 decode 发起的 abort(最常见的一种)启用保持,不过大多数释放仍会等待超时。在 v0.5.20 上,还要设置 `SGLANG_MOONCAKE_MAX_TRANSFER_BATCH_INDICES`([#32758](https://github.com/sgl-project/sglang/pull/32758))和一个宽松的 `SGLANG_DISAGGREGATION_DEFERRED_DECODE_KV_RELEASE_TIMEOUT`(60–120 秒),大小按 Mooncake Transfer Batch Size Tuning 中所述确定。这两者都不能替代升级,因为在 v0.5.21 之前,除 decode 发起的 abort 之外的故障类型仍不受保护。在当前构建上,默认值是正确的,这些旋钮只是增加余量。
- **上线后关注 `sglang:decode_deferred_kv_release_total{outcome}`**。该指标随 v0.5.22 发布([#41128](https://github.com/sgl-project/sglang/pull/41128))。健康的部署几乎所有的保持都通过 `drained` 释放。超时占比过高意味着 ack 路径需要关注,且每次 abort 都在等待完整的保持期。
## 结论
这些规则适用于 KVCache 异步移动的任何场景,无论是在引擎之间、内存层级之间,还是流水线阶段之间:
1. 绝不要释放异步引擎可能仍会访问的槽位。
2. 保持到出现正向 drain 信号或达到有界超时。
3. 让每一种降级都延长保持时间。
4. 每个修复的发布粒度要足够小,使其回归测试恰好证明一件事。
如果 PD 部署出现这些修复无法解释的输出损坏,或任何看起来属于这一类的问题,请在 [sgl-project/sglang](https://github.com/sgl-project/sglang/issues) 上开 issue,附上拓扑(prefill/decode 并行度、传输后端、版本)、事件前后的请求模式,以及可用的 decode\_deferred\_kv\_release 指标。维护者会优先处理这类报告。
## 致谢
我们感谢报告此问题的研究人员识别出该暴露面、提供可复现的报告并协调披露。我们特别感谢 Cloudflare,其工程师分享了生产环境背景、开发了最初的应急补丁,并与我们一起完成了上游修复和加固审计。
我们感谢以下人员在这些修复上的工作:
- SGLang Community/RadixArk:Shangming Cai、Banruo Liu、Wenbo Jiang、Cheng Wan、Shiyang Chen 以及许多其他人
- Cloudflare:Kevin Flansburg、Kastan Day
我们还感谢 [@AsterWang](https://github.com/AsterWang) 和 [@KastanDay](https://github.com/KastanDay),他们早期的 PR([#24580](https://github.com/sgl-project/sglang/pull/24580)、[#32564](https://github.com/sgl-project/sglang/pull/32564))指出了这一竞态,并提议在复用前保持页面。
### Pull Request 列表
- 路由器 abort 配对:[#41612](https://github.com/sgl-project/sglang/pull/41612)、[#39461](https://github.com/sgl-project/sglang/pull/39461)、[#19524](https://github.com/sgl-project/sglang/pull/19524)、[#29017](https://github.com/sgl-project/sglang/pull/29017)
- 准入修复:[#36988](https://github.com/sgl-project/sglang/pull/36988)、[#38935](https://github.com/sgl-project/sglang/pull/38935)
- 中止通知发送至 prefill:[#27372](https://github.com/sgl-project/sglang/pull/27372)
- 延迟释放 decode 侧 KVCache:[#35049](https://github.com/sgl-project/sglang/pull/35049)、[#35360](https://github.com/sgl-project/sglang/pull/35360)
- 能力门控与默认开启:[#37454](https://github.com/sgl-project/sglang/pull/37454)、[#41023](https://github.com/sgl-project/sglang/pull/41023)
- Ack 快速路径:[#40645](https://github.com/sgl-project/sglang/pull/40645)、[#41402](https://github.com/sgl-project/sglang/pull/41402)
- 对所有失败类型均保持等待:[#41404](https://github.com/sgl-project/sglang/pull/41404)
- Mooncake 发送失败时绝不按已排空确认:[#43719](https://github.com/sgl-project/sglang/pull/43719)
- FakeKVReceiver 回归修复:[#41618](https://github.com/sgl-project/sglang/pull/41618)
- 带代际标记的 ack 与 MORI 支持:[#37077](https://github.com/sgl-project/sglang/pull/37077)
- 跨后端协议对齐:[#35281](https://github.com/sgl-project/sglang/pull/35281)、[#36612](https://github.com/sgl-project/sglang/pull/36612)
- Hicache 撤回竞态:[#24226](https://github.com/sgl-project/sglang/pull/24226)、[#30899](https://github.com/sgl-project/sglang/pull/30899)
- Hicache 恢复 rank 共识:[#43562](https://github.com/sgl-project/sglang/pull/43562)
- Mooncake 传输批大小:[#32758](https://github.com/sgl-project/sglang/pull/32758)
- PD 校验和:[#39500](https://github.com/sgl-project/sglang/pull/39500)
- 发布指标:[#41128](https://github.com/sgl-project/sglang/pull/41128)
- 早期提案、复现器与审计(未合并):[#24580](https://github.com/sgl-project/sglang/pull/24580)([@AsterWang](https://github.com/AsterWang))、[#32564](https://github.com/sgl-project/sglang/pull/32564)([@KastanDay](https://github.com/KastanDay))、[#38961](https://github.com/sgl-project/sglang/pull/38961)([@kflansburg](https://github.com/kflansburg))
## 参考
\[1\] Mooncake:面向 LLM 服务的以 KVCache 为中心的解耦架构。[https://github.com/kvcache-ai/Mooncake](https://github.com/kvcache-ai/Mooncake)
\[2\] SGLang PD 解耦文档。[https://docs.sglang.io](https://docs.sglang.io/)
\[3\] 在 96 块 H100 GPU 上以 PD 解耦和大规模专家并行部署 DeepSeek。[https://www.lmsys.org/blog/2025-05-05-large-scale-ep/](https://www.lmsys.org/blog/2025-05-05-large-scale-ep/)
## 附录
### 相关 issue 与修复
下表汇总了上文讨论的 issue 与修复,从请求路由开始,再到 KV cache 处理。
| 层 | 问题 | 修复 |
| --- | --- | --- |
| 路由器 | 单侧中止:decode 已终止而 prefill 仍在推送(或反之,双方各自等待已失效的对端 300 秒) | PD 路由器取消该配对:`sgl-router` 中的 [#41612](https://github.com/sgl-project/sglang/pull/41612)(基于 [#39461](https://github.com/sgl-project/sglang/pull/39461));`sgl-model-gateway` 中的 [#19524](https://github.com/sgl-project/sglang/pull/19524) 和 [#29017](https://github.com/sgl-project/sglang/pull/29017) |
| 准入 | 在入口被拒绝的请求仍进入 bootstrap、预留缓冲区并执行了一次 stub forward | 在准入环节拒绝并清理,此时尚不存在任何交接状态([#38935](https://github.com/sgl-project/sglang/pull/38935),基于 [#36988](https://github.com/sgl-project/sglang/pull/36988),后者在 forward 之后清理已中止的请求) |
| 停止新写入 | prefill 未感知 decode 中止,继续排队 chunk | decode 侧中止通知每个 prefill rank([#27372](https://github.com/sgl-project/sglang/pull/27372)) |
| 保持等待直至排空 | 已启动的写入无法取消 | 延迟释放槽位,直到每个 rank 排空 `ABORT_ACK` 或超时([#35049](https://github.com/sgl-project/sglang/pull/35049)、[#35360](https://github.com/sgl-project/sglang/pull/35360)) |
| 保持快速路径存活 | ack 因 `clear()` 竞态丢失 → 每次中止都要付出完整超时代价 | 保留 ack 目标,直到未完成的写入排空([#40645](https://github.com/sgl-project/sglang/pull/40645)) |
| | 每个 room 只有一个 ack 槽位 → MLA fan-out rank(prefill TP < decode TP)永远收不到 ack | 将 ack 扇出到该 room 已知的每个 decode 对端([#41402](https://github.com/sgl-project/sglang/pull/41402)) |
| | ack 在 decode tracker 就绪前到达 → 被丢弃 | 在发送 `ABORT` _之前_就绪 tracker([#41402](https://github.com/sgl-project/sglang/pull/41402)) |
| 所有路径均保持等待 | 等待仅针对 decode 发起的中止启用;传输/prefill 故障在 t=0 即释放 | 对每种失败类型都延迟([#41404](https://github.com/sgl-project/sglang/pull/41404)) |
| 协议卫生 | 复用的 room id 可能消费到过期的 ack;MORI 则完全没有 ack | 在公共层使用带代际标记的 `ABORT`/`ABORT_ACK`;MORI 选择启用等待([#37077](https://github.com/sgl-project/sglang/pull/37077)) |
| 兄弟管道 | hicache 撤回路径在设备→主机卸载拷贝进行中释放了设备槽位,导致被污染的 KVCache 持久化到主机层 | 撤回会等待卸载排空([#24226](https://github.com/sgl-project/sglang/pull/24226),[#30899](https://github.com/sgl-project/sglang/pull/30899)) |
| Rank 共识 | 在经 MIN 归约的 KVCache 轮询之后,TP rank 会依据 rank 本地的 hicache 恢复状态产生分支。一个 rank 的恢复失败导致该组失去同步(队列长度不匹配、调度器崩溃),其准入记账在下一个 tick 出现偏差 | 将恢复结果并入已经归约的 KVCache 轮询,使每个 rank 只做一次决策([#43562](https://github.com/sgl-project/sglang/pull/43562)) |
| 检测 | 绕过预防的损坏不可见 | 可选的传输校验和([#39500](https://github.com/sgl-project/sglang/pull/39500));曾评估提交时的长度校验,但未采纳(见 PD Checksum) |
| 观测 | 无法判断 ack 快速路径在生产环境中是否存活 | 保留计数、时长以及 `drained`/`timeout`/`error` 结果指标([#41128](https://github.com/sgl-project/sglang/pull/41128)) |
中间几行来自一次社区审计。该审计于 2026 年 9 月 11 日发布([#38961](https://github.com/sgl-project/sglang/pull/38961)),[@kflansburg](https://github.com/kflansburg) 在其中发现了另外两种可能让该竞态重新出现的途径。排空 ack 经常未能到达,因此释放操作回退到超时。并非由 decode 自身发起的故障从未触发 hold,从而通过另一条路径重新打开了最初的 bug。我们于 2026 年 9 月 22 日至 28 日期间在 main 上修复了这两处:现在 hold 会在每种故障类型上触发,排空 ack 现在也能到达 decode。审计还标记了 prefill 自身的源页面,而我们的分析表明该路径本就安全,因为这些字节只能落入 decode hold 正在隔离的槽位中。2026 年 10 月 11 日合并的 [#43562](https://github.com/sgl-project/sglang/pull/43562) 修复了另一项发现,即 rank 分歧的 HiCache 恢复。
### 披露时间线
| 日期 | 事件 |
| --- | --- |
| 2026-09-09 | Cloudflare 向 SGLang 维护者通报了生产环境中一次跨请求 KVCache 暴露,并在 [#38704](https://github.com/sgl-project/sglang/pull/38704) 中分享了后续安全检查 |
| 2026-09-10 | SGLang 维护者与 Cloudflare 会面,就上游修复达成一致 |
| 2026-09-11 | 加固审计([#38961](https://github.com/sgl-project/sglang/pull/38961))发现 ack 快速路径在生产形态下经常回退到超时 |
| 2026-09-16 | 准入修复([#38935](https://github.com/sgl-project/sglang/pull/38935))和可选 PD 校验和([#39500](https://github.com/sgl-project/sglang/pull/39500))合并 |
| 2026-09-22 → 09-28 | 快速路径恢复([#40645](https://github.com/sgl-project/sglang/pull/40645),[#41402](https://github.com/sgl-project/sglang/pull/41402));每种故障类型都会延迟([#41404](https://github.com/sgl-project/sglang/pull/41404));默认开启([#41023](https://github.com/sgl-project/sglang/pull/41023));hicache 修复([#30899](https://github.com/sgl-project/sglang/pull/30899)) |
| 2026-10-02 | v0.5.21 发布,默认受保护(Mooncake、NIXL) |
| 2026-10-03 | MORI 和带代际标记的 ack 合并([#37077](https://github.com/sgl-project/sglang/pull/37077));随 v0.5.22 发布 |
| 2026-10-11 | 针对 decode HiCache 恢复的 rank 共识修复合并([#43562](https://github.com/sgl-project/sglang/pull/43562)) |
| 2026-10-12 | 失败的 Mooncake 发送不再以已排空状态 ack([#43719](https://github.com/sgl-project/sglang/pull/43719));随 v0.5.22 发布 |
| 2026-10-22 | 失败的 NIXL 和 MORI 发送不再以已排空状态 ack([#45973](https://github.com/sgl-project/sglang/pull/45973));随 v0.5.23 发布 |
| 2026-10-26 | 由 SGLang 和 Cloudflare 协调披露。 |
## 正文 · 原文
On September 9, 2026, Cloudflare notified us of cross-request KV cache exposure in its GLM-5.3 deployment on Workers AI. We worked with Cloudflare to address the request-rejection path involved and strengthen related cache-lifetime handling in disaggregated serving.
We are publishing this disclosure in coordination with Cloudflare and the researcher who reported the issue through Cloudflare’s HackerOne program. Cloudflare’s report covers its production incident and customer impact, and the researcher’s report (available soon) will describe the discovery and external validation. This post focuses on the underlying mechanisms, the changes we made, and guidance for SGLang operators.
[Read Cloudflare’s report here.](https://blog.cloudflare.com/kv-cache-exposure-remediation/)
## Impact and affected deployments
Cloudflare reports that the incident affected `@cf/zai-org/glm-5.3` on Workers AI between August 27 and September 9, 2026. Its investigation found no other Workers AI model affected.
Cloudflare’s investigation found that requests exceeding the prefill KV cache capacity could return content from another user’s session. For details on the incident’s scope and customer impact, see Cloudflare’s incident report.
The issue occurred in disaggregated prefill/decode serving, where separate workers process the prompt and generate the response. The single-node rejection path returned an error instead.
## What happened
Cloudflare’s deployment had a configured context length of 1,048,576 tokens, but its prefill KV cache pool could hold 681,280 tokens. A request could therefore fit the configured context window while exceeding the memory available to process it.
In SGLang’s disaggregated prefill/decode path, a request marked for rejection could still proceed through the handoff. Decode could then read cache pages that had not been populated for that request. The configuration mismatch made the rejection path reachable. The admission fix described below ensures that a rejected request returns an error before entering the disaggregated handoff. It complements deployment checks that compare pool capacity with the configured context length.
Cloudflare disabled public access to the model on September 9. It increased the prefill pool to accommodate the configured context window and deployed a Cloudflare-authored engine patch that returned errors for rejected requests. Cloudflare reports that it validated these changes against the researcher’s reproduction and its own canary-marker tests before restoring the model that evening.
The upstream fix, [#36988](https://github.com/sgl-project/sglang/pull/36988), retires aborted requests when their final prefill result is processed. Follow-up change [#38935](https://github.com/sgl-project/sglang/pull/38935) stops intake-rejected requests earlier, before they enter the disaggregated handoff, avoiding unnecessary work and delays.
Alongside the admission fix, we worked with Cloudflare on how SGLang handles KV cache transfers when requests are interrupted.
In prefill-decode (PD) disaggregated serving, an aborted request can still have RDMA writes in flight. If decode releases and reassigns the destination slots before those writes complete, the late writes can overwrite another request’s KVCache and corrupt its output.
Cloudflare’s report also describes our joint work on related KV cache lifetime issues. The following sections explain the deferred-release mechanism, additional transfer safeguards, and integrity checks, along with our validation results and guidance for operators.
## Background
In PD-disaggregated serving, the prefill and decode phases run on separate engines, and each request’s KVCache must move from the prefill engine into the decode engine’s memory pool. For performance, this transfer is asynchronous. Transfer engines such as Mooncake, NIXL, or MORI issue it as one-sided RDMA WRITE operations that stream KVCache chunks into pre-reserved destination slots while prefill computation is still in progress.
Because the transfer is asynchronous, a request can die while some of its writes are in flight. Prefill can then drop the chunks still waiting in its software queues, but it cannot stop a write it has launched toward the destination, such as the chunk in progress at that moment. In earlier implementations, when a request terminated mid-transfer (a client disconnect, a timeout, a transport fault), decode returned its destination slots to the allocator at termination, without waiting for such a write to finish. If the allocator reassigned the slots before the write landed, the dead request’s KVCache landed in another request’s KVCache. In typical workloads the corruption was rare, although long contexts make it far more likely (see fact 4 below). Each occurrence was silent. The transfer reported success, and no component raised an exception or logged an error. A single contaminated slot derails an entire generation, because attention reads every position.
As a result, a request asking the model to count from 1 to 15 comes back as an incoherent stream of unrelated tokens. We then set out to make the race reproducible on demand and to fix it end to end, from the transfer engine layer up to the serving engine.
## The Problem: When Is a KVCache Slot Actually Free?
The failure takes four steps:
1. A client disconnect, an admission rejection, or a transfer fault aborts request A mid-transfer while at least one launched write has not yet landed.
2. Decode frees A’s KVCache slots at abort time. The allocator marks them free while the remote writer is still writing to them.
3. The allocator hands those slots to victim request B.
4. A’s launched writes land inside B’s KVCache, and B decodes on foreign keys and values with no error.
These steps apply to any PD implementation that moves KVCache between engines with asynchronous DMA, push or pull, because each one has to decide when a slot is free. We diagnosed and fixed the family in SGLang.

_**Figure 1:** The race. The abort frees A’s slots while A’s launched writes are still landing; B inherits the slots and the late writes land inside its KVCache._
Four facts about the system make each step possible.
**1\. The allocator can mark a slot free while writes to it are in flight.** In PD disaggregation, the decode engine pre-reserves destination KVCache slots for each incoming request, and its allocator tracks them as allocated or free. But “free” has a second, hidden condition: A slot is safe to reuse only when _no remote writer can still touch it_. The bugs in this class come from the gap between these two definitions. Each one is a distributed use-after-free, where the dangling pointer is a write request sitting in a NIC queue on another machine. The fix makes the second condition explicit, observable, and bounded.
**2\. Launched RDMA work cannot be cancelled, for push and pull alike.** SGLang’s KVCache transfer is push-based \[3\]. Prefill _writes_ KVCache into decode’s memory, because prefill knows the moment it produces each chunk and can overlap transfer with ongoing computation, chunk by chunk. At and below the NIC, cancellation is hard to achieve without hurting availability. The RDMA API has no call that cancels an individual work request already posted to the NIC’s queue pair. Python cannot interrupt the transfer engine’s blocking C++ call (e.g., Mooncake’s `batch_transfer_sync`) mid-call. If that call times out, its work requests stay posted with no completion accounting, so those writes may land at an unknown later time. One-sided RDMA WRITE bypasses the receiver’s CPU, so decode gets no signal for each write that it could wait on. The NIC also places data per packet: A write interrupted mid-flight still leaves a partially written prefix at the destination, so even the paths that look like cancellation can mutate destination memory.
A pull-based design would not escape this. If decode issued RDMA READs instead, an abort arriving mid-pull would leave decode’s _own_ posted reads landing in slots it just freed, because a posted READ is as impossible to take back as a posted WRITE. Under pull, the hazard also runs the other way: If prefill frees its source pages at abort time and the allocator reassigns them while a READ is still in flight, that read fetches another request’s KVCache into decode. Freed-then-reassigned slots corrupt the transfer at either end.

_**Figure 2:** The transfer stack below the Python scheduler. An abort can stop future chunks, but everything already handed to the C++ engine or the NIC is beyond recall, and the receiving CPU never sees the writes at all._
**3\. Complete liveness handling starts at the router but cannot end there.** An abort is only safe when _both_ halves of a P/D pair learn about it. A well-designed PD router cancels the pair together, so neither side waits out a 300-second timeout for a peer that is never coming. In `sgl-router`, [#41612](https://github.com/sgl-project/sglang/pull/41612) builds the PD pairing on top of the abort-on-disconnect of [#39461](https://github.com/sgl-project/sglang/pull/39461). The pair shares one router-minted request id. A client disconnect aborts decode, a prefill failure drops the decode request so the engine aborts it, and prefill learns of the abort through decode’s engine-side `ABORT` notification. `sgl-model-gateway` pairs the same two events ([#19524](https://github.com/sgl-project/sglang/pull/19524) for disconnects, [#29017](https://github.com/sgl-project/sglang/pull/29017) for prefill failures). Before that engine-side notification existed, a router that aborted only one side guaranteed the race: Decode released its slots while prefill, never told, kept pushing. Even a perfect router only narrows the window. The abort reaches prefill asynchronously, after some scheduling and network delay, and whatever the transfer engine already launched (an in-flight C++ call, posted work requests) continues regardless. The router decides who dies (abort orchestration), and the engine decides when memory is safe (drain correctness).
**4\. The race window scales with prefill context length.** SGLang chunks transfers and overlaps them with prefill computation, so the exposure window spans much of the prefill itself. A short prompt is in flight for milliseconds. A 128K–1M-token context, at tens to hundreds of kilobytes of KVCache per token depending on the model, keeps writes in flight for seconds to minutes, across many chunks and many posted work requests. The tail _after_ the abort grows with it. A long context means more chunks in flight and more work requests still draining at the moment the abort lands, so launched writes can keep landing well after the allocator has reclaimed the freed slots. Under memory pressure, aborting a long request makes a write-after-reuse the expected outcome. Long-context workloads are also the ones where clients time out and disconnect mid-request. Long contexts therefore widen the window, lengthen the post-abort tail, and raise the abort rate all at once, so this bug class surfaces as deployments move to long-context serving.
### Impact and Exposure
The aborted-transfer race described here requires three conditions:
1. PD disaggregation is enabled with a KVCache transfer backend (Mooncake, NIXL, MORI, or a custom `CommonKVManager` backend). A single-instance deployment without this cross-engine transfer path is not affected by this specific race.
2. A request terminates mid-transfer (client disconnect, router cancel, timeout, transport fault, prefill failure, or hicache restore failure) while at least one of its writes is launched and not yet landed.
3. The allocator reassigns the freed slots to another request before the launched writes land, which under memory pressure happens in milliseconds.
Remove any one of the three and the corruption cannot occur. Within PD deployments, default configurations were affected from the v0.4.x line (early 2025), when PD disaggregation shipped with RDMA backends, until v0.5.21, or v0.5.22 for MORI. A separate variant with the same signature required decode-side hierarchical cache offload, which is not on by default (fixed by [#24226](https://github.com/sgl-project/sglang/pull/24226) and [#30899](https://github.com/sgl-project/sglang/pull/30899)).
**Deterministic or probabilistic?** In production the race is probabilistic, so operators saw occasional corrupted outputs that they could not reproduce. Once the window is held open, it reproduces on demand (our harness in the next section, and the community reproducer in [#32564](https://github.com/sgl-project/sglang/pull/32564)). A client can raise the odds on purpose by sending long-context requests and disconnecting mid-transfer, which maximizes the race window and abort rate. The client cannot choose the victim or the landing slots, and the flow runs one way. The aborted request’s KVCache lands _in_ another request’s KVCache, and nothing flows back to the aborter. The realistic impact is therefore one-way cross-request leakage, in which fragments of the aborted request’s context can surface in another client’s output, as they did in our reproduction. The aborter learns nothing, but in multi-tenant serving this is a data-leakage concern as well as an integrity one.
**Can a deployer tell whether exposure already occurred?** A deployer has no reliable way to tell, because the corruption leaves no error or log line. The opt-in PD checksum ([#39500](https://github.com/sgl-project/sglang/pull/39500)) can catch some late writes (see PD Checksum). After upgrading, use the [#41128](https://github.com/sgl-project/sglang/pull/41128) metrics to confirm the protection is working (see Recommended actions for operators).
## Deterministic Reproduction
We needed to reproduce the race before we could claim a fix, so we built a harness that triggers it on demand on a 1P1D pair. An env-gated delay inside prefill’s `send_kvcache` holds a launched write in flight, `--max-total-tokens 96` forces immediate slot reuse, and instrumentation logs every request’s slot range and every free-or-hold decision.
With the window held open, the overlap appears every time: Decode frees request A’s slots `[14..44]` while its write is in flight, request B allocates `[1..32]`, and the two ranges collide on `[14..32]`. A software delay cannot make the write itself land late on a single P/D pair, because the transfer queue serializes writes to the same decode engine. To show _wrong output_, not only overlapping slot numbers, we copied the aborted request’s freed page over the overlap at the victim’s commit point, which is what a late write does. Without the fix, the victim’s output was garbage. With the fix, decode held the donor’s slots, the victim got fresh slots, and the output was identical to the baseline.
The real late landing also reproduces end to end on live hardware once two prefill engines share one decode, so nothing serializes their writes. We ran it on a 2×H200 pair (Mooncake over RDMA) with the hold disabled. A one-sided abort freed the donor’s 430 slots, the victim preallocated `[1..105]` inside them, and the donor’s launched write landed mid-decode. The victim counted correctly up to 80, then its output turned into the donor’s prompt text: Another request’s KVCache, read back out of its generation. With the hold enabled and the same timing, decode logged hold, drain-ack, and release, and the victim waited for the release and then produced output identical to the baseline, byte for byte. The full recipe (harness patch, launch script, driver, and the observed outputs) is in a [public gist](https://gist.github.com/ShangmingCai/bdcfb3131b000a17810b4dca256d40cf).
## The Fix: Deferred KVCache Release
Decode cannot cancel prefill’s launched writes, so it keeps the aborted request’s slots reserved for longer than those writes can keep landing:
```
T_hold > t_abort_propagation + t_remaining_chunks + t_in_flight_call
```
The fix is safe only if this inequality holds on every abort path. The rest of the design makes every abort path hold the slots (the left side) and bounds each term on the right (detailed in [#44186](https://github.com/sgl-project/sglang/issues/44186)). Waiting for completion also fits the hardware: Reliable-connection RDMA executes work requests in order, so “all writes drained” is a single completion-queue observation on the sender.
### Deferred Release
We built the mechanism in three steps:
- **Notify prefill on abort** ([#27372](https://github.com/sgl-project/sglang/pull/27372)). On a decode-side abort, decode sends `ABORT` to every prefill rank, and each rank drops every chunk still waiting in the software queue for that room. That cuts `t_remaining_chunks` to near zero. Without the notification, every remaining chunk of the transfer launches and lands, so the term covers the entire rest of the prefill. `t_in_flight_call` is a separate term for the call already executing when the abort takes effect. On Mooncake’s serial per-room queue, the notification leaves nothing beyond that call, while a backend that posts chunks asynchronously, like NIXL, can still have several launched chunks outstanding. Either way, launched work keeps writing; the notification stops only what has not launched.
- **Deferred decode-side KVCache release** ([#35049](https://github.com/sgl-project/sglang/pull/35049), NIXL port [#35360](https://github.com/sgl-project/sglang/pull/35360)). On abort, decode _holds_ everything the request owns instead of freeing it: its KVCache slots, its row in the request pool, and its metadata buffer entry. That makes writer liveness explicit. Each prefill rank tracks its outstanding work per room and counts chunks at dequeue, so a count of zero proves that nothing already dequeued is still in flight. Chunks still waiting in the queue are safe for a different reason: Prefill marks the room failed before it arms the ack, so the worker drops them at dequeue. The rank replies `ABORT_ACK` only when that count drains to zero. Decode releases only when every notified rank has acked (the positive drain signal) or when a timeout fires (the bounded fallback). The timeout is a practical backstop, not a proof. Its length (30 seconds by default in [#35049](https://github.com/sgl-project/sglang/pull/35049)) matches the transfer engine’s per-call budget, which starts before the abort, and on a healthy path every launched write lands well inside it. A call that times out on a degraded path can leave writes that the engine is still retrying, so prefill never acknowledges a failed call as drained ([#43719](https://github.com/sgl-project/sglang/pull/43719)), and the release waits out the full timeout. Decode never inserts slots released this way into the radix tree; it quarantines a failed room’s KVCache and discards it unread.
- **Capability gating and default-on** ([#37454](https://github.com/sgl-project/sglang/pull/37454), [#41023](https://github.com/sgl-project/sglang/pull/41023)). The hold only helps on backends that ack. It shipped as an opt-in with a per-backend gate and became the default in v0.5.21, once Mooncake and NIXL both supported the drain ack.
In [#35049](https://github.com/sgl-project/sglang/pull/35049), all chunks for a room go to the same transfer queue, served by a single worker thread. That worker is the room’s only writer, and it sends `ABORT_ACK` itself rather than leaving that to the bootstrap thread. Decode counts acknowledgments from dummy ranks too; those ranks acknowledge immediately. Once all expected acknowledgments arrive, no prefill rank can still write to the slots.
Room IDs are reused across requests, so decode creates a new ack tracker for each abort and ignores acks for rooms it isn’t tracking. [#37077](https://github.com/sgl-project/sglang/pull/37077) later added generation tags to the messages themselves.
We tested this on 2×H200 by aborting 132 requests mid-transfer through the hold path. There were no crashes, and every held slot was returned once its hold resolved.

_**Figure 3:** The fix. On abort, decode holds A’s slots instead of freeing them. Every prefill rank drains its launched writes and replies `ABORT_ACK`, and the slots are released on the last ack, in milliseconds. The timeout is a bounded fallback that can only lengthen the hold, never shorten it._
Decode retains the slots until all required drain acknowledgements arrive or the release timeout expires. A missing acknowledgement makes decode wait for the timeout. Timeout-based release is safe only if outstanding writes have stopped by then; expiry alone does not confirm that they have drained. The mechanism holds slots only on the abort path (see Performance Impact). Even on the abort path, the hold costs nothing unless the pool is near exhaustion and an incoming request cannot allocate until those slots return; then it shows up as a short admission delay. A feature under development will cut this cost further: Decode will receive the KVCache into host memory first and copy it into VRAM only when the transfer is done and the request is about to start decoding. The hold will then occupy host memory instead of GPU KVCache capacity.
#### Failed Mooncake Sends
[#43719](https://github.com/sgl-project/sglang/pull/43719), merged on October 12, 2026, closes an early-release path in the Mooncake drain ack. Before it, if a submitted Mooncake send returned an error, prefill marked the room failed and still acked it as drained once decode’s `ABORT` arrived. An error from `batch_transfer_sync` does not mean the writes stopped: After a timeout, the slices keep retrying, and after a failed batch, other tasks can still be in flight. Decode freed the slots on that ack, and the leftover writes could land in another request’s KVCache. #43719 withholds the drain ack for any room whose KVCache, state, or aux send failed, so decode holds the slots until its release timeout. Every failed send now costs that hold, until Mooncake can report whether a failed batch has drained. #43719 is on main, for release in v0.5.22.
### Admission Fix
Admission control starts at the router. A PD router sends each request to a prefill worker and a decode worker together, and decode cannot produce a token without prefill’s KVCache. If prefill rejected or failed a request, the decode half used to wait out its 300-second timeout while it held a decode slot, so a burst of prefill failures could pin many decode slots at once. [#29017](https://github.com/sgl-project/sglang/pull/29017) in `sgl-model-gateway` and [#41612](https://github.com/sgl-project/sglang/pull/41612) in `sgl-router` now drop the decode request as soon as prefill returns an error, and the decode engine aborts it within seconds.
Inside the engines, [#38935](https://github.com/sgl-project/sglang/pull/38935), merged on September 16, 2026, retires a request that a worker rejects at intake when it enters either admission queue (`PrefillBootstrapQueue.add` or `DecodePreallocQueue.add`), before any handoff state exists. Before it, a rejected request still reported itself as live, so it completed a bootstrap handshake, reserved a metadata buffer, and ran a forward pass on a one-token stub; [#36988](https://github.com/sgl-project/sglang/pull/36988) retired it only after that pass. If both workers reject a request, decode now returns 400 at once instead of timing out after 300 seconds. This keeps rejected requests out of the disaggregation handoff.
### Mooncake Transfer Batch Size Tuning
Two settings guard a Mooncake transfer call. The first is Mooncake’s own `MC_TRANSFER_TIMEOUT` (30 seconds by default), which fails any `batch_transfer_sync` call that takes longer. A call that fails this way can still leave writes in flight (fact 2), which is why prefill withholds the drain ack after a failed send.
The second is `SGLANG_MOONCAKE_MAX_TRANSFER_BATCH_INDICES`, added in [#32758](https://github.com/sgl-project/sglang/pull/32758) (merged September 8, 2026), which caps the data volume, and with it the duration, of each call. At its default of 0, SGLang sends a chunk’s whole KVCache in one call, the efficient path. A positive value splits the transfer into ordered calls of at most that many indices, so a hung call strands at most one batch of posted writes, and the first failed call ends the loop before the remaining batches post. Smaller batches cost throughput and TTFT, so lower the limit only as an extra safeguard for extreme context lengths. In #32758’s tests, a single call for an 800K-token request carried 30 GB.
`SGLANG_DISAGGREGATION_DEFERRED_DECODE_KV_RELEASE_TIMEOUT` then sets `T_hold`. A value of 60 to 120 s gives the hold a wide margin. A larger value adds little safety and multiplies held KVCache during abort storms. If releases fall to the timeout, held capacity is about abort rate × `T_hold` × pages per request, so size the decode pool for that product.
### PD Checksum
[#39500](https://github.com/sgl-project/sglang/pull/39500), merged on September 16, 2026, adds an optional integrity check for the transferred KVCache, off by default. Prefill computes an Adler-32 checksum on the GPU, over contiguous regions and over indexed KVCache and state pages, and sends it with the final chunk. Decode records it when it commits the transfer and checks the received KVCache against it before the request joins a running batch, so it catches late writes that land before that point; the hold keeps later writes out of reused slots. Decode keeps the checksum across retraction. A mismatch aborts the request and increments the transfer-failure counter; in CI, a mismatch raises an error instead. Health-check requests skip the check.
To turn it on, pass `--disaggregation-enable-kv-checksum` to both the prefill and decode engines; their KVCache layouts must match. The flag adds an optional metadata buffer. It stays off by default because every transfer pays for an extra GPU read of the KVCache and an extra synchronization point when flipped on.
We also evaluated a commit-time check of the transferred length and declined it: The published length derives from the same prefill accounting that gates Success, so the two sides can only agree.
## Performance Impact
Across 21 runs on 2×H100 (1P1D Mooncake, 1024 requests at concurrency 256, 7 repetitions per arm), throughput was statistically indistinguishable before the patch and with the feature enabled or disabled. All deltas were within ±2%, and every 95% confidence interval spanned zero. Five weeks before the default flipped, a separate 16-run sweep on 2×H200 measured the cost of defaulting the feature on at −0.38% (CI −2.12 to +1.36, not significant).
On the abort path itself, the cost depends on the ack fast path. With it alive, a held request releases milliseconds after the drain; with it dead, every abort pays the full release timeout.
## Recommended actions for operators
For PD deployments, we recommend the following:
- **Upgrade.** v0.5.21 protects Mooncake and NIXL deployments by default, for every failure kind. We have not backported this fix line to earlier release branches. MORI deployments, and the generation-tagged ack hygiene, need a build at or past the merge of [#37077](https://github.com/sgl-project/sglang/pull/37077) (2026-10-03, in v0.5.22). In practice, cut the build past [#41128](https://github.com/sgl-project/sglang/pull/41128), merged twelve hours later the same day, so it also carries the release metrics described below. When prefill and decode use different acknowledgement formats, the drain handshake can fail and releases fall back to the configured timeout. A new decode drops generation-less acknowledgements from an old prefill, and an old decode receives no acknowledgement from a new prefill. In these cases, safe release depends on outstanding writes stopping within the hold period.
- **Run a router that pairs aborts.** Engine-side holds protect a deployment only after the abort reaches the engine. A router that cancels both halves of the pair (as `sgl-router`’s PD mode does) is the first layer of defense.
- **On v0.5.19 or v0.5.20**, set `SGLANG_DISAGGREGATION_DEFERRED_DECODE_KV_RELEASE=1` on both prefill and decode servers. This enables the hold for decode-initiated aborts (the most common kind), though most releases wait out the timeout. On v0.5.20, also set `SGLANG_MOONCAKE_MAX_TRANSFER_BATCH_INDICES` ([#32758](https://github.com/sgl-project/sglang/pull/32758)) and a generous `SGLANG_DISAGGREGATION_DEFERRED_DECODE_KV_RELEASE_TIMEOUT` (60–120 s), sized as described under Mooncake Transfer Batch Size Tuning. Neither replaces upgrading, because failure kinds other than decode-initiated aborts stay unprotected before v0.5.21. On current builds the defaults are correct, and the knobs add margin.
- **Watch `sglang:decode_deferred_kv_release_total{outcome}`** after rollout. The metric ships in v0.5.22 ([#41128](https://github.com/sgl-project/sglang/pull/41128)). A healthy deployment releases almost all holds through `drained`. A timeout-heavy ratio means the ack path needs attention and every abort is waiting out the full hold.
## Conclusion
These rules apply wherever KVCache moves asynchronously, whether between engines, memory tiers, or pipeline stages:
1. Never free a slot an async engine may still touch.
2. Hold until a positive drain signal or a bounded timeout.
3. Make every degradation lengthen the hold.
4. Ship each fix small enough that its regression test proves exactly one thing.
If a PD deployment shows output corruption that these fixes do not explain, or anything that looks like this family, open an issue on [sgl-project/sglang](https://github.com/sgl-project/sglang/issues) with the topology (prefill/decode parallelism, transfer backend, version), the request pattern around the incident, and the decode\_deferred\_kv\_release metrics if available. The maintainers treat this class of report with priority.
## Acknowledgement
We thank the reporting researcher for identifying the exposure, providing a reproducible report, and coordinating disclosure. We are especially grateful to Cloudflare, whose engineers shared the production context, developed the initial incident patch, and worked with us through the upstream fixes and hardening audit.
We thank the following people for their work on these fixes:
- SGLang Community/RadixArk: Shangming Cai, Banruo Liu, Wenbo Jiang, Cheng Wan, Shiyang Chen, and many others
- Cloudflare: Kevin Flansburg, Kastan Day
We also thank [@AsterWang](https://github.com/AsterWang) and [@KastanDay](https://github.com/KastanDay), whose early PRs ([#24580](https://github.com/sgl-project/sglang/pull/24580), [#32564](https://github.com/sgl-project/sglang/pull/32564)) called out this race and proposed holding pages before reuse.
### Pull Request List
- Router abort pairing: [#41612](https://github.com/sgl-project/sglang/pull/41612), [#39461](https://github.com/sgl-project/sglang/pull/39461), [#19524](https://github.com/sgl-project/sglang/pull/19524), [#29017](https://github.com/sgl-project/sglang/pull/29017)
- Admission fix: [#36988](https://github.com/sgl-project/sglang/pull/36988), [#38935](https://github.com/sgl-project/sglang/pull/38935)
- Abort notification to prefill: [#27372](https://github.com/sgl-project/sglang/pull/27372)
- Deferred decode-side KVCache release: [#35049](https://github.com/sgl-project/sglang/pull/35049), [#35360](https://github.com/sgl-project/sglang/pull/35360)
- Capability gating and default-on: [#37454](https://github.com/sgl-project/sglang/pull/37454), [#41023](https://github.com/sgl-project/sglang/pull/41023)
- Ack fast path: [#40645](https://github.com/sgl-project/sglang/pull/40645), [#41402](https://github.com/sgl-project/sglang/pull/41402)
- Hold on every failure kind: [#41404](https://github.com/sgl-project/sglang/pull/41404)
- Failed Mooncake send never acks as drained: [#43719](https://github.com/sgl-project/sglang/pull/43719)
- FakeKVReceiver regression fix: [#41618](https://github.com/sgl-project/sglang/pull/41618)
- Generation-tagged acks and MORI support: [#37077](https://github.com/sgl-project/sglang/pull/37077)
- Cross-backend protocol alignment: [#35281](https://github.com/sgl-project/sglang/pull/35281), [#36612](https://github.com/sgl-project/sglang/pull/36612)
- Hicache retraction race: [#24226](https://github.com/sgl-project/sglang/pull/24226), [#30899](https://github.com/sgl-project/sglang/pull/30899)
- Hicache restore rank consensus: [#43562](https://github.com/sgl-project/sglang/pull/43562)
- Mooncake transfer batch size: [#32758](https://github.com/sgl-project/sglang/pull/32758)
- PD checksum: [#39500](https://github.com/sgl-project/sglang/pull/39500)
- Release metrics: [#41128](https://github.com/sgl-project/sglang/pull/41128)
- Early proposals, reproducer, and audit (not merged): [#24580](https://github.com/sgl-project/sglang/pull/24580) ([@AsterWang](https://github.com/AsterWang)), [#32564](https://github.com/sgl-project/sglang/pull/32564) ([@KastanDay](https://github.com/KastanDay)), [#38961](https://github.com/sgl-project/sglang/pull/38961) ([@kflansburg](https://github.com/kflansburg))
## Reference
\[1\] Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. [https://github.com/kvcache-ai/Mooncake](https://github.com/kvcache-ai/Mooncake)
\[2\] SGLang PD Disaggregation documentation. [https://docs.sglang.io](https://docs.sglang.io/)
\[3\] Deploying DeepSeek with PD Disaggregation and Large-Scale Expert Parallelism on 96 H100 GPUs. [https://www.lmsys.org/blog/2025-05-05-large-scale-ep/](https://www.lmsys.org/blog/2025-05-05-large-scale-ep/)
## Appendix
### Related issues and fixes
The table below summarizes the issues and fixes discussed above, starting with request routing and moving through KV cache handling.
| Layer | Problem | Fix |
| --- | --- | --- |
| Router | One-sided aborts, where decode dies and prefill keeps pushing (or vice versa, each waiting 300 s for a dead peer) | The PD router cancels the pair: [#41612](https://github.com/sgl-project/sglang/pull/41612) in `sgl-router` (on top of [#39461](https://github.com/sgl-project/sglang/pull/39461)); [#19524](https://github.com/sgl-project/sglang/pull/19524) and [#29017](https://github.com/sgl-project/sglang/pull/29017) in `sgl-model-gateway` |
| Admission | Requests rejected at intake still entered bootstrap, reserved buffers, and ran a stub forward | Reject and clean up at the admission doors, before any handoff state exists ([#38935](https://github.com/sgl-project/sglang/pull/38935), building on [#36988](https://github.com/sgl-project/sglang/pull/36988), which cleans up aborted requests after the forward pass) |
| Stop new writes | Prefill, unaware of a decode abort, keeps queueing chunks | Decode-side abort notifies every prefill rank ([#27372](https://github.com/sgl-project/sglang/pull/27372)) |
| Hold until drain | Launched writes cannot be cancelled | Deferred slot release until per-rank drain `ABORT_ACK` or timeout ([#35049](https://github.com/sgl-project/sglang/pull/35049), [#35360](https://github.com/sgl-project/sglang/pull/35360)) |
| Keep the fast path alive | Ack lost to a `clear()` race → every abort pays the full timeout | Keep the ack target until outstanding writes drain ([#40645](https://github.com/sgl-project/sglang/pull/40645)) |
| | One ack slot per room → MLA fan-out ranks (prefill TP < decode TP) never acked | Fan the ack out to every decode peer the room knows ([#41402](https://github.com/sgl-project/sglang/pull/41402)) |
| | Ack arrives before the decode tracker is armed → dropped | Arm the tracker _before_ sending `ABORT` ([#41402](https://github.com/sgl-project/sglang/pull/41402)) |
| Hold on every path | The hold armed only for decode-initiated aborts; transport/prefill faults released at t=0 | Defer on every failure kind ([#41404](https://github.com/sgl-project/sglang/pull/41404)) |
| Protocol hygiene | A reused room id could consume a stale ack; MORI had no ack at all | Generation-tagged `ABORT`/`ABORT_ACK` in the common layer; MORI opts into the hold ([#37077](https://github.com/sgl-project/sglang/pull/37077)) |
| Sibling pipes | The hicache retraction path freed device slots under an active device→host offload copy, so poisoned KVCache persisted into the host tier | Retraction waits for the offload to drain ([#24226](https://github.com/sgl-project/sglang/pull/24226), [#30899](https://github.com/sgl-project/sglang/pull/30899)) |
| Rank consensus | TP ranks branched on the rank-local hicache restore status after the MIN-reduced KVCache poll. One rank’s failed restore desynchronized the group (queue-length mismatch, scheduler crash), and its admission accounting diverged one tick later | Fold the restore outcome into the already-reduced KVCache poll, so every rank makes one decision ([#43562](https://github.com/sgl-project/sglang/pull/43562)) |
| Detect | Corruption that slips past prevention is invisible | Opt-in transfer checksums ([#39500](https://github.com/sgl-project/sglang/pull/39500)); a commit-time length validation was evaluated and declined (see PD Checksum) |
| Observe | No way to tell if the ack fast path is alive in production | Hold count, duration, and `drained`/`timeout`/`error` outcome metrics ([#41128](https://github.com/sgl-project/sglang/pull/41128)) |
The middle rows came from a community audit. In the audit, published on September 11, 2026 ([#38961](https://github.com/sgl-project/sglang/pull/38961)), [@kflansburg](https://github.com/kflansburg) found two more ways the race could reopen. The drain ack often failed to arrive, so releases fell back to the timeout. A fault that decode did not itself initiate never armed the hold, which reopened the original bug by another path. We closed both on main between September 22 and 28, 2026: The hold now arms on every failure kind, and the drain ack now reaches decode. The audit also flagged prefill’s own source pages, and our analysis showed that path was already safe, because those bytes can only land in slots the decode hold is quarantining. [#43562](https://github.com/sgl-project/sglang/pull/43562), merged on October 11, 2026, fixes a further finding, a rank-divergent HiCache restore.
### Disclosure Timeline
| Date | Event |
| --- | --- |
| 2026-09-09 | Cloudflare notifies the SGLang maintainers of a cross-request KVCache exposure in production and shares follow-up safety checks in [#38704](https://github.com/sgl-project/sglang/pull/38704) |
| 2026-09-10 | SGLang maintainers meet with Cloudflare to agree on upstream fixes |
| 2026-09-11 | Hardening audit ([#38961](https://github.com/sgl-project/sglang/pull/38961)) finds the ack fast path often falling back to the timeout in production shapes |
| 2026-09-16 | Admission fix ([#38935](https://github.com/sgl-project/sglang/pull/38935)) and opt-in PD checksum ([#39500](https://github.com/sgl-project/sglang/pull/39500)) merged |
| 2026-09-22 → 09-28 | Fast path restored ([#40645](https://github.com/sgl-project/sglang/pull/40645), [#41402](https://github.com/sgl-project/sglang/pull/41402)); every failure kind defers ([#41404](https://github.com/sgl-project/sglang/pull/41404)); default-on ([#41023](https://github.com/sgl-project/sglang/pull/41023)); hicache fix ([#30899](https://github.com/sgl-project/sglang/pull/30899)) |
| 2026-10-02 | v0.5.21 released, protected by default (Mooncake, NIXL) |
| 2026-10-03 | MORI and generation-tagged acks merged ([#37077](https://github.com/sgl-project/sglang/pull/37077)); ships in v0.5.22 |
| 2026-10-11 | Rank-consensus fix for decode HiCache restore merged ([#43562](https://github.com/sgl-project/sglang/pull/43562)) |
| 2026-10-12 | Failed Mooncake sends no longer ack as drained ([#43719](https://github.com/sgl-project/sglang/pull/43719)); ships in v0.5.22 |
| 2026-10-22 | Failed NIXL and MORI sends no longer ack as drained ([#45973](https://github.com/sgl-project/sglang/pull/45973)); ships in v0.5.23 |
| 2026-10-26 | Coordinated disclosure by SGLang and Cloudflare. |