# Microsoft Research Asia 开源 Agent Lightning v1.0:3,500 行代码的真实 harness 智能体 RL 训练框架
> 原标题:Agent Lightning v1.0: A 3,500-Line Lightweight Agentic RL Framework for Training Agents with Real Harnesses
- 来源:Microsoft Research 博客
- 发布时间:2026-10-07T16:00:00.000Z
- AIHOT:https://aihot.news/items/t9wypjd9cbb42ttuq7vrpp07l
- 原文:https://www.microsoft.com/en-us/research/blog/agent-lightning-v1-0-a-3500-line-lightweight-agentic-rl-framework-for-training-agents-with-real-harnesses/
## 摘要
Microsoft Research Asia 提出 Harnessed Agentic RL 训练范式并开源重建的 Agent Lightning v1.0,让部署时使用的同一 agent harness 直接参与强化学习,无需在训练框架内重写 agent。
## 推荐理由
框架让部署中的真实 agent harness 直接参与 RL 训练,无需重写 agent 代码,还给出了端到端编码智能体训练效果作参考。
## 正文 · 中文译文

## 概览
- 驾驭式智能体强化学习:微软亚洲研究院提出了一种训练范式,让部署时使用的同一智能体框架直接参与强化学习,无需在训练框架内重新实现智能体。
- 轻量设计:Agent Lightning v1.0 用约 3,500 行代码实现了完整的智能体强化学习控制平面。
- 原生 Kubernetes 支持:智能体以标准 Kubernetes 作业的形式运行在自管理集群、云 Kubernetes 或本地基础设施上,不依赖付费的商业沙箱服务。
- 数据高效的训练方案:一个端到端编码智能体流水线将 Qwen3.5-9B 在 SWE-bench Verified 上的 Pass@1 从 41.8% 提升至 56.4%,提高了 14.6 个百分点,且仅使用了约 6,000 条基于开源数据集的训练样本。
AI 智能体已从单一模型演变为由模型、工具和执行环境构成的复杂全栈系统。其能力越来越依赖于在模型之外协调它们的智能体框架。强化学习(RL)是一种让 AI 系统通过试错学习的方法,以对其行动的奖励和惩罚为指导。RL 可以让这些智能体变得更好,但大多数智能体 RL 系统要求开发者在训练框架内重新实现智能体。这成本高昂,而且意味着被训练的智能体并非最终部署的那个智能体。
为解决这一问题,微软亚洲研究院的研究人员提出了驾驭式智能体强化学习(Harnessed Agentic RL)训练范式,并开源了完全重建的 [Agent Lightning v1.0(在新标签页中打开)](https://github.com/microsoft/agent-lightning)。与原始版本相比,Agent Lightning v1.0 更强调保持轻量、与真实框架集成,以及提供完整、可复现的智能体 RL 训练流水线。
Agent Lightning v1.0 围绕驾驭式智能体强化学习进行了重建,主要改进包括:
- **轻量**:整个框架约 3,500 行代码。Agent Lightning v1.0 在一个足够小巧、清晰、易于理解、修改和扩展的代码库中实现了完整的驾驭式智能体强化学习系统。
- **在真实智能体框架上训练**:智能体通过 Agent Lightning v1.0 中的大语言模型(LLM)代理访问模型,现有框架代码保持不变。
- **原生 Kubernetes 支持**:智能体直接以 Kubernetes 作业的形式运行,无需外部商业沙箱服务。自管理集群和本地基础设施均可支持大规模 rollout。
- **完整的编码智能体训练示例**:一个基于 Qwen3.5-9B 构建的端到端流水线将 SWE-bench Verified 上的 Pass@1 从 41.8% 提升至 56.4%,绝对提升 14.6 个百分点,且仅使用了约 6,000 条训练样本。
## 传统智能体强化学习的局限
传统智能体强化学习假设训练框架拥有与环境的交互循环。在 ReAct 风格的循环中,模型生成一个动作,环境返回一个观察结果,观察结果被追加到上下文中,模型再生成下一个动作,因此整个 rollout 映射为一条连续的 token 轨迹。verl、AReaL 和 slime 等早期 RL 系统就是这样构建的,这意味着训练智能体需要在 RL 框架内重建其循环。
真实的 harness 已经超出了这一假设。mini-SWE-agent、OpenHands、OpenCode、Claude Code 和 Codex 等编码智能体各自带来自己的上下文管理、工具协议、执行逻辑和依赖,通用智能体系统也是如此。为训练重建一个 harness 成本高昂,而且重建后的智能体行为可能不再与部署的智能体一致。
Agent Lightning 走了一条不同的路。它在智能体与模型之间放置一个 LLM 代理。智能体照常运行:只需将原先调用模型 API 的端点指向 Agent Lightning,训练框架就能观察并记录其模型调用。在 v1.0 中,研究者更进一步,将这一范式正式定义为 Harnessed Agentic RL:部署时使用哪个智能体 harness,训练时就由哪个 harness 直接参与强化学习(图 1)。

图 1. 传统 agentic RL 与 Harnessed Agentic RL 的对比。在传统 agentic RL 中,训练框架管理环境和智能体循环。在 Harnessed Agentic RL 中,harness 同时管理这两者。
## 使用真实 harness 训练中的四个挑战
Harnessed Agentic RL 与传统 agentic RL 的一个核心区别在于,环境交互循环由智能体 harness 处理,而非训练框架。训练系统只能观察到一系列 LLM 请求与响应对,因此一次 rollout 可能被拆分成数量不定的训练样本。这带来了四个关键挑战:
- **重新分词与样本合并**:harness 以文本形式保存上下文,但 RL 训练需要 rollout 期间采样的 token ID。将文本再次经过 chat template 和 tokenizer 可能会改变 token 边界,因此相邻调用并不总能合并为一个样本。
- **优势计算**:重新分词、子智能体和上下文摘要可能将一次 rollout 拆分成多个样本。直接在样本级别计算基线和优势,会导致产生更多样本的 rollout 被重复计数,从而改变 rollout 级别原有的统计关系。
- **损失归一化**:按样本数平均损失会给产生更多样本的 rollout 更大权重。由于样本数往往只是 harness 行为的产物,损失归一化也必须避免受其扭曲。
- **训练后端调度**:样本数量和长度只有在 harness 完成后才可知,而 GPU 数量和数据/张量并行配置通常是固定的。后端必须将可变的工作负载映射到固定的资源上。
## 用 3,500 行代码构建完整的智能体 RL 控制平面
在系统设计上,Agent Lightning v1.0 将简洁性作为首要原则。整个框架约 3,500 行代码,包含三个核心组件:API Gateway、Rollout Controller 和 Customized Trainer(图 2)。
API 网关存储 rollout、模型和事件,并充当兼容 OpenAI 的 LLM 代理。它将 harness 中的每次模型调用与其 rollout 关联起来,并记录训练所需的提示、响应和 log probabilities。rollout 控制器启动并管理 agent 执行,既可以作为本地进程,也可以作为标准 Kubernetes 作业,从而将 agent 执行与训练器分离。基于 verl 构建的定制训练器创建 rollout、等待其完成、收集样本,并通过样本适配器组装最终的训练样本。因此,对于现有的 agent harness,只需将模型端点指向 Agent Lightning 代理,通常就足以快速接入 RL 训练。

图 2. Agent Lightning v1.0 系统架构,展示了 API 网关、rollout 控制器和定制训练器。
## 共置异步 RL
不同 agent 的 rollout 时间差异很大。同步 RL 会等待批次中最慢的 agent,导致 GPU 空闲,而完全异步 RL 虽然提高了利用率,但需要为 rollout 和训练分别设置 GPU 池。为此,Agent Lightning v1.0 引入了共置异步 RL,让 rollout 和模型更新共享同一组 GPU。
一旦系统收集到足够的 rollout,更新就会开始:API 网关暂停接受新请求,并等待已在进行的请求完成,更新完成后 rollout 恢复。整个状态转换对外部 agent harness 是透明的。在实验中,这种方法相比同步 RL 实现了约 2 倍的端到端加速,同时使用的 GPU 比传统异步 RL 更少(图 3)。

图 3. 同步 RL、异步 RL 和共置异步 RL 的对比。共置异步 RL 提高了利用率,同时占用的 GPU 更少。
## 在 Kubernetes 上运行 agent
收集足够的 rollout 意味着要同时运行许多 agent,这会消耗大量 CPU、内存和计算资源。其他 Harnessed Agentic RL 框架通常将这些 agent 托管在 Modal Sandbox 或 E2B 等商业沙箱服务上,成本会随规模迅速上升。相反,Agent Lightning v1.0 将它们作为标准 Kubernetes 作业运行,复用现有的自管理集群、云 Kubernetes 或本地基础设施(图 4)。现有计算资源得到更高效的利用,大规模 rollout 成本更低,整个流水线保持开源且可复现。

图 4. Agent Lightning v1.0 中的 Rollout Controller 提供原生 Kubernetes 支持,将 agent 直接作为标准 Kubernetes 作业运行。
## 6,000 个训练样本,性能提升 14.6 个百分点
为了测试该方法,研究人员在 SWE-smith、mini-SWE-agent 和 Qwen3.5-9B 上构建了完整流水线,涵盖数据清洗、环境构建、reward-hacking 防护和 RL 训练。训练集包含约 6,000 个样本,无需大规模计算。仅 RL 训练就将 Qwen3.5-9B 在 SWE-bench Verified 上从 41.8% 提升到 56.4%,提高了 14.6 个百分点。
编码 agent 实验进一步证实了此前对两个挑战的分析:优势计算和损失归一化。与样本级处理相比,rollout 级优势结合 rollout 级归一化实现了更高的验证奖励,并在训练期间保持策略熵更稳定(图 5)。

图 5. Qwen3.5-9B 在 SWE-smith 验证集上的通过率和策略熵。
[技术报告](https://www.microsoft.com/en-us/research/publication/agent-lightning-v1-0-towards-harnessed-agentic-rl/)
[GitHub 项目](https://github.com/microsoft/agent-lightning)
在新标签页中打开
文章 [Agent Lightning v1.0:一个 3,500 行的轻量级智能体强化学习框架,用于通过真实测试框架训练智能体](https://www.microsoft.com/en-us/research/blog/agent-lightning-v1-0-a-3500-line-lightweight-agentic-rl-framework-for-training-agents-with-real-harnesses/) 首次出现在 [Microsoft Research](https://www.microsoft.com/en-us/research)。
## 正文 · 原文

## At a glance
- Harnessed Agentic RL: Microsoft Research Asia introduces a training paradigm in which the same agent harness used in deployment participates directly in reinforcement learning, removing the need to reimplement the agent inside the training framework.
- Lightweight by design: Agent Lightning v1.0 delivers a complete agent RL control plane in roughly 3,500 lines of code.
- Native Kubernetes support: agents run as standard Kubernetes jobs on self-managed clusters, cloud Kubernetes, or local infrastructure, with no dependency on paid commercial sandbox services.
- Data-efficient training recipe: an end-to-end coding agent pipeline raised Qwen3.5-9B from 41.8% to 56.4% Pass@1 on SWE-bench Verified, a 14.6 percentage point gain, using only about 6,000 training samples based on open sourced dataset.
AI agents have evolved from single models to complex full-stack systems built from models, tools, and execution environments. Their capabilities increasingly depend on the agent harness that coordinates them from outside the model. Reinforcement learning (RL) is an approach where AI systems learn through trial and error, guided by rewards and penalties for their actions. RL can make those agents better, but most agent RL systems require developers to reimplement the agent inside the training framework. That is costly, and it means the agent being trained is not quite the agent that gets deployed.
To address this, researchers at Microsoft Research Asia have introduced the Harnessed Agentic RL training paradigm and open-sourced a fully rebuilt [Agent Lightning v1.0 (opens in new tab)](https://github.com/microsoft/agent-lightning). Compared with the original, Agent Lightning, v1.0 puts more emphasis on staying lightweight, on integrating with real harnesses, and on a complete, reproducible agent RL training pipeline.
Agent Lightning v1.0 was rebuilt around Harnessed Agentic RL, with key improvements:
- **Lightweight**: the entire framework is about 3,500 lines of code. Agent Lightning v1.0 implements a complete Harnessed Agentic RL system in a codebase that is small and clear enough to understand, modify, and extend.
- **Training on a real agent harness**: agents reach the model through the large language model (LLM) proxy in Agent Lightning v1.0, leaving existing harness code unchanged.
- **Native Kubernetes support**: agents run directly as Kubernetes jobs, without external commercial sandbox services. Self-managed clusters and local infrastructure alike can support rollouts at scale.
- **A complete coding agent training example**: an end-to-end pipeline built on Qwen3.5-9B raised Pass@1 on SWE-bench Verified from 41.8% to 56.4%, an absolute gain of 14.6 percentage points, using only about 6,000 training samples.
## The limits of traditional agentic RL
Traditional agentic RL assumes the training framework owns the interaction loop with the environment. In a ReAct-style loop, the model generates an action, the environment returns an observation, the observation is appended to the context, and the model generates the next action, so the whole rollout maps onto one continuous token trajectory. Early RL systems such as verl, AReaL, and slime were built this way, which meant training an agent required rebuilding its loop inside the RL framework.
Real harnesses have outgrown that assumption. Coding agents such as mini-SWE-agent, OpenHands, OpenCode, Claude Code, and Codex each bring their own context management, tool protocols, execution logic, and dependencies, as do general-purpose agent systems. Rebuilding one for training is expensive, and the rebuilt agent may no longer behave in the same way as the deployed agent.
Agent Lightning takes a different route. It places an LLM proxy between the agent and the model. The agent continues to run as before: simply point the endpoint that previously called the model API at Agent Lightning, and the training framework can observe and record its model calls. In v1.0, the researchers go further and formally define this paradigm as Harnessed Agentic RL: whichever agent harness is used in deployment is the harness that takes part directly in reinforcement learning during training (Figure 1).

Figure 1. Traditional agentic RL compared with Harnessed Agentic RL. In traditional agentic RL, the training framework manages the environment and the agent loop. In Harnessed Agentic RL, the harness manages both.
## Four challenges in training with real harnesses
A core difference between Harnessed Agentic RL and traditional agentic RL is that the environment interaction loop is handled by the agent harness rather than the training framework. The training system can only observe a series of LLM request and response pairs, so a single rollout may be split into a variable number of training samples. This brings four key challenges:
- **Retokenization and sample merging**: Harnesses keep context as text, but RL training needs the token IDs sampled during the rollout. Passing text through the chat template and tokenizer again can shift token boundaries, so adjacent calls cannot always be merged into one sample.
- **Advantage calculation**: Retokenization, subagents and context summarization can split one rollout into several samples. Computing baselines and advantages directly at the sample level causes rollouts that produce more samples to be counted repeatedly, which alters the original statistical relationships at the rollout level.
- **Loss normalization**: Averaging loss by sample count gives more weight to rollouts that produce more samples. Since sample count is often just a product of harness behavior, loss normalization must also avoid being distorted by it.
- **Training backend scheduling**: Sample count and length are known only after the harness finishes, while GPU counts and data/tensor parallel configurations are usually fixed. The backend has to map a variable workload into fixed resources.
## Building a complete agent RL control plane in 3,500 lines of code
In system design, Agent Lightning v1.0 treats simplicity as its first principle. The entire framework is about 3,500 lines of code, with three core components: the API Gateway, the Rollout Controller, and the Customized Trainer (Figure 2).
The API gateway stores rollouts, models, and events, and serves as an OpenAI-compatible LLM proxy. It links every model call from the harness to its rollout and records the prompts, responses, and log probabilities that training needs. The rollout controller starts and manages agent execution, either as local processes or as standard Kubernetes jobs, keeping agent execution separate from the trainer. The customized trainer, built on verl, creates rollouts, waits for them to finish, collects samples, and assembles the final training samples through a sample adapter. As a result, for an existing agent harness, simply pointing the model endpoint at the Agent Lightning proxy is usually enough to connect quickly to RL training.

Figure 2. The Agent Lightning v1.0 system architecture showing the API gateway, rollout controller, and customized trainer.
## Collocated async RL
Rollout times vary widely across agents. Synchronous RL waits for the slowest agent in a batch and leaves GPUs idle, while fully asynchronous RL raises utilization but needs separate GPU pools for rollout and training. In response, Agent Lightning v1.0 introduces Collocated Async RL, which lets rollout and model updates share the same set of GPUs.
Once the system has collected enough rollouts, the update begins: the API Gateway pauses accepting new requests and waits for requests already in progress to finish, and rollout resumes after the update completes. The entire state transition is transparent to the external agent harness. In experiments, this approach achieved about a 2x end-to-end speedup over synchronous RL while using fewer GPUs than conventional asynchronous RL (Figure 3).

Figure 3. Synchronous RL, asynchronous RL, and Collocated Async RL compared. Collocated Async RL raises utilization while occupying fewer GPUs.
## Running agents on Kubernetes
Collecting enough rollouts means running many agents at once, which consumes substantial CPU, memory, and compute resources. Other Harnessed Agentic RL frameworks often host those agents on commercial sandbox services such as Modal Sandbox or E2B, where cost climbs quickly with scale. Instead, Agent Lightning v1.0 runs them as standard Kubernetes jobs, reusing existing self-managed clusters, cloud Kubernetes, or local infrastructure (Figure 4). Existing compute resources are used more efficiently, large rollouts cost less, and the whole pipeline stays open source and reproducible.

Figure 4. The Rollout Controller in Agent Lightning v1.0 provides native Kubernetes support, running agents directly as standard Kubernetes jobs.
## 6,000 training samples, a 14.6-point performance gain
To test the approach, researchers built a full pipeline on SWE-smith, mini-SWE-agent, and Qwen3.5-9B, covering data cleaning, environment construction, reward-hacking safeguards, and RL training. The training set holds about 6,000 samples and needs no large-scale compute. RL training alone raised Qwen3.5-9B from 41.8% to 56.4% on SWE-bench Verified, a gain of 14.6 percentage points.
The coding agent experiments further confirm the earlier analysis of two challenges: advantage calculation and loss normalization. Compared with sample-level handling, rollout-level advantage combined with rollout-level normalization achieves a higher validation reward and keeps policy entropy more stable during training (Figure 5).

Figure 5. Pass rate and policy entropy for Qwen3.5-9B on the SWE-smith validation set.
[Technical report](https://www.microsoft.com/en-us/research/publication/agent-lightning-v1-0-towards-harnessed-agentic-rl/)
[GitHub project](https://github.com/microsoft/agent-lightning)
Opens in a new tab
The post [Agent Lightning v1.0: A 3,500-Line Lightweight Agentic RL Framework for Training Agents with Real Harnesses](https://www.microsoft.com/en-us/research/blog/agent-lightning-v1-0-a-3500-line-lightweight-agentic-rl-framework-for-training-agents-with-real-harnesses/) appeared first on [Microsoft Research](https://www.microsoft.com/en-us/research).