# OpenRouter 重建默认路由算法:TTFT 降 16.9%、429 错误减少 67%
> 原标题:How we rebuilt provider routing for better, faster inference
- 来源:OpenRouter:Announcements
- 发布时间:2026-10-28T00:00:00.000Z
- AIHOT:https://aihot.news/items/l4znfnzfs6rz6vorl1wyj90gb
- 原文:https://openrouter.ai/blog/announcements/how-we-rebuilt-provider-routing/
## 摘要
OpenRouter 宣布重建默认 provider 路由算法,从只按价格排序改为按 Routing Cost 综合 effective price、TTFT、吞吐、错误率和输出质量选路。
## 推荐理由
OpenRouter 自己公布的路由算法实测数据与机制细节,读者可据此评估默认路由切换对现有调用成本和延迟的实际影响。
## 正文 · 中文译文
今天,我们宣布对 OpenRouter 自成立以来对提供商路由算法所做的最大改进。
此前,我们的默认路由器主要只奖励提供商一件事:低价。新的路由器奖励的是真正对你最有利的推理:便宜,同时快速、高吞吐且可靠。能提供更多价值的提供商会自动赢得更多流量,因此每个提供商都有理由持续改进。这意味着你通过 OpenRouter 获得的推理会越来越快、越来越可靠、越来越划算,而你无需改动一行代码。
它已经见效了。新算法带来了显著提升:TTFT(16.9%)、吞吐量(16.3%)以及容量错误(大幅降低 67%!),而成本仅小幅上升(6.0%)。这还是在提供商尚未有时间针对新算法进行优化_之前_的结果。更多细节请查看完整的[实验结果](https://openrouter.ai/blog/announcements/how-we-rebuilt-provider-routing/#results)。
工作原理:
- **它衡量你实际得到的东西。**对于每个模型,它会跟踪每个端点的实际价格、延迟、吞吐量和错误率,并为每个请求挑选成本与性能的最佳组合。
- **它实时做出反应。**当某个端点出现问题时,流量会从它那里转移,直到它恢复。
- **它奖励更好的推理。**进行创新的提供商会自动获得更多流量作为奖励。
路由仍然会感知缓存,因此一旦我们将你发送到某个提供商,只要缓存仍然有效,你的会话就会留在那里。
想要仅按价格路由?你仍然可以这样做。请参阅[保持不变的部分](https://openrouter.ai/blog/announcements/how-we-rebuilt-provider-routing/#what-stays-the-same)中关于固定和排序提供商的内容。
## 我们此前的状况:仅按价格路由的局限
OpenRouter 上的大部分流量(约 77% 的请求)流向由多个提供商提供的模型。当请求未包含明确的排序或提供商列表时,我们需要选择一个提供商来发送请求。此前,我们几乎完全基于输入 token 价格做出这一选择。提供商越便宜,获得的流量比例就越高,任何失败的请求都会被重新路由到下一个最便宜的选项。
这种仅按价格的方法为用户带来了生态系统中最低的价格,但也造成了一些意想不到的后果:
- **便宜者胜,即使它很差。**回退机制会捕获失败,但用户要等待它们完成,而速度的提升并未得到奖励。
- **静态排名。**该算法对实时性能变化视而不见,因此提供商必须退化到被完全降级的地步,才会反映在流量分配中。
- **利润消失。**提供商赢得流量的唯一方式就是降价,这迫使它们减少对确保可靠性和质量的基础设施的投资。
- **定价游戏。**我们用来排名的价格几乎全是提示输入价格,因此提供商可以通过奇怪的定价组合来保住流量份额,比如列出人为压低的输入价格,然后用定价过高的缓存或补全 token 来弥补差额。
我们希望确保这样一个市场:相互竞争的提供商通过提供更好的推理来赢得流量,而不仅仅是更便宜的推理。
## 新算法:针对一个优质答案的“成本”进行优化
新的路由算法会持续监控每个提供商的表现,计算出实时的路由成本,用来衡量用户为获得一个好答案所付出的代价。路由成本涵盖实际美元金额的价格、等待响应所花费的时间成本,以及提供商失败或返回不可用响应时重试的成本。
简单来说:一次尝试要花多少钱,它又有多大概率成功?


## 一次尝试的成本:价格加等待
对于每个请求,路由器会根据该模型近期的流量,估算它将使用多少输入和输出 token。这一估算同时决定了一次尝试的两个部分:你要支付的价格和你需要等待的时间。
### 有效价格
有效价格是提供商实际会向你收取的价格,根据近期实测流量计算得出。它反映了提供商真实的缓存命中率以及模型输入与输出 token 的平均构成,因此可能与标价相差很大。这确保提供商无法通过把昂贵的缓存或输出 token 隐藏在低输入价格之下来赢得流量。




路由器按每个提供商的有效价格对你的请求定价,并优先选择最便宜的选项。
### 吞吐量
吞吐量是与价格相权衡的因素:更快的提供商可以收取更高费用,却仍能赢得你的请求。它以端到端的每秒输出 token 数来衡量,因此包含首 token 时间加上提供商流式传输完整输出响应所需的时间。路由器会使用按模型设定的换算率,将该等待时间计入路由成本,换算率基于其提供商通常收取的费用及其速度。


### 首 token 时间
首 token 时间(TTFT)是响应开始流式传输之前你需要等待的时长。它单独计入路由成本,尽管它已经计入吞吐量,因为交互式使用最依赖它。在智能体会话中,模型可能连续发起许多工具调用,而每一次都要等待首 token。


### 速率限制
速率限制(HTTP 429)意味着提供商容量已满,拒绝了你的请求。你需要等待这次拒绝,然后你的完整请求会被发往另一个提供商。路由器会将这段等待时间加上一项惩罚计入该提供商的路由成本,并按它触发速率限制的频率加权,因此影响通常很小。反复出现的速率限制更多是通过下文所述的流量上限和退避机制产生影响。


## 它有多大概率成功:可靠性与质量
一次失败的尝试仍然要付出其价格和等待时间,因此路由成本会计算必须重试直到成功为止的预期成本。如果每 5 次尝试中有 1 次失败,那么每个好答案你要为 1.25 次尝试付费。
### 可靠性
可靠性错误是指因提供商而失败的请求。你会损失这次尝试并不得不重试,因此每个错误都会在路由成本中带来沉重的惩罚。


如果某个提供商在同一模型上的错误率远高于其他提供商,路由器会停止向其发送请求,直到它恢复。
### 质量
质量衡量提供商的响应是否可用。目前主要的检查是工具调用是否以良好格式返回,我们还在增加更多检查。每个糟糕的响应都会提高该提供商的路由成本,因为一个廉价但错误的答案代价高昂。


被标记为工具调用异常,或基准准确率远低于该模型应有得分的提供商,会远远落后,并且只会获得少量测试流量,直到标记被清除。
如果将所有请求都发送给单一最佳提供商,会使其过载,因此流量需要共享。更快的提供商可以承担更高的路由成本并保持竞争力。**竞争优势**范围内的所有提供商共享大部分流量,稍微落后于最佳提供商只会带来少量损失。超出该范围后,流量份额会迅速下降。


在此基础上还有三条保护措施:


- **下限:**每个提供商都保留少量流量,这样当表现不佳的端点有所改善时我们能及时发现。新端点从模型的典型数值开始,因此不需要流量历史记录。
- **退避:**开始失败或触发速率限制的提供商会获得更少流量,直到其恢复。
- **上限:**路由器会学习每个提供商在开始返回 429 之前能承受多少负载,并将剩余流量溢出到其他提供商。增加容量后,上限会自动提高。
## 保持不变的部分
- 设置了 `provider.order` 的请求会绕过新的路由逻辑,提供商按指定顺序排列(默认仍和以前一样基于价格)。设置 `provider.only` 或 `provider.ignore` 时,如果剩余提供商多于一个,将使用新算法从中选择。
- 显式指定 `provider.sort`(`price`、`throughput` 或 `latency`)会使用该排序方式,而不是新算法。
- 对话中的第一个请求才会由算法选择提供商。进行中的对话会固定到其原始提供商,以保持提示缓存处于热状态。如果该提供商变得不可用,对话会自动回退到另一个提供商。
- 如果某次尝试失败,请求会按排名顺序用下一个提供商重试。
## 我们如何衡量
为了衡量新路由算法的影响,我们必须超越标准的 A/B 测试,因为路由行为的变化很难与控制组隔离。例如,如果新算法能更好地将流量从表现不佳的提供商处引开,那么旧算法发往该提供商的请求会发现更多可用容量并变得更快,从而扭曲结果。请求形态和流量模式的巨大变异性进一步增加了难度。单个鲸鱼用户向单一提供商发起大量高负载请求,无论我们使用哪种算法路由其请求,都可能人为抬高该提供商的指标。
因此我们进行了一项切换实验:将每天分成 3 小时的时间块,在每对匹配的时间块之间切换新路由器和旧路由器,每次切换后有 1 小时的预热期并丢弃数据,并在整整一周内对具有代表性的开源模型样本测量每个模型的影响。
## 结果
在实验的一周内,我们在新旧路由算法之间均匀分配了超过 2400 万次请求和 1030 亿个输入 token,测量了对关键性能指标的影响。
| 指标 | 旧版 | 新算法 | 新 vs 旧 | 95% 区间 |
| --- | --- | --- | --- | --- |
| TTFT p50 | 1,007 ms | 837 ms | **降低 16.9%** | 降低 13.9% 至 19.8% |
| TTFT p95 | 4,601 ms | 3,733 ms | **降低 18.9%** | 降低 15.7% 至 21.9% |
| 吞吐量 p50 | 62.4 tok/s | 72.6 tok/s | **提高 16.3%** | 提高 13.8% 至 18.9% |
| 每千次请求成本 | $1.25 | $1.32 | **提高 6.0%** | 提高 5.1% 至 7.0% |
| 每百万 token 成本 | $0.301 | $0.324 | **提高 7.3%** | 提高 4.9% 至 9.6% |
| 调用方 TTFT p50 | 1,130 ms | 941 ms | **降低 16.7%** | 降低 13.8% 至 19.7% |
| 调用方 TTFT p95 | 6,032 ms | 4,504 ms | **降低 25.3%** | 降低 21.6% 至 29.0% |
| 整个请求 p50(到最后一个 token 的时间) | 4.00 秒 | 3.24 秒 | **降低 19.0%** | 降低 14.2% 至 23.6% |
| 整个请求 p95 | 41.5 秒 | 34.9 秒 | **降低 16.0%** | 降低 12.2% 至 19.7% |
| 至少遇到一次 429 | 7.1% | 3.8% | **−3.4 个百分点**
**减少 67%** | −4.2 至 −2.5 个百分点
减少 62% 至 72% | | 需要重试 | 10.4% | 6.9% | **−3.5 个百分点**
**减少 47%** | −4.4 至 −2.6 个百分点
减少 41% 至 52% | | 每个请求在失败尝试中花费的时间 | 349 毫秒 | 242 毫秒 | **−106 毫秒**
**减少 55%** | −181 至 −35 毫秒
减少 48% 至 60% | ## 开始使用 如果你使用默认路由,那就无需再做什么。为每个请求提供服务的提供商信息会包含在响应中,也会显示在你的活动页面上。 要保留原来仅按价格的路由,请将 `provider.sort` 设置为 `"price"`,在模型 slug 后追加 `:floor`,或在[账户设置](https://openrouter.ai/settings/preferences)中将默认排序设为价格。要自行选择提供商,请使用 `provider.order` 或 `provider.only`。请参阅[提供商路由指南](https://openrouter.ai/docs/guides/routing/provider-selection)。 ## 常见问题 ### 我需要修改代码才能使用新路由吗? 不需要。未显式指定提供商排序,且没有提供商顺序、only 或 ignore 列表的请求,会在已启用新路由的模型上自动使用新路由器。你的 API 密钥、请求体和模型 ID 保持不变。 ### 如何保留仅按价格的路由? 在请求体中将 provider.sort 设为 price,在模型 slug 后追加 :floor,或在账户设置中将默认提供商排序设为价格。任何显式排序都会绕过新路由器。 ### 新路由会覆盖我的提供商顺序或 only 列表吗? provider.order 会绕过新路由器,并按你指定的顺序路由。provider.only 和 provider.ignore 仍会筛选候选集合,但新算法会在剩余提供商中进行选择。 ### 新路由会比仅按价格的路由成本更高吗? 可能会,因为它有时会选择更快或更可靠的提供商,而不是最便宜的。在我们的线上实验中,每个请求的成本增加了 6.0%,同时 TTFT 提升了 16.9%,吞吐量提升了 16.3%,429 错误率下降了 67%。 ### 在新路由下,提供商如何获得更多流量? 改善路由器评分的信号:有效价格、首 token 时间、吞吐量、错误率和限流率,以及输出质量。 ## 正文 · 原文 Today we’re announcing the biggest improvement to our provider routing algorithm since the founding of OpenRouter. Until now, our default router mostly rewarded providers for one thing: a low price. The new router rewards the inference that is actually best for you: cheap, but also fast, high-throughput, and reliable. Providers that deliver more value automatically win more traffic, so every provider has a reason to keep improving. That means the inference you get through OpenRouter keeps getting faster, more reliable, and more cost-effective, and you don’t have to change a line of code. It’s already working. The new algorithm has unlocked significant improvements in TTFT (16.9%), throughput (16.3%) and capacity errors (a remarkable 67% reduction!), with only minor increases in cost (6.0%). That’s _before_ providers have had any time to optimize for the new algorithm. See our full [experimental results](https://openrouter.ai/blog/announcements/how-we-rebuilt-provider-routing/#results) for more details. How it works: - **It measures what you actually get.** For every model, it tracks each endpoint’s realized price, latency, throughput, and error rate, and picks the best blend of cost and performance for each request. - **It reacts in real time.** When an endpoint has trouble, traffic moves away from it until it recovers. - **It rewards better inference.** Providers that innovate are automatically rewarded with more traffic. Routing is still cache-aware, so once we’ve sent you to a provider, your session stays put as long as the cache is warm. Want price-only routing? You can still have it. See pinning and sorting providers in [What stays the same](https://openrouter.ai/blog/announcements/how-we-rebuilt-provider-routing/#what-stays-the-same). ## Where we were: the limitations of Price-Only Routing Most traffic (~77% of requests) on OpenRouter goes to models served by more than one provider. When a request doesn’t include an explicit sort or provider list, we need to pick a provider to send it to. Until now, we made that choice based almost entirely on input token price. The less expensive a provider was, the greater percentage of traffic it would receive, and any failed request was re-routed to the next cheapest option. This price-only approach got users the lowest prices in the ecosystem, but it created some unintended consequences: - **Cheap won even when it was bad.** Fallbacks caught failures, but users waited through them, and speed improvements weren’t rewarded. - **Static rankings.** The algorithm was blind to real-time performance variations, so a provider had to degrade to the point of being de-ranked entirely before it was reflected in traffic distribution. - **Vanishing margins.** The only way for providers to win traffic was to cut price, putting pressure on them to reduce their investments in the very infrastructure that ensures reliability and quality. - **Pricing games.** The price we ranked by was almost all prompt input price, so a provider could keep traffic share with weird pricing cocktails like listing an artificially low input price and then making back the difference with overpriced cached or completion tokens. We wanted to ensure a marketplace where competing providers won traffic by serving better inference, not just cheaper inference. ## The New Algorithm: optimizing for what a good answer “costs” The new routing algorithm constantly monitors the performance of each provider to calculate a real-time Routing Cost that captures what a user pays for a good answer. The Routing Cost encompasses the price in actual dollars and cents, the cost of time spent waiting for a response, and the cost of retries when a provider fails or provides an unusable response. To put it simply: what does one try cost, and how often does it work?   ## What one try costs: price plus waiting For each request, the router estimates how many input and output tokens it will use from the model’s recent traffic. That one estimate sets both parts of a try: the price you would pay and the time you would wait. ### Effective price Effective price is the price that the provider would actually charge you, calculated from recent measured traffic. It reflects the provider’s real cache hit rate and the model’s average mix of input and output tokens, so it can differ a lot from the list price. This ensures that a provider can’t win traffic by hiding expensive cached or output tokens behind a low input price.     The router prices your request at each provider’s effective price, and prioritizes the cheapest options. ### Throughput Throughput is what trades off against price: a faster provider can charge more and still win your request. It’s measured as output tokens per second end-to-end, so it includes the time to first token plus the time it takes the provider to stream the entire output response. The router adds that wait to the Routing Cost using a conversion rate set per model, based on what its providers typically charge and how fast they are.   ### Time to first token Time to first token (TTFT) is how long you would wait before the response started streaming. It adds to the Routing Cost on its own, even though it already counts toward throughput, because interactive use depends on it most. In an agent session, the model may make many tool calls in a row, and each one waits for a first token.   ### Rate limits A rate limit (HTTP 429) means the provider is out of capacity and turned your request away. You would have to wait for the rejection, then your full request would go to another provider. The router adds that wait plus a penalty to the provider’s Routing Cost, weighted by how often it rate limits requests, so the effect is usually small. Repeated rate limits matter more through the traffic caps and backoff described below.   ## How often it works: reliability and quality A failed try still costs its price and its wait, so the Routing Cost calculates the expected cost of having to retry until one works. If 1 in 5 tries fails, you pay for 1.25 tries per good answer. ### Reliability A reliability error is a request that fails because of the provider. You would lose the attempt and have to retry, so each error carries a heavy penalty in the Routing Cost.   If a provider’s error rate climbs far above the other providers for the same model, the router stops sending it requests until it recovers. ### Quality Quality measures whether a provider’s responses are usable. Today the main check is whether tool calls come back well formed, and we’re adding more. Each bad response raises the provider’s Routing Cost, because a cheap wrong answer is expensive.   A provider flagged for broken tool calls, or for benchmark accuracy well below what the model should score, drops far behind and only gets a small amount of test traffic until the flag clears. Sending every request to the single best provider would overload it, so traffic is shared. Faster providers can carry a higher Routing Cost and stay competitive. Everything inside the **competitive edge** shares most of the traffic, and being a little behind the best costs a little. Past the edge, traffic share drops off fast.   Three guardrails sit on top:   - **Floor:** every provider keeps a trickle of traffic, so we can notice when struggling endpoints improve. New endpoints start from the model’s typical numbers, so they don’t need a traffic history. - **Back off:** a provider that starts failing or rate limiting gets less until it recovers. - **Cap:** the router learns how much load each provider can take before it returns 429s, and spills the rest to others. Add capacity and the cap rises on its own. ## What stays the same - Requests that set `provider.order` bypass the new routing logic, with providers ordered as specified (default is price-based as before). Setting `provider.only` or `provider.ignore` will use the new algorithm to select from the remaining providers if there are more than one. - An explicit `provider.sort` (`price`, `throughput`, or `latency`) uses that sort instead of the new algorithm. - Only the first request in a conversation will have a provider algorithmically selected. An ongoing conversation is pinned to its original provider so that the prompt cache stays warm. If that provider becomes unavailable, the conversation will fall back to another provider automatically. - If a try fails, the request is retried with the next highest ranked provider. ## How we measured it In order to measure the impact of the new routing algorithm, we had to go beyond a standard A/B test, because changes in routing behavior can’t easily be isolated from the control group. For example, if the new algorithm did a better job of routing traffic away from a struggling provider, the old algorithm’s requests to that provider would find more capacity available and get faster, skewing the results. Further compounding the difficulty is the enormous variability in request shape and volume patterns. A single whale user that spins up an automated flurry of high effort requests to a single provider can artificially inflate the metrics for that provider regardless of which algorithm we used to route their requests. So we ran a switchback experiment: We split each day into 3-hour blocks and flipped each matched pair of blocks between the new and old router, with a one-hour burn-in period of discarded data each changeover, and measured the per-model impact across a representative sample of open source models over one full week. ## Results Over the week of the experiment, we split over 24 million requests with 103 billion input tokens evenly between the old and new routing algorithms, measuring the impact on key performance metrics. | Measure | Legacy | New Algorithm | New vs legacy | 95% interval | | --- | --- | --- | --- | --- | | TTFT p50 | 1,007 ms | 837 ms | **16.9% lower** | 13.9% to 19.8% lower | | TTFT p95 | 4,601 ms | 3,733 ms | **18.9% lower** | 15.7% to 21.9% lower | | Throughput p50 | 62.4 tok/s | 72.6 tok/s | **16.3% higher** | 13.8% to 18.9% higher | | Cost per K requests | $1.25 | $1.32 | **6.0% higher** | 5.1% to 7.0% higher | | Cost per M tokens | $0.301 | $0.324 | **7.3% higher** | 4.9% to 9.6% higher | | Caller TTFT p50 | 1,130 ms | 941 ms | **16.7% lower** | 13.8% to 19.7% lower | | Caller TTFT p95 | 6,032 ms | 4,504 ms | **25.3% lower** | 21.6% to 29.0% lower | | Whole request p50 (time to last token) | 4.00 s | 3.24 s | **19.0% lower** | 14.2% to 23.6% lower | | Whole request p95 | 41.5 s | 34.9 s | **16.0% lower** | 12.2% to 19.7% lower | | Hit at least one 429 | 7.1% | 3.8% | **−3.4 pt**
**67% fewer** | −4.2 to −2.5 pt
62% to 72% fewer | | Needed a retry | 10.4% | 6.9% | **−3.5 pt**
**47% fewer** | −4.4 to −2.6 pt
41% to 52% fewer | | Time spent in failed attempts, per req | 349 ms | 242 ms | **−106 ms**
**55% less** | −181 to −35 ms
48% to 60% less | ## Get started If you use default routing, you’re done. The provider that served each request is in the response and on your activity page. To keep the old price-only routing, set `provider.sort` to `"price"`, append `:floor` to the model slug, or set your default sort to price in [account settings](https://openrouter.ai/settings/preferences). To pick providers yourself, use `provider.order` or `provider.only`. See the [provider routing guide](https://openrouter.ai/docs/guides/routing/provider-selection). ## Frequently asked questions ### Do I need to change my code to use the new routing? No. Requests with no explicit provider sort and no provider order, only, or ignore list use the new router automatically on the models where it's enabled. Your API key, request body, and model IDs stay the same. ### How do I keep price-only routing? Set provider.sort to price in the request body, append :floor to the model slug, or set your default provider sort to price in account settings. Any explicit sort bypasses the new router. ### Does the new routing override my provider order or only list? provider.order bypasses the new router and routes in the order you specify. provider.only and provider.ignore still filter the candidate set, but the new algorithm picks among the providers that remain. ### Will the new routing cost more than price-only routing? It can, since it will sometimes pick a faster or more reliable provider over the cheapest one. In our live experiment, cost per request increased by 6.0%, while realizing a 16.9% improvement in TTFT, a 16.3% improvement in throughput, and a 67% decrease in 429 error rate. ### How does a provider get more traffic under the new routing? Improve the signals the router scores: effective price, time to first token, throughput, error and rate-limit rates, and output quality.
**减少 67%** | −4.2 至 −2.5 个百分点
减少 62% 至 72% | | 需要重试 | 10.4% | 6.9% | **−3.5 个百分点**
**减少 47%** | −4.4 至 −2.6 个百分点
减少 41% 至 52% | | 每个请求在失败尝试中花费的时间 | 349 毫秒 | 242 毫秒 | **−106 毫秒**
**减少 55%** | −181 至 −35 毫秒
减少 48% 至 60% | ## 开始使用 如果你使用默认路由,那就无需再做什么。为每个请求提供服务的提供商信息会包含在响应中,也会显示在你的活动页面上。 要保留原来仅按价格的路由,请将 `provider.sort` 设置为 `"price"`,在模型 slug 后追加 `:floor`,或在[账户设置](https://openrouter.ai/settings/preferences)中将默认排序设为价格。要自行选择提供商,请使用 `provider.order` 或 `provider.only`。请参阅[提供商路由指南](https://openrouter.ai/docs/guides/routing/provider-selection)。 ## 常见问题 ### 我需要修改代码才能使用新路由吗? 不需要。未显式指定提供商排序,且没有提供商顺序、only 或 ignore 列表的请求,会在已启用新路由的模型上自动使用新路由器。你的 API 密钥、请求体和模型 ID 保持不变。 ### 如何保留仅按价格的路由? 在请求体中将 provider.sort 设为 price,在模型 slug 后追加 :floor,或在账户设置中将默认提供商排序设为价格。任何显式排序都会绕过新路由器。 ### 新路由会覆盖我的提供商顺序或 only 列表吗? provider.order 会绕过新路由器,并按你指定的顺序路由。provider.only 和 provider.ignore 仍会筛选候选集合,但新算法会在剩余提供商中进行选择。 ### 新路由会比仅按价格的路由成本更高吗? 可能会,因为它有时会选择更快或更可靠的提供商,而不是最便宜的。在我们的线上实验中,每个请求的成本增加了 6.0%,同时 TTFT 提升了 16.9%,吞吐量提升了 16.3%,429 错误率下降了 67%。 ### 在新路由下,提供商如何获得更多流量? 改善路由器评分的信号:有效价格、首 token 时间、吞吐量、错误率和限流率,以及输出质量。 ## 正文 · 原文 Today we’re announcing the biggest improvement to our provider routing algorithm since the founding of OpenRouter. Until now, our default router mostly rewarded providers for one thing: a low price. The new router rewards the inference that is actually best for you: cheap, but also fast, high-throughput, and reliable. Providers that deliver more value automatically win more traffic, so every provider has a reason to keep improving. That means the inference you get through OpenRouter keeps getting faster, more reliable, and more cost-effective, and you don’t have to change a line of code. It’s already working. The new algorithm has unlocked significant improvements in TTFT (16.9%), throughput (16.3%) and capacity errors (a remarkable 67% reduction!), with only minor increases in cost (6.0%). That’s _before_ providers have had any time to optimize for the new algorithm. See our full [experimental results](https://openrouter.ai/blog/announcements/how-we-rebuilt-provider-routing/#results) for more details. How it works: - **It measures what you actually get.** For every model, it tracks each endpoint’s realized price, latency, throughput, and error rate, and picks the best blend of cost and performance for each request. - **It reacts in real time.** When an endpoint has trouble, traffic moves away from it until it recovers. - **It rewards better inference.** Providers that innovate are automatically rewarded with more traffic. Routing is still cache-aware, so once we’ve sent you to a provider, your session stays put as long as the cache is warm. Want price-only routing? You can still have it. See pinning and sorting providers in [What stays the same](https://openrouter.ai/blog/announcements/how-we-rebuilt-provider-routing/#what-stays-the-same). ## Where we were: the limitations of Price-Only Routing Most traffic (~77% of requests) on OpenRouter goes to models served by more than one provider. When a request doesn’t include an explicit sort or provider list, we need to pick a provider to send it to. Until now, we made that choice based almost entirely on input token price. The less expensive a provider was, the greater percentage of traffic it would receive, and any failed request was re-routed to the next cheapest option. This price-only approach got users the lowest prices in the ecosystem, but it created some unintended consequences: - **Cheap won even when it was bad.** Fallbacks caught failures, but users waited through them, and speed improvements weren’t rewarded. - **Static rankings.** The algorithm was blind to real-time performance variations, so a provider had to degrade to the point of being de-ranked entirely before it was reflected in traffic distribution. - **Vanishing margins.** The only way for providers to win traffic was to cut price, putting pressure on them to reduce their investments in the very infrastructure that ensures reliability and quality. - **Pricing games.** The price we ranked by was almost all prompt input price, so a provider could keep traffic share with weird pricing cocktails like listing an artificially low input price and then making back the difference with overpriced cached or completion tokens. We wanted to ensure a marketplace where competing providers won traffic by serving better inference, not just cheaper inference. ## The New Algorithm: optimizing for what a good answer “costs” The new routing algorithm constantly monitors the performance of each provider to calculate a real-time Routing Cost that captures what a user pays for a good answer. The Routing Cost encompasses the price in actual dollars and cents, the cost of time spent waiting for a response, and the cost of retries when a provider fails or provides an unusable response. To put it simply: what does one try cost, and how often does it work?   ## What one try costs: price plus waiting For each request, the router estimates how many input and output tokens it will use from the model’s recent traffic. That one estimate sets both parts of a try: the price you would pay and the time you would wait. ### Effective price Effective price is the price that the provider would actually charge you, calculated from recent measured traffic. It reflects the provider’s real cache hit rate and the model’s average mix of input and output tokens, so it can differ a lot from the list price. This ensures that a provider can’t win traffic by hiding expensive cached or output tokens behind a low input price.     The router prices your request at each provider’s effective price, and prioritizes the cheapest options. ### Throughput Throughput is what trades off against price: a faster provider can charge more and still win your request. It’s measured as output tokens per second end-to-end, so it includes the time to first token plus the time it takes the provider to stream the entire output response. The router adds that wait to the Routing Cost using a conversion rate set per model, based on what its providers typically charge and how fast they are.   ### Time to first token Time to first token (TTFT) is how long you would wait before the response started streaming. It adds to the Routing Cost on its own, even though it already counts toward throughput, because interactive use depends on it most. In an agent session, the model may make many tool calls in a row, and each one waits for a first token.   ### Rate limits A rate limit (HTTP 429) means the provider is out of capacity and turned your request away. You would have to wait for the rejection, then your full request would go to another provider. The router adds that wait plus a penalty to the provider’s Routing Cost, weighted by how often it rate limits requests, so the effect is usually small. Repeated rate limits matter more through the traffic caps and backoff described below.   ## How often it works: reliability and quality A failed try still costs its price and its wait, so the Routing Cost calculates the expected cost of having to retry until one works. If 1 in 5 tries fails, you pay for 1.25 tries per good answer. ### Reliability A reliability error is a request that fails because of the provider. You would lose the attempt and have to retry, so each error carries a heavy penalty in the Routing Cost.   If a provider’s error rate climbs far above the other providers for the same model, the router stops sending it requests until it recovers. ### Quality Quality measures whether a provider’s responses are usable. Today the main check is whether tool calls come back well formed, and we’re adding more. Each bad response raises the provider’s Routing Cost, because a cheap wrong answer is expensive.   A provider flagged for broken tool calls, or for benchmark accuracy well below what the model should score, drops far behind and only gets a small amount of test traffic until the flag clears. Sending every request to the single best provider would overload it, so traffic is shared. Faster providers can carry a higher Routing Cost and stay competitive. Everything inside the **competitive edge** shares most of the traffic, and being a little behind the best costs a little. Past the edge, traffic share drops off fast.   Three guardrails sit on top:   - **Floor:** every provider keeps a trickle of traffic, so we can notice when struggling endpoints improve. New endpoints start from the model’s typical numbers, so they don’t need a traffic history. - **Back off:** a provider that starts failing or rate limiting gets less until it recovers. - **Cap:** the router learns how much load each provider can take before it returns 429s, and spills the rest to others. Add capacity and the cap rises on its own. ## What stays the same - Requests that set `provider.order` bypass the new routing logic, with providers ordered as specified (default is price-based as before). Setting `provider.only` or `provider.ignore` will use the new algorithm to select from the remaining providers if there are more than one. - An explicit `provider.sort` (`price`, `throughput`, or `latency`) uses that sort instead of the new algorithm. - Only the first request in a conversation will have a provider algorithmically selected. An ongoing conversation is pinned to its original provider so that the prompt cache stays warm. If that provider becomes unavailable, the conversation will fall back to another provider automatically. - If a try fails, the request is retried with the next highest ranked provider. ## How we measured it In order to measure the impact of the new routing algorithm, we had to go beyond a standard A/B test, because changes in routing behavior can’t easily be isolated from the control group. For example, if the new algorithm did a better job of routing traffic away from a struggling provider, the old algorithm’s requests to that provider would find more capacity available and get faster, skewing the results. Further compounding the difficulty is the enormous variability in request shape and volume patterns. A single whale user that spins up an automated flurry of high effort requests to a single provider can artificially inflate the metrics for that provider regardless of which algorithm we used to route their requests. So we ran a switchback experiment: We split each day into 3-hour blocks and flipped each matched pair of blocks between the new and old router, with a one-hour burn-in period of discarded data each changeover, and measured the per-model impact across a representative sample of open source models over one full week. ## Results Over the week of the experiment, we split over 24 million requests with 103 billion input tokens evenly between the old and new routing algorithms, measuring the impact on key performance metrics. | Measure | Legacy | New Algorithm | New vs legacy | 95% interval | | --- | --- | --- | --- | --- | | TTFT p50 | 1,007 ms | 837 ms | **16.9% lower** | 13.9% to 19.8% lower | | TTFT p95 | 4,601 ms | 3,733 ms | **18.9% lower** | 15.7% to 21.9% lower | | Throughput p50 | 62.4 tok/s | 72.6 tok/s | **16.3% higher** | 13.8% to 18.9% higher | | Cost per K requests | $1.25 | $1.32 | **6.0% higher** | 5.1% to 7.0% higher | | Cost per M tokens | $0.301 | $0.324 | **7.3% higher** | 4.9% to 9.6% higher | | Caller TTFT p50 | 1,130 ms | 941 ms | **16.7% lower** | 13.8% to 19.7% lower | | Caller TTFT p95 | 6,032 ms | 4,504 ms | **25.3% lower** | 21.6% to 29.0% lower | | Whole request p50 (time to last token) | 4.00 s | 3.24 s | **19.0% lower** | 14.2% to 23.6% lower | | Whole request p95 | 41.5 s | 34.9 s | **16.0% lower** | 12.2% to 19.7% lower | | Hit at least one 429 | 7.1% | 3.8% | **−3.4 pt**
**67% fewer** | −4.2 to −2.5 pt
62% to 72% fewer | | Needed a retry | 10.4% | 6.9% | **−3.5 pt**
**47% fewer** | −4.4 to −2.6 pt
41% to 52% fewer | | Time spent in failed attempts, per req | 349 ms | 242 ms | **−106 ms**
**55% less** | −181 to −35 ms
48% to 60% less | ## Get started If you use default routing, you’re done. The provider that served each request is in the response and on your activity page. To keep the old price-only routing, set `provider.sort` to `"price"`, append `:floor` to the model slug, or set your default sort to price in [account settings](https://openrouter.ai/settings/preferences). To pick providers yourself, use `provider.order` or `provider.only`. See the [provider routing guide](https://openrouter.ai/docs/guides/routing/provider-selection). ## Frequently asked questions ### Do I need to change my code to use the new routing? No. Requests with no explicit provider sort and no provider order, only, or ignore list use the new router automatically on the models where it's enabled. Your API key, request body, and model IDs stay the same. ### How do I keep price-only routing? Set provider.sort to price in the request body, append :floor to the model slug, or set your default provider sort to price in account settings. Any explicit sort bypasses the new router. ### Does the new routing override my provider order or only list? provider.order bypasses the new router and routes in the order you specify. provider.only and provider.ignore still filter the candidate set, but the new algorithm picks among the providers that remain. ### Will the new routing cost more than price-only routing? It can, since it will sometimes pick a faster or more reliable provider over the cheapest one. In our live experiment, cost per request increased by 6.0%, while realizing a 16.9% improvement in TTFT, a 16.3% improvement in throughput, and a 67% decrease in 429 error rate. ### How does a provider get more traffic under the new routing? Improve the signals the router scores: effective price, time to first token, throughput, error and rate-limit rates, and output quality.