Meta 详解 Prometheus 吉瓦级超集群:一期已完成并用于训练 Muse 模型

来源:Meta Engineering Blog(RSS) 2026年10月19日 21:00 AIHOT 评分:66 精选
摘要:Meta 介绍其位于俄亥俄州 New Albany 的吉瓦级超集群 Prometheus,一期已建成并用于训练 Muse 系列基础模型,目标扩展到约 100 万张 GPU。
推荐理由:Meta 官方拆解其俄亥俄超集群的网络、散热与可靠性工程,读者可了解吉瓦级训练集群的实际运行细节。
# Meta 详解 Prometheus 吉瓦级超集群:一期已完成并用于训练 Muse 模型 > 原标题:Inside Prometheus: Meta’s Journey to a Gigawatt-Scale Supercluster - 来源:Meta Engineering Blog - 发布时间:2026-10-19T13:00:10.000Z - AIHOT:https://aihot.news/items/ebl13m9gich6kfjixzvfiw5bq - 原文:https://engineering.fb.com/2026/10/19/data-center-engineering/inside-prometheus-meta-gigawatt-scale-supercluster/ ## 摘要 Meta 介绍其位于俄亥俄州 New Albany 的吉瓦级超集群 Prometheus,一期已建成并用于训练 Muse 系列基础模型,目标扩展到约 100 万张 GPU。 ## 推荐理由 Meta 官方拆解其俄亥俄超集群的网络、散热与可靠性工程,读者可了解吉瓦级训练集群的实际运行细节。 ## 正文 · 中文译文 - 随着 Muse 系列基础模型最近的发布,我们想带你深入了解让这一切成为可能的基础设施。 - Prometheus 是我们吉瓦级超级集群,其第一阶段已经完成,我们正努力将其扩展至约 100 万块 GPU 的规模。 - Prometheus 还采用了闭环液冷等技术,以帮助其高效运行。 目前,在俄亥俄州新奥尔巴尼,Meta 正在开展其迄今为止最具雄心的 AI 基础设施项目。Prometheus,[我们的吉瓦级超级集群,](https://www.youtube.com/watch?v=lvieN3sJTUk)是我们基础设施的一次进步,它正在训练下一代基础模型,这些模型将为数十亿人带来全新的 AI 体验。 两年前,新奥尔巴尼还是一个承载网络流量的常规数据中心站点。借助硬件、网络和建筑设计方面的进步,我们改造了这个站点,使其成为前沿模型训练项目的所在地。我们正在分阶段建设这座超级集群。第一阶段已经完成,并用于训练我们的 Muse 基础模型,包括 [Muse Wonder](https://research.meta.ai/blog/muse-wonder) 模型——这是我们新的前沿模型,在个人和专业智能体任务上实现了最先进的性能。 大规模训练任务需要一切以最优状态运行才能维持其性能;单台主机上的问题就可能拖慢或中止整个任务,降低其“有效算力利用率”(goodput)。通过专注于提升系统的稳定性和可靠性,我们在 Muse Wonder 的预训练运行中实现了超过 85% 的 goodput,在强化学习(RL)运行中实现了超过 90% 的 goodput。在这些训练运行期间,我们实现了约 47k GPU 小时的整体挂钟平均无中断时间(MTBI)。此后,我们在整个集群上实施了更多变更,包括固件修复,将 MTBI 提升至超过 54K GPU 小时,并且我们还在持续改进。 现在,我们正着眼于 Meta Superintelligence Labs 的下一代前沿模型,继续扩展 Prometheus。 _Prometheus 超级集群位于 Meta 的俄亥俄州新奥尔巴尼数据中心。_ ## 超级集群:释放全新 AI 能力 如今,Prometheus 由分布在多栋互连建筑中的数十万块 GPU 组成,这些建筑包括传统数据中心建筑、AI 优化数据中心建筑以及快速部署结构,全部通过数英里的光纤紧密互连,作为一台单一机器运行。  这一容量使我们能够追求此前不切实际的更大模型和更长上下文窗口。在满规模状态下,Prometheus 将包含约 100 万块分布式 GPU,全部同步运行。  ## 多代 AI 硬件 Prometheus 目前部署了两个 NVIDIA GPU 平台,第三个平台(基于 NVIDIA 的 [Vera Rubin NVL72](https://www.nvidia.com/en-us/data-center/vera-rubin-nvl72/))即将到来。每个平台都代表着性能和效率的代际飞跃,依托我们的 NVL72 机架架构,其中 72 块 GPU 在同一网络结构内作为单个加速器运行。 ### Catalina:NVIDIA GB200 我们的首个平台[Catalina](https://engineering.fb.com/2024/10/15/data-infrastructure/metas-open-ai-hardware-vision/)采用 NVIDIA GB200 NVL72。我们部署的是 Ariel 变体,这是一种双机架配置,每个机架中的 36 块 GPU 通过 NVLink 互连,构成一个统一的 72-GPU 域。与 [Grand Teton H100](https://engineering.fb.com/2024/10/15/data-infrastructure/metas-open-ai-hardware-vision/) 相比,Catalina 在原始算力和内存容量方面均有提升。它采用我们的 [Open Rack V3 (ORv3)](https://engineering.fb.com/2022/10/18/open-source/ocp-summit-2022-grand-teton/) 高功率机架(HPR)结构,这是一种贡献给 [Open Compute Project (OCP)](https://www.opencompute.org/?ref=engineeringatmeta) 的开放硬件设计。  GB200 NVL72 的 Ariel 变体为每块 NVIDIA Blackwell GPU 搭配一颗 NVIDIA Grace CPU 和 2 块 NVIDIA ConnectX-7 400G 网卡,CPU 与 GPU 的比例为 1:1,使我们能够比业界其他厂商更早地为每块 GPU 部署 800G 的 I/O。我们选择 Ariel,是因为相对于 NVIDIA H100,GB200 NVL72 在几乎所有其他维度上都实现了扩展:FLOPS、HBM 带宽、scale-up 带宽和 scale-up 域规模,而每块 GPU 的 scale-out 带宽原本会停留在 400G。我们预计这种不均衡会损害性能,因此选择了为每块 GPU 提供 800G scale-out 的配置。这也让我们在区域扩展、800G scale-out 变得可用时,拥有更统一的规格。 _ Catalina GB200 NVL72 具备:_ - _**72 块 NVIDIA Blackwell GPU**,配备 13.4 TB HBM3e(每块 GPU 192 GB)。_ - _**720 峰值 PFLOPS 的 FP8 训练算力**(FP4 下为 1,440 PFLOPS)。_ - _**域内 130 TB/s 峰值 NVLink 带宽**。_ - _**576 TB/s 峰值聚合内存带宽。**_ - _**约 120 kW 系统总功耗**,覆盖这对双机架。_ ### Clemente:NVIDIA GB300 NVL72 我们的第二个平台 Clemente 转向 NVIDIA GB300 NVL72,在 GPU 和平台层面均对 Catalina 有所改进。GB300 将每块 GPU 的 HBM 容量提升 1.5 倍,从 192 GB 增至 288 GB,密集 FP4 吞吐量也提升约相同倍数,同时注意力(softmax)加速大约翻倍。在平台层面,scale-out 不再是 GB200 NVL72 上的制约因素:NVIDIA ConnectX-8 SuperNIC 原生提供每块 GPU 800G,因此我们早期在 Ariel 上部署的能力成为基线。再加上 2:1 的 GPU 与 CPU 比例,Clemente 能够将完整的 72-GPU 域装入单个机架,简化了我们的物理部署。Clemente 可采用我们的模块化风助液冷(AALC)部署,也可在我们新设计的数据中心中使用设施液冷(FLC)。由于线缆减少,可维护性也大幅提升。 _Clemente GB300 NVL72 具备:_ - _**72 块 NVIDIA Blackwell Ultra GPU**,配备 20.7 TB HBM3e(每块 GPU 288 GB——比 GB200 多 50%)。_ - _**超过 1 exaFLOP 的峰值 FP4 算力。**_ - _**130 TB/s 峰值 NVLink 带宽**(每块 GPU 双向 1.8 TB/s)。_ - _**约 135 kW 机架功耗。**_ - _**性能较 GB200 NVL72 提升 5-10%**。_ ## 连接超级集群 在传统 Web 服务中,数据中心网络在实例通常不需要高带宽、高性能以及跨多台服务器紧密锁步协调的服务之间传输请求和响应。大规模 AI 训练需要所有这些特性,这意味着连接它们的网络如今已成为 AI 工作负载的重要组成部分。为最好地解决这一问题,除了连接服务器的标准数据中心网络外,几年前我们还新增了一个基于以太网的[专用第二数据网络](https://engineering.fb.com/2024/03/12/data-center-engineering/building-metas-genai-infrastructure/),称为后端(BE)网络,用于在数据中心建筑内部和跨建筑实现 GPU 到 GPU 的直接通信。 大型预训练任务将训练划分为数据并行的模型副本,从而在网络使用上形成层次结构:对延迟敏感和对带宽敏感的操作集中在副本内部,而对较高延迟和较低带宽容忍度更高的流量则在副本之间调度。我们的网络设计旨在匹配这种层次结构,在建筑内部采用低延迟、高带宽的交换结构,并通过聚合网络连接各建筑的交换结构。 我们与 Meta Superintelligence Labs 紧密合作,确保并行设计、流量模式以及通信算法和软件经过优化,以尽量减少任何网络通信的暴露。这使 GPU 尽可能保持忙碌,而不是等待网络。 Meta 团队在过去几年中已将这些网络概念应用于早先的[建筑级 24K GPU 集群](https://engineering.fb.com/2024/03/12/data-center-engineering/building-metas-genai-infrastructure/)和[我们的区域级 100K+ GPU 集群](https://engineering.fb.com/2025/09/29/data-infrastructure/metas-infrastructure-evolution-and-the-advent-of-ai/),并在 Prometheus 中将其提升到新的规模和距离水平。 ### 网络交换结构 在建筑层面,每个基于以太网的交换结构通过高带宽、低延迟的多层拓扑互连多达 20,736 个 GPU。我们最初的设计使用[解耦调度交换结构(DSF)](https://engineering.fb.com/2025/10/20/data-center-engineering/disaggregated-scheduled-fabric-scaling-metas-ai-journey/)来缓解训练集合通信造成的网络拥塞,这类通信通常具有少量大型、长生命周期的网络流,在传统负载均衡技术下会发生冲突;DSF 通过在交换结构中利用子数据包级多路径路由和数据包重排序,以及虚拟输出队列和基于信用的端到端流控制来实现这一点。 随着以太网 NIC 技术的进步,在端点实现数据包重排序成为可能,我们得以升级到[非调度交换结构(NSF)](https://engineering.fb.com/2025/10/13/data-infrastructure/ocp-summit-2025-the-open-future-of-networking-hardware-for-ai/#nsf),使用浅缓冲 51.2T 以太网交换机,例如带动态负载均衡的 Minipack3,以及基于 NVIDIA Spectrum-4 交换机 ASIC、在 ConnectX NIC 上使用 NVIDIA 自适应路由的[Minipack3N](https://engineering.fb.com/2025/10/13/data-infrastructure/ocp-summit-2025-the-open-future-of-networking-hardware-for-ai/) 。DSF 和 NSF 均基于 Meta 的[FBOSS 网络栈](https://engineering.fb.com/2025/10/13/data-infrastructure/ocp-summit-2025-the-open-future-of-networking-hardware-for-ai/#nsf),使我们能够控制并快速迭代和优化 Prometheus 中的交换结构,并具备大规模运营这些集群的可观测性。 ### 聚合 [后端聚合(BAG)](https://engineering.fb.com/2026/02/09/data-center-engineering/building-prometheus-how-backend-aggregation-enables-gigawatt-scale-ai-clusters/)网络是一个多层 hub-and-spoke 网络,连接 Prometheus 集群中各建筑级 fabric,跨度从 2 公里到 100 公里以上。它采用 Arista 7800 系列机箱,搭载深度缓冲的 Broadcom Jericho3 芯片,每个区域对可提供数 Pbps 带宽。 ![](https://engineering.fb.com/wp-content/uploads/2026/10/Meta-Prometheus-BAG-3.png) 在该网络上运行预训练需要与 Meta Superintelligence Labs 进行专门的优化工作,以确保训练不被拖慢,尤其是考虑到这些距离下的光速限制。 ![](https://engineering.fb.com/wp-content/uploads/2026/10/Meta-Prometheus-BAG-1.png) 综合来看,Prometheus 网络的复杂性涉及数百万英里的光纤、数万台交换机以及数百万个光模块——从建筑内部使用的 2×400G FR4 LITE 短距低成本可插拔光模块,到约 10 公里以上链路使用的 800G-ZR 相干光模块。 ![](https://engineering.fb.com/wp-content/uploads/2026/10/Meta-Prometheus-BAG-2.png) ## 在每个层面保障工程可靠性 在 Prometheus 的规模下,每个组件都需要进行可靠性监控。任何地方出现单一故障都可能中断整个训练任务,并浪费宝贵的计算资源。  在这一层面维持可靠性,需要与 NVIDIA 建立超越传统供应商关系的协同工程模式。团队深度嵌入我们的数据中心,共享实时遥测数据,并开展联合故障排查会议,按根本原因分解每一次系统停机。 这种以归因为驱动的责任模型确保双方都专注于有针对性的改进。Meta 优化调度和基础设施,NVIDIA 则处理固件、信号诊断和内存管理。这种透明度消除了模糊性,并推动两家组织更快修复问题。 ### 72-GPU 故障域 从 H100 世代的 NVL8 架构过渡到 NVL72,从根本上重新定义了我们应对基础设施韧性的方式。在 NVL8 时代,scale-up 域——即通过 NVLink 紧密耦合用于张量并行的 GPU 组——被限制为 8 个单元。借助 NVL72,该域扩展到 72 个 GPU,使故障影响范围扩大 9 倍。 更大的 scale-up 域正是 NVL72 带宽优势的来源,但它也使 72-GPU 机架成为[我们规划容量所依据的分配单元](https://openreview.net/pdf?id=18jPgte2tM)。训练并行性最高效地映射到 2 的幂组大小上,因此每个 NVL72 域运行一个 64-GPU 切片,并将剩余 8 个 GPU 保留为机架内备用。当某个 GPU 发生故障时,备用 GPU 立即接替,因此任务从上一个检查点重启,而不是等待硬件维修,机架也继续留在任务中。机架内冗余将有效机架故障率降至由可能移除整个 pod 的故障所设定的下限;对于更罕见的整机架故障,我们保留约 2% 的额外机架作为储备。对于 32,768 个活跃 GPU 的训练运行,这意味着需要配置 523 个 NVL72 机架,即 37,656 个 GPU,比活跃占用规模高出约 13%。这一余量的大部分(11%)来自 64-of-72 切片几何结构,而不是绝对的可靠性成本。 物理复杂性进一步加剧了这些可靠性要求。单个 NVL72 域包含 5,184 条以 200 Gbps 运行的调谐差分对信号链路,每条链路都经过多个潜在故障点,包括 GPU SerDes、计算托盘 PCB、线缆盒和交换机托盘接口。在如此密集的互连结构中,以每通道 200 Gbps 保持一致的信号完整性,在此规模下代表着前所未有的工程挑战。 ### 故障恢复  故障恢复涉及我们如何应对故障,以尽量减少对训练时间的影响。根据故障的范围(即影响范围),我们通过两种机制来处理: **并行感知容错(**[**PAFT**](https://arxiv.org/pdf/2602.00277)**):** 当故障可以隔离到某个数据并行模型副本时,我们允许将单个故障副本下线,而其余集群重复该步骤并继续有效训练,从而降低硬件故障的影响。受影响的切片只需在可用的备用主机上重启并重新同步,将潜在的整个系统停机转化为聚合吞吐量的轻微、暂时下降。 **检查点**:当故障影响范围大于一个数据并行模型副本时,整个作业会重启并回退到上一个检查点;频繁设置检查点可减少这些事件中丢失的工作量。我们的分布式检查点架构将模型状态(包括梯度和优化器数据)异步保存到多层存储。我们使用增量检查点和内存暂存,确保这些保存对训练吞吐量的影响最小。 这两种故障恢复机制与一种备用策略相结合,该策略利用机架内预分配的备用 GPU 托盘以及备用机架,以尽量减少更换硬件的启动时间,并减少因硬件故障而丢失的工作量 ### 系统性减少中断 在训练前沿模型时,基础设施可靠性需要超越简单可用性的视角。根本挑战在于保持训练势头,避免频繁的系统停机。在 Prometheus 中,我们将任何迫使作业暂停并启动恢复周期的故障——从 GPU 硅缺陷和网络结构异常到存储瓶颈或热限制——归类为_中断_。 这些中断会带来巨大的运维开销,因为我们的工作负载紧密相互依赖;72-GPU 域内的单个故障就可能影响整个集群的聚合吞吐量。这种性能损失直接影响我们的工程迭代速度,导致计算效率降低,并为数十亿人提供先进 AI 体验的时间线延长。 ![](https://engineering.fb.com/wp-content/uploads/2026/10/Meta-Prometheus-GB200-GB200-interruptions.png) 在 2025 年中期 Prometheus 的初始阶段,我们的中断率峰值达到每 8K GPU 每天近 1,000 次事件。经过一年的严格跨职能工程,我们已将中断减少了 99%,使集群能够并行维持数万个 GPU 的多个训练作业。 实现这种稳定性需要对每个中断源进行细粒度分解,然后针对每个故障类别部署有针对性的工程解决方案: **NVLink 信号完整性:**最初是 GB200 部署的主要瓶颈。在超过 600,000 个 NVLink 连接的情况下,即使信号出现轻微劣化,也会触发频繁的 NVLink 恢复周期。我们实施了持续遥测和自动标记,利用 L11 诊断烧录在生产前隔离弱链路。我们的分类工具可区分瞬态故障与持续性信号完整性故障,指导有针对性的维修和组件更换,并在修复后的系统返回生产前进行验证。  **内存可靠性(HBM 和 LPDDR):**随着我们以更高内存容量扩展 GB300,内存相关故障成为中断的主要来源。我们与 NVIDIA 紧密合作,利用集群遥测和详细的错误分析来识别反复出现的故障特征,并通过改进的 HBM 固件设置、内存巡检、改进的 LPDDR 纠错算法以及基于模式的预测检测来缓解这些问题,在性能下降的设备影响作业吞吐量之前将其退役。我们正在整个集群中部署这些变更,并衡量其对中断率和硬件更换需求的影响。 **NVLink 交换机与 Fabric 健康:**单个交换机故障可能危及整个 NVL72 域。我们以毫秒级分辨率监控 Fabric 健康状态,利用自动恢复路径隔离故障组件,并在其余基础设施之间重新平衡流量,而不会中断作业。 **后端网络事件:**在 BAG 超级骨干层,链路事件通过基于硬件的故障检测进行透明管理。我们的平面拓扑确保单个光纤或端口故障被重新路由,而不会中断同步的 GPU 流量突发。 持续改进由我们的运营基础设施驱动:New Albany 站点的同址分诊团队、3 月份全面同步全栈修复的稳定性冲刺,以及严格的归因驱动问责框架。每次维修都必须经过强制隔离,以在返回计算池之前验证系统完整性。 **我们的可靠性北极星仍然是每 8K GPU 每天两次中断,即约 100k GPU 小时的 MTBI。将中断率从峰值降低 99% 表明,在 Prometheus 的规模下,稳定性是一项工程挑战,且具有可解决的工程成果。** ## 在集群规模上成熟硬件 部署数十万个第一代加速器会暴露出在较低规模下隐藏的质量问题。GB200 部署已将硬件成熟化转变为我们团队和合作伙伴的一项主要工程学科。 工程领域包括: - **快速固件包部署:**我们构建了可靠且快速的固件包自动化,能够在几小时内向超过 100k GPU 部署固件和驱动程序修复。我们将单 Pod 升级时间的 P90 从硬件引入初期的超过七小时降低到两小时。这使我们能够将针对主要中断类别的快速修复推送到我们的集群。 - **集群健康:** 我们关联多机架系统的遥测数据,并对每一次作业中断进行分类,以指导软件修复、硬件维修和更换决策。我们跟踪 MTBI 以及可用容量和有效训练吞吐量,并建立了稳健的反馈循环,将硬件和软件可靠性改进转化为更高的计算产出。  - **跨合作伙伴认证:** 与多家制造合作伙伴实施并行验证,通过来自生产集群的持续反馈确保质量一致。 - **自动化信号筛查:** 在路由任何生产流量之前,使用诊断压力测试验证超过 600,000 条 NVLink 路径。在此规模下,即使微小的缺陷率也需要主动识别。 - **供电压力测试:** 研究 PSU 模块和电压调节器在单个 GPU 1,400W 负载下的相互作用,以揭示仅在大规模部署中才会出现的故障模式。 - **精密液冷:** 对冷板和歧管组件执行严格的质量关卡,以维持最佳结温并防止整个集群出现泄漏。 - **集群维护:** 主动准备、快速处置、修复优先级排序以及备件预置。  - **健康检查:** 使用先进的接入/老化测试和作业前健康检查来识别健康节点,以便纳入大型训练作业。 每一代新的平台架构——从 Blackwell 到 Vera Rubin——都融入了这些真实世界的运营洞察,在每一步收紧我们的筛查和设计规范。 ## 优化功耗与管理瞬态 [最大化吞吐量](https://arxiv.org/pdf/2605.24461) 在给定功率包络下,最小化搁浅功率,而同步分布式训练带来了供电难题。当数万个 GPU 齐步执行计算密集型阶段和通信周期时,会触发巨大的功率瞬态——集群内负载突然波动 30-40%。启动或暂停大型训练作业可以在几秒内转移数十兆瓦以上的负载。 我们通过 [Dynamo Dimmer](https://arxiv.org/pdf/2605.24461) 来解决这个问题,这是我们的集群级电源管理框架,可实现智能功率封顶。Dimmer 不会硬性停止工作负载,而是在集群接近电力上限时逐步调整 GPU 功率限制,确保训练不中断地继续。该系统由处理即时尖峰的机架级电池备份(BBU)模块、用于峰值平滑的电容托盘(CBU)、设施级电池储能系统(BESS)以及与公用事业提供商的深度协调来强化,以管理这些独特的 AI 负载曲线。  ## 管理异构集群中的冷却 虽然我们的分布式方法使我们能够快速构建 Prometheus,但这也意味着要在不同地点应对特定站点的供电和冷却限制。  我们的物理基础设施融合了多代冷却技术,以提供这些高功率机架所需的液冷能力。Prometheus 运行在混合的模块化辅助液冷(AALC)方案之上,包括我们的自然冷却数据中心建筑和快速部署结构。我们专为 AI 优化的定制建筑现在采用设施级液冷(FLC),以实现最大功率密度。  除了自有建筑外,我们还整合来自战略合作伙伴的租赁容量,从而加快部署时间表并提升总可部署容量。同时,我们的工业级快速部署结构重新定义了建设模式,以更快的周期交付可直接投入生产的算力。 ## 高效运营  在新奥尔巴尼,我们长期与当地公用事业公司合作,确保为我们的站点支付全部能源成本,且不会给当地电网增加任何不必要的负担。我们还与能源开发商合作,使我们的能源使用与新的清洁和可再生能源相匹配。  Meta 最新的 AI 优化数据中心采用[闭环液冷](https://about.fb.com/news/2026/08/closed-loop-cooling-explained-the-plumbing-behind-metas-ai/),因为这是冷却 GPU 服务器最高效的方式,并支持水资源的高效利用。芯片直连的闭环液冷通过让我们在单个服务器机架上容纳更多 GPU,有助于节省空间。我们还通过[OCP](https://engineering.fb.com/2026/10/12/open-source/ocp-summit-2026-powering-ai-with-open-innovations-in-scale-up-scale-out-and-scale-across-networks/)与更广泛的行业分享[IcePack](https://engineering.fb.com/2026/10/12/open-source/ocp-summit-2026-powering-ai-with-open-innovations-in-scale-up-scale-out-and-scale-across-networks/)等系统的设计,以延续我们的可持续发展工作;IcePack 是我们的液冷网络机架。 ## 超越 Prometheus 我们将继续与 NVIDIA 紧密合作,通过混合部署 NVIDIA GB300 NVL72 和 NVIDIA Vera Rubin NVL72 来扩展 Prometheus 的容量。 NVIDIA 的下一代[Vera Rubin NVL72](https://www.nvidia.com/en-us/data-center/vera-rubin-nvl72/?ref=engineeringatmeta)平台将在每 GPU 性能上实现代际飞跃,并将直接集成到 Prometheus 现有的网络架构中。随着集群成熟,它将通过我们即将推出的 San Miguel 平台,以 Vera Rubin 扩充其容量(主要是互连的 GB300),从而提供更高的每瓦算力。 ![](https://engineering.fb.com/wp-content/uploads/2026/10/Vera-Rubin-Prometheus.jpg) _San Miguel Vera Rubin NVL72 将具备:_ - **72 块 NVIDIA Vera Rubin GPU**,每块 GPU 配备 288 GB HBM4 和 19.2 TB/s 峰值带宽。 - **每机架 2.52 峰值 EFLOPS 的 FP4 训练算力** **1.26 峰值 EFLOPS FP8 训练算力**。 - **216 TB/s 峰值 NVLink 带宽**(每 GPU 双向 3 TB/s)。 - **约 1,500 TB/s 峰值聚合内存带宽。** - **全液冷**计算托盘,每机架功率超过 170 kW。 - **每瓦性能较 GB300 NVL72 提升 10-20%**。 Prometheus 中的技术成果帮助建立了蓝图,将指导我们未来如何应对超级集群设计的复杂性。我们即将在路易斯安那州里奇兰教区建设的超级集群 Hyperion,正从零开始设计,并充分考虑了来自 Prometheus 的洞见。它将采用闭环液冷、[与我们的公用事业合作伙伴 Entergy 实现全面电网连接](https://www.entergy.com/news/entergy-louisiana-announces-a-new-agreement-with-meta-that-will-deliver-an-additional-2b-in-customer-savings)(相关费用由 Meta 全额支付),以及下一代网络设计。  _了解更多关于我们在[新奥尔巴尼的社区工作。](https://datacenters.atmeta.com/2026/10/new-albany-where-ohios-community-spirit-meets-the-frontier-of-ai/)_ 文章 [深入 Prometheus:Meta 迈向吉瓦级超级集群的历程](https://engineering.fb.com/2026/10/19/data-center-engineering/inside-prometheus-meta-gigawatt-scale-supercluster/) 首次发布于 [Engineering at Meta](https://engineering.fb.com/)。 ## 正文 · 原文 - With the recent release of the Muse family of foundational models, we want to take you inside the infrastructure that has made it all possible. - The first phase of Prometheus, our gigawatt-scale supercluster, is already complete and we’re working toward scaling it to comprise  ~1 million GPUs. - Prometheus also features technologies like closed-loop liquid cooling to help it operate efficiently. Right now, in New Albany, Ohio, Meta is undertaking its most ambitious AI infrastructure project to date. Prometheus, [our gigawatt-scale supercluster,](https://www.youtube.com/watch?v=lvieN3sJTUk) is an advancement in our infrastructure that is training the next generation of foundational models that will bring new AI experiences to billions of people. Two years ago, New Albany was a conventional data center site running web traffic. With advancements in hardware, networking and building design, we reworked this site to be home to training programs for frontier models. We’re building the supercluster in phases. The first phase has already been completed and was used to train our Muse foundational models, including the [Muse Wonder](https://research.meta.ai/blog/muse-wonder) models, our new frontier models that deliver state-of-the-art performance on personal and professional agentic tasks. Large training jobs require everything to work optimally to sustain their performance; an issue on a single host can slow down or stop the entire job, lowering its ‘goodput’. By focusing on improving our systems’ stability and reliability we were able to achieve more than 85% goodput for pre-training runs and more than 90% goodput for reinforcement learning (RL) runs of Muse Wonder. During these training runs, we were able to achieve an overall wall-clock mean time between interruptions (MTBI) of ~47k GPU hours. Since then, we have landed further changes across our fleet including firmware fixes to increase MTBI to more than 54K GPU hours, and we are constantly improving it. Now, we’re continuing to scale Prometheus with our next frontier models from Meta Superintelligence Labs in mind. _The Prometheus supercluster is housed in Meta’s New Albany, Ohio, data center._ ## The Supercluster: Unlocking New AI Capabilities Today, Prometheus comprises an array of hundreds of thousands of GPUs across multiple interconnected buildings, including traditional and AI-optimized data center buildings and rapid deployment structures, all tightly interconnected by miles of fiber and functioning as a single machine.  This capacity allows us to pursue larger models and longer context windows than were previously practical. At full scale, Prometheus will comprise roughly 1 million distributed GPUs, all functioning in lockstep.  ## Multiple Generations of AI Hardware Prometheus currently deploys two NVIDIA GPU platforms, with a third (based on NVIDIA’s [Vera Rubin NVL72](https://www.nvidia.com/en-us/data-center/vera-rubin-nvl72/)) arriving soon. Each represents a generational leap in performance and efficiency, leveraging our NVL72 rack architecture where 72 GPUs act as a single accelerator within the same network fabric. ### Catalina: NVIDIA GB200 Our first platform, [Catalina](https://engineering.fb.com/2024/10/15/data-infrastructure/metas-open-ai-hardware-vision/), leverages the NVIDIA GB200 NVL72. We deploy the Ariel variant, a two-rack configuration in which 36 GPUs per rack are interconnected via NVLink to form a unified 72-GPU domain. Catalina provides improvements in raw compute and memory capacity compared to [Grand Teton H100](https://engineering.fb.com/2024/10/15/data-infrastructure/metas-open-ai-hardware-vision/). It leverages our [Open Rack V3 (ORv3)](https://engineering.fb.com/2022/10/18/open-source/ocp-summit-2022-grand-teton/) high-power rack (HPR) structure, an open hardware design contributed to the [Open Compute Project (OCP)](https://www.opencompute.org/?ref=engineeringatmeta).  The Ariel variant of GB200 NVL72 pairs each NVIDIA Blackwell GPU with an NVIDIA Grace CPU and 2x NVIDIA ConnectX-7 400G NICs at a 1:1 CPU-to-GPU ratio, letting us deploy 800G of I/O per GPU earlier than the rest of the industry. We chose Ariel because the GB200 NVL72 scaled up nearly every other dimension relative to the NVIDIA H100: FLOPS, HBM bandwidth, scale-up bandwidth, and scale-up domain size, while scale-out bandwidth per GPU would have stayed at 400G. We expected that imbalance to cost performance, so we opted for a configuration that delivers 800G of scale-out per GPU. This also gave us more uniformity of spec as the region expanded over time and 800G scale-out became available. _ The Catalina GB200 NVL72 features:_ - _**72 NVIDIA Blackwell GPUs** with 13.4 TB of HBM3e (192 GB per GPU)._ - _**720 peak PFLOPS of FP8 training compute** (1,440 PFLOPS at FP4)._ - _**130 TB/s peak NVLink bandwidth** within the domain._ - _**576 TB/s peak aggregate memory bandwidth.**_ - _**~120 kW total system power** across the two-rack pair._ ### Clemente: NVIDIA GB300 NVL72 Our second platform, Clemente, moves to the NVIDIA GB300 NVL72 and improves on Catalina at both the GPU and the platform level. GB300 raises HBM capacity per GPU by 1.5x, from 192 GB to 288 GB, and dense FP4 throughput by a similar factor, while roughly doubling attention (softmax) acceleration. At the platform level, scale-out is no longer the constraint it was on the GB200 NVL72: NVIDIA ConnectX-8 SuperNICs deliver 800G per GPU natively, so the capability we deployed early on Ariel becomes the baseline. That, together with a 2:1 GPU-to-CPU ratio, allows Clemente to pack the full 72-GPU domain into a single rack, streamlining our physical deployment. Clemente can be deployed with our modular Air-Assisted Liquid Cooling (AALC) or use Facility Liquid Cooling (FLC) in our newly designed data centers. Serviceability also improves substantially, with reduced cabling. _The Clemente GB300 NVL72 features:_ - _**72 NVIDIA Blackwell Ultra GPUs** with 20.7 TB of HBM3e (288 GB per GPU — 50% more than the GB200)._ - _**Over 1 exaFLOP of peak FP4 compute.**_ - _**130 TB/s peak NVLink bandwidth** (1.8 TB/s bidirectional per GPU)._ - _**~135 kW rack power.**_ - _**5-10% performance improvement** over the GB200 NVL72._ ## Connecting the Supercluster In traditional web services, the data center network transports requests and responses between services whose instances typically don’t require high bandwidth, high performance, and tight, lock-step coordination across many servers. Large-scale AI training requires all these, meaning the network that connects them is now an important part of AI workloads. To best address this, in addition to the standard data center network connecting servers, several years ago we added a [second dedicated data network](https://engineering.fb.com/2024/03/12/data-center-engineering/building-metas-genai-infrastructure/) based on Ethernet, called the Back-End (BE) network, to enable direct GPU-to-GPU communication both within and across data center buildings. Large pre-training jobs partition the training into data-parallel model replicas, creating a hierarchy in network usage: Latency-sensitive and bandwidth-sensitive operations concentrate within a replica, while traffic that is more tolerant of higher latency and less bandwidth is scheduled between replicas. Our network is designed to match this hierarchy, with low-latency, high-bandwidth fabrics within buildings and an aggregation network connecting the building fabrics.  We work closely with Meta Superintelligence Labs to ensure the parallelism design, traffic patterns, and communication algorithms and software are optimized to minimize exposure of any network communication. This keeps the GPUs as busy as possible and not waiting on the network. The Meta team has employed these networking concepts in earlier [building-scale, 24K GPU clusters](https://engineering.fb.com/2024/03/12/data-center-engineering/building-metas-genai-infrastructure/) and [our region-scale, 100K+ GPU cluster](https://engineering.fb.com/2025/09/29/data-infrastructure/metas-infrastructure-evolution-and-the-advent-of-ai/) over the last several years, and we took them to a new level of scale and distance in Prometheus. ### Network Fabrics At the building level, each Ethernet-based fabric interconnects up to 20,736 GPUs through a high-bandwidth, low-latency multi-layer topology. Our initial design used a [Disaggregated Scheduled Fabric (DSF)](https://engineering.fb.com/2025/10/20/data-center-engineering/disaggregated-scheduled-fabric-scaling-metas-ai-journey/) to mitigate the network congestion caused by training collectives that typically have a small number of large, long-lived network flows that collide under traditional load-balancing techniques; it does this by utilizing sub-packet-level multipath routing and packet re-ordering in the fabric along with virtual output queues and credit-based end-to-end flow control.  As Ethernet NIC technology advanced, enabling packet re-ordering at the endpoint, we were able to update to a [Non-Scheduled Fabric (NSF)](https://engineering.fb.com/2025/10/13/data-infrastructure/ocp-summit-2025-the-open-future-of-networking-hardware-for-ai/#nsf) utilizing shallow-buffer 51.2T Ethernet switches, such as Minipack3 with Dynamic Load Balancing and the NVIDIA Spectrum-4 switch ASIC-based [Minipack3N](https://engineering.fb.com/2025/10/13/data-infrastructure/ocp-summit-2025-the-open-future-of-networking-hardware-for-ai/) with NVIDIA’s Adaptive Routing on ConnectX NICs. Both DSF and NSF are based on Meta’s [FBOSS networking stack](https://engineering.fb.com/2025/10/13/data-infrastructure/ocp-summit-2025-the-open-future-of-networking-hardware-for-ai/#nsf), giving us the control to rapidly iterate on and optimize the fabrics in Prometheus, and the visibility to operate these clusters at scale. ### Aggregation The [Backend Aggregation (BAG)](https://engineering.fb.com/2026/02/09/data-center-engineering/building-prometheus-how-backend-aggregation-enables-gigawatt-scale-ai-clusters/) network is a multi-layer hub-and-spoke network that connects building-level fabrics across the Prometheus cluster, spanning 2 km to over 100 km. It is built with Arista 7800-series chassis running deep-buffered Broadcom Jericho3 silicon, delivering multiple Pbps of bandwidth per region pair. ![](https://engineering.fb.com/wp-content/uploads/2026/10/Meta-Prometheus-BAG-3.png) Running pretraining over this network required a dedicated optimization effort with Meta Superintelligence Labs to ensure training was not slowed down, especially given speed-of-light constraints at these distances. ![](https://engineering.fb.com/wp-content/uploads/2026/10/Meta-Prometheus-BAG-1.png) Pulling this all together, the complexity of the Prometheus network involves millions of miles of fiber strand, tens of thousands of switches, and millions of optics — from 2×400G FR4 LITE short-reach, low-cost pluggables inside the buildings to 800G-ZR coherent optics for links beyond ~10 km. ![](https://engineering.fb.com/wp-content/uploads/2026/10/Meta-Prometheus-BAG-2.png) ## Engineering Reliability at Every Level At Prometheus’ scale, every component needs to be monitored for reliability. A single fault anywhere can halt an entire training job and waste valuable compute resources.  Maintaining reliability at this level requires a co-engineering model with NVIDIA that goes beyond a traditional vendor dynamic. Teams are deeply embedded at our data centers, sharing real-time telemetry and running joint triage sessions to decompose every system halt by its root cause. This attribution-driven accountability model ensures both sides are focused on targeted improvements. While Meta optimizes scheduling and infrastructure, NVIDIA addresses firmware, signal diagnostics, and memory handling. This transparency eliminates ambiguity and drives faster fixes across both organizations. ### The 72-GPU Failure Domain The transition from the H100 generation’s NVL8 architecture to NVL72 has fundamentally redefined our approach to infrastructure resilience. In the NVL8 era, the scale-up domain — the group of GPUs tightly coupled via NVLink for tensor parallelism — was limited to eight units. With NVL72, that domain expands to 72 GPUs, increasing the fault blast radius by 9X. The larger scale-up domain is what gives NVL72 its bandwidth advantage, but it also makes the 72-GPU rack the [unit of allocation we plan capacity around](https://openreview.net/pdf?id=18jPgte2tM). Training parallelism maps most efficiently onto power-of-two group sizes, so each NVL72 domain runs a 64-GPU slice and keeps its remaining eight GPUs as in-rack spares. When a GPU fails, a spare takes its place immediately, so the job restarts from its last checkpoint instead of waiting on hardware repair, and the rack stays in the job. The in-rack redundancy pushes the effective rack fault rate down to the floor set by failures that can remove an entire pod; for rarer whole-rack faults we hold roughly 2% additional racks in reserve. For a 32,768 active-GPU training run, this means provisioning 523 NVL72 racks, or 37,656 GPUs, about 13% above the active footprint. Most of that margin (11%) falls out of the 64-of-72 slice geometry rather than being an absolute reliability tax. Physical complexity further intensifies these reliability requirements. A single NVL72 domain incorporates 5,184 tuned differential-pair signaling links operating at 200 Gbps, each traversing multiple potential points of failure, including GPU SerDes, compute tray PCBs, cable cartridges, and switch tray interfaces. Maintaining consistent signal integrity across this dense fabric — at 200 Gbps per lane — represents an unprecedented engineering challenge at this scale. ### Fault Recovery  Fault recovery addresses how we react to faults to minimize the impact on training time. Depending on the scope (i.e., blast area) of a fault we address it through two mechanisms: **Parallelism-Aware Fault Tolerance (**[**PAFT**](https://arxiv.org/pdf/2602.00277)**):** When a fault can be isolated to a data-parallel model replica, we reduce the impact of a hardware failure by allowing a single faulty replica to be taken offline, while the remaining cluster repeats that step and continues productive training. The compromised slice simply restarts on available spare hosts and resynchronizes, converting a potential total system halt into a minor, temporary reduction in aggregate throughput. **Checkpointing**: When a fault blast area is larger than a data-parallel model replica, the entire job is restarted back to the last checkpoint; frequent checkpointing reduces the amount of work lost in these events. Our distributed checkpointing architecture asynchronously saves the model state — including gradients and optimizer data — to multi-tier storage. We use delta checkpointing and in-memory staging to ensure these saves happen with minimal impact on training throughput. These two fault recovery mechanisms combine with a sparing strategy that utilizes pre-allocated spare GPU trays within a rack, as well as spare racks, to minimize the time to bring up replacement hardware and reduce the amount of work lost to hardware failures ### Systematic Reduction of Interruptions When training frontier models, infrastructure reliability requires a perspective beyond simple availability. The fundamental challenge lies in maintaining training momentum without frequent system halts. Within Prometheus, we classify an _interruption_ as any fault — from GPU silicon defects and network fabric anomalies to storage bottlenecks or thermal limits — that forces a job to pause and initiate a recovery cycle. These interruptions introduce a large operational overhead because our workloads are tightly interdependent; a single fault within a 72-GPU domain can affect the entire cluster’s aggregate throughput. This performance loss directly impacts our engineering iteration speed, resulting in less effective compute and longer timelines for delivering advanced AI experiences to billions of people. ![](https://engineering.fb.com/wp-content/uploads/2026/10/Meta-Prometheus-GB200-GB200-interruptions.png) During the initial phase of Prometheus in mid-2025, our interruption rates reached a peak of nearly 1,000 events per 8K GPUs daily. Through a year of rigorous cross-functional engineering, we have achieved a 99% reduction in interruptions, enabling the cluster to sustain multiple training jobs of tens of thousands of GPUs in parallel. Attaining this stability required a granular decomposition of every interruption source, followed by the deployment of targeted engineering solutions for each failure category: **NVLink Signal Integrity:** initially the primary bottleneck for GB200 deployments. With over 600,000 NVLink connections, even marginal signal degradation triggers frequent NVLink recovery cycles. We implemented continuous telemetry and automated flagging, utilizing L11 diagnostic burns to isolate weak links before production. Our classification tools distinguish transient faults from persistent signal integrity failures, guiding targeted repairs and component replacement, and we validate repaired systems before returning them to production.  **Memory Reliability (HBM and LPDDR):** Memory-related faults became a major source of interruptions as we scaled GB300 with higher memory capacities. Working closely with NVIDIA, we used fleet telemetry and detailed error analysis to identify recurring failure signatures, which are mitigated via refined HBM firmware settings, memory scrubbing, an improved error correction algorithm for LPDDR, and pattern-based predictive detection to retire degrading devices before they impact job throughput. We are deploying these changes across the fleet and measuring their impact on interruption rates and hardware replacement demand. **NVLink Switch and Fabric Health:** A single switch fault can compromise an entire NVL72 domain. We monitor fabric health at millisecond resolution, utilizing automated recovery paths to isolate faulty components and rebalance traffic across the remaining infrastructure without a job interruption. **Backend Network Events:** At the BAG super-spine layer, link events are managed transparently via hardware-based fault detection. Our planar topologies ensure that individual fiber or port failures are rerouted without interrupting the synchronized GPU traffic bursts. Sustained improvement is driven by our operational infrastructure: A colocated triage team at the New Albany site, a comprehensive stability sprint in March that synchronized fixes across the stack, and a rigorous attribution-driven accountability framework. Every repair undergoes a mandatory quarantine to validate system integrity before returning to the compute pool. **Our reliability north star remains two interruptions per 8K GPUs per day, which is ~100k GPU hours MTBI. Driving a 99% reduction from our peak interruptions rate demonstrates that, at Prometheus’ scale, stability is an engineering challenge with solvable engineering outcomes.** ## Maturing Hardware at Fleet Scale Deploying hundreds of thousands of first-generation accelerators reveals quality issues that are hidden at lower volumes. The GB200 deployment has transformed hardware maturation into a primary engineering discipline for our teams and partners. Areas of engineering include: - **Rapid Firmware Bundle Deployment:** We built reliable and fast firmware bundle automation, capable of deploying firmware and driver fixes to more than 100k GPUs within a few hours. We’ve brought down the P90 of single-pod upgrade times from more than seven hours in early hardware introduction to two hours. This has let us push rapid fixes for the top interruption categories to our fleet. - **Fleet Health:** We correlate telemetry across multi-rack systems and classify every job interruption to guide software fixes, hardware repairs, and replacement decisions. We track MTBI alongside available capacity plus effective training throughput, and have a robust feedback cycle for hardware and software reliability improvements to translate into more productive compute.  - **Cross-Partner Qualification:** implementing parallel validation with multiple manufacturing partners to ensure consistent quality through continuous feedback from our production fleet. - **Automated Signal Screening:** validating over 600,000 NVLink paths using diagnostic stress tests before any production traffic is routed. Even minor defect rates at this scale require proactive identification. - **Power Delivery Stress Testing:** investigating how PSU modules and voltage regulators interact under the 1,400W load of individual GPUs to surface failure modes that only appear at high volumes. - **Precision Liquid Cooling:** enforcing strict quality gates for cold plate and manifold assembly to maintain optimal junction temperatures and prevent leaks across the fleet. - **Fleet Maintenance:** proactive preparation, rapid disposition, prioritization for fix-repair, and prepositioning spares.  - **Health Checks:** advanced ingestion/burn-in and pre-job health checks used to identify healthy nodes for inclusion in large training jobs. Each new platform architecture — from Blackwell to Vera Rubin — incorporates these real-world operational insights, tightening our screening and design specifications at every step. ## Optimizing Power and Managing Transients [Maximizing throughput](https://arxiv.org/pdf/2605.24461) for a given power envelope, minimizing stranded power, and synchronous distributed training introduces a power delivery hurdle. When tens of thousands of GPUs execute compute-heavy phases and communication cycles in unison, they trigger large power transients — sudden load fluctuations of 30-40% within the cluster. Initiating or pausing a major training job can shift loads of more than tens of megawatts in just seconds. We address this with [Dynamo Dimmer](https://arxiv.org/pdf/2605.24461), our cluster-wide power management framework that enables intelligent power capping. Instead of hard-stopping workloads, Dimmer gradually adjusts GPU power limits as the cluster nears its electrical ceiling, ensuring training continues uninterrupted. This system is reinforced by rack-level battery backup (BBU) modules that handle immediate spikes, capacitor trays (CBUs) for peak smoothing, facility-level battery energy storage systems (BESS), and deep coordination with utility providers to manage these unique AI profiles.  ## Managing Cooling in a Heterogenous Fleet While our distributed approach has allowed us to build Prometheus quickly, it has also meant navigating site-specific power and cooling constraints across different locations.  Our physical infrastructure incorporates multiple cooling generations to deliver the liquid cooling required by these high-power racks. Prometheus runs on a mix of modular air-assisted liquid cooling (AALC), including our free-air data center buildings and our rapid-deployment structures. Our purpose-built AI-optimized building now uses facility liquid cooling (FLC) for maximum power density.  By integrating leased capacity from strategic partners in addition to our own buildings, we accelerate our deployment timelines and total deployable capacity. Meanwhile, our industrial-grade rapid-deployment structures have redefined the construction playbook, delivering production-ready compute with faster timelines. ## Operating Efficiently  In New Albany, we’ve long partnered with local utilities to ensure we are paying the full cost of energy for our site and not adding any unnecessary strain to the local grid. We’ve also partnered with energy developers to match our energy use with new clean and renewable energy.  Meta’s newest AI-optimized data centers use [closed-looped, liquid cooling](https://about.fb.com/news/2026/08/closed-loop-cooling-explained-the-plumbing-behind-metas-ai/) as it is the most efficient way to cool GPU servers, and supports efficient use of water. Direct-to-chip, closed-loop liquid cooling helps save space by allowing us to fit more GPUs on a server rack. We also work to extend our sustainability work by sharing designs for systems like [IcePack](https://engineering.fb.com/2026/10/12/open-source/ocp-summit-2026-powering-ai-with-open-innovations-in-scale-up-scale-out-and-scale-across-networks/), our liquid-cooled network rack, with the wider industry through [OCP](https://engineering.fb.com/2026/10/12/open-source/ocp-summit-2026-powering-ai-with-open-innovations-in-scale-up-scale-out-and-scale-across-networks/). ## Beyond Prometheus We’ll continue to work closely with NVIDIA to expand Prometheus’ capacity with a mix of NVIDIA GB300 NVL72s and NVIDIA Vera Rubin NVL72s. NVIDIA’s next-generation [Vera Rubin NVL72](https://www.nvidia.com/en-us/data-center/vera-rubin-nvl72/?ref=engineeringatmeta) platform will be a generational leap in per-GPU performance and will integrate directly into Prometheus’s existing network fabric. As the cluster matures, it will deliver even more compute per watt by augmenting its capacity (primarily interconnected GB300s) with Vera Rubin via our upcoming San Miguel platform. ![](https://engineering.fb.com/wp-content/uploads/2026/10/Vera-Rubin-Prometheus.jpg) _The San Miguel Vera Rubin NVL72 will feature:_ - **72 NVIDIA Vera Rubin GPUs** with 288 GB of HBM4 and 19.2 TB/s peak bandwidth per GPU. - **2.52 peak EFLOPS of FP4 training compute** **1.26 peak EFLOPS FP8 training** per rack. - **216 TB/s peak NVLink bandwidth** (3 TB/s bidirectional per GPU). - **~1,500 TB/s peak aggregate memory bandwidth.** - **Fully liquid-cooled** compute trays at over 170 kW per rack. - **10-20% improvement in performance per watt** over the GB300 NVL72. The technological accomplishments within Prometheus have helped establish the blueprint that will inform how we approach the complexity of supercluster design going forward. Hyperion, our upcoming supercluster in Richland Parish, Louisiana, is being designed from the ground up with insights from Prometheus in mind. It will feature closed-loop liquid cooling, [full grid connectivity with our utility partner Entergy](https://www.entergy.com/news/entergy-louisiana-announces-a-new-agreement-with-meta-that-will-deliver-an-additional-2b-in-customer-savings), for which Meta pays the full cost, and next-generation network designs.  _Learn more about our [community efforts in New Albany.](https://datacenters.atmeta.com/2026/10/new-albany-where-ohios-community-spirit-meets-the-frontier-of-ai/)_ The post [Inside Prometheus: Meta’s Journey to a Gigawatt-Scale Supercluster](https://engineering.fb.com/2026/10/19/data-center-engineering/inside-prometheus-meta-gigawatt-scale-supercluster/) appeared first on [Engineering at Meta](https://engineering.fb.com/).