Olmo-core 3 released: open, scalable MoE training infrastructure
Olmo-core 3 is an open training stack for large mixture-of-experts (MoE) models, designed to scale MoE training into the trillion-parameter range and to support MXFP8 and distributed parallelism.
What changed
Replaces FSDP with a DDP-based MoE stack; expert pool scaled from 8 to 128 with four experts per token; capacity rose from 4.6B to 47B while throughput fell by less than 5%.
Why it matters
Developers and researchers can use the open stack to train larger MoEs (tested up to 1.2-trillion parameters) and gain higher throughput—e.g., a 47B MoE ran 52,000 tokens/sec per NVIDIA B300 GPU.
Who should care
DevelopersResearchersEnterprises
Worth trying
Yes — Open code on GitHub; supports MXFP8 and NVIDIA B300 benchmarks.