Abu Dhabi’s Institute of Foundation Models has released K2-Horizon, a fleet of six fully open language models ranging from a 0.9B dense model to a 375B mixture-of-experts flagship. The models target reasoning and agentic work, from math and science to software engineering and deep research.
What sets the release apart is how much comes with the weights. IFM published intermediate checkpoints from every training stage, the pre-training data, training logs, and all training, evaluation and infrastructure code. It also documented the pipelines that built the data, a part of development rarely disclosed.
A New Attention Trick
The flagship’s headline is Mixture-of-Value Attention, or MoVA, which applies mixture-of-experts to attention value projections, plus a softplus attention gate. The 36B-A4B model uses both and nearly matches a 32B dense model trained on the same data, using one-seventh the active parameters.
IFM also introduced Uno, lossless diffusion adapters that speed up decoding by up to 2.2 times without a separate draft model. The adapters are distilled from each model, so the speedup comes without the extra serving machinery that speculative decoding normally needs.
The Benchmark Claims
Four of the models share one 22T-token pre-training corpus and recipe, letting IFM compare scale and sparsity cleanly. The 375B flagship ranks first or second on most agentic and coding benchmarks against four recent open models, and it leads on Toolathlon-Verified with a score of 65.3.
The same model beats OpenAI’s GPT-5.6 Terra on Toolathlon-Verified and comes close to GPT-5.6 Luna, and it exceeds both GPT-5.6 variants on a banking benchmark. Its main gap is expert-level scientific reasoning, where it trails on the HLE and CritPt benchmarks the paper names.
Smaller models punch above their weight too. The 7B beats similarly sized open models on most benchmarks, and both the 7B and 3.7B lead SWE-bench Verified by wide margins. The 36B-A4B stays within a few points of the dense 32B on every benchmark the paper reports.
Our Read
The openness package is the real story here, not the parameter count. Weights alone are now routine; a full development history with data, checkpoints, logs and every pipeline is not. NVIDIA’s Nemotron diffusion family opened its weights too, but K2-Horizon goes further on reproducibility.
The benchmark claims deserve the usual caution: they are the authors’ own evaluations, with no independent verification yet. Still, the Toolathlon and banking results against GPT-5.6 variants are striking for a smaller open model, and the gap on expert science benchmarks is stated plainly in the paper.
For researchers, the value is the controlled experiment: four models on the same 22T-token corpus, with identical data batches for two pairs, make it possible to study scale and sparsity directly. For builders, the small models and the Uno speedup matter more than the flagship.
Why It Matters
Open-weight releases keep getting more transparent, and K2-Horizon pushes that further than most. A 10T-token synthetic-data pipeline, documented end to end, gives other labs a recipe rather than just a model. If the benchmarks hold up, the 375B flagship is a genuine rival to the best open models.
Background
IFM’s LLM360 family has made openness its calling card since K2-V2, and K2-Horizon extends that approach across six models. Tencent opened the 770B Hy4 weights this week too, in a month crowded with large open releases. The models carry Apache 2.0 licences, with one dataset under CC BY 4.0.
Source: the paper and IFM’s release page


0 Comments