On 5 November, Google open-sourced the inference engine that already answers more than six million queries every second inside its own products. XLA Serving Engine, a C++ runtime for the OpenXLA ecosystem, moves machine-learning models out of Python and into production serving, and Google says it powers YouTube Shorts, Google Ads, Search and parts of Gemini Omni. The release turns a year of internal production use into public infrastructure anyone can now run.
What XLA Serving Engine does
Most machine-learning engineers train models in Python, using frameworks such as JAX and PyTorch, but many production environments refuse Python because of its deployment and operational complexity. XLA Serving Engine bridges that gap with what Google calls a “Train in Python, Serve in C++” workflow, exporting a model into a compiled format that a native C++ runtime can execute directly.
The export format is an XLA SavedModel, which stores the model’s computation as StableHLO, a portable intermediate representation, while Orbax checkpointing stores the weights. Once exported, the XLA Serving Runtime loads the artifact, wraps the PJRT execution interface directly, and applies the XLA compiler’s optimisations, cutting out the Python interpreter and its global interpreter lock entirely.
The numbers Google puts behind it
For short requests, Google reports that the engine delivers two to five times the throughput of the best Python server its team could build, and eight to thirty-six times that of a straightforward one. It uses thirty to seventy percent less CPU time per prediction and roughly a quarter of the memory, though these are Google’s measurements and remain unverified.
Scale is the stronger claim. Google began deploying core components of the engine in its internal production infrastructure last year, and it now serves over six million queries per second. Google names YouTube Shorts, Google Ads, Search and parts of Gemini Omni among the products it serves — proven infrastructure, not a laboratory prototype.
Who is already using it
Three private-preview partners put their names to the launch. Arm says the engine proves production JAX and XLA serving on Arm Neoverse platforms, including Google Cloud Axion, for vision, embedding and recommendation workloads. Intel points to native JAX inference on Xeon processors accelerated by AMX, from cloud to edge to on-premises deployments.
The most striking testimony comes from Moloco, whose demand-side advertising platform processes over 650 billion daily inferences and peaks at 10.5 million queries per second. Its engineers call the engine the high-performance, C++-based JAX serving solution they had been waiting for, one that complements their Cloud TPU Trillium investment while staying framework agnostic.
What it cannot do — and what comes next
Google is candid about the engine’s limits. It is a lightweight, one-shot inference engine, and for large autoregressive language models with long context lengths, the company recommends established projects such as vLLM or SGLang, which handle LLM-specific memory management. The engine is aimed at vision, embedding and recommendation models rather than chatbots.
Google plans TorchTPU integration so PyTorch users can export models to the same format and serve them on TPUs in C++, plus ahead-of-time compilation to cut cold-start times and a C++ data preprocessor for teams wanting preprocessing inside the model graph. Docker images tested on CPU, GPU and TPU, with simple gRPC APIs, are available now.
Our Read
The way to read this release is as Google publishing its own plumbing. Six million queries a second across Shorts, Ads, Search and Gemini Omni is a scale claim no rival serving project can match, and the Apache 2.0 licence means vendors can now build on the same runtime Google runs internally. That is a genuine gift, not a gesture.
But the gift has a shape, and the shape is Google’s. The engine is built for the workloads Google runs at scale — short, high-volume predictions — and its headline speedups are Google’s own measurements against Python servers it built itself. Teams serving large language models will find little here, and the test is whether anyone outside Google adopts it.
Why It Matters
Inference, not training, is where most AI spending lands, and whoever controls the serving layer controls the cost of every AI feature. By open-sourcing the runtime behind its own search and advertising pipelines, Google is trying to set the default for that layer, as it did with Kubernetes. If the engine spreads, Google’s hardware partners and Cloud TPU customers benefit.
There is also a defensive logic. Google is spending heavily on AI infrastructure, from a $17 billion Wyoming data center to a $15 billion investment in Malaysia, and cheaper inference stretches every dollar of that build-out further. Open-sourcing the engine invites the wider industry to optimise it for free, while keeping Google’s own serving costs falling as adoption grows.
Background
XLA Serving Engine sits inside OpenXLA, the open-source compiler ecosystem Google has built around its XLA compiler, and it follows Google’s earlier open-sourcing of TorchTPU for PyTorch on TPUs. The release arrives as rival serving stacks such as vLLM and SGLang dominate language-model deployment, leaving Google to claim the non-LLM serving ground instead.
Source: Google’s announcement and the XSE repository
0 Comments