Google has open-sourced the engine behind some of its busiest AI services, releasing XLA Serving Engine, or XSE, as a free C++ runtime that serves machine-learning models without Python in the loop.

The release was announced on 5 November under the Apache 2.0 licence. Google describes XSE as a general-purpose model export and inference runtime for the OpenXLA ecosystem, built around a simple idea: train in Python, serve in C++. Developers train a model in JAX or PyTorch, export it into an XLA SavedModel containing StableHLO, an intermediate representation of the model's function, and the C++ runtime loads and executes it, with no Python interpreter involved at serving time.

Already running at Google scale

This is not untested code. Google began deploying core components of XSE inside its own production infrastructure in 2025, and says the engine now serves over six million queries per second, powering some of its highest-volume inference pipelines, including YouTube Shorts, Google Ads, Search and parts of Gemini Omni. The Register reports that in Google's own tests on short requests, XSE delivered two to five times the throughput of the best Python server the company could build, using a quarter of the memory.

Built for small, fast models

XSE is aimed at a specific workload: one-shot inference, where a model answers a single short request, such as ranking a recommendation or classifying an image, under a tight latency budget. It runs on CPUs, NVIDIA GPUs and Google Cloud TPUs, and Google says it uses 30 to 70 per cent less CPU per prediction than Python implementations.

Part of the OpenXLA ecosystem

XSE sits inside the OpenXLA Project, the industry-backed effort to compile machine-learning computations into native code for different processors. It plugs into existing OpenXLA tooling, including the XLA compiler and StableHLO, so teams already using that stack can adopt it without rebuilding their models. Google says the engine is lightweight and suited to recommendation systems, vision models and other compact prediction calls, where a request must be answered in milliseconds and thousands of them arrive every second.

For developers, the practical change is that the boundary between research and production gets thinner. A model prototyped in a notebook can be exported and dropped into a C++ service without being rewritten, and the same compiled artefact can run on a laptop CPU, a data-centre GPU or a Cloud TPU. That portability is the point of the OpenXLA approach, and XSE is the piece that carries a model across the finish line into production.

It is not a rival to the big language-model serving engines. Google positions XSE for smaller, sharply task-oriented models, while large language-model inference is better served by systems such as vLLM or SGLang. The quick-start instructions are published with the code at github.com/openxla/xla-serving, and the figures quoted here are Google's own measurements, not independently verified results.