Google has open-sourced TorchTPU, a native PyTorch backend that lets existing PyTorch code run on its Tensor Processing Units with minimal changes. Announced at PyTorch Conference 2026 on Tuesday, October 20, the release makes TPUs a native PyTorch device alongside GPUs and CPUs — and lands one day after Amazon released its own PyTorch backend for its Trainium chips.
TorchTPU was co-developed with the PyTorch team at Meta and is published on GitHub under the Apache 2.0 licence. Google says it replaces PyTorch/XLA, the older integration, and recommends it for all new TPU workloads.
TorchTPU: A Native PyTorch Device, Not A Translation Layer
The technical shift is in how code executes. PyTorch/XLA recorded lazy-tensor graphs and often required developers to mark compilable regions, and it supported only pure SPMD distributed code. TorchTPU plugs into PyTorch through the standard PrivateUse1 interface, so tensors are ordinary PyTorch tensors addressed as torch.device("tpu"), with no subclasses or wrappers. It preserves tensor views and in-place mutations, runs one process per device, and also handles divergent MPMD execution.
Developers get three eager execution modes — a default Strict mode with standard asynchronous semantics, a synchronous Debug mode with tensor bounds checking, and a DeferAndFuse mode that fuses runs of operators to cut memory traffic. For production throughput, torch.compile captures the graph and lowers it through StableHLO to OpenXLA, with compiled executables cached across processes and restarts. Custom Pallas kernels are callable as ordinary PyTorch functions, and the distributed layer supports DDP, FSDP2 and DTensor over a topology-aware mesh.
The first release is version 0.2.0 on PyPI, for Python 3.11 and later. Google says it has been tested on TPU generations from v5e up to Ironwood, and its serving stack accepts FP8, MXFP4 and NVFP4 checkpoints. The design mirrors the approach Amazon took with TorchNeuron for Trainium, which reached developers a day earlier.
TorchTPU Integrations: vLLM, SGLang, TorchTitan And LiteRT
Google shipped the ecosystem pieces with the backend rather than after it. Native PyTorch backends for the vLLM serving engine, built with Inferact, and for SGLang, built with RadixArk, keep the upstream schedulers and API servers while adding continuous batching, quantisation and multi-chip parallelism on TPUs. Hugging Face Transformers v5 deploys models to a TPU by passing device_map="tpu", and Google runs nightly testing of the Transformers suite on TPU hardware.
For training, TorchTitan carries a production-grade TorchTPU integration with recipes for models including Llama 3 70B and 405B, Qwen 3 8B and 32B, the Qwen 3 235B mixture-of-experts model and FLUX.1. Google reports up to 52% model flops utilisation training Qwen 3 8B, with near-linear scaling to 8,192 Ironwood chips — figures the company reported itself. The llm-d 1.0 release adds Kubernetes-native routing across TPU and GPU fleets, VeRL gains a TPU backend for reinforcement-learning post-training in preview, and Ray can schedule pipeline stages across CPUs, GPUs and TPUs in one workflow.
Two additions widen the surface further. LiteRT, formerly TensorFlow Lite, joins the PyTorch ecosystem, converting a PyTorch model for phones, laptops, browsers and embedded devices with a single call. And the TPU agent suite, in early preview on GitHub, plugs into coding harnesses to automate onboarding, diagnostics and inference performance tuning for PyTorch models moving to TPUs.
TorchTPU And The Fight Over Nvidia’s Software Moat
PyTorch is the toolkit most AI applications are built with — TechCrunch puts its share at roughly 85% of AI apps — and because Nvidia’s GPUs were the default hardware when the framework matured, much of its tooling grew up around Nvidia’s stack. That software gravity, more than raw chip performance, is what has kept developers on Nvidia hardware, a dynamic explored in our explainer on why Nvidia powers the AI boom.
Google and Amazon are now attacking that layer directly, a day apart, with the same strategy: meet developers inside PyTorch and make the hardware a configuration change. Nvidia is meanwhile still winning new CPU business at agent hosts, as Modal’s adoption of the Vera CPU showed this week — evidence that the contest is over default choices at every layer of the stack, not a single chip generation.
Why It Matters
TorchTPU’s central attribute is portability with production proof points attached. For AI startups and research teams, a native backend removes the rewrite that previously made TPUs a specialist choice, which puts Google Cloud’s Ironwood capacity into direct price competition with GPU instances for both training and inference. For Meta, which co-developed the backend, the payoff is hardware diversity for its own LLM, ads and recommendation workloads without leaving PyTorch. For Nvidia, the risk is cumulative rather than immediate: every framework layer that treats hardware as interchangeable weakens the CUDA-centred lock-in that supports its margins, and with Amazon’s TorchNeuron doing the same for Trainium, two of the largest cloud operators are now funding that erosion openly.
Background
Google introduced TPUs in 2016 as its in-house answer to GPU-based training, but for years PyTorch developers reached them only through the PyTorch/XLA bridge, while Google’s own teams favoured JAX — a split that kept TPUs peripheral for the largest developer community in AI. PyTorch itself, created by Meta in 2016 and handed to the Linux Foundation in 2022, became the industry’s common language in the meantime. TorchTPU closes that gap: a decade after the TPU’s debut, Google is meeting PyTorch developers on their own ground instead of asking them to move.
Source: Google Cloud Blog — TorchTPU is now open source; MarkTechPost; TechCrunch
0 Comments