Google open-sources TPU Raiden library to rival Nvidia’s inference stack

Google Alphabet Inc. ↘ 0.88% has open-sourced TPU Raiden, an inference optimization library that handles KV-cache data movement between chips during large language model serving. The repository is live on GitHub under the Apache-2.0 license, marking one of the clearest signals yet that Google is willing to externalize infrastructure that was long treated as proprietary to its TPU ecosystem.officechai

The release was first flagged by SemiAnalysis, which noted that Raiden occupies the same layer of the inference stack as NVIDIA's ↗ 2.27% NIXL library, providing KV-cache transfer between prefill and decode instances along with primitives for cache offloading.x+1

What Raiden Does

Modern large-scale inference splits work into two phases: prefill, which processes the input prompt, and decode, which generates tokens one at a time. Running both on the same chip is inefficient, so production systems disaggregate them across separate hardware pools. When prefill completes, it produces a KV-cache — the model's running memory of its input — that must be handed off quickly to decode chips.officechai

Raiden manages that handoff. According to its repository documentation, the library includes modules for direct chip-to-chip transfer within a single machine, cross-VM transfer over the network, and offloading cache blocks from TPU memory to host RAM. A shared-memory mode lets the cache persist in DRAM even if the model server restarts, useful during routine production updates.officechai

Google's documentation notes the project is still under active development and not yet intended for general production use.

Strategic Context

Google has never sold TPUs as standalone hardware, but the customer base for its custom silicon has expanded. Anthropic has secured access to a million TPUs, and OpenAI has begun using Google TPUs for parts of its operations.officechai

Each of those customers needs software tooling comparable to what has grown around NVIDIA GPUs. Frameworks like vLLM and SGLang are deeply integrated with NIXL; if Raiden achieves similar integration, companies already running NVIDIA-based disaggregated serving could point workloads at TPUs with less engineering overhead.officechai

The timing also coincides with NVIDIA's own open-source push. On August 4, NVIDIA announced it would open its cuFile APIs and storage stack through a new cross-vendor GitHub organization, with Google, Intel ↗ 1.84%, and Meta ↗ 0.37% listed among the initial maintainers.igorslab

An Ecosystem Race

The parallel moves suggest an industry-wide shift toward externalizing inference infrastructure that vendors once kept closed. For Google, the calculus is straightforward: open tooling lowers the barrier for outside teams to adopt TPUs for production workloads rather than defaulting to the NVIDIA stack simply because it is better documented in the open.officechai

NVIDIA has previously argued its GPUs remain a generation ahead and are the only platform running every AI model everywhere computing happens. Whether that position holds as Google continues to open pieces of its stack remains the central question for enterprises choosing where to run inference at scale.officechai