Google joins Amazon in luring AI developers from Nvidia

Google Alphabet Inc. ↘ 1.11% said on Tuesday, October 20, that TorchTPU, its native PyTorch backend for its Tensor Processing Units, is now open source. Amazon Amazon.com, Inc. ↗ 0.07% had unveiled a similar tool, TorchNeuron, a day earlier. Both companies are targeting the software advantage that keeps most AI apps tied to Nvidia ↘ 0.29% graphics chips.techcrunch.com

PyTorch is the open-source framework most AI developers use to build and run models. Many of its features were built around Nvidia hardware. Each tool lets developers move existing code to a rival chip with a configuration change instead of a rewrite.techcrunch.com

Google's TorchTPU goes public

Google built TorchTPU with Meta's ↘ 1.59% PyTorch team and made it public at PyTorch Conference North America in San Jose. The code is on GitHub under the Apache 2.0 license. Google said it worked with Meta, Hugging Face, vLLM maker Inferact and SGLang maker RadixArk so that common tools behave on TPUs as they do in standard PyTorch. It also launched a TPU agent suite to help developers migrate apps and fix errors. The SGLang team said its TorchTPU serving engine is already running in production at Meta. Google said its new vLLM and SGLang engines are serving preview customers at scale. Google first described TorchTPU in April and had been testing it with select customers.lmsys.org+4

Amazon's TorchNeuron targets Trainium

Amazon announced TorchNeuron on Monday, October 19, and released it as an open-source public beta on Tuesday. The tool runs on Trainium2 and Trainium3 instances. Developers enable it by changing the PyTorch device setting to "neuron". Amazon is also sending changes to the main PyTorch project that remove code tying the framework to specific chips. Before TorchNeuron, developers had to rewrite PyTorch code to call interfaces specific to Trainium, said Colin Brace, an executive at Amazon's Annapurna Labs chip unit.aws.amazon.com+2

Gaps remain

The first release has limits. TorchNeuron does not yet support some features, such as dynamic input shapes, and about 22,000 PyTorch tests are skipped. Google has also said that developers may still need to adjust model design or write custom code to get the best performance from TPUs. Nvidia's lead rests partly on CUDA, its software platform for running models on its GPUs. Its libraries and debugging tools form an ecosystem that rival chipmakers also have to compete with. Being able to switch chips easily does not mean developers will leave Nvidia hardware in large numbers. Still, Amazon and Google are clearly working to end Nvidia's advantage.thenextweb.com+2