Brain implant first to decode speech and gestures at once

Researchers at the University of California, San Francisco have demonstrated for the first time that a single brain-computer interface can translate a paralyzed person's intended speech and upper-body gestures simultaneously, allowing them to control a full-body virtual avatar that talks and moves in tandem. The findings, published Monday in Nature Neuroscience, mark a step toward restoring the richness of natural human communication for people with severe paralysis.medicalxpress+1

Decoding Two Channels at Once

Led by neurosurgeon Edward Chang, the team implanted thin electrode arrays on the brain's sensorimotor cortex of three patients with varying degrees of vocal tract and bodily paralysis caused by conditions such as ALS and brainstem stroke. The 253-electrode electrocorticography arrays captured neural signals as participants attempted to speak phrases and perform common gestures — a hand wave, a thumbs-up — both in isolation and together.nature+1

Previous BCIs have enabled either speech or movement independently, but combining the two degraded performance. Chang's team discovered that the brain signals produced during simultaneous speech and gesture are not simply a sum of the two individual signals. "Conversation is about much more than the words being spoken. It's a multilayered, dynamic process involving the whole motor cortex," Chang said. Machine-learning decoders trained specifically on data from concurrent speech and gesture outperformed those trained on isolated tasks, and successfully translated the thoughts of two participants into avatar commands.medicalxpress+1

Beyond Eye-Tracking

The current standard of care for people with severe paralysis — eye-tracking systems that enable text-to-speech — is slow, offers limited expression, and can be physically exhausting. The new BCI aims to provide a more natural alternative by restoring nonverbal cues alongside words. "These promising results give me hope that in the future, patients with severe paralysis will be able to recapture the holistic nature of human communication," said Debara Tucci, director of NIH's National Institute on Deafness and Other Communication Disorders.medicalxpress+1

What Comes Next

The system used in this study relied on wired connections between implanted sensors and external processing units. Chang said his team will soon test a fully implantable, wireless version with better prospects for long-term, everyday use. The research builds on the lab's earlier work, which in 2023 demonstrated that a brain implant could drive a digital avatar's speech and facial expressions for a single paralyzed patient.techxplore+2