Scientists have developed a device that translates paralysed people’s brain signals
into words at faster speeds than before, with plans to improve this even further, and enable people – who can no longer talk after strokes, brain disease or paralysis – to communicate their thoughts in real time.
Another team has enabled a paralysed woman to speak through an avatar.
Both findings have been reported in two papers in the journal Nature.
US woman Pat Bennett (68) who has motor-neurone disease (MND), tested the technology and said it could help her stay connected to the world, as the implants in her brain decode the words she wants to say, reports BBC News.
She was diagnosed in 2012 with a disease that attacks areas of the brain that control movement, causing eventual paralysis. Her speech was the first thing affected.
For the Stanford University research, a surgeon implanted four sensors the size of
pills into her brain, in areas key to producing speech.
When she tells her lips, tongue and jaw to make sounds to form words, an algorithm decodes information coming out of her brain.
“This system is trained to know what words should come before other ones, and which phonemes make what words,” said Dr Frank Willett, co-study author.
“If some were wrongly interpreted, it can still take a good guess.”
After four months of training the software to interpret Bennett’s speech, her brain activity was being translated into words on a screen at 62 words per minute, about three times the speed of previous technology.
Normal conversations are about 160 words per minute, the researchers say, but they are yet to produce a device people can use in everyday life.
One in 10 words was wrong in a vocabulary of 50 words and there were errors in a quarter of Bennett’s 125 000-word vocabulary.
“But it’s a big advance toward restoring rapid communication to people with paralysis who can’t speak,” Willett said.
In another study, from the University of California San Francisco (UCSF), a second woman, who has severe paralysis from a stroke, was able to speak through a digital avatar, complete with her own facial expressions.
Scientists decoded signals from more than 250 paper-thin electrodes implanted on the surface of her brain and used an algorithm to recreate her voice, based on a recording of her speaking at her wedding.
The system reached nearly 80 words per minute and made fewer mistakes than previous methods, with a larger vocabulary.
“It’s what gives a user the potential, in time, to communicate almost as fast as we do and to have much more naturalistic and normal conversations,” said researcher Sean Metzger, who helped develop the technology.
Study author Dr Edward Chang was “thrilled” to see the success of the brain interface in real time.
Improvements in artificial intelligence (AI) had been “really key”, he said, and there were now plans to look at turning the technology into a medical device.
Current technology allows some people with MND to bank their voice before it’s lost, and then use their eyes to select the words or letters they want to say on a screen, but it can be time-consuming.
Study 1 details
A high-performance speech neuroprosthesis
Francis Willett, Erin Kunz, Chaofei Fan, Jaimie Henderson, et al.
Published in Nature on 23 August 2023
Abstract
Speech brain–computer interfaces (BCIs) have the potential to restore rapid communication to people with paralysis by decoding neural activity evoked by attempted speech into text or sound. Early demonstrations, although promising, have not yet achieved accuracies sufficiently high for communication of unconstrained sentences from a large vocabulary. Here we demonstrate a speech-to-text BCI that records spiking activity from intracortical microelectrode arrays. Enabled by these high-resolution recordings, our study participant – who can no longer speak intelligibly owing to amyotrophic lateral sclerosis – achieved a 9.1% word error rate on a 50-word vocabulary (2.7 times fewer errors than the previous state-of-the-art speech BCI2) and a 23.8% word error rate on a 125 000-word vocabulary (the first successful demonstration, to our knowledge, of large-vocabulary decoding). Our participant’s attempted speech was decoded at 62 words per minute, which is 3.4 times as fast as the previous record and begins to approach the speed of natural conversation (160 words per minute). Finally, we highlight two aspects of the neural code for speech that are encouraging for speech BCIs: spatially intermixed tuning to speech articulators that makes accurate decoding possible from only a small region of cortex, and a detailed articulatory representation of phonemes that persists years after paralysis. These results show a feasible path forward for restoring rapid communication to people with paralysis who can no longer speak.
Study 2 details
A high-performance neuroprosthesis for speech decoding and avatar control
Sean Metzger, Kaylo Littlejohn, Alexander Silva, Edward Chan, et al.
Published in Nature on 23 August 2023
Abstract
Speech neuroprostheses have the potential to restore communication to people living with paralysis, but naturalistic speed and expressivity are elusive1. Here we use high-density surface recordings of the speech cortex in a clinical-trial participant with severe limb and vocal paralysis to achieve high-performance real-time decoding across three complementary speech-related output modalities: text, speech audio and facial-avatar animation. We trained and evaluated deep-learning models using neural data collected as the participant attempted to silently speak sentences. For text, we demonstrate accurate and rapid large-vocabulary decoding with a median rate of 78 words per minute and median word error rate of 25%. For speech audio, we demonstrate intelligible and rapid speech synthesis and personalisation to the participant’s pre-injury voice. For facial-avatar animation, we demonstrate the control of virtual orofacial movements for speech and non-speech communicative gestures. The decoders reached high performance with less than two weeks of training. Our findings introduce a multimodal speech-neuroprosthetic approach that has substantial promise to restore full, embodied communication to people living with severe paralysis.
Nature article – A high-performance speech neuroprosthesis (Open access)
Nature article – A high-performance neuroprosthesis for speech decoding and avatar control
BBC Health article – Brain advance gives voice hope to paralysed (Open access)
See more from MedicalBrief archives:
New drug helps to preserve brain cells for a while after stroke
UK scientist with MND converts himself into a cyborg
First human trial: Tiny device translates thought into action
New research offers hope for motor neurone disease sufferers