So far, it just sounds you hear – it can’t read your mind or anything… yet
AI system- AI algorithms can help scientists process brain waves and convert them directly into speech, according to new research.
“Our voices help connect us to our friends, family and the world around us, which is why losing the power of one’s voice due to injury or disease is so devastating,” said Nima Mesgarani, senior author of the paper published in Scientific Reports and a researcher at Columbia University. “With today’s study, we have a potential way to restore that power. We’ve shown that, with the right technology, these people’s thoughts could be decoded and understood by any listener.”
Neurons in our brain’s auditory cortex are excited whenever we listen to people speak – or even imagine people speaking. How exactly the brain makes sense of the jumble of sound waves or constructs a facsimile of the process when we imagine people speaking is still unknown. However, neuroscientists have shown that brain patterns emitted during a task can be pieced together to reconstruct the words being spoken. It has propelled the idea of building neuroprosthetics, devices that act as brain-computer interfaces.
The group of researchers tried to advance the technique known as auditory stimulus reconstruction using a neural network. First, an autoencoder was trained to convert audio signals to spectrograms, detailing different frequencies in the sounds, from 80 hours of speech recordings.
Next, the researchers placed electrodes directly onto the brains of five participants undergoing brain surgery for epilepsy to record electrical activity. All of them had normal hearing. They all listened to a recital of short stories for 30 minutes. The stories were randomly paused, and they were asked to recite the last sentence to train a vocoder. The vocoder was taught to map specific brain patterns to audible speech.
The participants listened to a string of 40 digits – zero to nine – being recited. The recorded brain signals were run through the vocoder to produce audio signals, and these samples were then fed back to the autoencoder for analysis so that the system could repeat the digits being reconstructed.