by the skillnumber of machines for read our minds has steadily progressed over the past few years. Now researchers have used AI video generation technology to give us a window into the mind’s eye.
The main driver behind attempts to interpret brain signals is the hope that one day we may be able to provide new windows of communication for people in comas or with various forms of paralysis. But there is also hope that the technology can create more intuitive interfaces between humans and machines that could also have applications for healthy people.
Until now, most research has focused on efforts to recreate the inner monologue.s of patients, using AI systems to choose which words they are thinking of. The most promising results have also come from invasive brain implants, which are unlikely to be a practical approach for most people.
Now, though, researchers at the National University of Singapore and the Chinese University of Hong Kong have shown they can combine non-invasive brain scans and AI imaging technology to create short snippets of video that are eerily similar to the clips subjects were watching. . when your brain data was collected.
The work is an extension of research by the same authors published at the end of last year, where they showed that they could generate still images that roughly corresponded to the images shown to participants. This was achieved by first training a model on large amounts of data collected using fMRI brain scanners. This model was then combined with AI Stable Diffusion open source imaging to create the images.
In a new role published in prepress server arXiv, the authors take a similar approach, but adapt it so that the system can interpret brain data streams and convert them into videos instead of photos. First, they trained a model on large amounts of fMRI so that it could learn the general characteristics of these brain scans. This was then scaled up so that it could process a succession of fMRI scans rather than individual ones, and then retrained on combinations of fMRI scans, the video snippets that elicited that brain activity, and text descriptions.
Separately, the researchers adapted the pre-trained Stable Diffusion model to produce video instead of still images. It was then retrained on the same videos and text descriptions that the first model had been trained with. Finally, the two models were combined and fitted into fMRI scans and their associated videos.
The resulting system was able to take new fMRI scans that had not been seen before and generate videos that broadly resembled clips of human subjects had was watching at the time. While far from a perfect match, the AI output was generally very close to the original video, accurately recreating crowd scenes or horse herds and generally matching the color palette.
To evaluate their system, the researchers used a video classifier designed to assess how well the model had understood the semantics of the scene – for example, whether it perceived that the video was of fish swimming in an aquarium or a family walking along a path – even if the pictures were slightly different. His model scored 85 percent, which is a 45 percent improvement over the state of the art.
While AI-generated videos are still flawed, the authors say this line of research could have applications in both basic neuroscience and future brain-machine interfaces. However, they also acknowledge potential downsides to the technology. “Government regulations and efforts by the research communities are needed to ensure the privacy of biological data and prevent any malicious use of this technology,” they write.
This is likely a nod to concerns that the combination of AI brain-scanning technology could make it possible for people to intrusively record other people’s thoughts without their consent. Aanxieties were also expressed earlier this year, when researchers used a similar approach to essentially create an outline transcription of the voice inside people’s headsalthough experts have pointed out that this would be impractical if not impossible for the near future.
But whether you see it as a creepy invasion of your privacy or an exciting new way to interact with technology, it looks like machine mind readers are getting closer to reality.
Image credit: Claudia Dewald of Pixabay
