Mind Reading  

A newly developed neural network method now makes it easier and more accurate to decode fMRI scans. Using deep learning, researchers were able to reconstruct images a viewer was looking at through brain scan data analysis. The 'deep generative multiview model' also learned to correlate the data so that the accompanying standard fMRI noise could be accounted for in the generation of the reconstructed images.

One of the far reaching goals of neuroscience is to be able to read a person's thoughts. Such a technology is the inspiration, in part for Elon Musk's new company, Neuralink, along with other brain-machine interface ventures. So far, for data coming from functional magnetic resonance imaging (fMRI) scans, the task has proven to be very challenging.

fMRI scans are inherently noisy, and the activity in one voxel is well known to be influenced by activity in other voxels. This kind of correlation is computationally difficult and expensive to manage. Most work in this area has simply not dealt with it. This has significantly reduces the quality of the image reconstructions they produce.

Now, Changde Du at the Research Center for Brain-Inspired Intelligence in Beijing, China, and they his research team have developed a better ways to process data from fMRI scans to produce more accurate brain-image reconstructions. The team's research has been published online.

Their method uses deep learning techniques that handle nonlinear correlations between voxels more capably. The result is a much better way to reconstruct the way a brain perceives images.

Changde used several data sets of fMRI scans of the visual cortex of a human subject looking at a simple image—a single digit or a single letter. Each data set consists of the scans and the original image. They mapped the data to find a way to use the fMRI scans to reproduce the viewer's perceived image. In total, the team has access to over 1,800 fMRI scans and original images.

Related articles

• Google's Inceptionism Lets Us Look at an Artificial Intelligence Hallucination 
• Stanford Researchers and Google Create World's Largest Artificial Neural Network 
• The US Army Tries Mind Reading 

According to the researchers it was a straightforward deep learning task. They used 90 percent of the data to train the network to understand the correlation between the brain scan and the original image. Next, they tested the neural network on the remaining data by feeding it the scans and asking it to reconstruct what the viewed images were.

This approach had the advantage of having the network learn which voxels were used to reconstruct the image, avoiding the need to process the data from them all.

The neural network also learned how the data from the fMRI data was correlated. This was an important part of the research, because if the correlations are ignored, they end up being treated like noise and discarded. So the new approach—the so-called deep generative multiview model or DGMM—exploits these correlations and distinguishes them from real noise.  

The team compared their results from those of a number of other brain image reconstruction techniques. (See image at top). Generally, the reconstructed images are clear representations of the originals, and were for the most part superior to those derived by other methods.

“Extensive experimental comparisons demonstrate that our approach can reconstruct visual images from fMRI measurements more accurately,” write the study authors.

The research may have other implications other than regenerating what a view sees by interpreting a brain scan. "Although we focused on visual image reconstruction problem in this paper, our framework can also deal with brain encoding tasks," write the study authors.

The next steps for the research will include ways to analyze scenes more complex than simple numbered text and possibly moving images.

SOURCE  MIT Technology Review

By  33rd Square	Embed

Mind Reading  

Researchers using electrodes in patients’ temporal lobes have found that the signals carry information that let scientists predict what object patients are seeing, almost in real time.

Using electrodes implanted in the temporal lobes of awake patients, scientists have decoded brain signals at nearly the speed of perception. Further, analysis of patients’ neural responses to images of faces and houses enabled the scientists to subsequently predict which images the patients were viewing, and when, with better than 95 percent accuracy.

The research has been published in PLOS Computational Biology.

University of Washington computational neuroscientist Rajesh Rao and University of Washington Medicine (UW) neurosurgeon Jeff Ojemann, working their student Kai Miller and with colleagues in Southern California and New York, conducted the study.

Rao has also attained notoriety lately for his experiments in the field of brain-brain communication.

“We were trying to understand, first, how the human brain perceives objects in the temporal lobe, and second, how one could use a computer to extract and predict what someone is seeing in real time?” explained Rao, a UW professor of computer science and engineering.  Rao also directs the National Science Foundation’s Center for Sensorimotor Engineering, headquartered at UW.

In the image above, the numbers 1-4 denote electrode placement in temporal lobe, and neural responses of two signal types being measured.

Related articles

• Brain-Computer Interface Developments Progress 
• Brain Surgeons Find The Neurological Basis of Human Speech 
• Mind-Reading Devices May Be On the Horizon According to Stanford Scientists 

“Clinically, you could think of our result as a proof of concept toward building a communication mechanism for patients who are paralyzed or have had a stroke and are completely locked-in,” he said.

The study centered around seven epilepsy patients receiving care at Harborview Medical Center in Seattle. Each was experiencing epileptic seizures not relieved by medication, Ojemann said, so each had undergone surgery in which their brains’ temporal lobes were implanted – for about a week – with electrodes to try to locate the seizures’ focal points. 

“They were going to get the electrodes no matter what; we were just giving them additional tasks to do during their hospital stay while they are otherwise just waiting around,” Ojemann said.

In the experiment, the electrodes from multiple temporal-lobe locations were connected to powerful computational software that extracted two characteristic properties of the brain signal: “event-related potentials” and “broadband spectral changes.”

Rao characterized the former as likely arising from “hundreds of thousands of neurons being co-activated when an image is first presented,” and the latter as “continued processing after the initial wave of information.”

The subjects, watching a computer monitor, were shown a random sequence of pictures of human faces and houses, interspersed with blank gray screens. Their task was to watch for an image of an upside-down house. 

Neuroscientist Rajesh Rao and neurosurgeon Jeff Ojemann 

“We got different responses from different (electrode) locations; some were sensitive to faces and some were sensitive to houses,” Rao said.

"Our result as a proof of concept toward building a communication mechanism for patients who are paralyzed or have had a stroke and are completely locked-in."

The software used sampled and digitized the brain signals 1,000 times per second to extract their characteristics. The software also analyzed the data to determine which combination of electrode locations and signal types correlated best with what each subject actually saw.

In that way it yielded highly predictive information.

By training an algorithm on the subjects' responses to the known set of images, the researchers could examine the brain signals representing the final third of the images, whose labels were unknown to them, and predict with 96 percent accuracy whether and when (within 20 milliseconds) the subjects were seeing a house, a face or a gray screen.

This accuracy was attained only when event-related potentials and broadband changes were combined for prediction, which suggests they carry complementary information.

“Traditionally scientists have looked at single neurons,” Rao said. “Our study gives a more global picture, at the level of very large networks of neurons, of how a person who is awake and paying attention perceives a complex visual object.”

The scientists' technique, he said, is a steppingstone for brain mapping, in that it could be used to identify in real time which locations of the brain are sensitive to particular types of information.

“The computational tools that we developed can be applied to studies of motor function, studies of epilepsy, studies of memory. The math behind it, as applied to the biological, is fundamental to learning,” Ojemann said.

SOURCE  The University of Washington

By 33rd Square

Embed

Neuroscience  

The US Army is testing mind reading technology on soldiers at their MIND Lab facility. The intent is to speed up image analysis, for now.

At the US Army Research Laboratory facility called "The MIND Lab," a computer system was able to accurately determine what target image a soldier was thinking about.

MIND, which stands for "Mission Impact Through Neurotechnology Design," has been developed by Dr. Anthony Ries.

Ries is a cognitive neuroscientist studying visual perception and target recognition. He connected a test soldier to an electroencephalogram and then had him sit in front of a computer to look at a series of images that would flash on the screen.

The soldier was asked to choose one of five categories of images: five categories of images: boats, pandas, strawberries, butterflies and chandeliers, but keep the choice to himself. Then images flashed on the screen at a rate of about one per second. Each image fell into one of the five categories. The Soldier didn't have to say anything, or click anything. He had only to count, in his head, how many images he saw that fell into the category he had chosen.

When the experiment was over, the computer revealed that the subject had chosen to focus on the "boat" category. The computer accomplished that feat by analyzing brainwaves from the soldier. When a picture of a boat had been flashed on the screen, the Soldier's brain waves appeared different from when a picture of a strawberry, a butterfly, a chandelier or a panda appeared on the screen.

"Our ability to collect and store imagery data has been surpassed by our ability to analyze it."

Ries said that a big problem he sees for the intelligence community is the vast amount of image information coming in to be analyzed - imagery from unmanned aerial vehicles or satellites or surveillance aircraft, for instance. Everything must be looked at and evaluated.

Related articles

• Mini Human Brain Grown In Petri Dish 
• Synaptic Transistors Point To Computer Hardware That Learns 
• Researchers Develop Neuromorphic Chip With Cognitive Abilities 

"Our ability to collect and store imagery data has been surpassed by our ability to analyze it," Ries said.

Ries thinks that one day the intelligence community might use computers and brainwaves, or "neural signals," to more rapidly identify targets of interest in intelligence imagery, in much the same way the computer in his lab was able to identify pictures of "boats" as targets of interest for the soldier who had chosen to focus on the "boats" category.

"What we are doing is basically leveraging the neural responses of the visual system," he said. "Our brain is a much faster image processor than any computer is. And it's better at detecting subtle differences in an image."

The automated system could greatly reduce the amount of time it takes to process an image, and that means that a larger number of images - more of that gathered intelligence data - can be processed sooner.

Ries' particular research is finding out how other things an analyst might be doing as he does image analysis might affect the neural signal his brain generates.

SOURCE  U.S. Army

By 33rd Square

Embed

 Neuroscience  

It has now been shown, for the first time that is possible to reconstruct basic speech units like words, and complete sentences directly from brain waves and to generate the corresponding text. The research could be the first step in systems that can read your mind.

Speech is mainly produced in the human cerebral cortex. For many years, brain waves associated with speech processes can be directly recorded with electrodes located on the surface of the cortex.

It has now been shown, for the first time that is possible to reconstruct basic speech units like words, and complete sentences of continuous speech from these brain waves and to generate the corresponding text.

Researchers at KIT and the Wadsworth Center, have published their research in the scientific journal Frontiers in Neuroscience.

"It has long been speculated whether humans may communicate with machines via brain activity alone,” says Tanja Schultz, who conducted the present study with her team at the Cognitive Systems Lab of KIT. "As a major step in this direction, our recent results indicate that both single units in terms of speech sounds as well as continuously spoken sentences can be recognized from brain activity."

"These results were obtained by an interdisciplinary collaboration of researchers of informatics, neuroscience, and medicine. In Karlsruhe, the methods for signal processing and automatic speech recognition have been developed and applied," states Schultz.

Related articles Researchers Turn One Kind Of Neuron Into Another In A Living Brain  Scientists Create New Memories by Directly Changing the Brain  Wireless Brain Sensor Developed At Brown University

"Along with decoding speech from brain activity, our models allow for a detailed analysis of the brain areas involved in speech processes and their interaction,” outline Christian Herff und Dominic Heger, who developed the Brain-to-Text system within their doctoral studies.

The present work is the first that decodes continuously spoken speech and transforms it into a textual representation. For this purpose, cortical information is combined with linguistic knowledge and machine learning algorithms to extract the most likely word sequence. Currently, Brain-to-Text is based on audible speech. However, the results are an important first step for recognizing speech from thought alone.

"Our recent results indicate that both single units in terms of speech sounds as well as continuously spoken sentences can be recognized from brain activity."

The brain activity was recorded in the USA from seven epileptic patients, who participated voluntarily in the study during their clinical treatments. An electrode array was placed on the surface of the cerebral cortex (electrocorticography, or ECoG) for their neurological treatment.

While the patients read aloud sample texts, the ECoG signals were recorded with high resolution in time and space. Later on, the researchers in Karlsruhe analyzed the data to develop Brain-to-Text. In addition to basic science and a better understanding of the highly complex speech processes in the brain, Brain-to-Text might be a building block to develop a means of speech communication for locked-in patients in the future.



SOURCE  Karlsruhe Institute of Technology

By 33rd Square

Embed

 Mind Reading  

Researchers have found the first solid evidence that the pattern of brain activity seen in someone performing a mathematical exercise under experimentally controlled conditions is very similar to that observed when the person engages in thinking about numbers and quantities in general.

A brain region activated when people are asked to perform mathematical calculations in an experimental setting is similarly activated when they use numbers — or even imprecise quantitative terms, such as “more than”— in everyday conversation, according to a study by Stanford University School of Medicine scientists.

Using a new method, the researchers collected the first solid evidence that the pattern of brain activity seen in someone performing a mathematical exercise under experimentally controlled conditions is very similar to that observed when the person engages in quantitative thought in the course of daily life.

“We’re now able to eavesdrop on the brain in real life,” said Josef Parvizi, MD, PhD, associate professor of neurology and neurological sciences and director of Stanford’s Human Intracranial Cognitive Electrophysiology Program. Parvizi is the senior author of the study, published in Nature Communications. The study’s lead authors are postdoctoral scholar Mohammad Dastjerdi, MD, PhD, and graduate student Muge Ozker.

The finding could lead to “mind-reading” applications that, for example, would allow a patient who is rendered mute by a stroke to communicate via passive thinking. Conceivably, it could also lead to more dystopian outcomes: chip implants that spy on or even control people’s thoughts.

“This is exciting, and a little scary,” said Henry Greely, JD, the Deane F. and Kate Edelman Johnson Professor of Law and steering committee chair of the Stanford Center for Biomedical Ethics, who played no role in the study but is familiar with its contents and described himself as “very impressed” by the findings. “It demonstrates, first, that we can see when someone’s dealing with numbers and, second, that we may conceivably someday be able to manipulate the brain to affect how someone deals with numbers.”

The researchers monitored electrical activity in a region of the brain called the intraparietal sulcus, known to be important in attention and eye and hand motion. Previous studies have hinted that some nerve-cell clusters in this area are also involved in numerosity, the mathematical equivalent of literacy.

However, the techniques that previous studies have used, such as functional magnetic resonance imaging, are limited in their ability to study brain activity in real-life settings and to pinpoint the precise timing of nerve cells’ firing patterns. These studies have focused on testing just one specific function in one specific brain region, and have tried to eliminate or otherwise account for every possible confounding factor. In addition, the experimental subjects would have to lie more or less motionless inside a dark, tubular chamber whose silence would be punctuated by constant, loud, mechanical, banging noises while images flashed on a computer screen.

Related articles Stanford Neuroscientists Develop Technique To Read Mouse Minds  Human Memory Implants May Soon Be Tested  Scientists Use Light To Reach New Understandings of the Brain

“This is not real life,” said Parvizi. “You’re not in your room, having a cup of tea and experiencing life’s events spontaneously.” A profoundly important question, he said, is: “How does a population of nerve cells that has been shown experimentally to be important in a particular function work in real life?”

His team’s method, called intracranial recording, provided exquisite anatomical and temporal precision and allowed the scientists to monitor brain activity when people were immersed in real-life situations. Parvizi and his associates tapped into the brains of three volunteers who were being evaluated for possible surgical treatment of their recurring, drug-resistant epileptic seizures.

The procedure involves temporarily removing a portion of a patient’s skull and positioning packets of electrodes against the exposed brain surface. For up to a week, patients remain hooked up to the monitoring apparatus while the electrodes pick up electrical activity within the brain. This monitoring continues uninterrupted for patients’ entire hospital stay, capturing their inevitable repeated seizures and enabling neurologists to determine the exact spot in each patient’s brain where the seizures are originating.

During this whole time, patients remain tethered to the monitoring apparatus and mostly confined to their beds. But otherwise, except for the typical intrusions of a hospital setting, they are comfortable, free of pain and free to eat, drink, think, talk to friends and family in person or on the phone, or watch videos.

The electrodes implanted in patients’ heads are like wiretaps, each eavesdropping on a population of several hundred thousand nerve cells and reporting back to a computer.

In the study, participants’ actions were also monitored by video cameras throughout their stay. This allowed the researchers later to correlate patients’ voluntary activities in a real-life setting with nerve-cell behavior in the monitored brain region.

As part of the study, volunteers answered true/false questions that popped up on a laptop screen, one after another. Some questions required calculation — for instance, is it true or false that 2+4=5? — while others demanded what scientists call episodic memory — true or false: I had coffee at breakfast this morning. In other instances, patients were simply asked to stare at the crosshairs at the center of an otherwise blank screen to capture the brain’s so-called “resting state.”

Consistent with other studies, Parvizi’s team found that electrical activity in a particular group of nerve cells in the intraparietal sulcus spiked when, and only when, volunteers were performing calculations.

Afterward, Parvizi and his colleagues analyzed each volunteer’s daily electrode record, identified many spikes in intraparietal-sulcus activity that occurred outside experimental settings, and turned to the recorded video footage to see exactly what the volunteer had been doing when such spikes occurred.

They found that when a patient mentioned a number — or even a quantitative reference, such as “some more,” “many” or “bigger than the other one” — there was a spike of electrical activity in the same nerve-cell population of the intraparietal sulcus that was activated when the patient was doing calculations under experimental conditions.

That was an unexpected finding. “We found that this region is activated not only when reading numbers or thinking about them, but also when patients were referring more obliquely to quantities,” said Parvizi.

“These nerve cells are not firing chaotically,” he said. “They’re very specialized, active only when the subject starts thinking about numbers. When the subject is reminiscing, laughing or talking, they’re not activated.” Thus, it was possible to know, simply by consulting the electronic record of participants’ brain activity, whether they were engaged in quantitative thought during nonexperimental conditions.

Any fears of impending mind control are, at a minimum, premature, said Greely. “Practically speaking, it’s not the simplest thing in the world to go around implanting electrodes in people’s brains. It will not be done tomorrow, or easily, or surreptitiously.”

Parvizi agreed. “We’re still in early days with this,” he said. “If this is a baseball game, we’re not even in the first inning. We just got a ticket to enter the stadium.”

SOURCE  Stanford School of Medicine

By 33rd Square

Subscribe to 33rd Square

 Mind Reading  

Using a mathematical model, researchers in The Netherlands have reconstructed thoughts from data collected from fMRI test subjects - essentially they read the minds of the participants.

By analysing MRI images of the brain with an elegant mathematical model, researchers from Radboud University Nijmegen have reconstruct thoughts more accurately than ever before. In this way, they have succeeded in determining which letter a test subject was looking at.

The researchers work has been published in the  journal Neuroimage.

Functional MRI scanners have been used in cognition research primarily to determine which brain areas are active while test subjects perform a specific task. The question is simple: is a particular brain region on or off? A research group at the Donders Institute for Brain, Cognition and Behaviour at Radboud University has gone a step further: they have used data from the scanner to determine what a test subject is looking at.

The researchers 'taught' a model how small volumes of 2x2x2 mm from the brain scans -- known as voxels -- respond to individual pixels. By combining all the information about the pixels from the voxels, it became possible to reconstruct the image viewed by the subject. The result was not a clear image, but a somewhat fuzzy speckle pattern. In this study, the researchers used hand-written letters.

Related articles Breakthrough Technique Gives Unprecedented View Inside The Brain  BigBrain Presents An Ultrahigh-Resolution 3D Model of the Human Brain  Researchers Demonstrate Matrix-Style Learning

"After this we did something new", says lead researcher Marcel van Gerven. "We gave the model prior knowledge: we taught it what letters look like. This improved the recognition of the letters enormously. The model compares the letters to determine which one corresponds most exactly with the speckle image, and then pushes the results of the image towards that letter. The result was the actual letter, a true reconstruction."

"Our approach is similar to how we believe the brain itself combines prior knowledge with sensory information. For example, you can recognize the lines and curves in this article as letters only after you have learned to read. And this is exactly what we are looking for: models that show what is happening in the brain in a realistic fashion. We hope to improve the models to such an extent that we can also apply them to the working memory or to subjective experiences such as dreams or visualisations. Reconstructions indicate whether the model you have created approaches reality."

In other words, the researchers claim to be very close to the ability to read your mind with their technique.  Such an understanding may also open up the possibility of implanting thoughts, knowledge or, on a more sinister level, control the actions of individuals without their authority.

"In our further research we will be working with a more powerful MRI scanner," explains Sanne Schoenmakers, who is working on a thesis about decoding thoughts. "Due to the higher resolution of the scanner, we hope to be able to link the model to more detailed images. We are currently linking images of letters to 1200 voxels in the brain; with the more powerful scanner we will link images of faces to 15,000 voxels."


SOURCE  Radboud University Nijmegen

By 33rd Square

Subscribe to 33rd Square