Chat2Brain: A Method for Mapping Open-Ended Semantic Queries to Brain Activation Maps

by   Yaonai Wei, et al.

Over decades, neuroscience has accumulated a wealth of research results in the text modality that can be used to explore cognitive processes. Meta-analysis is a typical method that successfully establishes a link from text queries to brain activation maps using these research results, but it still relies on an ideal query environment. In practical applications, text queries used for meta-analyses may encounter issues such as semantic redundancy and ambiguity, resulting in an inaccurate mapping to brain images. On the other hand, large language models (LLMs) like ChatGPT have shown great potential in tasks such as context understanding and reasoning, displaying a high degree of consistency with human natural language. Hence, LLMs could improve the connection between text modality and neuroscience, resolving existing challenges of meta-analyses. In this study, we propose a method called Chat2Brain that combines LLMs to basic text-2-image model, known as Text2Brain, to map open-ended semantic queries to brain activation maps in data-scarce and complex query environments. By utilizing the understanding and reasoning capabilities of LLMs, the performance of the mapping model is optimized by transferring text queries to semantic queries. We demonstrate that Chat2Brain can synthesize anatomically plausible neural activation patterns for more complex tasks of text queries.


page 3

page 4

page 7


NeuroQuery: comprehensive meta-analysis of human brain mapping

Reaching a global view of brain organization requires assembling evidenc...

Complex Coordinate-Based Meta-Analysis with Probabilistic Programming

With the growing number of published functional magnetic resonance imagi...

Neuroimaging Meta Regression for Coordinate Based Meta Analysis Data with a Spatial Model

Coordinate-based meta-analysis combines evidence from a collection of Ne...

Mapping cognitive ontologies to and from the brain

Imaging neuroscience links brain activation maps to behavior and cogniti...

ExpertoCoder: Capturing Divergent Brain Regions Using Mixture of Regression Experts

fMRI semantic category understanding using linguistic encoding models at...

Text-mining the NeuroSynth corpus using Deep Boltzmann Machines

Large-scale automated meta-analysis of neuroimaging data has recently es...

Semantic Folding Theory And its Application in Semantic Fingerprinting

Human language is recognized as a very complex domain since decades. No ...

Please sign up or login with your details

Forgot password? Click here to reset