Connectionists: [PhD] Inria Bordeaux, FR: From Reservoir Transformers to BrainGPT

[Xavier Hinaut]

PhD position available at Inria, Bordeaux, France: "From Reservoir Transformers to BrainGPT"

# Keywords
Transformers; Reservoir Computing; Computational Neuroscience

# Duration & start date
3 years starting 1st October 2023 (estimated)

# Context
This PhD thesis is part of the BrainGPT "Inria Exploratory Action" project. In the wake of the emergence of large-scale language models such as ChatGPT, the BrainGPT project is at the forefront of research in Artificial Intelligence and Computational Neuroscience. These models, although remarkably powerful, do not reflect the way in which our brains process and learn language. BrainGPT is rising to the challenge by focusing on the development of models that are more faithful to human cognitive functioning, inspired by data on brain activity during listening or reading. The ambition is to create more efficient models that are less dependent on intensive calculations and massive volumes of data. BrainGPT will open up new perspectives on our understanding of language and cognition.

# Project and Work description
The rapid rise in performance of language models, as demonstrated by the recent appeal of ChatGPT, is undeniable. However, the computational cost and environmental impact associated with such models are often overlooked [1]. These models rely on Transformers [2], which facilitate unsupervised learning on a large volume of data. These same models are used to predict brain activity from functional magnetic resonance imaging (fMRI) or magnetoencephalography (MEG), an application our team also exploits [3].

The main ambition of the BrainGPT project is to combine the explainability of mechanistic models with the predictive power of Transformers to analyze brain imaging data. Today, on the one hand we have explanatory but less predictive mechanistic models, such as those based on Reservoir Computing, and on the other hand, high-performance predictive models, but not explanatory, like Transformers. Our goal is to combine the best of these two approaches, to develop more efficient ("sample efficient") models inspired by Transformers, which more faithfully reflect the how the brain works, while improving the predictive power of mechanistic models.

Towards this ambition, the BrainGPT project seeks to identify the key mechanisms that allow Transformers to predict brain activity. Furthermore, our project strives to build models that are more biologically plausible than Transformers, incorporating the most relevant components for predicting brain activity, while integrating constraints derived from human cognition studies.

The long-term objectives of the BrainGPT project are as follows:
- Making Transformers more biologically plausible, which could improve the prediction of brain activity by imaging (fMRI, MEG, etc.).
- Proposing new perspectives and computing paradigms that do not rely exclusively on gradient backpropagation, given its high computational and energy cost.
- Reducing the energy footprint of Transformers by minimizing the computational costs associated with their learning.

In summary, the thesis will mainly consist of developing new bio-inspired models inspired by the mechanisms, learning methods, and emerging behaviors of Large Language Models (LLMs) and Transformers. Subsequently, in collaboration with our collaborators, these models will be tested to assess their ability to predict brain activity from imaging data.

[1] Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021, March). On the dangers of stochastic parrots: Can language models be too big?🦜. In Proc. of the 2021 ACM conference on fairness, accountability, and transparency (pp. 610-623).
[2] Vaswani, A. et al. (2017) Attention is all you need. In Proc. of Advances in neural information processing systems.
[3] Oota, S. R., Trouvain, N., Alexandre, F., & Hinaut, X. (2023, August). MEG Encoding using Word Context Semantics in Listening Stories. In Proc. of INTERSPEECH 2023.

# Contact & Application
Informal contact/questions and application: email to xavier dot hinaut at inria dot fr
Application & More info: https://recrutement.inria.fr/public/classic/en/offres/2023-06611
All positions available: https://github.com/neuronalX/phd-and-postdoc-positions
Deadline: Applications will be considered on a rolling basis. A candidate will be selected as soon as a suitable profile is found.


# Skills
Ideal Candidate Profile:
- Holds an engineering or scientific degree and/or a PhD in digital sciences (computer science, automation, signal processing).
- Has a first professional experience (6 months of internship or more) in Machine Learning and Python development. Especially in one or more of the following: Recurrent Neural Networks (in particular Reservoir Computing), Transformers, Large Language Models.
- Possesses strong expertise in the scientific Python software and scientific stack (numpy/scipy).
- Demonstrates a solid grasp of linear algebra concepts.
- Proficiency in technical English is crucial, as it enables efficient collaboration with our international partners and effective presentations at conferences.
- Has proven experience with version management, familiarity with Git, and proficiency in using the GitHub platform.

# Advisor & Location
Xavier Hinaut
Inria Bordeaux & Institute for Neurodegenerative diseases (Pellegrin Hospital Campus, Bordeaux).


Best regards,

Xavier Hinaut
Inria Research Scientist
www.xavierhinaut.com -- +33 5 33 51 48 01
Mnemosyne team, Inria, Bordeaux, France -- https://team.inria.fr/mnemosyne
& LaBRI, Bordeaux University --  https://www4.labri.fr/en/formal-methods-and-models
& IMN (Neurodegeneratives Diseases Institute) -- http://www.imn-bordeaux.org/en
---
Our Reservoir Computing library: https://github.com/reservoirpy/reservoirpy