Connectionists: [internship] Animal vocalization analysis and annotation tool (Bordeaux, France)

[Xavier Hinaut]

**AI internship offer at Inria and Bordeaux Neurocampus (France) on Canapy: an Animal vocalization analysis and annotation tool**

Application and more info: https://github.com/neuronalX/internships/blob/main/2022-2023_MSc-or-BSc_Trouvain-Leblois-Hinaut_Canapy_Songbird-GUI_EN.pdf

The main objectives of the internship will be:
1. to develop a graphical interface to train vocalization annotation models, to visualize their performance and to re-annotate parts of the dataset accordingly (in a similar fashion as semi-supervised learning);
2. to develop the corresponding software backend: data management (audio and annotations), serving and local persistence of the models (MLOps);
3. to collaborate with the project members to define the needs, establish the specifications or integrate pre-existing tools. This objective also implies collaborating with international researchers, and making an open source tool available to the public.
The development will be incremental: a first prototype will allow to train models and to present their evaluation on the interface. A second prototype will offer advanced editing possibilities of the dataset (re-annotation of parts of the audio according to the results of the model), and the final version will integrate advanced analysis tools (dataset errors detection, spectrograms dimensionality reduction for visualization and/or clustering, syntactic analysis of song sequences, ...)

The student will have to develop an interface, preferably web, in javascript/typescript (React...) or directly in Python (bokeh/panel/holoviz...). 
The software backend will serve Machine Learnnig models defined in Python (type scikit-learn/reservoirpy at first, eventually type tensorflow/pytorch).
The tool could be inspired by or integrated with the VocalPy initiative [3]. The student will be encouraged to collaborate with the project collaborators. For example, the data could follow the convention defined by the VocalPy crowsetta package.

Once complete, the tool will be made public, on Github, along with its documentation. The goal is to impact a large international community, like ReservoirPy [4], a library already developed in the Mnemosyne team for the ML community.

[1] N. Trouvain et X. Hinaut, « Canary Song Decoder: Transduction and Implicit Segmentation with ESNs and LTSMs », in ICANN 2021 - 30th International Conference on Artificial Neural Networks, Bratislava, Slovakia, sept. 2021, vol. 12895, p. 71 82. doi: 10/gq43sk.
[2] Y. Cohen, D. A. Nicholson, A. Sanchioni, E. K. Mallaber, V. Skidanova, et T. J. Gardner, « Automated annotation of birdsong with a neural network that segments spectrograms », eLife, vol. 11, p. e63853, janv. 2022, doi: 10/gq43sd.
[3] « VocalPy ». https://github.com/vocalpy
[3] « ReservoirPy ». https://github.com/reservoirpy/reservoirpy

Best regards,

Xavier Hinaut
Inria Research Scientist
www.xavierhinaut.com -- +33 5 33 51 48 01
Mnemosyne team, Inria, Bordeaux, France -- https://team.inria.fr/mnemosyne
& LaBRI, Bordeaux University --  https://www4.labri.fr/en/formal-methods-and-models
& IMN (Neurodegeneratives Diseases Institute) -- http://www.imn-bordeaux.org/en