Continual Pre-training via Sparse Updates

Applicant Profile

• Education: You are pursuing an engineering degree (M1/M2) or a Master’s program specializing in a scientific field related to machine learning, computer science, applied statistics or signal processing. You have followed theoretical courses in machine learning and gained practical experience in deep learning related projects.
• Technical Skills:
  Proficiency in Python and familiarity with deep learning libraries such as TensorFlow/Keras or PyTorch. Experience in computer vision and/or NLP.
• Other Skills:
  – Fluent in French and/or English (spoken and written) , Strong analytical abilities, Ability to communicate about your work, Ability to work autonomously and take the initiative.

Objectives

The aim of the internship is to review, implement, benchmark, and improve state-of-the-art continual learning methods that rely on parsimonious updates.

Context

see details in PDF

Keywords

Parameter-Efficient Fine-Tuning, Self-Supervised Learning, Automatic Speech processing.

Contacts

• Eva Feillet, Université Paris-Saclay, LISN, eva.feillet@lisn.fr
• Sahar Ghannay, Université Paris-Saclay, LISN, sahar.ghannay@lisn.fr

Stipends

Around 680 €/months + access to staff dining facilities and the campus’s many sports facilities