Towards multimodal assessment of L2 level: speech and eye tracking features in a cross-cultural settingVers l’évaluation multimodale du niveau L2: caractéristiques de la parole et du mouvement des yeux dans un cadre multimodal

Committee

• Véronique DELVAUX, Professor, Université de Mons
• Sophie HERMENT, Professor, Université d’Aix-Marseille
• Ralph ROSE, Professor, Waseda University
• Sébastien LALLÉ, Associate Professor, Sorbonne Université
• Nicolas AUDIBERT, Associate Professor, Université Sorbonne Nouvelle
• Cédric Gendrot, Professor, Université Sorbonne Nouvelle

Abstract

In recent years, the world of education has undergone critical changes, especially with the system’s massive digitalization in 2020, as well as the advancement of generative AI technologies. LeCycl (Learning Cyclotron) [Vargo et al., 2023] project is a part of this scientific trend and its aim is to accelerate the knowledge flow. It takes into consideration 3 main processes of learning: perception, mastering and transfer. This thesis, as part of the LeCycl project, focuses on exploring second language (L2) oral reading strategies and analyzing difficulties faced by representatives of different cultures and their techniques of coping with them. For this purpose, we are relying on multimodal cues including speech and eye tracking, as well as an original protocol that introduces nudges (represented by comic books) for the cultural adaptation [Hutin et al., 2023]. For this purpose, we developed a protocol involving the collection of reading aloud data from both native and non-native English speakers (French and Japanese speakers) [Kobylyanskaya, 2022]. We analyzed speakers’ performance through
acoustic and linguistic measures (phoneme realization, prosody and disfluencies such as pauses, hesitations, truncations), as well as eye movement measures [El Baha et al. 2022; Kobylyanskaya et al., 2023]. Then, we used
machine learning methods to define the speaker’s L2 level based on the extracted measures. Finally, we evaluate the contribution of comic books images on speakers’ oral reading performance. The results highlight that the representatives of different cultures face different challenges when reading in a foreign language and employ different strategies to overcome them, which are translated both at verbal and ocular levels. Our results underline the need for culturally adapted learning tools and the challenges involved in developing them.