Talks

Utterance fluency, particularly in speaking, has been demonstrated to be a robust indicator of L2 proficiency (Tavakoli et al., 2020), strongly linked to automaticity (Suzuki & Révész, 2023) and, as such, an advantageous aspect of performance to assess (de Jong, 2018). Recently, studies have also looked at writing fluency to determine how much it relates to performance (Aldridge & Fontaine, 2019) and proficiency (Révész et al., 2022). The present study attempts to compare speaking and writing fluency based on conversational interactions with automated agents or chatbots to determine how speaking and writing-based fluency measures correlate or diverge from each other and how they predict—or not—proficiency. To the best of our knowledge, this will be the first study to look at L2 writing fluency in the context of spontaneous, immediate conversational tasks in the context of synchronous computer-mediated communication (SCMC), as all previous studies on L2 writing fluency have been focused on relatively long, non-interactional forms of writing (Suzuki & Révész, 2023). We collected spontaneous speech and written performance from N=215 teenage learners of French through a computer-assisted language learning (CALL) game constructed around guided conversations with characters. The speaking performance was a computer-delivered interview consisting of 27 items (short questions). We computed most speaking fluency metrics known to be good predictors of L2 proficiency, such as speech and articulation rate, length of runs in syllables and duration, silent pausing rate, etc. (Suzuki & Révész, 2023; Segalowitz et al., 2017). We also obtained a proxy of L2 proficiency for the participants through a vocabulary size test (Peters et al., 2019). In parallel, we collected keystroke logs of the written conversational interactions of 81 participants in the CALL game. The keylogs allow us to compute a series of writing fluency metrics, particularly speed and pausing, such as length of P-bursts in characters, characters over time, pause number per minute and median pause length (see also Révész et al., 2022). We will report how results from the same participants correlate across modes and across aspects of utterance fluency and how much they can predict, separately and jointly, L2 proficiency. References - Aldridge, M., & Fontaine, L. (2019). Using keystroke logging to capture the impact of cognitive complexity and typing fluency on written language production. In E. Lindgren & K. P. H. Sullivan (Eds.), Observing Writing (pp. 285–305). Brill. https://doi.org/10.1163/9789004392526_014 - de Jong, N. H. (2018). Fluency in second language testing: Insights from different disciplines. Language Assessment Quarterly, 15(3), 237–254. https://doi.org/10.1080/15434303.2018.1477780 - Peters, E., Velghe, T., & Van Rompaey, T. (2019). The VocabLab tests: The development of an English and French vocabulary test. ITL - International Journal of Applied Linguistics, 170(1), 53–78. https://doi.org/10.1075/itl.17029.pet - Révész, A., Michel, M., Lu, X., Kourtali, N., Lee, M., & Borges, L. (2022). The relationship of proficiency to speed fluency, pausing, and eye-gaze behaviours in L2 writing. Journal of Second Language Writing, 58, 100927. https://doi.org/10.1016/j.jslw.2022.100927 - Segalowitz, N., French, L., & Guay, J.-D. (2017). What features best characterize adult second language utterance fluency and what do they reveal about fluency gains in short-term immersion? Canadian Journal of Applied Linguistics / Revue Canadienne de Linguistique Appliquée, 20(2), 90–116. https://doi.org/10.7202/1050813ar - Suzuki, S., & Révész, A. (2023). Measuring speaking and writing fluency: A methodological synthesis focusing on automaticity. In Practice and Automatization in Second Language Research. Routledge. - Tavakoli, P., Nakatsuhara, F., & Hunter, A.-M. (2020). Aspects of fluency across assessed levels of speaking proficiency. Modern Language Journal, 104(1), 169–191. https://doi.org/10.1111/modl.12620

L’émergence des intelligences artificielles (IA) génératives et des grands modèles de langue ouvre de très nombreuses questions, opportunités et inquiétudes en éducation, en didactique des langues et ailleurs. Cette intervention tentera de clarifier ce qui change du point de vue technologique, ce que cela implique du point de vue pédagogique, et ce que les apprenants et enseignants de FLE peuvent en faire. Nous explorerons les usages de l’IA en termes de planification, d’interaction, de génération ou adaptation de matériaux et d’évaluation, tout en proposant un rapport critique et éthique à l’IA.

CONTEXTE. Les outils d’intelligence artificielle (IA) offrent de plus en plus d’opportunités d’apprendre et de pratiquer les langues étrangères de manière autonome. Le fait de pouvoir interagir librement, sur n’importe quel thème, est souvent présenté comme un levier majeur de motivation et un atout d’une IA conversationnelle (p. ex. ChatGPT, Alexa…). Mais quelle interactivité importe le plus pour la perception et l’efficacité de ses systèmes pour l’apprentissage du FLE ?

MÉTHODOLOGIE. Nous avons mené une étude expérimentale auprès de 215 adolescents néerlandophones apprenant le français, qui ont interagi avec deux versions d’un jeu sérieux conçu autour de conversations guidées avec des personnages parlants basés sur l’IA : LanguageHero (Linguineo). Les deux versions différaient en termes d’interactivité, l’une étant entièrement libre, tandis que l’autre reproduisait une tâche de complétion de dialogue. Nous avons mesuré les perceptions des apprenants, leur engagement et les effets d’apprentissage en termes de vocabulaire et de développement de la fluidité orale.

RÉSULTATS. Contrairement à nos attentes, le système de dialogue libre n’a pas été perçu comme significativement différent par rapport à la tâche de complétion du dialogue. Cependant, il y avait des différences significatives dans la perception d’une version pilote du système dépourvue de mécanismes d’étayage productif et de feedback correctif. Parallèlement, l’interactivité du système de dialogue a augmenté l’engagement comportemental et la production par le biais d’essais et d’erreurs encouragés par le feedback correctif du système. L’apprentissage du vocabulaire montre néanmoins des effets relativement similaires entre les deux conditions.

DISCUSSION. Au-delà de certaines limites propres à la conception de l’étude, nous émettons l’hypothèse que l’interactivité comprise comme la possibilité d’influencer librement le dialogue, comme elle est souvent conçue dans le contexte des jeux vidéos, n’est pas aussi importante du point de vue motivationnel qu’on pourrait le supposer. Par contre, l’interactivité telle que conçue dans les théories cognitives sur l’apprentissage des langues, conçue comme espace de négociation sur la forme et le sens fait d’étayage et de feedback correctif, semble essentielle pour la motivation d’une activité d’apprentissage perçue comme telle, pour l’engagement des apprenants dans l’activité et, in fine, pour les développements langagiers qui en découlent.

L’émergence des intelligences artificielles (IA) génératives et des grands modèles de langue ouvre de très nombreuses questions, opportunités et inquiétudes en éducation, en didactique des langues et ailleurs. Cette intervention tentera de clarifier ce qui change du point de vue technologique, ce que cela implique du point de vue pédagogique, et ce que les apprenants et enseignants de FLE peuvent en faire. Nous explorerons les usages de l’IA en termes de planification, d’interaction, de génération ou adaptation de matériaux et d’évaluation, tout en proposant un rapport critique et éthique à l’IA.

Les avancées récentes des intelligences artificielles (IA) génératives ont donné lieu à de nombreuses discussions sur leur rôle en éducation (Romero et al, 2023), menaçant même pour certains le rôle de l’enseignant. Si l’utilisation d’IA en apprentissage des langues n’est pas neuve (Heift & Schulze, 2015), l’accessibilité des systèmes conversationnels basés sur de grands modèles linguistiques (LLM) met à portée de main des utilisateurs nombre d’usages auparavant seulement hypothétiques. La question des rôles à assigner à l’apprenant, à l’enseignant et aux outils-systèmes comme l’IA se pose dès lors, renouvelée, dans ce nouveau cadre technologique.

Cette communication interroge d’une part la littérature scientifique récente sur les systèmes conversationnels pour l’apprentissage des langues. D’autre part, nous explorons les pratiques effectives d’apprenants de français langue étrangère au départ d’enquêtes et d’entretiens.

Face à la large diversité des usages possibles pour l’apprenant — du générateur de texte au correcteur, en passant par l’interlocuteur, le générateur d’exercices, le fournisseur d’explications — et ceux possibles pour l’enseignant — évaluateur, générateur, adaptateur —, les usages réellement adoptés par les apprenants sont plus circonscrits. Ils contredisent une vision où la technologie joue le premier rôle et démontrent au contraire la centralité de la relation enseignant-apprenant.

This breakout session provides a brief overview of scientific research into the use of conversational AI (such as ChatGPT) in language education. The session will consist of two parts. In the first part, you will learn about the different applications of conversational AI for language learners and teachers. Moreover, you will gain insight into the effectiveness of conversational AI for language learning. In the second part, we take a closer look at how technologies such as chatbots and generative language models work and how they are developed and evaluated.

BACKGROUND. Interactivity has been defined in various ways: in instructed SLA, as distinguishing dialogic tasks from monologic ones (Ellis et al., 1994); in game design, as involving agency and user control. We propose a bidimensional model of interactivity in conversational AI, structured around user control and bi-directional adaptivity. This study addresses the question of how these different dimensions of interactivity are operationalized in conversational AI for language learning, i.e., when learners interact with a chatbot to practice an L2 (Bibauw et al., 2019; 2022), and how much they influence the learning experience.

METHODS. We conducted an experimental study with 215 teenage learners of French, who interacted with two versions of a serious game designed around guided conversations with AI-based talking characters. The two versions differed in interactivity, one being entirely free while the other replicated a dialogue completion task. We measured learners’ perceptions, engagement, and learning effects on vocabulary knowledge.

RESULTS. Contrary to our expectations, the free dialogue system was not perceived as significantly different compared to the dialogue completion task. However, there were significant differences in the perception of a pilot version of the system devoid of scaffolding and feedback mechanisms. On the other hand, the interactivity of the dialogue system increased behavioral engagement and production through trials-and-errors incentivized by the system feedback. Vocabulary learning shows similar effects for both conditions in receptive (d = 1.16) and productive knowledge (d = 0.59).

DISCUSSION. Beyond certain limitations proper to the study design, we hypothesize that interactivity, understood as a set of game-like motivational characteristics such as agency, might have less impact on perceptions and effectiveness than expected. Our results refocus our understanding of the benefits of interactivity towards differences in engagement, feedback, and scaffolding, and concretely realized in conversational AI through negotiation of form and meaning.

Deze sessie biedt een beknopt overzicht van het wetenschappelijk onderzoek naar het gebruik van chatbots en generatieve taalmodellen (zoals ChatGPT) in het (taal)onderwijs.

De sessie verloopt als volgt:

• Je krijgt een korte, technische introductie in dialoogsystemen.
• Je leert over het gebruik van dialoogsystemen in het taalonderwijs. Je verkrijgt meer inzicht in de verschillende types systemen en hun effectiviteit voor spreek- en schrijfoefeningen. Dit deel wordt in het Engels gepresenteerd.
• Verder leer je meer over generatieve taalmodellen en hun gebruik voor educatieve toepassingen. De sessie zal zich in het bijzonder toespitsen op de evaluatie van schrijftaken en het gebruik van artificieel gegenereerde teksten.

The number of times a word appears in the input, i.e., the frequency of occurrence, is a well-known factor influencing incidental vocabulary acquisition (Webb, 2020). However, paradoxically, the frequency of occurrence has only a medium predicting power in vocabulary learning, around r = .34 (Uchihara et al., 2019). Our study proposes to look at how productive uses of the target words help understand and predict incidental vocabulary acquisition, complementing the vision of frequency of occurrence in input by looking at output too in a productive task.

We conducted a large-scale experimental study on the effectiveness of dialogue-based CALL (here, a game allowing players to discuss with computer-controlled conversational agents) for vocabulary learning. In a pretest-posttest experiment, 215 Dutch-speaking teenage learners of French were tested on their receptive and productive knowledge of target words encountered in the dialogue-based CALL interactions.

Results showed significant improvement in vocabulary knowledge for participants in the experimental condition in both the meaning-recognition (d = 1.16) and form-recall tests (d = 0.59). The frequency of occurrence in input (actual encounters for each participant) was the best predictor for receptive vocabulary learning (r = .27 ***), but the frequency of use in output (actual instances in output for each participant) was the best predictor for productive vocabulary learning (r = .28 ***). Integrating all results in mixed-effects models, we were able to determine that not only does the frequency of occurrence in output predicts more strongly the posttest scores than input encounters, it also complements it and raises the model predictive power (R² = .66). We propose a model of incidental vocabulary learning in dialogue-based CALL connecting all variables and conclude on the importance of productive practice in lexical acquisition.

Dialogue systems, such as chatbots and dialogue-based games, provide an opportunity for safe, meaningful conversational practice to foreign language learners. We evaluated how much the speaking fluency of A1-A2 teenage learners of French (N = 164) was impacted by a very short-term synchronous written practice with a dialogue-based CALL game. Pre-post comparisons of various semi-automatized fluency measures confirm that such precise metrics can detect short-term L2 fluency development. The detected effects are promising, even if the limited time on task, the shortcomings of the instructional design, and the limited transfer of learning from writing to speaking do not allow the effect to significantly outperform the test-retest training effect.

Automatizing testing and computation of multiple variations (pruning, normalization…) of L2 utterance fluency metrics, we collected pre/post oral interview responses from N=215 learners of French, and compared each metric’s variation with external proficiency estimates, to determine which operationalizations best predict proficiency and detect very-short-term developmental changes in L2 fluency.

¿Por qué utilizar tecnología en la educación? ¿La tecnología mejora significativamente el aprendizaje? ¿Qué ofrece como posibilidades con más potencial pedagógico? ¿Qué conservar de la educación virtual?

En ciencias sociales, particularmente en educación, una parte importante de la investigación cuantitativa busca medir la efectividad de ciertas intervenciones. Este seminario busca introducir los enfoques cuantitativos, particularmente el enfoque experimental, con sus oportunidades y sus limitaciones.

Presentaré ejemplos de diseños correlacionales y experimentales en el marco del aprendizaje de idiomas, identificando instrumentos de recolección de datos y técnicas dominantes de análisis estadístico, a través de software libre cómo R y Jamovi. Mencionaré la tendencia a ir más allá de las pruebas de significancia estadística (hipótesis nula, p-value) hacía los tamaños de efecto. Este tipo de estudio también presenta una oportunidad importante de sintésis de los resultados científicos a través del proceso de meta-análisis.

Les systèmes de dialogue pour l’apprentissage des langues regroupent différentes applications permettant à un apprenant de langue étrangère de converser, oralement ou par écrit, avec un interlocuteur automatisé (chatbot, robot, interface vocale, personnage non-joueur, etc.). À partir d’une synthèse systématique de la littérature scientifique sur ces systèmes, je présenterai un tour d’horizon des différentes expériences de développement, débouchant sur une typologie des systèmes de dialogue pour l’apprentissage des langues. Je mettrai l’accent sur les approches technologiques utilisées, avec leurs implications et apports respectifs.

Dans un deuxième temps, je montrerai comment l’efficacité de ces différents systèmes a été évaluée empiriquement. Dans des études antérieures d’abord, à travers une méta-analyse des effets sur l’apprentissage des langues. Dans une étude expérimentale ensuite, avec un jeu dialogique pour l’apprentissage du français, dont nous avons mesuré les effets en termes de développement du vocabulaire et de la fluence, dans un processus semi-automatisé de passation et d’analyse d’entretiens oraux. Je discuterai des leçons de cette étude et des pistes les plus prometteuses qu’elle ouvre pour la recherche et les applications d’apprentissage des langues.

Chatbots and conversational agents have recently seen major developments in many domains, including education. In foreign language learning, dialogue-based computer-assisted language learning (CALL) applications allow a learner to practice a foreign language by having a conversation with a computer, be it orally or in writing. I will first present the results of our multilevel meta-analysis on the effectiveness of dialogue-based CALL for second language acquisition, including how exploratory moderator analyses give directions for the design of future conversational applications.

Subsequently, I will present the methodology and preliminary results from a randomised controlled study on the effects of LanguageHero, a dialogue-based CALL game developed by the Leuven-based startup Linguineo. The study measured the effects of three sessions of practice with the game on fluency and vocabulary development in 11 class groups of elementary learners of French across 4 Flemish schools (N = 215 participants). More specifically, we measured how practice with the chatbot affected spoken fluency in a computer-delivered interview, and how the in-game written fluency could be related to it. We also compared two distinct versions of the game (dialogue system dynamic interaction vs. dialogue completion activity) to establish whether the level of interactivity and spontaneity of the writing activity affected the learning outcomes.

Dialogue-based CALL systems allow a learner to practice meaningfully an L2 with an automated agent, through an oral (spoken dialogue systems) or written interface (chatbots) (Bibauw, François, & Desmet, 2015). In order to obtain a better comprehension of their effects on L2 proficiency development, we conducted a multilevel meta-analysis on all the experimental studies measuring an impact of such systems on language learning outcomes (40 publications). Effect sizes for each variable and group under observation were systematically computed ($k=96$). By combining all studies into a multilevel linear model, we observed a significant medium effect of dialogue-based CALL on general L2 proficiency development ($d=.61$). By integrating moderator variables into our statistical model, we are able to provide insights on the relative effectiveness of certain technological and instructional characteristics (spoken vs. written, task-oriented vs. open-ended, form-focused vs. meaning-focused) on different learning outcomes (writing vs. speaking vs. comprehension skills, complexity, accuracy and fluency measures…) and different samples of populations (L2 proficiency, age, context…), as well as to model the effect of treatment duration (number of sessions and time on task) and spacing on these outcomes, to better inform future system and research design.”

Dialogue-based computer-assisted language learning (CALL) systems allow a learner to practice meaningfully a foreign language (L2) with an automated agent, whether orally or through text chat. With the aim of establishing the impact of such systems on L2 proficiency development, we conducted a meta-analysis of empirical research on dialogue-based CALL. Twenty-nine effect sizes from effectiveness studies on various systems were included into a multilevel model, with moderator variables allowing for fine-grained analysis. Our findings establish a significant effect ($d=.64$) of dialogue-based CALL on general proficiency development. The comparisons of system characteristics, testing conditions and language learning outcome measurements allow to draw conclusions and recommendations for future system and research design on the topic.

Dialogue-based computer-assisted language learning (CALL) encompasses applications that allow a learner to practice a foreign language by carrying a conversation with a computer through unconstrained input. Such systems, whether speech or text-based, present various challenges to the CALL developer, both with the instructional design (degree of openness of the interaction, types of prompt, etc.) and the technological design (rules-driven vs. data-driven system, complexity level of the natural language processing, etc.). We propose a general task design framework for dialogue-based CALL, and align it with recommendations for natural language processing (NLP) techniques to tackle such challenge.

Dialogue-based CALL covers applications and systems allowing a learner to practice the target language in a meaning-focused conversational activity with an automated agent. We first present a common definition for dialogue-based CALL, based on three features: dialogue as the activity unit, computer as the interlocutor and negotiation of outcome through open learner production. We conducted a systematic literature review on the main scientific databases resulting, after filtering, in 138 relevant papers, which were analyzed and coded. Results show a scattered research field, with four different disciplinary approaches. We conclude with observations regarding the remaining challenges and opportunities for ICALL research.

Conversational agents are seen as a way to offer an anxiety-free L2 practice environment for learners, in the hope that it would produce similar proficiency improvements as those obtained in a chat with a human interlocutor. Various approaches have been used to develop conversational agents: pattern matching (from a knowledge base obtained either manually, from a corpus or with crowdsourcing), AI planners and script-based frameworks. However current agents still present important shortcomings for applications in language learning and very few researchers have ever evaluated their impact on L2 development. Following Petersen’s (2010) successful experiment, we will present a design for a task-based agent that would provide semantically and pragmatically consistent answers inside a specific goal-oriented conversation with a learner.