Dialogue systems for language learning: a meta-analysis

Abstract

The present study offers a meta-analysis of effectiveness studies on dialogue-based CALL, i.e., systems allowing a learner to practice a second language (L2) by interacting with a conversational agent (“bot”). Through a systematic inclusion and exclusion process, screening 419 publications, we identified 17 relevant meta-analysable studies. We made use of Morris and DeShon’s (2002) formulas to compute comparable effect sizes across designs, including $k = 100$ individual effect sizes, which were analysed through a multilevel random-effects model.

Results confirm that dialogue-based CALL practice has a significant medium effect size on L2 proficiency development ($d = 0.59$). We performed extensive moderator analyses to explore the relative effectiveness on several learning outcomes of different types and features of dialogue-based CALL (type of interaction, modality, agent embodiment, etc.). In particular, our study confirms the effectiveness of form-focused and goal-oriented systems, system-guided interactions, corrective feedback provision, and gamification features. Significant effects for lower proficiency learners, and on vocabulary, morphosyntax, holistic proficiency and accuracy are established. Finally, we discuss evolutions to be expected in dialogue-based CALL and the language learning opportunities it offers.

Dataset available on OSF

R code for the meta-analysis available on OSF

Appendices:

• A1_pool_of_publications.csv
• A2_coding-scheme.csv
• A3_list_of_effects.csv
• A4_studies_included.csv