Dialogue-based CALL: a multilevel meta-analysis

Abstract

Dialogue-based CALL systems allow a learner to practice meaningfully an L2 with an automated agent, through an oral (spoken dialogue systems) or written interface (chatbots) (Bibauw, François, & Desmet, 2015). In order to obtain a better comprehension of their effects on L2 proficiency development, we conducted a multilevel meta-analysis on all the experimental studies measuring an impact of such systems on language learning outcomes (40 publications). Effect sizes for each variable and group under observation were systematically computed ($k = 96$). By combining all studies into a multilevel linear model, we observed a significant medium effect of dialogue-based CALL on general L2 proficiency development ($d = .61$). By integrating moderator variables into our statistical model, we are able to provide insights on the relative effectiveness of certain technological and instructional characteristics (spoken vs. written, task-oriented vs. open-ended, form-focused vs. meaning-focused) on different learning outcomes (writing vs. speaking vs. comprehension skills, complexity, accuracy and fluency measures…) and different samples of populations (L2 proficiency, age, context…), as well as to model the effect of treatment duration (number of sessions and time on task) and spacing on these outcomes, to better inform future system and research design.

Date
Event
LEAD Summer School in L2 Acquisition
Location
Tübingen, Germany
Links

Poster

Download the poster

Dialogue-based CALL

Dialogue-based CALL systems involve

  • a dialogue (i.e., sequence of conversational turns)
  • with an automated agent (chatbot, robot, voice assistant, non player character…)
  • as a language learning task (≠scaffolding).

See Bibauw, François & Desmet, 2015 & under review for a full discussion.

Methods

Formulas for effect size calculation, on a single “raw” metric (aligned to between-groups effects) across experimental designs, from Morris & DeShon (2002):

$$d_{ \text{PP} } = J ( df_{\text{PP}} ) \left( \frac { M_{ \text{post,E} } - M_{ \text{pre,E} } } { \mathit{SD}_{ \text{pre,E}} } \right) $$

$$d_{\text{ECPP}} = J (df_{\text{ECPP}}) \left(\frac{M_{\text{post,E}}-M_{\text{pre,E}}}{\mathit{SD}_{\text{pre,E}}}-\frac{M_{\text{post,C}}-M_{\text{pre,C}}}{\mathit{SD}_{\text{pre,C}}}\right)$$

In previous Equations, we use Hedges’ $J$ as a correction function for small sample size bias (the original formula rather than the commonly used approximation) in order to obtain a more accurate estimate of the effect size (Hedges & Olkin, 1985):

$$J(df)=\frac{\Gamma{\left(df/2\right)}}{\sqrt{df/2}\ \Gamma{\left[\left(df-1\right)/2\right]}}$$

where $df$ corresponds to the degrees of freedom, calculated from the subsample sizes ($n$) in each study as $df_{\text{PP}}=n_{\text{E}}-1$ and $df_{\text{ECPP}}=n_{\text{E}}-1 + n_{\text{C}}-1$.

Multilevel meta-analysis modeling

Level Number of clusters/items Source of variance
Subjects $k=96$ ($n=804$) Random sampling variance
Effect sizes $k=96$ Variation within study
Studies $k_{studies}=17$ Variation between studies

All computations were done in R, with the metafor package, using the rma.mv() function (Viechtbauer, 2010):

rma.mv(di, vi, data = dataset, random = ~1|Paper/Effect)

See Van den Noortgate, López-López, Marín-Martínez & Sánchez-Meca (2013) for a discussion of multilevel modeling in meta-analyses.

Results

Mean effect of use of dialogue-based CALL for L2 development: $d = .61$ (95% CI: $[.373, .831]$).

Moderator analyses

Significant moderators:

  • L2 proficiency level
  • Outcome variable type (production vs. comprehension) and dimension (accuracy, complexity, fluency…)


Many interesting exploratory results needing to be confirmed in future research.

References

Bibauw, S., François, T., & Desmet, P. (2015). Dialogue-based CALL: an overview of existing research. In F. Helm, L. Bradley, M. Guarda, & S. Thouësny (Eds.), Critical CALL – Proceedings of the 2015 EUROCALL Conference, Padova, Italy (pp. 57–64). Dublin: Research-publishing.net.

Bibauw, S., François, T., & Desmet, P. (under review). Discussing with a computer to practice a foreign language: from a conceptual framework to a research agenda for dialogue-based CALL.

Mackey, A., & Goo, J. (2007). Interaction research in SLA: A meta-analysis and research synthesis. In A. Mackey (Ed.), Conversational interaction in second language acquisition: A collection of empirical studies (pp. 407–452). Oxford: Oxford University Press.

Morris, S. B., & DeShon, R. P. (2002). Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychological Methods, 7(1), 105‑125.

Van den Noortgate, W., López-López, J. A., Marín-Martínez, F., & Sánchez-Meca, J. (2013). Three-level meta-analysis of dependent effect sizes. Behavior Research Methods, 45(2), 576–594.

Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3).