Coding scheme for papers
variable description values
1 authors Main authors (first author and second author or 'et al')
2 year Year of publication
3 title Title of the publication
4 doi DOI
5 pub_type Type of publication (see 2.2.4) Abstract, Book chapter, Conference paper, Dissertation, Journal article, Magazine article, Master thesis, Preprint, Student report
6 source Journal title or conference acronym
7 strand Strand/field of publication, based on the main references cited, main keywords used and orientation of the content (see 3.3 and 3.4) Chatbots, Games, SDS/CA, CAPT, ICALL, Pedagogical agents, NA
8 pass_exclusion_criteria Selected after evaluating exclusion criteria (see 2.2) ✔/✘
9 available Full-text document of the publication is available ✔/✘
10 not_duplicate Publication is not a duplicate of another document in the same dataset ✔/✘
11 pub_language Publication language was understandable by our team de, en, es, et, kr
12 peer_reviewed_pub Published in a peer-reviewed source (journal paper, conference paper, book chapter or doctoral disseration) (see 2.2.4) ✔/✘
13 pass_inclusion_criteria Selected after evaluating inclusion criteria (see 2.2) ✔/✘
14 interactions The referred system or application involves interactions ✔/✘
15 with_system Interactions with the system (e.g., with a virtual agent) ✔/✘
16 natural_language Interactions in natural language ✔/✘
17 for_language_learning The system or the study are applied to language learning (either L1 or L2) false, L1, L2
18 enough_on_ds The paper is specifically about the interaction/dialogue system (opposite to a general review or description only mentionning very briefly a potential dialogue system) ✔/✘
19 pass_dbcall_criteria Satisfies the criteria for dialogue-based CALL (see 3.1. An operational definition) ✔/✘
20 interlocutor Who is the interlocutor for the interaction (human peer, native speaker, or system agent) human, human+system, system
21 interaction_unit The interaction unit (dialogue or item-based) dialogue, item
22 interaction_role The role of interaction/dialogue, as the activity/task itself or as a scaffolding element scaffolding, task
23 system
24 system_type Type of system as per the proposed dialogue-based CALL typology (see 4.2. A typology of dialogues and systems) Narrative, Form-focused, Goal-oriented, Reactive
25 dialogue_type Type of instructional dialogue as per the proposed dialogue-based CALL typology (see 4.2. A typology of dialogues and systems) Branching, Form-focused exercises, Elicited, Meaning-constrained, Form-constrained, Contextualized, Free
26 research_type Type of study (see 5.1. Types of study) Review, System description, Technical evaluation, Observational study, Survey study, Effectiveness study
27 n_tot Total number of subjects/learners in the empirical study (sample size)
28 proficiency_level Proficiency level of the learners (in the studied sample) A0, A1, A2, B1, mixed
29 L1 First language of the learners de, en, jp, kr, mixed, nl, pt, zh
30 context Context of use of the system in the empirical study high school, laboratory, language school, middle school, military, primary school, university
31 age Age range of the learners 11-12, 13-17, 14-15, 15-18, 18+, 6-11, 9-13
32 treatment_duration_weeks Duration of the treatment in weeks 1, 4, 5, 10, 16
33 treatment_duration_hours Duration of the treatment in hours (time on task) 1, 2, 3
34 qualitative Qualitative study (non quantitative description of behaviours and outcomes of the study) ✔/✘
35 quantitative Quantitative study ✔/✘
36 during Measurements during the use of the system (e.g., engagement, description of linguistic behavior of the users) ✔/✘
37 post Measurements after the use of the system (e.g., survey on users' perception) ✔/✘
38 prepost Measurements before and after the use of the system (pre-test and post-test) ✔/✘
39 delayed_post Delayed measurements after the use of the system (delayed post-test) ✔/✘
40 comp_condition Various groups with compared conditions (e.g., experimental group and control group, or alternate condition) ✔/✘
41 linguistic_sys Analysis of the linguistic behaviour of the system (e.g., descriptive statistics on certain linguistic characteristics of the utterances from the system) ✔/✘
42 sys_accuracy Measurements of the (technical/NLP) accuracy of the system (e.g., word error rate in recognition tasks) ✔/✘
43 linguistic_perf_user Measurements of the linguistic behaviour or performance of the users ✔/✘
44 attitude_tw_system Measurements of the users' attitude towards the system ✔/✘
45 perc_easeofuse Measurement of perceived ease-of-use of the system ✔/✘
46 perc_usefulness Measurement of perceived usefulness of the system ✔/✘
47 engagement_usage Measurement of engagement of users or usage of the system ✔/✘
48 motivational_effects Measurement of motivational effects of the system (i.e., impact of system use on certain motivational variables) ✔/✘
49 cognitive_effects Measurement of cognitive effects of the system (i.e., impact of system use on certain learning variables) ✔/✘
50 productive_skills Measurement of effects on productive language skills (writing/speaking) ✔/✘
51 speaking Measurement of effects on speaking skills ✔/✘
52 writing Measurement of effects on writing skills ✔/✘
53 holistic_proficiency Measurement of effects on productive skills evaluated as holistic proficiency (i.e., ±subjective global evaluation of skill) ✔/✘
54 complexity_voc Measurement of effects on productive skills evaluated through complexity measures or vocabulary/lexical measures ✔/✘
55 accuracy Measurement of effects on productive skills evaluated through accuracy measures (e.g., number of errors in a defined task) ✔/✘
56 fluency Measurement of effects on productive skills evaluated through fluency measures ✔/✘
57 receptive_skills Measurement of effects on receptive skills (listening/reading) ✔/✘
58 others_exam Measurement of effects through other tests, among others school exams or teaching-focused tests ✔/✘
59 reliable_ES The publication provides enough data (e.g., sample size, mean and standard deviation of at least two measurement points with a clearly designed and described evaluation instrument) on the results of the experimental study to compute reliable effect sizes ✔/✘
60 keyword Main keyword used in the publication to refer to dialogue-based CALL system (usually in the title or the abstract)