The study reports on an EFL writing test validation project. Subjects from diverse educational backgrounds responded to two tasks in front of an eye tracker. They produced stimulated recalls which were combined with measures of proficiency, typing speed and eye movements to map participants’ cognitive processes across levels and tasks.
This paper presents results from a study that aimed at using insights and methodologies from SLA research with the purpose of validating a new computer-based general EFL writing test. By triangulating data from eye-tracking, keystroke logging, text complexity analyses of the written performances (Coh-Metrix), as well as stimulated recalls from 15 participants (N=30, 15 participants x 2 tasks), the study investigated the following research question: How do cognitive processes, writing behaviours and features of text quality vary across two writing tasks and different levels of test taker ability? A group of L2 learners (N=32, L1=German) from various educational backgrounds responded to both parts of a general English writing test, as well as completing a typing speed test and a standalone receptive language proficiency test. While writing their responses, participants' eye movements and writing behaviour were recorded with a Tobii TX300 eye tracker and Inputlog. After each writing task, 15 subjects also produced stimulated recalls prompted by a replay of their eye movements while typing. The verbal recalls were transcribed and coded for evidence of cognitive processing as described in the literature (Révész, Michel, & Lee, 2017), which offered insights into the cognitive processes candidates employed. Selected metrics from the eye-tracking were used in correlational analyses and comparison of metrics across tasks and in relation to test taker proficiency and writing outcome. While our findings contribute to a particular test's validation, the presentation will also highlight some of the challenges of integrating complex research methods frequently employed by SLA research into test validation studies. In our view, building a strong alliance between the two related branches of Applied Linguistics also appears beneficial for SLA research in that insights from language testing can enrich theories of task effects and task complexity.
