Using human raters is a major validity concern for language testers and researchers. This mixed-methods study examined the role of psychological and cognitive attributes in rater cognition when assessing EFL speaking performances. It investigated whether decision-making styles, cognitive preferences or cognitive capacities were related to rating quality and rating behaviours.
In SLA research designs as well as the field of language testing, the quality of rating procedures constitutes a validity concern. The argument-based approach to validation (Kane, 1992; 2006) places rating scale design and rating procedures almost at the beginning of the interpretive argument (Knoch & Chapelle, 2017). Thus, any systematic weakness of the rating process affects the meaning of scores. While there has been an increased interest in rater cognition (Bejar, 2012), the field is yet to engage fully with research on judgment and decision-making – important areas of scholarship in cognitive psychology and economics. Findings from this literature suggest that complex decision-making tasks are greatly influenced by a range of factors including processing capacities, perception, the interplay of deliberate and automated thinking, and metacognitive control (Newell and Bröder, 2008). Exploring these factors would prove useful in modelling the task of assessing second-language speech in real time. Set within the context of a school-leaving EFL oral examination, the current study investigated the influence of decision-making style, preferred cognitive style, working memory and executive function on rater cognition (i.e., rating behaviour and rater severity, accuracy and consistency). Following a training session, 39 pre-service EFL teachers rated a set of video-recorded speaking performances (N=30), filled out two validated psychological questionnaires, and completed a battery of cognitive tests. Data analysis combined Many-Facets Rasch Measurement (MFRM) and inferential statistics to investigate (a) rater fit and alignment with a set of benchmark ratings and (b) relationships between individual attributes and measures of rater cognition.Findings show considerable individual differences among participants on some variables despite recruiting from a homogeneous group. Relationships between variables suggest a complex association between the attributes measured and rater cognition. The need to expand the range of cognitive attributes currently considered as influential for rating processes will be discussed.
