This paper presents how a polytomous cognitive diagnostic model is used in EFL writing assessment for Chinese EFL learners. It developed, modified and validated a descriptor-based multi-level rating scale by which a Q-Matrix was established and Sequential-DINA Model was applied to produce a fine-grained diagnostic score report, entirely different from the traditional one. It significantly facilitates assessment for learning and personalized instruction.
Diagnostic assessment is widely accepted as an interface between assessment and learning (Alderson, 2005). Cognitive diagnostic assessment (CDA), a newly developed psychometrics theory, has proved to be able to identify learners' strengths and weaknesses in latent cognitive attributes underlying their performance score. Previous studies on CDA-based writing assessment only used dichotomous rating checklist in assessing essays in order to meet dichotomous cognitive diagnostic models (CDMs). However, the checklist is limited to either Yes or No rating without indicating the in-between level of essay writing. Therefore, this study aims to construct a polytomous cognitive diagnostic model for Chinese College English Writing (CCEW) by using a polytomous descriptor-based rating scale modified on the basis of the previously developed checklist, so as to produce fine-grained, detailed cognitive diagnostic score reports for Chinese college EFL learners. Through mixed methods such as the Think-aloud protocol (TAP) and a questionnaire survey for extracting descriptors for the multi-level-rating scale, Multifaceted Rasch Analysis for its validation, expert-coding/judgement for constructing Q-Matrix and sequential-GDINA (a polytomous CDM model) used for model-fit analysis and generating the diagnostic profile, we obtained the following results: 1) The descriptor-based polytomous rating scale was reliable and valid, which could distinguish rating performance across raters; 2) The Q-Matrix coded by the experts proved to be acceptable in model-data-fitness; 3) Sequential GDINA Model had a better performance for relative fitness compared to P-DINA Model. As a result, the newly constructed model produced finer-grained diagnostic information with A2 (organization), A5 (mechanics) and A1 (content) as better mastery attributes and A4 (language use) and A3 (vocabulary) as poor mastery ones at group level. Individual diagnostic reports with strengths and weaknesses in attributes mastery patterns are available to each student in the form of descriptive evaluation as well as bar charts, making it possible for individual students to receive the targeted remedial instruction against their weaknesses. It shed some light on personalized assessment and "tailored" EFL writing instruction.
