National Adult Language, Literacy and Numeracy Assessment Conference

Believe it or not – computers can be used to score writing!

Believe it or not – computers can be used to score writing!

Presenters: Martina Bovell, ACER

Friday, 10 May 2013

Breakout Session 4, 1.30 – 2:30pm

Automated scoring is becoming increasingly common in the assessment of writing ability and the technology has been utilitised by ACER in the development of the Core Skills Profile for Adults (CSPA) Writing Assessment. This assessment is delivered, scored and reported directly online. The reports are returned immediately to users and provide summative and formative information, referenced to the ACSF, about individual and group performance.

For any assessment, it is incumbent on test developers to demonstrate whether the score a student is assigned bears a relationship with the actual performance on the test Although there is an increasing body of evidence that the automated scoring of writing can replicate human raters’ scoring at least as well as human raters can replicate each other’s scoring, there are widespread concerns in the research community about its use. For this reason, it is particularly important that the quality of score replication for all contexts in which automated scoring is used is examined and reported.

This presentation describes the approach taken to develop the CSPA Online Writing Assessment. It seeks to answer such questions as:

How effective is a machine at replicating the scores of human raters? Are there differences in its effectiveness across different task types and across different criteria?
What are the drawbacks of machine scoring of writing?

Back to the conference program

Sponsors and Supporters

Vocational, Adult and Workplace Education

For more information about ACER's expertise in Adult and Vocational Education, including Research and Assessment Services, visit the VAWE page

AQTF Quality Indicator Service