In search of Osiris: random item selection, fairness and defensibility in high-stakes e-assessment

Authors

  • John Dermo University of Bradford

Abstract

This paper explores issues of fairness and reliability in e-assessments where objectively marked questions are randomly selected from an item bank .In the UK Higher Education sector, there are several compelling reasons for wanting to use such assessments for summative purposes, but randomly selected items do raise concerns for students, instructors and institutions. Drawing upon latent trait analysis and computer adaptive testing, the paper advocates the use of calibrated item banks and proposes the innovative 'OSIRIS' (objective standardisation in random item selection) method of modifying student grades based on the difficulty level of the questions they received, demonstrating this with data from a large scale online assessment. The paper concludes that the OSIRIS modified grades method, in conjunction with following best practice for quality assurance in item banking, may mean that random item selection could in fact enhance the fairness and reliability of assessments, rather than being viewed as a risk.

Downloads