Technology - Data: the golden opportunity
E-assessment
Equal measures?
Comparability between testing methods is crucial to public confidence in the exams system. We look at the latest attempts at levelling the field
A major stumbling block has been comparability - how to ensure that paper tests and e-tests are on an equal footing; that each measures the same skills and knowledge; and that neither method offers an advantage to one candidate over the other. It is something Ofqual and e-assessment managers from English, Welsh and Northern-Irish qualifications regulators, and GCSE and GCE awarding bodies have been keenly studying over 15 months as part of research into technical issues around e-assessment.
At this year's CIEA annual conference, Andrew Boyle, programme leader for e-assessment at Ofqual, unveiled a 'consensus statement' from the group including a definition of comparability to which all have signed up: "test outcomes can be said to be comparable if features... of the score and grade distributions of candidates on both modes appear to be similar for equivalent groups of candidates, taking into account other known evidence about the test takers' knowledge, skills and understanding (such as information about their prior attainment)."
Why is comparability important? For Ofqual it is about underpinning public confidence in the whole exam system. Specifically, comparability is a key issue when people use results for a certain purpose - as university admissions criteria or to judge job applicants, for example - and do not differentiate between tests that have been assessed on paper or electronically. Comparability between these is essential if all candidates are to be treated fairly. Boyle explains: "Imagine that a paper-and-pencil test and an e-assessment existed in parallel within a qualification. If two people went for a job, the employer wouldn't want to have to think, 'Well, did the person sitting the e-assessment get an easier or harder test?'"
The consensus statement discusses other difficulties with the widespread roll-out of e-assessment, notably that it demands that PCs or laptops be made available in huge numbers at one time. As few schools or colleges can currently do that, the obvious alternative is for testing to be done in batches. But this would open up the potential for cheating if awarding bodies used the same test several times. A solution is to use different questions - but doesn't this raise further questions about comparability and equality if students are being judged together but have answered different questions? Boyle says not.
"I don't believe this approach causes major comparability concerns: even if tests had different questions they would be written to the same specification - curriculum balance, level of difficulty, and so on. Also, the statistical models available to compare question and test difficulty are pretty robust nowadays. Finally, it is possible that this will be an interim issue, and once schools have enough cheap laptops for every student to have their own, we can go back to holding tests all on the same day at the same time."
Others appear less confident than Boyle that the technology necessary to roll out e-assessment on a grand scale will be available in schools and colleges any time soon. David Glover, qualifications technology manager for awarding body OCR, says: "To a large extent, the technology that an awarding body would need to have in place in order to offer a broad range of e-assessments at both GCSE and A-level is available, and could be used more widely now. The main obstacle is the need for more technical infrastructure - high-speed internet connectivity, large numbers of PCs and laptops, computer rooms that are suitable for high-stakes testing, in schools and colleges." As such, he adds, it is difficult to predict how long it might take for e-assessment to become the norm within high-stakes national tests.
He adds: "While schools and colleges have successfully introduced technology into teaching through the use of virtual learning environments, and other e-learning technologies, it would be quite a significant additional step forward for many schools and colleges to be able to successfully administer potentially several hundred computer-based tests in a defined timetable window."
Peter Kent, head of Lawrence Sheriff School in Rugby, one of the top performing secondary schools in the country, agrees.
"A few years ago everyone seemed to be arguing that e-assessment was the way to go, but now this viewpoint is much less common because of the practical problems," he says. "It just would not work in almost any school that I can think of. You cannot both teach ICT/computing and offer e-assessment in a range of subjects - the logistical demands are too great unless you are not making an attempt to teach ICT.
"We have large groups studying both computing and ICT at both Key Stage 4 and 5, so e-assessment would decimate their teaching opportunities. The only way this could be done would be if assessment took place almost weekly. This is one of the reasons that initiatives such as functional skills have struggled. Originally it was thought that e-assessment could be used, but in practice this was found to be impossible."
Policy-makers and educationalists are divided as to whether the drive to introduce e-assessment universally is about making the system more efficient or empowering the learner.
Rose Clesham, assessment design and e-assessment manager with Edexcel, is adamant that efficiency is not the key driver. It is clear, she says, there is potential for e-assessment to increase efficiency in testing and assessment in terms of reducing the time taken for assessments to pass through the system and minimise transcription errors. But, like Heppell, she believes it offers opportunities to assess skills and qualities that cannot be assessed on paper, and to better reflect the learning and pedagogy used in education.
She adds: "Increasingly, students access and demonstrate learning using many forms of technology. We need to ensure that summative assessments reflect these skills. Efficiency is a driver here, but only if we can retain or enhance the validity of
our assessments."
Weblink
Consensus statement on technical issues in e-assessment:
www.ciea.org.uk/upload/conference_2009/presentations/consensus_statement.pdf
Expert view
Click onto the future
There are questions about whether e-testing may discriminate against some students, and not only those with physical disabilities but in unforeseen ways. Professor Stephen Heppell (pictured above), a leading authority on the use of technology in learning worldwide, points to the example of the driving test where the introduction of on-screen tests has effectively barred people with lower skills in literacy from taking the test.
"We have to be really careful about how we introduce this, in terms of inclusion especially,"
he says.
"It sounds very simple, but we have seen that it can be a real problem in some areas. We have seen some unexpected disasters with e-assessment. For example in the introduction of computer tests in the driving tests, which led to those who can't read and write not being allowed to drive - thus criminalising Travellers and others because they will continue to drive nevertheless. I don't think you need to be able to read and write in order to drive."
Nevertheless, he is a huge supporter of using technology in testing because it facilitates the kind of assessment that students will experience in real life.
"Technology lets us be relevant," he says. "When I talk to captains of industry, no one has done a timed essay in years - they do presentations. That combination of show-and-tell is a very appropriate way to assess people. The current [pencil and paper] model of assessment doesn't map on to where we use assessment in the real world.
"The traditional model of assessment of criteria and referencing worked very well in the last century because nothing changed very much. But it is under-ambitious because you are expected to do no better than what has gone before and we are not going to get better by doing that. If the Olympics was the same and you only ever threw the javelin as far as someone else had thrown it, it would be ludicrous."