E-assessment - E-standardisation
Support grows for online approach
Online technologies are a growing influence on assessment processes, from script distribution to marking, but does technology bring efficiency at the expense of reliability?
Words Julie Nightingale
As reported in the last issue, AQA has been examining the question of how reliable e-assessment can be in a large-scale research study of its online standardisation initiative, which began with pilots in summer 2007 and will be rolled out fully over the next three years.
Online standardisation takes the examiner training process online with scripts, marking tasks and feedback all channelled via a secure website (Make the Grade, summer 2008). Now the awarding body says it has gathered hard evidence that quashes fears that the electronic process is less reliable than the conventional form of standardisation.
The research
The pilots of summer 2007 and winter and summer 2008 have themselves already been scrutinised for indications of improved reliability or otherwise.
However the numbers are small and the data was insufficient to provide definitive conclusions.
The new, large-scale, designedstudy included 89 examiners, half of whom worked with the conventional standardisation process and half of whom used online standardisation. In effect the 'paper' examiners acted as a control group. Examiners in both groups worked on three questions from a GCSE History paper from summer 2007 that none of them had previously seen, requiring a mix of short and extended answers.
"We chose history to avoid any sense that the online method alters the nature of the questions," says Suzanne Chamberlain, senior research officer at AQA.
All participants marked the same set of 60 scripts in the same order - 30 before training and 30 after training. After training each participant also marked 10 scripts to practise their application of the mark scheme and received feedback.
The results showed that the face-to-face group was no more or less reliable in its marking than the online group. In other words, the quality of standardisation is not strained by putting it online.
Analysis of the online standardisation pilots had yielded similar results but the new research is more rigorous and more meaningful in terms of size, says Chamberlain. "It is a key finding for us."
On the surface, the face-to-face group had appeared to be out-performing colleagues in the online group, in that the accuracy of their marking was more improved. However, on closer inspection, the research team spotted that although the examiners were randomly allocated to either group, those in the online group coincidentally had a slightly better track record in terms of reliability.
"It was important to establish that in advance of the analysis because we needed to know what level they were all coming from," says Chamberlain. "Initially it seemed as if the face-to-face group had made a bigger leap but that was because they were coming from further away than the online group."
Following the research, AQA set up two focus groups to take further soundings from examiners involved in the study on how they felt about the online process. In the pilots, some examiners had voiced fears that the shift to online training, removing the need for examiners to meet principal examiners and colleagues face-to-face, would undermine their performance, while elsewhere, examiners have mourned the potential loss of the social side of standardisation meetings and the chance to have a day out of school.
"The face-to-face group was no more or less reliable in its marking than the online group "
In fact, says Chamberlain, the social aspect of the process was not an issue. But examiners were concerned about the prospect of losing the professional development opportunity that those faceto- face meetings provide.
"We talked to examiners particularly about the impact of this change on their professional identity. What came out most strongly was that examiners derive huge benefits from the role, most importantly, from the professional insight that they gain into the examination process. All the benefits of working with exam papers and the marks scheme are then transferred to their students and centres. Some feel their performance as teachers also improves through being an examiner."
Impact on the profession
To compensate for the loss of such an important professional development forum, AQA is planning to establish an online community similar to those offered in other professions, where examiners can pose questions and exchange ideas, allowing the exchange of professional insight to continue. The focus groups also expressed concerns about the move to online processes for examiners in general although Chamberlain attributes this in part to the fact that the standardisation process has remained unchanged for many years.
On the plus side, some examiners felt that the online method potentially offered them a greater sense of ownership of the process, in terms of being able to control where and when they carried out the work, she adds.
She brushes aside any suggestion that the research had to yield positive results, given that AQA is already well underway with online standardisation. "No, we were prepared for the finding that the online method wasn't reliable," she says. "We would not have invested the money and effort in a research exercise otherwise. It was vitally important for us to have this solid, robust evidence base."
AQA is pressing ahead with other electronic reforms, including e-portfolios, which should enable schools to submit any material that students have generated for assessment in any format.
Mike Cresswell, AQA's director general, is adamant that new technology is not in itself a threat to standards of assessment. "The primary consideration always has to be quality and validity of the assessment and there is nothing in the technologies that are coming through now that would undermine that. In the end, it's about how you use technology."
This is true even in the case of subjects which are especially complex to mark, such as History and English literature, he says.
"Work we have done suggests that with appropriate technology, which enables an examiner to annotate a script on screen, make comments and highlight errors, just as they would do with pencil and paper, you can do the subjects which require long answers."
That being the case, Cresswell says, "our aim is to e-standardise and e-mark everything, provided the quality is maintained."
Full marks for speed
Geoff, a principal examiner for AQA in GCE Geography B, has been involved in AQA's online standardisation pilots, and says his examiners appear to have found the process straightforward. One or two were slow and went beyond the 'standardisation window' - in other words the time allotted for the work - but the team did get on to the prime marking much faster than with conventional standardising.
He says that possible advantages of online standardisation include enabling a team leader and the assistant examiner to view scripts and the marks awarded simultaneously on screen while discussing them over the phone. Potential downsides include the loss of nuances.
"Subtle points that emerge during discussion can never completely be pre-empted," he says. "Assistant examiners learn a great deal during the face-toface meeting; they get a real 'feel' for the standards that I want them to apply. This is very difficult to impart online with five scripts."
The meeting of examiners is generally a "useful source of Inset [training]," he adds.
Online standardisation can be very effective, he emphasises. "The [Geography B] paper was marked within two and a half weeks; it works. But you do lose something; this 'something' is very difficult to quantify but it may be to do with the confidence that the members of the team have in dealing with the task."
He also suggests that the electronic method may also prove a less attractive proposition to new examiners.
"I fear that it may be more difficult to recruit and retain good examiners and good geographers if they only interact with a screen."