To view the CIEA response to the Sutherland Inquiry, in pdf format click here. It is also recreated below.

The response summarises a number of CIEA member responses to an e-mail in August of this year. Individuals have been quoted, although they have been kept anonymous so that no contractual obligations have been breached.

The CIEA has also commented on the training of markers as it sees appropriate.

 

 

The Sutherland Inquiry
Sanctuary Buildings
Great Smith street
London
SW1P 3BT

6th September 2008.

CIEA response to the Sutherland Inquiry

Dear Lord Sutherland,

INQUIRY INTO NATIONAL CURRICULUM TESTS 2008: CALL FOR EVIDENCE

Further to your letter dated 15 August 2008.

The Chartered Institute of Educational Assessors (CIEA), as the leading professional body representing examiners, markers and those involved in the educational assessment of others, is grateful for the opportunity to submit evidence to your inquiry into National Curriculum Tests (NCTs) for 2008.

As an independent charity the CIEA has been established with the support of DCSF and QCA to provide continuing professional development and accreditation to those involved in the educational assessment of others, including NCT markers, with the aim of improving the overall quality of assessment.

There is little doubt that any party involved in the NCT cycle this year would have wished for events to turn out the way they have. When ETS took over the marking contract from Edexcel, who like their predecessor AQA had carried out the contract to an exemplary high standard, ETS came in on a wave of goodwill with the intention of delivering an equally high quality service to previous test agencies and to that carried out in America and across the world.

However there are lessons to be taken from the events of this year which can positively build into a more robust assessment and testing system for subsequent years. I would therefore urge that where possible your inquiry adopts a positive and proactive tone in pursuing your investigation into the events of this year to ensure that public confidence is strengthened in our education system for the benefit of all learners over the longer-term.

It may also be an opportune time to stand back and reflect on the practicalities and desirability of the current model of a high stakes, high volume testing system. Although this matter may be beyond your remit, the scale, pace and cost of the current system are high and there are serious questions to be raised as to whether or not this testing regime, and the way it operates, is best suited to the needs of our young people. It is also a system at full stretch working to incredibly tight deadlines. Therefore any set back in delivery is likely to have high impact with little or no room for recovery.

In connection with your inquiry, CIEA members were keen to express their views individually through the Institute regarding their specific experiences of marking in this test cycle but have told the Institute that confidentiality clauses in their contracts prevented them from doing so publicly, even from commenting on information already in the public domain. Although these clauses are common today in contracts this may limit the type and availability of information submitted to your inquiry.

In response to the specific issues on which you have invited comment, questions 1 and 2 require intimate knowledge of QCA's internal processes that the Institute is not party to. It is therefore not possible for us to present meaningful comment. However the Institute would urge the inquiry to carefully consider the contingency planning arrangements, the risk management processes and the escalation systems that would allow for effective management of any contract.

The CIEA would therefore wish to submit the following response to the third area.

3. The delivery of the tests in 2008 by the National Assessment Agency (NAA), and ETS:

a) the nature and extent of the failures;
From feedback received by CIEA from members involved in the marking process, problems in the current situation focus on two areas:-

1. Marker management, including communications.
2. The training of markers.

I will deal with the training of markers under 3 (d), and will look at the issues relating to the management of markers under this point.

Marker Helpline

Members have told the CIEA that they felt the support systems designed to facilitate the free flow of information as a basis for decision-making could have been much improved.

The ETS helpline to monitor and manage issues arising from markers. was largely ineffective and severely hindered the marking process.. For example, one member commented:-

'Trying to iron out anomalies on the helpline was hopeless. The staff did not seem to understand the problems that I was presenting: pupils marked down as present, but no papers received - and mixed tiers or changes of surname seemed insurmountable. Eventually I sent an email listing all outstanding issues. I still received emails asking me to do extra administrative work after the marking deadline.'

Administrative changes

The use of an electronic student register to record which students had taken which test also seemed to raise some concerns as one member told us:-

'When the papers arrived there was no list of candidates who had taken the tests accompanying the papers - this was a mistake! Instead there was a list online with all students marked as present.'

Subsequently the list that appeared on the screen did not seem to tally with the list that I remember checking. Another examiner I spoke to had the same experience. In other words I was not sure if candidates had taken the test, were absent or if the papers had disappeared.

'I tested out the email helpdesk with a query about an additional candidate but received no reply at all - and still haven't!'

Entering Marking Data Online

CIEA members informed us that markers were asked to enter their marks for candidates work, after marking their scripts, online in a data capture system

This is usual practise for qualifications provided by awarding bodies but was new to some of the markers who marked NCT tests.

'Entering marks online was desperately slow. Evidently no-one had tested this on a large scale.' One member reported.

Positive Experiences

Having said this, members told us that there were also some positive experiences involved with the process and not all markers encountered the same problems. One member told us that:-

'The good news was that ETS were very prompt at paying marking fees. My team leader was superb throughout the whole process.'

Overall experiences tend to be mixed in the current test cycle and this example is perhaps typical of that experienced by markers.

'I was disappointed with the KS3 marking, because I felt I was marking to a high standard and so many people ended up disappointed, largely through a poorly set up administrative system.'

b) risk identification, management and contingency planning by ETS and NAA. The CIEA does not have any knowledge of this and therefore cannot comment.
c) the quality and use of management information provided by ETS to NAA, and NAA to Ofqual. The CIEA does not have any knowledge of this and therefore cannot comment.
d) the administration of the marking process, including the training and management of markers;

Quality of marking

The CIEA is aware of the concerns of some educational organisations and individuals reported in the media over recent months about the quality of marking. However at this point in time this is largely anecdotal.

Feedback from CIEA members involved in the marking of NCTs has indicated that the same individuals broadly have been involved in marking the current test cycle as in previous cycles.

Common sense would indicate that wide variations in marking quality should not be the case if the same markers have received consistently the same level of support and training to carry out their duties as in previous test cycles. If the quality of marking this year is seen to be at variance with that of previous years then questions have to be addressed about the format, type and nature of training and support provided to markers and whether this was fit for purpose.

Experiences of training among markers varied, with some markers feeling that the training was very rigorous, as one member told us:-

'The initial marker training.was very thorough. We were expected to undergo further training in our own time for no extra pay.I did this extra training, but it took at least a day to do this. I actually felt confident and quite conversant with the mark scheme.'

During the training some markers felt that the answers suggested in the online training modules were erroneous.

'I marked all my papers inspired by a £100 bonus. Again there were disagreements with the 'correct' answers when I did the online reviews of marking, but I felt my marking was quite tight and that I was marking to a high standard.'

NCT marker training

Having said this, the 'training' of markers is something of a misnomer. Training as it applies to NCT markers constitutes a meeting designed to standardise the behaviour of markers aligned to a common mark scheme.

However traditionally training in the wider sense of the word is far more strategic and far more wide-reaching. Training represents a sustained intervention over time which arises from the professional development needs of the individual for the long-term benefit of the organisation, the individual concerned and his or her working practices.

The CIEA would therefore argue that the 'training' of markers is limited, tactical and confined around a discrete activity for a given purpose which limits the variability of the individual marker's behaviour in line with prescribed expectations but does little to contribute to their own professional development over the longer-term.

The reason NCT marker training has developed in this way is mainly historical. Traditionally it has been the case that teachers typically undertake the marking of NCTs. This gave rise to the common perception that 'training' could be time-bound and limited in its scope given the nature of the role, as markers were already adept at marking students work within educational establishments.

This perception that all teachers can apply assessments in a consistent and reliable manner has been frequently questioned by Ofsted for example in their last annual report, which stated that variability of assessment practice both between teachers in individual schools, as well as between different teachers in different schools, is the single biggest factor impacting upon school performance.

An alternative approach to marker training

To address the shortcomings in training would require a mechanism for developing the skills, capabilities and knowledge of NCT markers in the round in a manner which contributes to their own professional development, but is able to support greater flexibility in the system during periods of extreme stress, regardless of the specific assessment instrument involved.

This approach would help to underpin reliability in NCT assessment outcomes by quality assuring outputs regardless of content or context so allowing for greater flexibility in a system already running at capacity. It would also help to support reliable and valid assessments during periods of uncertainty as has been experienced over recent months.

Professionalising NCT markers

The current system is reliant on short term contracts for markers drawn from the teaching profession. The majority are practicing teachers so the marking commitment is short term, working to tight deadlines and usually undertaken in addition to an already demanding full time job. As a consequence, training is highly focused on the task in hand and does little to enhance the individual's understanding of assessment or their personal capability. This situation is piecemeal and misses the opportunity to enhance the professional knowledge and understanding of teachers in a way that could benefit teaching and learning in our schools.

One way of addressing issues of marker 'training' would be to strengthen the requirements of any future national curriculum test contractor to provide access and encouragement for markers to engage in a structured programme of continuing professional development, independently accredited, that would ensure their skills, capability and knowledge of good assessment practice was maintained and enhanced from test cycle to test cycle regardless of the test instrument used.

Adopting this approach, the CIEA believes, would lead to a more flexible, more professional cadre of independently accredited markers who could provide a more consistent and reliable approach to assessment, whatever the assessment.

In response to this need to strengthen the overall professional development of markers, the CIEA has developed a Professional Framework of Assessment which underpins its programme of accreditation and helps to inform its membership grades.

This framework outlines the role, competencies and behaviours needed by those undertaking assessment tasks in awarding bodies, as well as internally within the classroom. The framework is available to view online at www.ciea.org.uk, and provides a structured approach to the process of assessment and the continuing professional development of those involved in assessment tasks so that, once assessment requirements have been identified, individuals can identify their own needs in developing and sustaining good practice.

Chartered Educational Assessor

By benchmarking the skills of individuals against a nationally-recognised framework, we can then look to develop accreditation for those individuals demonstrating the highest skills in assessment practice, that of Chartered Educational Assessor.

These qualified assessors would be accredited to national standards by an independent organisation such as the CIEA, They would be capable of making consistent judgements regarding student achievement over time. Such a system would provide assurances that assessments are carried out to national standards consistently over time and go some way to supporting greater public confidence in our exams and testing system.

e) communication with schools by ETS and NAA; and
Members have told the CIEA that in some local authorities schools received results after they had closed for the summer holidays, or had only received their results electronically, and were not in possession of the scripts when they received their results, and were unable to check pupil marks accurately because these were not returned at the same time.
Some other schools have received their pupil scripts but not a full release of data relating to the whole cohort.

This has led to some concerns among teaching and local authority staff about the accuracy of returned marking information so that schools might be more likely to consider marking reviews.

f) the effectiveness of regulation of National Curriculum tests by the QCA (as regulator) and subsequently Ofqual. The CIEA does not have any knowledge of this and therefore cannot comment.

It is clear that the experience of this year's test failure will leave deep scars on an already stretched system. It is a system already under extreme stress with very little flexibility or contingency. Whilst this system can deliver, as indeed it has in the past, it is clearly operating at full, or perhaps even over-, capacity. Whilst many of the processes can be fixed, it is time for a more fundamental inspection of the system to redefine the purposes of tests and assessments and to ensure that the chosen model is fit for purpose using up-to-date technology and a highly skilled work force.

Yours sincerely,

David Wright
CEO