IEA National Conference Proceedings

IEA Conference Day 1

Keynote Address: Assessment of, as and for learning. Can we have all three? - Issues raised at the subsequent workshop.

1. Accountable assessment takes place against a backdrop of competitiveness within a global economy. Does assessment in the UK therefore have to be placed in a global perspective?

• The currency achieved by learners need to be transferable across nations and learners need to understand that they live and compete in a global economy.
• Internationally, nations are looking to develop skilled workforces and assessment needs to take account of this.

2. For the changes envisaged by the speaker to be effective, there will need to be a long lead-time for new policies to be effective - perhaps 3 -4 years.

• Do we have the time for this?
• Are the current changes, such as the introduction of the new Diplomas and Progress Tests ways of embracing that change or are they just another initiative?
• Most learners still view the assessment system as a way of apportioning grades (This is what I have achieved) rather than viewing the system as a way of expressing skill sets (This is what I can do). The accountable system itself promotes this view and needs to change. It is embedded in the way we use grades as ways of entering university.

3. The current assessment system focuses on curriculum content and its mastery, rather than the skill sets needed by learners to be effective citizens. The view proposed by the speaker addresses this issue.

Seminar 1: Using e-assessment for low stakes formative purposes in primary schools

1. Background

• Seemingly irreconcilable forces appear to be working against each other
• The first is the easy access to computer technology and e-assessment that affords rapid analysis of data
• The other is the move towards personalised learning and assessment, embodied in the way assessment in the classroom allows teachers to focus more effectively on the learners' needs

2. Apparent contradictions

• The emphasis on teacher assessment is apparently at odds with the increasing use of e-assessment
• However, the two can be complementary when the data from e-assessment is used to help teachers focus on teaching.

3. The scope of the project

• Attractive and easy to use for the learner
• Reflective of a broad and rich curriculum
• Make full use of the dynamic power of e-assessment in a formative context

4. Summary

• Deriving useful formative assessment details for teachers and learners to analyse
• Learners have enjoyed the process
• Teachers have been supportive on the whole
• Technological problems have been the worst issue to resolve
• There is more work to be done on the reporting system

Seminar 2: KS3 ICT Tests - a formative assessment

1. Background

• This is a new remit for the KS3 ICT Project
• In line with NAA's vision for new assessments

2. Task overview

• Purpose behind the tasks is to define the level of the skill of the application and not the level of knowledge or understanding

3. What are we assessing?

• Recording and mapping progress to ICT capabilities
• Gathering evidence of ICT capability
• Reporting on that capability to pupils and to parents

Seminar 3. Practitioners give an overview of introducing Monitoring Children's Performance (MCP)

1. MCP provides data that teachers can use in a variety of situations, such as reporting to SMT or reporting to paents. MCP allows for progress within levels. The pilot has involved 100 schools in 12 Local Authorities.

2. The case study comes from a school that was put in special measures in 2005

• The school places a strong emphasis on pupil progress
• Contextual and NCT assessment data did not provide the school with the information it needed on a regular basis, but MCP did
• MCP has been so successful that the school was taken out of special measures within 10 months. MCP has all the advantages of a modern SAT NAV system, directing both teachers and learners in the right direction

3. The process was straightforward.:

• A moderator made an initial contact and each teacher chose 6 learners on whom to concentrate. The process was explained and emphasis was placed upon the assessment focus (AF), the importance of evidence and its use.
• Teachers used the techniques with other children
• Moderator's input has allowed staff to widen their focus, the original 6 acting as benchmarks

4. The next steps will be to implement the lessons learned from the MCP pilot to the whole school

• The pilot has been funded by DfES and will appear on the primary strategy website later in the year
• Further consultants will be trained and made available from Spring 2008
• MCP has proved a manageable and scaleable methodology

Keynote Address: E-Assessment and Formative Assessment

1. If national policy raises achievement and motivation, how can summative assessment be used formatively?

• How can we take the fear out of assessment for the learners?
• How can we use formative assessment diagnostically
• What benefits can grade descriptors show?

2. How can we move from the instrumentalist application of techniques, as currently happens, to the embedding of techniques that become fit for purpose?

• How can we ensure that this happens?
• What quality assurance procedures are needed?
• How can we stop political targets, for example for inclusion in FE, militating against good assessment practices that will ensure deep and meaningful learning?

Seminar 1: Bring back the APU! Is this invoking a golden age that did not really exist?

1. The original purpose of the APU was to provide :

• Formative data (diagnostic for both pupil and teacher)
• Summative data (feedback for both pupil and parent)
• Evaluative data (information at LEA and school level)
• Informative data (information at a national level)

2. The National curriculum Tests have steadily added different purposes

• Using NC Tests to measure standards over time has been frought with difficulty
• There is no stability in the curriculum content
• This has caused instability in standards within the testing system
• It has resulted in teachers teaching to the test

3. The APU

• Similar sampling models used across the world
• Low stakes testing that provided anonymity for pupils but a wealth of data for government
• Concentrated on /English, maths, science, modern foreign languages and design and technology
• Tests were short and administered by teachers
• Nevertheless there were issues with this methodology

4. Benefits

• Good comparative performance indicators over time

5. Disadvantages

• Difficult to measure and compare standards over time because of the context of the assessments, which changed over time

6. Key Lessons

• Establish a precise baseline
• Ensure stability of the test instrument
• Limit the purposes to which the resulting data is put

Seminar 2: The importance of accuracy in assessment for learners

Accuracy

• Meets student needs effectively
• Gives teachers confidence
• Important for accountability

Starting Point

• Set high expectations
• Offer structured support
• Development within a framework
• Manageability

Development

• Offer high quality training that is quality assured
• Offer differentiated support
• Develop rigorous monitoring
• Offer quality moderation processes
• Review working practices
• Develop independence of schools with support from LA

Key Information

• Support for schools offered through highlighting key features of accurate assessment
• Collate assessment calendars
• Circulate a fortnightly newsletter
• Dedicated area on the intranet
• Generic starter power point training slides
• Organise an assessment for learning conference where schools carried out an audit of current practice which led to the clustering of schools according to their current practices

Changes to KS1 from 2004

• Close consultation with headteachers
• Raise confidence levels of teachers
• Track the progress of learners within each school

Using Foundation Stage Data

• Not as a predictor of future attainment, but as a way of reviewing pupil progress and addressing issues of teaching and learning

Seminar 3: What works in assessment?

1. It is important to begin with the learning idea, rather than the subject area. There are two distinct approaches:

• The analytical approach
• The empirical approach

2. The analytical approach looks at models of cognitive learning. In e-learning the instructional strategy of the software is crucial

3. The empirical approach relies more heavily on self and peer assessments.

• Important because in the learners' lives, the teacher cannot always be there.
• Induction and team training are important facets of this approach
• The group of learners who have most need are those who do not know they need help or are frightened of failure.
• To help all learners in self and peer assessments, it is useful to allow learners to work in self-selecting pairs.
• Such collaboration is a useful skill to take into the adult working world.

IEA Conference Day 2

Keynote address: Why accountability in assessment is a crucial issue

1. What do stakeholders want and how do we make assessment appropriate so we are not teaching to the tests but are testing skills that are needed?

• Pupils- most important; leading establishment (VI form colleges); gaining knowledge; assessment appropriate t needs of learning;
• Parents/carers need to understand assessment. Receivers; educate; employers;
• First question is to ask what purpose of assessment is, then identify stakeholders; what their needs are and how we achieve them;
• The key stakeholder is the student

Do stakeholders know what skills they want to be assessed?

• Starts at the wrong end- stakeholders don't often know what skills to assess - quite a worry. Start with skills to be assessed (stakeholder need) then make assessment appropriate accordingly;
• Need to establish what skills the learners have acquired, this is the feedback is what's needed - too much focus on what learners cannot do;
• Working with stakeholders to identify the skills they are seeking
• Teaching to tests is an easy trap to fall into;
• Analysis of KS2 & 3 progression - detail of how students dealt with specific questions.
• Feedback to teachers would be helpful in terms of what is needed to progress;
• NAA 2008 to introduce item level data, easily accessed to replace current 'blunt' feedback;
• Receivers of students are in global economy so students will need different skills for workforce, life-long learning - i.e. 'solving' skills needed; - 2 way link to assessment needs fed by employers then assessment can reflect needs of workforce skills;
• Assessment is not a science but an art;
• Need to help teachers understand their role in this;
• A real need to support the assessor and provide confidence;
• Should be more in PGCE courses on assessment;
• We live in a voluntary system - the markers and moderators, it is a voluntary and not obligatory - many more non-teachers involved;
• Need to look at teacher contracts and their involvement;
• Issues needs to be challenged;
• Fewer and fewer practicing teacher practitioners in this arena;

2. What is the main purpose of assessment of school performance?

• Identify best way to take testing forward;
• Testing what we value and valuing what we assess;
• Accountability can drown out other agendas e.g. what student feedback is needed; is not most important; confusion over formative & summative assessment - terms not being used accurately recently;
• Rush of more bureaucratic system - continued summative assessment is not better than continuous formative assessment;
• Concern over 'progress tests' - term implies more assessment. Need alternatives from more assessment;
• Feedback make it positive rather than negative . success and feed this back;
• Validity vs. reliability - impact to students should not be forgotten;
• TA should be valid therefore do we need external tests to measure students progress;
• Support for examiners - high expectations but it has to be done in spare time -Secondments; professional support; financial support;
• PPA time for examiners;
• Moderator/coursework adviser - pupils should understand assessment objectives. Assessment is a 'celebration of achievement' over a period of years. Encourages teachers to ask students how to better deliver a course - students engaging in their own learning;
• Headteachers view - every 4-6 weeks teachers are producing valid assessment of pupils in secondary schools, not annually. Teachers (esp. NQTs) need assessment support. In school moderation only every 2 yrs, feedback from 4-6 wks; day to day or month to month is the real demand not national test data;
• More time given to implement tests not considering curriculum implications;
• Time; Money; Training to get design right and then will be easier to implement;
• Assessment - done for to students rather than for them including teachers;
• Consult; acknowledge teacher expertise and professionalism as assessment experience sharing good practice;
• Teachers seeing examining as area of expertise;
• Students driving their own learning and understanding the process;
• Formative assessment is now the measurement - teachers need to have evidence that they are making valid and reliable assessments;
• Day to day support;

Seminar 1: How would the use of overseas markers contribute to the accountability agenda?

1. This is a relatively new field of research

2. For tests and assessments to be seen as reliable, they have to be fit for purpose.

3. The UK uses a single marking system, unlike the US. The results of KS marking for maths were encouraging when looked at from the perspective of inter-marker reliability

• Research now needs to extend beyond maths to look at English
• There is a need to investigate what makes a good marker
• We need to ensure that the capacity for change exists. For example, will there be sufficient reliable markers for introducing Progress Tests in this country.
• There needs to be an investment in technology to facilitate the transfer of scripts

4. In a global economy, overseas markers could help global accountability and global currency in assessment. UK qualifications are used in over 100 countries. Those teachers who teach globally would participate in future marking opportunities if the technology and the regulations permitted.

Seminar 2: Selecting reliable markers - some studies in UK public examinations

1. What makes a good examiner?

Can we predict who will be a reliable marker!

Examiner criteria - delegate views:

• Knowledge
• Reliable
• Practising teachers
• Good time management
• Flexibility

Professional view:

• Very difficult to come up with factors for a good marker
• Relevant experience in subject - not sort after skills such as reliability/good time management;
• Some ideas on NAA and AQA websites;
• Are these criteria relevant especially around recruiting teachers;

2. What makes a good marker?

• Consistency
• Severity of leniency
• Severe but consistent = ok= often inexperienced need training;
• Inconsistent = unsatisfactory
• Examiners graded A - E by senior examiners; judged by
  • Return scripts on time
  • Severity
  • Consistency
• Length of time in jobs does not necessarily give a guide as to how good a marker is;
• Training is key to having good examiners;
• At KS3 English research suggests that in the main non-teachers can mark as well as teachers.
• Perhaps present criteria not appropriate;
• Good markers stick to mark scheme;

Question - are there differences across subjects?

Statement - importance of marking criteria.

• Background
• Subject knowledge
• Attitudes/motivation
• Personality traits - always important; markers do not need to have social skills;
• Age/gender - can have an effect but not generally a factor in consistent marking;
• Some evidence show that training can reduce consistency in marking shorter answers, but in longer answers training improved consistency;

In the research no evidence as to how long it took markers to complete marking scripts was taken into account and how this affected closeness to senior markers needs.

• Quote from delegate 'Conscientiousness can really screw a marker up!'
• Do factors identified in 3 day trial change over a longer ¾ week marking period;

Seminar 3: Introducing the specialised Diploma and its assessment regime

1. The purpose of the diploma:

• Breaking down of the vocational/academic divide;
• The diploma is an opportunity to bridge the divide;
• Why is it different?
• Range of new approaches being introduced;
• Applied and ac learning;
• Not competitive - employability not jobs - levers will be 'job ready' but not vocationally competent;
• Composite qualifications - to provide speciality and breadth
• Employers actively engaged with education in the development;

2. Who is it for?

• 3 levels for different learners - choice, coherence, appropriate curriculum;
• Level 1- provides the foundation to progress to level 2;
• Students more likely to be engaged by alternative programme;
• Both levels are part of post 16
• Level 3 - for those looking for different route into higher education, wanting comparative and applied curriculum;
• Being rolled out until 2013, then will be an entitlement;

3. GCSE/GCE - Diploma - Apprenticeship

• 14 lines of learning to be delivered by 2010; development been accelerated from timeline see out and White Paper, rolled out in 3 phases;
• Science may be added at 15th line of learning will be later than the others;

4. Components: Principal learning:

• Completely new, currently being developed by employers;
• Sector based
• At least 50% is applied learning;
• Breadth across the sector

Generic learning:

• Functional skills
• Personal, learning and thinking skills
• At least 10 days work experience
• A project
• Planning and reviewing

Additional and specialist learning:

• Allows learners to go deeper into a particular sector;
• Or may allow learner to add breadth e.g. by adding GCSE/GCE;
• KS4 delivered alongside diploma for post 16's;
• L1 - roughly the same size as 4 -5 GCSEs so 1 year course post 16, 2 year post 16;
• Progression to L2 usual route rather than employment;
• L2 - volume to 5 - 6 or possible 7 GCSEs - 1 -2 year programme leads to employment with training or full time work;
• L3 - comparable in the length of study of 3 A levels, leading to employment with training or further higher education;
  • L1 - 600 GLH (guided learning hours)
  • L2 - 800 GLH
  • L3 - 1080 GLH

Personal Learning and Thinking Skills (PLTS) platform for employability skills;

• Valued by employers;
• So much emphasis on this area;
• Gives diploma its distinctiveness;
• Embedded into principal learning;
• At the level of the qualification;
• If leaver passes principal learning, they will have passed all 6 PLTS (Personal learning and thinking skills);
• 60 GLH set aside for PLTS;
• Teachers will make PLTS evident within the learning;
• Phase 1 qualifications currently being accredited at QCA. They will enter the diploma catalogue by September 2007. These will be available to centre who have [assed the gateway process.
• Will be graded, based on principal learner and the project looking at whether additional and specialised learning can be included in the grading;

5. Assessment of PLTS:

L1 -30 GLH externally assessed

L2 - 60 GLH

L3 - 120 or 180 GLH depending on sector

• + Functional skills (discrete)
• + PLTS (within Pl)
• + Project internally assessed and moderated
• + Work experience (10 days and reported in the Diploma transcript)

6. Assessment of Additional and specialist learning:

• Follow existing assessment arrangements
• Implications for professionalising local assessment questions

7. Issues, questions and answers

• Where does BTEC fit in and will it still exist?

A & S learning will include best of existing qualifications, so BTEC and other good qualifications will be included in the diploma and still offered post 19.

QCA looking at characteristics of successful qualifications and incorporating them in diplomas.

• How far are the needs of the learners being built in, are they being consulted?

QCA represents the need of the learners; they are the key focus of QCA. National consultations taking place using focus groups, which include students.

• Concern about the number of 'hurdles' to be achieved to gain the diploma. Will this cause large number of students to fail?

It is a demanding qualification, however even if they stumble at one hurdle they will receive skills for what they achieved.

• Need for teaching materials, teacher and assessment training.

Who is leading the development of teaching for the diploma?

DfES have workforce development programme via QIA & SSAT, have also appointed 14 -19 coordinator;

QCA are producing curriculum models;

ABs are producing sample assessment materials

Keynote Address: Will good assessment practice ever tame the tiger of accountability?

1. Good assessment practice:

Model of KS1:

• No stress
• Moderated
• Teachers assessment trusted;
• Need to take in account 'love of learning'
• Reduction in summative assessment
• Evidence around accuracy of Teacher Assessments being as good as summative tests? = good news;
• 'Daily Mail effect'
• Examples around 21st century science - more 'student' based; teacher led subject and assessment;
• Example of CSE mode 3 - teacher devised; teacher led; teacher assessed;
• Bad effect of league tables on teaching and learning, leads to lack of trust between primary and secondary schools;

2. Can we design a supportive assessment system?

• Purpose - for learning and learners e.g. transfer etc
• Strong potential but weak in practice - there is no alternative, changes within assessment will be slow and must be linked to teacher CPD;
• Need for teachers to be given TIME in class to change a 'result driven' curriculum, need for teachers to change their attitudes to tests/exams/assessment;
• Need for task involvement;

3. Fit for purpose - assessment must be dependable

• Value for money/value added
• Synergy not contradict

4. Assessment for Learners;

• How reliable are present tests?
• Lack of reliability = misguided decisions/unfair to learner 30% errors?
• Making good progress - multiple attempts = better results;
• Many tests are not valid - they do not test what they try to test;
• Need to use TA's - just as reliable;
• At present teachers teach to test, need to improve summative skills;
• Also present system encourages pupils to be competitive; stress = low pupil esteem = poor performance;

5. What is it like now?

• Focus on accountability of institutions;
• Data of limited range of validity
• Monitoring of national standards is weak;
• Teachers skills and status devalued;
• Interest of schools (to produce good results) in conflict with interests of pupils (Some schools are focusing on 'borderline' students in year 11).
• Improvement - slow & long term;
• Strengthen formative - ongoing assessment

Seminar 1: Towards an integrated framework of CPD for teachers in Scotland: SQA's role

1. Putting the learner first has been the thrust in developments in Scotland, with the key players working in a coherent and collaborative manner.

2. The key players are:

• CPD Scotland
• GTC (S)
• Learning Teaching Scotland
• SQA

3. CPD built into a teacher's entitlement. This allows 35 hours per annum, which can be accessed in a number of ways to suit the individual.

4. GTC(S) has a close focus on standards within the teaching community - NQTs, Chartered Teachers, Chartered Headteachers. The GTC focuses on professional recognition. The Chartered status is an attempt to keep good practitioners in the classroom, enhancing both pay and status.

5. Learning Teaching Scotland focuses on teaching and assessment, auditing the school, auditing the practice of individuals. It provides a web portal that all student teachers can access and add to a CPD e-portfolio that links directly to training.

6. SQA is smaller than its English counterpart and aims to facilitate collaborative working. It has developed the SQA Academy - a virtual learning environment for both SQA staff as well as teachers. It contributes to the AifL (Assessment is for Learning) initiative. It promotes this initiative through its virtual learning environment

7. The issues that have been raised and addressed:

• National consensus is important for the development of effective CPD
• GTC(S) standards are recognised by everyone as being appropriate.
• There is a desire to work together and de3velop a coherent approach.

Seminar 2: The challenge of accountability from a regulatory perspective

1. Background

• QCA's remit is to keep under review all aspects of maintained schools curriculum and the national examination and assessment system
• QCA is required to show that assessments are fit-for-purpose

2. Problems

• Fitness for purpose in an assessment system is highly complex
• Communicating this to the public is therefore challenging
• Transparency and openness about the quality of the assessment system is therefore full of difficult issues.

3. Understanding purpose

• A range of purposes for both NC Tests and public examinations that lie beyond the original purpose
• Evaluation itself has a number of sub-sections
• How do we communicate all this to the public?

Seminar 3: The principles of assessment and Making Good Progress Tests pilots

1. Background

• Government concerns over the slow progress of some learners in English and maths

2. Proposals

• Assessment and testing for progress
• Progression tutoring
• Progression Targets
• Progression Premium

3. Purpose

• To align teacher assessments more closely to external tests
• To enable schools to focus on individual progress
• To allow all stakeholders to have a shared understanding
• To investigate different forms of assessment
• To provide reliable data