I want to take a bird’s eye view of the report and summary that Ofqual published HERE. So let’s remind ourselves “what’s it all about Alfie!” … I’ve listed the four studies with some of the key points from the findings relating to each “study” along with any comments from little ‘ol me. I’m not really sure where this post is going to take us … here we go … “hold on tight”!

I’ve gone back and added things from the final “discussion” so that they are under the relevant area of study, as some points were made in the final part but weren’t necessarily listed under the findings. I am also going to warn you that this is the longest post I’ve ever written!! Way too long!! It’s that flipping blogging Tourette’s … I swear I’ve tried to cut it down but feel all of it is relevant!

Buckle yourself in ….

Study 1: Compared the expected difficulty of all questions from exam boards’ sample assessment materials (SAMs) and also included a comparison with questions from recent GCSE (allegedly!) papers and with similar international qualifications.

• The current GCSEs were judged to be easier than similar international assessments (Mel: duh! .. isn’t that part of the reason driving these changes .. talk about stating the obvious!)
• I quote “in general, the expected difficulty of the reformed higher tier GCSE sample assessments was more in line with international assessments” … (Mel: ummm loose use of the phrase “in general” )
• There was a disclaimer in the “discussion” section about the danger of comparing internationally but went onto say “ That said, the expected difficulty of OCR and Pearson’s higher tier assessments was more in line with those of international jurisdictions such as Shanghai and Japan than with the expected difficulty of the current papers.”
• It is also worth noting (and no doubt I’ll mention it again) that the intention of the DfE was to bring us in line with international exams.
• There is a note that comparing the GCSE with similar international assessments will be the subject of a future report and I look forward to that.
• The difficulty of the OCR and Pearson SAMs was higher than that of their current exams (Mel: shouldn’t that be what we expected to happen?)
• AQA’s sample assessments were very similar to that of their current GCSE but does say that AQA current GCSE is harder than OCR and Edexcel
• AQA questions tended to be the easiest “whatever the assessment objective that they were intended to measure”.
• The differences in difficulty between boards were judged to be greatest for AO1 and AO2 rather than AO3.

 Study 2: Compared the difficulty of questions from the non-calculator sample assessments across the exam boards which included aggregation to the level of the whole question papers.

• Students performed better on AQA’s papers, on both old and new content (but they also has less new content in their SAMs)
• On the foundation tier AQA was easier
• On the higher tier Pearson was considerably more difficult (Mel: Define considerably?)
• Students found the papers difficult and as they progressed through the papers the non-response rate got worse BUT even content from the current GCSE was lower than “one would expect on a real examination”.
• The assessments failed to “sufficiently differentiate between students of differing levels of ability” and so would not be feasible to grade students on the basis of these assessments. The report states that it is impossible to disentangle the effects of motivation and difficulty on students’ scores. This part included all the assessments with “the possible exception of AQA foundation tier” … well, did it include it or not?
• Only the non-calculator papers were used … in fact this represents roughly 1/3 of the examinations (the other two papers are calculator based). This is a major issue for me to reconcile in my own mind as I know that (a) my students perform better on calculator papers in general and (b) there are a whole raft of topics that are calculator only!!

Study 3. Compared the extent to which questions elicited mathematical problem-solving across the exam boards.

• that there was less consensus between the judges as to which questions best elicited problem solving.
• The differences between exam boards were more substantial on the foundation tier papers than on the higher tier papers.
• Eduqas AO3 questions were better at eliciting mathematical problem solving than the other exam boards’ items on the foundation tier.
• On the higher tier papers, the judgements were far more similar across exam boards
• There was no relationship between the extent to which the items were judged as eliciting AO3 and their expected difficulty.

Study 4: Looked at the ways in which problem-solving questions vary across exam boards’ SAMs.

• Eduqas’s questions “were more likely to require open-ended answers, to require candidates to justify their answers and methods, to allow multiple approaches to solving a problem and to use demanding language (although not more so than OCR’s items)”
• AQA’s items included less text than the other boards’ questions (although not significantly more so than Pearson’s items) … which is interesting considering the claim from AQA about being that less text heavy that was being bandied around (I vaguely remember a figure of 22% less). I do know that at least one of the boards hadn’t yet put their papers past a language expert.
• In my opinion none of the boards came off well with regard to problem solving but a lot of that is down to the fact that there is no really clear definition of what problem solving is in an exam context (hence the clarification that is being sought through the latest consultation). I genuinely feel really sorry for Eduqas though who did better than the rest of the boards gets absolutely no credit for this in the summary

There is a section of the report titled “discussion” and not “conclusions” and opens with a reminder of the policy objectives of reforming GCSE maths from The Department for Education (DfE). This serves as a very real reminder of the issue of having two departments involved in this. I really feel for the exam boards … the goalposts have been set up by one government department and then moved by another (TBH I’m not even sure they are the same pair of goalposts!) This part of the report makes a point of reminding us that the subject aims were for students to have “confidence and competence with mathematical content when they can apply it flexibly to solve problems.” There is even a hat-tip to Gove’s letter to Ofqual back in Feb 2013 and refers to an ambition for an “increase in demand, to reflect that of high-performing jurisdictions.”

What is good is that there is an acknowledgment that the new GCSE maths covers more content than the current GCSE; however this is almost taken back with the phrase that “In one sense, the new maths GCSE is immediately more challenging in that more material must be covered.”

The fact that the foundation tiers showed a greater spread of difficulty than the higher tier is an important point I want to mention. I believe that until the DFE published further guidance (last year when the SAMs were going through the accreditation process) there was a belief that effectively part of the grade G equivalent was going to disappear – it is after all about raising attainment isn’t it? I genuinely think that is what Gove wanted … and it wasn’t until it was clarified that the bottom of a level 1 is the same as the bottom of a grade G that at least two of the boards were working on that basis. I genuinely feel that they interpreted this correctly … but when the SAMs got bounced back it was a result of someone realising that effectively if that were to happen effectively an extra approximately 3.8% of students would get a U. Imagine going into an election knowing that they would be failing an extra (lets be conservative with the estimate) 25,000 students . Very early on, after having seen the first lot of SAMs any maths teacher could have told you that the foundation tier for certain boards didn’t have stuff that could be accessed by the lower ability. It is also as a consequence of the content shift from higher to foundation tier of topics that have traditionally been higher tier that this has happened and this hasn’t been mentioned greatly. …. Teaching these traditionally higher tier topics to foundation students requires a mind-set shift. I have to be honest we went down the higher tier route for most students years ago (as we didn’t want to limit the access to the more exciting maths) and I could share with you stats that show great progress (dare I say exceptional!) from students that are classes as lower attainers as a result. I have written loads of times about why we’ve use this strategy and it’s led to amazing results for those students.

The key thing to highlight is that if the intention was to make GCSEs harder, then the skew that has happened in OCR, Pearson and Eduqas’ SAMs with regard to the student testing should absolutely be expected, but no reference to this is made in the Ofqual statement, in fact it has been used as a reason for why they are not adequate in some way.  It seems there has been no reference to the fact that the exams are not aimed at the current cohort … I refer you to what I’ve always said “don’t think about these changes in the context of our current year 11 but about what you can do with a year 7 given time to teach them the new content”.

The report then goes onto talk about the adjustment of grade boundaries being necessary to ensure fairness and mentions that any “adjustment of grade boundaries to compensate for differences in difficulty is a necessary feature of an exams system in which the (resource intensive) pre-testing of items is not conducted.”. I’ve always said easier papers = higher boundaries and tough papers = lower boundaries. Anyone that thinks they should be fixed is living in “ga-ga” land.

In the summary the report states that there is a difference is difficulty across the exam boards (MEL: duh! We could have told you that last year … and could have saved you a fortune!) and also “AQA have not increased the difficulty of their sample assessments” and as a result in the difference in difficulty there could be “negative consequences for teaching and learning.”

Following the publication of the repot and summary there have been several press stories – there are some really media-savvy people out there and in the same way that I’ve tried to pick out the main “findings”, so have they. I’ve added my thoughts, but this is a blog where I can do that and I IMPLORE YOU TO DRAW YOUR OWN CONCLUSIONS don’t just choose an exam board based on choosing the one that has got it “less wrong” than the other boards. Make up your own mind whether this it could be argued that they may have done so by not actually reforming their GCSE much! I have no issue with AQA – they were accredited in the same way the other boards were and therein lies the issue.

Anyway, I suspect that the “hoorays” may have peed off someone else because the day after reports were published an additional statement an additional statement appeared on the Ofqual website.

However on the Ofqual blog (and in the letter that was emailed to lots of schools) what they should have updated is the nuance in the final paragraph – stating that AQA need to lift “to some extent” the foundation papers. Be clear .. they either do or they don’t!.

So in short … and if you’ve got this far well done! A great piece of data-rich research with some shortfalls … the issue is now about how these have been interpreted and how people are “cherry-picking” statements for their own means. I genuinely don’t blame the exam boards in any of this, after all they have either (a) a lot to lose or (b) a lot to gain … this all lies at the feet of Ofqual and the Dfe. There is no doubt … it has been a debacle from start to finish.

Despite all that my main concern remains one of “running out of time” … this has been going on far too long. We don’t have the time to digest the new SAMs (expected by 30^th June) and what happens if they don’t get it right next time.

Do the right thing … defer it a year.