Yesterday Dr Vanessa Pittard (Assistant Director, Curriculum and Standards) gave a DFE update as the opening speech at a maths conference – a section of this was devoted to showing our position in the PISA tests. Not a biggie? How worried should we be that international surveys of student achievement are making such appearances?

Not much would be my response, if they were a fair reflection. What is worrying, apart from their validity is the fact that they are being used to drive through policy changes that are far reaching – many of which are made, by people who don’t question, or understand the limitations of the findings when they are presented to them.

When the reports are published in the media, they are jumped on and part of the reason why the media is all over the results like a rash is, I think, because they are presented in quite a media friendly manner, and whilst the limitations are there for all to see (they publish disclaimers!) most people take it at face value, and then subsequently most people who read the papers, believe EVERY SINGLE WORD without questioning it.

So what are “PISA” tests?

Since 2000, every three years 15 year old students take allegedly “curriculum independent” tests in three key areas: reading, maths and science. As part of this cycle there is a major and minor emphasis so that one of the three areas is the main focus effectively every nine years. In 2012 (the year when maths was the major focus) some countries also took part in optional assessments for Problem Solving and Financial Literacy.

The tests are 2 hours long and contain a mixture of multiple choice and open ended questions. A common test is not given to all students as they may be given a different combination of different tests. Additionally questionnaires are completed by students and also school leadership staff to provide background and more contextual information.

PISA rules require a minimum sample size of 4500 students except for very small countries where all students are included. About 510,000 students from 65 countries take part, so the number of students from in the UK who sit the tests is so very small (approximately 520,000 students in an annual cohort) and I suspect that they aren’t representative of the entire population.

Problems also arise from different cultures, and different attitudes to education in general and tests in particular – for example some critics would argue that East Asian countries do well because of their “deference to authority” and a drive to be successful which leads to lots of out of school tuition. Others would argue that there is scope for gaming these tests and some countries are suspected of excluding their weak performers.

Implications

I am suggesting that PISA (and the other international tests) have had a considerable impact on education in the UK. I’m not saying it is useless just that we should be taking it with a pinch of salt and not using it as a stick to beat schools with and we definitely shouldn’t be using it to validate changes to our education system that politicians want to make. I’ve even wondered whether other countries that take part have the same reaction to the results when they are published.

You only need to google “education minister response to PISA” to see lots of rhetoric and the results being used to drive through policy changes and this is dangerous, as they are treating the tests as authoritative and unquestionable. The man who is in charge at OECD (Organisation for Economic Cooperation and Development) for the PISA tests has allegedly been called “the worlds’ schoolmaster” and how worrying is that? One man with all that power! By some estimates, half the countries that have taken Pisa tests since they started in 2000 have reformed their education systems in light of the results. To give you two examples: (1) Germany’s poor performance in the first tests shocked the country and all but one of its states introduced a school-leaving exam, several states abandoned the traditional school half-day and other states modified their secondary schooling system. (2) The US embarked on the development of “common core standards” as a result of their ranking amongst other reasons.

I know that PISA results are anxiously awaited by governments, education ministers, and journos, but I don’t get the extent to which they are cited countless policy reports and the way countries are overhauling their education systems in the hopes of improving their rankings. There is so much more to education.

Why do we crave to compare ourselves to Shanghai?

Shanghai isn’t China – it is one city in a vast country. Yet its position in PISA is often used to maintain that China’s entire education system is far superior to ours, and this is missing the point – education isn’t separate from the culture in which it operates and it is wrong for us to search through the PISA report for the panacea to “fix” our system.

Even if it were right for us to compare ourselves to Shanghai (and I don’t think it is!! I suspect we should be comparing ourselves to countries closer to home or possibly Canada … don’t hold me to that, as it’s just a finger in the air thought) there are some well documented differences between the UK and Shanghai (I would love to get hold of some data comparing us to China and not just Singapore), and I could go on, but just to give you a flavor:

• There is an undeniable respect for teachers as professionals in Shanghai (can we say the same about the UK? Especially given all the in-fighting and media bashing?)
• Teacher CPD is continual and never ending in Shanghai. I know that as teachers we (I mean you longer serving teachers) fought for designated CPD time, but I’m not convinced that we embrace this yet and certainly haven’t got the right model yet.
• Shanghai teachers have way less contact time than we do – they seemingly only spend half of their time in the classroom, with the rest of the time spent working on ways to improve their teaching.
• Shanghai abolished end of primary school exams but in England until recently there has been an excessive focus on testing. . Schools could then “focus on deeper learning rather than teaching to the exams.”

So are they useless?

No. To be fair to OECD, they always publish caveats and “health warnings”  and they themselves acknowledge that they aren’t exact (See below). Professor David Spiegelhalter (of “frequency tree gate!”) has also written previously about his concerns if you want to look at this in more depth – it really is eye opening. Some of it is beyond my little brain, I have to admit!

Ok, so they aren’t precise but they aren’t useless ether. What shouldn’t be happening is an education minister (or opposition minister) using them to provide ammunition in their political war. For example, Gove used the 2009 results to claim that English schools had gone “down, down, down” since 2000, when test results were better and 15-year-olds had mostly been educated under the Tories. This is just plain wrong.

What the results do show, along with TIMMS and the other one (PIRLS???) are trends over time and I would suggest that in the case of PISA, 13 years isn’t such a long time – given that schools/education are used as a political football and changes are made all the time moving those proverbial goalposts.

What gets my goat is that education is about so much more than league tables and jumping through hoops and what we have here is a global league table, driving national tables having far reaching implications right down to each and every single one of our students.

As an aside, and if I didn’t have misgivings about the report: why didn’t this make the headlines?