Although this argument is directed at Norway, the same criticisms also apply to any country where PISA international rankings have influenced education policy – Paul Beaumont, PhD Fellow, Noragric

At the end of the 18^th century, France introduced a property tax whereby each house would be taxed according to the number of windows and doors. However, what began as a near perfect proxy for size of the house, quickly ‘colonized’ the behaviour it sought to measure. Over the course of the next decade, the number of doors and windows in French houses dropped sharply. The French had discovered the Hawthorne Effect; the tendency for people to respond to being observed, and screwing up the validity of the measure.

All this is to say, Senterpartiet – though it pains me to say– are right about OECD’s PISA tests, the latest round of which was released last week: Norway should indeed pull out. Senterpartiets Marit Knutsdatter Strand, is quite correct that PISA only “provides a superficial picture. Teachers spend a lot of time completing the test itself, and at the same time prepare students for that kind of testing. It’s the wrong use of resources”.

Of course, Norwegian education experts and teachers have long lamented PISA’s malign influence on education, however with SV also now opposed to PISA and Arbeidpartiet’s (characteristic) ambivalence, Norway seems closer than ever to escaping PISA’s tyranny.

For anyone not paying attention, PISA tests come around every three years and seek to measure the literacy, science, and mathematics skills of 15 year olds. By putting several thousand teenagers  in each participating country through tests in science, literacy and maths, PISA seeks to provide an ongoing assessment of international education levels and provide countries with comparative data to circulate good practice and help countries’ improve their education systems. Every third year in December, the scores are publicized, politicized and a competition among countries is produced, whereby countries can bask in the glory of winning PISA (e.g. Finland) and those lower down may face domestic criticism and even suffer a Pisa shock for their poor performance. This in turn is intended to inspire reforms aimed to improve education performance. Sounds sensible, somewhat scientific, and is no doubt well-intentioned.

Indeed, when I first started studying PISA, I thought it was probably a good thing. My wider research investigates how states pursue international status by doing highly visible, costly and difficult activities. Such activities generate a symbolic utility on top of the primary utility. For instance, a Ferrari has the primary utility of getting one from A to B, but also provides its owner with the symbolic utility of owning a supercar. Historically, at the country level, symbolic utility has incentivised the oversupply of nuclear weapons and opera houses at the expense of less visible but more pro-social activities like education investment. By virtue of its publicity, and the fact that only one country can ‘win’, PISA generates symbolic utility for education investment that I once believed would help harness status concerns for the public good. I have since changed my mind and here is why.

The Law of Perverse Effects

When quantified performance metrics are tied to rewards they tend to produce perverse effects. This tendency is so strong in sociology it is near enough a law.  It is not surprising then that PISA produces perverse effects: Pisa is heavily politicized around the world and governments, schools, and teachers, thus face powerful incentives to perform well in the rankings. Like our French property owners, PISA participants do not sit like vegetables waiting to be measured.

There are a number of ways to game PISA. The most gratuitous is by excluding weak students from taking the tests: Shanghai topped tests in 2012, but it excluded significant numbers of students – likely low-achieving students – because of internal migration rules. Less egregious but also widespread is the practice of teaching to the test: Instead of teaching students how to read, one can teach them how to score well on a PISA test for reading. The two are not the same thing. Whenever you see a reference to PISA average, remember it is always partly a function of such gaming activities.

Conversely, PISA tests are quite pointless for the students involved. They receive no feedback, it wastes their time, and thus provides them with no obvious educational reward. Norwegian students in the mid 2000s organised a boycott of national tests for similar reasons.  Internationally, one study estimated that student motivated accounted for 19 percent of the variation in the average PISA test scores between countries. Developing critical thinking skills to avoid exerting oneself in an international vanity competition is exactly what I would want a schooling system do for my child, yet protesting PISA in such a manner gets punished by PISA with poor scores.

One way to interpret PISA scores then – and the PISA average –  is that it captures to some unknown extent a) a country’s willingness to exert time and effort preparing students for a pointless test, and b) those student’s willingness to exert time and energy upon an exam that is pointless.  Neither factor is significant for gauging the quality of an education system, quite the opposite.

Myopic Picture of Education

Indeed, the rankings that PISA encourages a myopic picture of education. PISA – like all quantified rankings –embodies a tacit theory of the phenomenon it claims to measure. All rankings involve deciding what to measure and how to weight it, and thus what to exclude. PISA is no different: via tests of literacy, science and maths skills, PISA claims to measure the ability of 15 year olds’ ‘ability to meet real life challenges’. Meanwhile, the school’s system is treated as the crucial factor at play.

No doubt the education system is important for determining education outcomes, yet it ignores crucial structural factors that are often at least as important: social-economic circumstances of the parents are a strong predictor of education outcomes, meanwhile immigrants – by virtue of a number of factors outside the control of schools – often struggle in such tests. Indeed, part of the reason of Finland’s ‘miracle’ was because it had relatively lower levels of non-native speakers: one fifth of the OECD average [note to FrP – this is an argument for ignoring PISA not for cutting migration)]. Meanwhile, South Korea’s high scores are far from miraculous once one realises that around half of their students receive private tutoring costing more than two thirds of what the country spends on public education.

Given the tiny margins of countries around the OECD average, it is very likely the difference in position can be accounted for factors quite outside the schools. Or put differently, it is highly likely that schools with better education systems finish below schools with worse education systems. Yet, as anyone who has followed the Norwegian PISA debate knows, PISA scores begets a narrow debate about schools and reforms of schools.

Undemocratic Agenda Setting

Last but not least, Norway opted into PISA with little or no debate yet it has set the agenda for Education reform for the past 20 years. As Professor of Education at UiO,  Svein Sjøberg, has noted, given the cross-party concern with PISA, and the countless references to PISA tests and OECD experts in parliamentary reports and school  documents, it “does not require profound analysis to suggest they have had an enormous influence on policy making”.

This political influence is not an accident. The results of PISA are routinely presented in relative league tables that accentuate statistically insignificant differences and turn them into politically salient problems. Indeed, the difference between 10-20 place is substantially tiny, yet the press have tended to report on it breathlessly: Norway goes down, or in the infamous words of Høyre’s former Education minister finishing around the middle is: “like coming home from the winter Olympics without any medals”. Yet, given the tiny differences involved, to extend one Norwegian scholar’s metaphor, paying attention to changes in position in PISA is akin to becoming emotionally invested in small movements in the Peloton in the Tour de France.

Paying for Privilege of Pernicious Policy Pressure

Ultimately, by signing up to PISA, Norway pays for the privilege of outsourcing value judgements about what constitutes good education to unelected international technocrats, whose evaluation, as we saw, suffers from serious validity problems. Further, Norway pays for the privilege of this dubious evaluation to be circulated with the intention of presenting insignificant changes in relative ‘performance’ in a manner designed to generate attention and ferment reform in education policy. This is not a good a deal.

Indeed, Senterpartiet wanting to opt out of PISA does not make them opposed to knowledge about schools, as Høyre – the original abusers of PISA rankings back in 2001— would have it, but puts them broadly in line with leading international education researchers. Heck, the harm obsessing over PISA has done to Norwegian education has even been given the name PISA Syndrome. Thus. Senterpartiet and SV are not opposed to knowledge about schools, they are opposed to the illusion of knowledge driving education policy.

Some may object that PISA is useful because it highlights real problems. For instance, this year’s PISA round highlighted a troublingly large gender literacy gap: 26% of Norwegian boys placed at the lowest level compared to 12% of girls. Yet, it should not require an international test to establish this gap. In fact, we would likely get a far more precise picture if one instructed Norwegian researchers to design and implement such a test for its students, one that covered performance at each stage in their development instead of just a one off test at 15. Moreover, it would also clearly be better to develop a test tailored to best fit the country’s language in question and the school curriculum, rather than designed mainly so pupils from different cultures can equally (mis)understand it.

Norway should not require international relative rankings to make education performance politically salient. Indeed, instead of just pulling out of PISA, I propose that Norway use the money it saves from opting out of PISA on its own bespoke studies carried out by Norwegian education experts. Then, on the day of the PISA announcement, when the rest of the OECD is busying itself squawking about marginal changes in their relative performance, Norway could set aside a week to publicly reflect upon its own education research. This would surely prove much more valuable in the long-term to helping to formulate democratic, evidence-driven policy priorities.  Certainly this would surely be better than the current direction whereby Norway risks moving towards an education system tantamount to 19^th century France’s windowless houses.

Paul David Beaumont is currently finalising his PhD dissertation ‘The Grammar of International Status Competition’ at the Department of International Environmental and Development Studies at the Norwegian University of Life Sciences. Twitter: @BeaumontPaul​.