The most recent international league tables of pupil performance differ considerably. England languishes well down the list in PISA 2009, stars in the Pearson Global Index 2012, and lies somewhere in-between in TIMSS 2011. This report seeks to explain the differences and highlight some underlying consistencies.
There are three main reasons for the different rankings: Countries are ranked on scores which may not be different; Different countries are involved; The tests differ and some countries are ahead on one but not the other. There is a further reason for the difference between the Pearson Index and the tests: The Index uses additional data.
Secondary school pupils
We can see how these differences play out if we look in detail at the maths performance of secondary school pupils as an example. PISA 2009 has England joint 27 out of 65 countries and TIMSS 2011 tenth out of 42. If we want to be at least 95% sure that a country has performed above England, then there are 20 above England in PISA and six in TIMSS.
- Of those countries, five are above in both: Japan, Hong Kong, Singapore, South Korea and Taiwan.
- Eleven countries were above England in PISA, but did not take part in TIMSS: Belgium, Canada, Denmark, Estonia, Germany, Iceland, Liechtenstein, Macao, the Netherlands, Shanghai, and Switzerland.
- Four countries were above in PISA, but not in TIMSS: Australia, Finland, New Zealand and Slovenia.
- Russia was above England in TIMSS, but not PISA.
Primary school pupils
- The differences between TIMSS 2011 for primary school pupils and PIRLS 2011 are not so sharp since they are from the same stable.
- Five of the countries doing better than England on at least two out of maths, science and reading have a familiar ring to them: Japan, Hong Kong, Singapore, South Korea and Taiwan.
- To them can be added Russia which tends to do well in TIMSS-type tests and Finland which does better at TIMSS primary than secondary.
Pearson Global Index
England is sixth in the world for education according to the new Global Index published by Pearson. But this is derived mainly from PISA 20092 where the combined reading, maths and science scores place it joint 18th. One-third of the Pearson ranking is contributed by graduation rates from upper secondary school and university, where England is second behind South Korea. These data, however, are incomplete, based on different definitions, come from different sources, and are more a matter of policy than educational attainment. If we discount these data , England ranks 12th in the Pearson Index. The difference from PISA is explained almost entirely by the fact that five countries above England in PISA are not included in the Index.
Changes over time
While England‟s performance in PISA maths appears to have declined markedly since 2000, there seems to have been a dramatic improvement in TIMSS maths. However, these are artefacts explainable in terms of participation and response rates. The number of countries significantly above England in maths increased from two in PISA 2000 to 20 in PISA 2009. England‟s sample in 2000, however, was biased to high performing schools through a poor response rate. The OECD has declined to use it as a baseline. Without it, we are left with PISA 2006 and PISA 2009 where there is a difference of only two countries due to two top performers taking part for the first time in 2009.
The number of countries above England in TIMSS maths fell from 14 in 1999 to six in 2011. The difference is largely explained by five of the countries on the first occasion not taking part on the second. England did, however, appear to improve relative to three countries: Finland, Hungary and Malaysia.
A long tail of under-performance?
England is often charged with having a long tail of underachievement. TIMSS/PIRLS 2011 do show that there were more poorly performing primary school children in England than in the leading countries and there was a wider spread of scores. In this sense, there was underachievement, but fewer also reached the highest benchmark, in spite of the inclusion of independent and grammar schools.
The spread of scores in the top-performing countries, except Finland, increased in secondary education to become more like that in England. But England still had fewer at the highest level, a lower mean, and more at the lowest level. This would indicate that bringing England‟s performance up to the best requires improvement across the piece, not just levering up from the bottom.
Although it looks from media coverage as though there are big discrepancies in the results of PISA, TIMSS and the Pearson Index, there is, in fact, an underlying consistency. It is, however, the differences which have been highlighted. This is because league tables are popular. But there is also the spin that has been put upon them by politicians of all parties.
When the results of the 2000 round of PISA became available in 2001 the Labour Government was looking for evidence that its reforms were succeeding. England’s unusually high position led the Government to attach greater importance to the results than they deserved given the disappointing response rate.
The current Coalition Government has been seeking justification for the changes it wishes to make to the education system. It has offered a gloomy interpretation of the results even when, as in the TIMSS/PIRLS 2011, England, on the surface, appears to have done quite well.
The value of the international comparisons risks being lost if the findings are continually subsumed into convenient political narratives.
Cutting through the spin, there are five Asian countries (Hong Kong, Japan, Singapore, South Korea, and Taiwan) which have consistently performed above England in PISA, TIMSS and PIRLS. Other Asian countries are prominent when they take part. The tests are designed to enable education systems to be compared, and it is easy to assume that the differences reflect the quality of the education. But this is not necessarily the case.
There are other possible explanations. Among the suggestions that have been made are: a culture of hard work and effort, a trait of quiet persistence and parenting style. The success of Chinese children is portable since they also shine in England’s education system. Besides cultural and personal differences, there are many other factors that could come into play, for example, the importance of the results to a country, and the extent of preparation and practice for the tests. This is not to say that the schools in these countries are not of high quality; only that there may not be a magic bullet which can be incorporated into England‟s education system.
There is a group of countries, more culturally similar to England, that consistently do better on PISA. Some, New Zealand and Australia for example, do significantly worse than England on TIMSS. Whether we wish to follow them will depend on whether we value the “literacies‟ of PISA tests more than the “knowledge and understanding‟ of TIMSS.
Many of those above England on PISA were absent from TIMSS. Among them were some of our nearest neighbours. Belgium, Germany, the Netherlands and Switzerland were all above us in maths. Their approaches should be examined closely to see if there is anything that can be learned from them to improve our pupils’ grasp of maths, which is in urgent need of attention. It is where England‟s record is poorest.
If there is a lesson to be drawn from these analyses it is: don’t leap to conclusions based on a country’s apparent ranking in league tables. As presented, the messages are decidedly mixed. Any differences do not have to be mainly to do with the schools. The data could be invaluable, but they need to be interpreted with great care.