PISA, Piece by Piece

December 04, 2013

Multiple Pages
PISA, Piece by Piece

With the release of new PISA test scores for 65 countries’ 15-year-olds this week, it’s worth taking a look at TIME reporter Amanda Ripley’s latest book The Smartest Kids in the World: And How They Got that Way.

Ripley came up with the clever idea of following three American high schoolers as exchange students in Finland, South Korea, and Poland. She chose Finland and South Korea because they are perennial PISA powerhouses, while Poland has improved its ranking significantly in this century.

Her sample size of three American kids abroad is hardly foolproof, and yet it’s a start. Everybody has opinions on schooling, but few people have firsthand experience with different countries’ school systems because it’s immensely time-consuming to sit in on classrooms long enough that the teacher runs out of her dog-and-pony shows for visitors and finally gets down to normal business.

Having only recently become interested in the topic of education, Ripley is a true believer in PISA scores. (By the way, PISA stands for Programme for International Student Assessment, and it’s sponsored by the OECD, the club for rich countries.)

While I don’t know what scores will be revealed this week from the triennial PISA, I can anticipate the tenor of American media coverage by recalling the 2010 brouhaha over how badly American students scored. In reality, American kids did surprisingly well on that last PISA, as long as you adjust for race. As I wrote in 2010, the main focus of the 2009 test was reading, and American students read relatively well:

• Asian Americans outscored every Asian country, and lost out only to the city of Shanghai, China’s financial capital.

• White Americans outperformed the national average in every one of the 37 historically white countries tested, except Finland.

• Hispanic Americans beat all eight Latin American countries.

• African Americans would likely have outscored any sub-Saharan country, if any had bothered to compete.

Keep in mind, however, that American students didn’t do quite as well on math and science. And Americans did better in this last PISA go-round than in the previous three, so they may regress back toward their mean this time.

“PISA’s global results are largely the American bell curve writ large, so nobody notices.”

More broadly, can you rely upon PISA scores? In truth, nobody seems to really know how much to trust PISA and its ace salesman Andreas Schleicher.

PISA is a professional product, but recently there has been arcane criticism of the mathematics of its sophisticated item response theory.

Yet even without delving into Gödel’s Incompleteness Theorem (as one of PISA’s critics has), the sheer logistical challenge of what PISA attempts to do should raise common-sense questions about how equitably 65 countries can be compared. Translation of tests, selection of representative samples, and prevention of local authorities putting their thumbs on the scale are challenges so daunting to get exactly equal around the world that most observers just seem to hope for the best and trust that Schleicher has somehow devised a globally level playing field.

Social science history shows that Pareto’s 80-20 principle applies to the accuracy of testing. It’s not that hard to concoct exams that are roughly accurate, but it’s quite difficult to fine-tune the process to give reliable answers to the most interesting questions, such as whether kids learned more in the 2012 PISA than the 2009 PISA, or whether Poland is really doing better than, say, the Czech Republic.

A big question is how motivated the test-takers are. PISA tests aren’t supposed to have any consequences for the students, so lazy ones might be inclined to bubble in their answers after they get bored or tired. Perhaps some countries do a more effective job than others of motivating test-takers to work the full two hours for national pride? Or are their students just more cooperative? Ripley mentions that Finns, Koreans, and Poles were exceptionally dutiful about filling in their PISA demographic questionnaires, so that may offer a clue to their high scores.

At the continental level, PISA (and the competing TIMSS test) show the same general broad patterns as seen in countless tests in the US: On average, Northeast Asians are out front, then Europeans, then Latin Americans, then blacks. So the PISA results have some prima facie validity. Of course, it’s discouraging to learn that PISA’s global results are largely the American bell curve writ large, so nobody notices.

Instead, what everybody wants to know about are World Cup-style matchups: Did France beat Germany? Did Japan outscore Taiwan? Yet those are precisely the kind of comparisons where the margin for error may well overwhelm the reliability of the results.

Ripley simply accepts the 2009 PISA scores at face value and focuses on an American exchange student in each of three countries: up and comer Poland, South Korea (which rather than being an “educational superpower” appears to be a fairly standard northeast Asian culture), and Finland. That far northern land has consistently done the best of the white countries on the PISA. (It also done well on the competing TIMSS test.)

First, a few general observations on being an exchange student. While it can sound exciting to bored teens, it can turn out to be a depressing experience. Countries that score well on the PISA tend to be fairly far north and thus have limited daylight during the school year. Moreover, kids who want to get away from their own high school’s social grind sometimes find that no matter where you go, there you are. The lonely Oklahoma girl who went to a small town in Finland found winter there to be darker, colder, and more introverted than back home. She almost wound up being sent home for depression.

Moreover, none of Ripley’s three American visitors spoke the local language, so it’s not clear how much they learned at school. And their hazy perceptions of the schooling they sat through are less than definitive. Still, their recollections are better than listening to adults theorize.

In general, two semesters as an exchange student seem about twice as long as is useful. Diminishing returns set in. Eric, the Minnesota lad who went to South Korea, grew bored and went home early.

You can’t blame him. Koreans score high, both in South Korea and in the US, but their educational system is absurd. Here is Ripley’s believe-it-or-not tale of Eric’s first-ever class period in Korea:

A third of the class was asleep. Not nodding off, but flat-out, no apology sleeping with their heads down on the desks. One girl actually had her head on a special pillow that slipped over her forearm. This was pre-meditated napping.

What’s going on? Most of Eric’s classmates spend from 8AM to 9PM at school, primarily test prepping, then go to private hagwon cram schools until curfew at 11PM. They nap through the day on the assumption that school is less important than evening test prep:

He had found one possible explanation for Korea’s PISA scores, and it was depressing. Kids learned a lot, but they spent a ridiculous amount of time doing so.…He was astounded by the inefficiency of it all. In Korea, school never stopped.

Ripley doesn’t mention another reason Koreans tend to score well on tests: organized cheating. For example, a recent SAT test was called off because the test booklet was being sold in Korea for thousands of dollars.

In contrast, Finnish education is famously laid-back. It’s the anti-Korea. This Nordic country is the closest thing to a real-life Lake Wobegon, where all the children are above average. Students don’t start first grade until age seven, aren’t given enormous amounts of homework, aren’t tested often, and don’t test-prep much at all. Finland doesn’t pay schoolteachers all that much in money, but they are respected, both in the culture and (crucially) in the classroom.

Finnish teachers are instructed to keep the left half of the bell curve up to speed so they don’t drag down the class, while the smarter kids are expected to read ahead in the textbook on their own. Whether this would work in the US, where the bell curve is wider and the students less disciplined and cooperative, is another question.

But the Finns have a system that works well for Finns, and good for them.

Polish schooling isn’t anything special, but it has by all accounts improved significantly since the 1990s. Considering what a mess communism left in Eastern Europe, the rapidity with which Poland has been progressing toward “normal country” status is impressive.

What do Finland, Poland, and South Korea have in common?

First, all three achieved their independence only within the last 100 years, and they were each invaded at Stalin’s command in 1939-1950. They remain relatively patriotic in the old-fashioned nationalist modernizer mode that did so much to lift educational levels beginning in Prussia. They see education as a way for their people to get ahead in the world.

In contrast, India—even though it is always grouped with China in President Obama’s rhetoric about America’s main educational rivals—has dropped out of the PISA after a disastrous trial run in which two Indian states performed at sub-Saharan levels. India has been the darling of Davos Man for its seeming postmodern post-nationalism, but unfashionable old nationalism retains the best track record at uplifting the masses.

Second, none of the three had colonies and therefore don’t feel any imperial guilt.

Third, the three countries have very few students of first- or second-generation immigrant background. A recent PISA study showed Finland’s student body is only 3 percent non-Finnish (compared to 19 percent immigrant in the US), while Poland’s and South Korea’s immigrant percentages rounded down to zero.

Ripley visits a Finnish school that has many immigrants and breathlessly reports that everybody is above average. In reality, children of immigrants score well over a half standard deviation below average in Finland. And the three largest sources of immigrants in Finland remain neighboring Estonia, Russia, and Sweden.

In 2009, immigrants dragged down average PISA scores in almost all countries except Australia, which works hard to keep out illegal immigrants and uses a points system to help pick promising legal ones.

Fourth, even though Ripley is excited to see some Somalis in one Finnish school, none of these countries has many blacks. What giant pandas are to the cause of biodiversity preservation, blacks are to the reigning dogma of diversity: “charismatic megafauna.” Thus, not only do blacks tend to lower test scores and disrupt classrooms, they also tend to warp elite thinking and public policy.
For example, the Obama Administration has been persecuting local school districts that suspend higher percentages of black students. Now, you know and I know and even Barack Obama knows that the reason for this is because on average black students cause more trouble. But in America today you aren’t supposed to know that, so the federal government is pushing a policy on school districts: If they want to stay out of trouble with the feds, they should ignore black troublemakers.

This won’t do the non-troublemakers in the classroom much good, but who cares about them? If the students who just wish to learn want the federal government’s concern, they’ll need to become charismatic megavictims too. The important thing in 2013 America is to hunt down all those white racist schoolteachers and administrators who cause The Gap.

How that will help get us to Finnish levels of classroom order and quality teachers remains a puzzle.

Daily updates with TM’s latest