Helsinki Cathedral and monument to Alexander II, Finland

Source: Shutterstock

With the release of new PISA test scores for 65 countries’ 15-year-olds this week, it’s worth taking a look at TIME reporter Amanda Ripley’s latest book The Smartest Kids in the World: And How They Got that Way.

Ripley came up with the clever idea of following three American high schoolers as exchange students in Finland, South Korea, and Poland. She chose Finland and South Korea because they are perennial PISA powerhouses, while Poland has improved its ranking significantly in this century.

Her sample size of three American kids abroad is hardly foolproof, and yet it’s a start. Everybody has opinions on schooling, but few people have firsthand experience with different countries’ school systems because it’s immensely time-consuming to sit in on classrooms long enough that the teacher runs out of her dog-and-pony shows for visitors and finally gets down to normal business.

Having only recently become interested in the topic of education, Ripley is a true believer in PISA scores. (By the way, PISA stands for Programme for International Student Assessment, and it’s sponsored by the OECD, the club for rich countries.)

While I don’t know what scores will be revealed this week from the triennial PISA, I can anticipate the tenor of American media coverage by recalling the 2010 brouhaha over how badly American students scored. In reality, American kids did surprisingly well on that last PISA, as long as you adjust for race. As I wrote in 2010, the main focus of the 2009 test was reading, and American students read relatively well:

“€¢ Asian Americans outscored every Asian country, and lost out only to the city of Shanghai, China’s financial capital.

“€¢ White Americans outperformed the national average in every one of the 37 historically white countries tested, except Finland.

“€¢ Hispanic Americans beat all eight Latin American countries.

“€¢ African Americans would likely have outscored any sub-Saharan country, if any had bothered to compete.

Keep in mind, however, that American students didn’t do quite as well on math and science. And Americans did better in this last PISA go-round than in the previous three, so they may regress back toward their mean this time.

More broadly, can you rely upon PISA scores? In truth, nobody seems to really know how much to trust PISA and its ace salesman Andreas Schleicher.

PISA is a professional product, but recently there has been arcane criticism of the mathematics of its sophisticated item response theory.

Yet even without delving into Gödel’s Incompleteness Theorem (as one of PISA’s critics has), the sheer logistical challenge of what PISA attempts to do should raise common-sense questions about how equitably 65 countries can be compared. Translation of tests, selection of representative samples, and prevention of local authorities putting their thumbs on the scale are challenges so daunting to get exactly equal around the world that most observers just seem to hope for the best and trust that Schleicher has somehow devised a globally level playing field.

Social science history shows that Pareto’s 80-20 principle applies to the accuracy of testing. It’s not that hard to concoct exams that are roughly accurate, but it’s quite difficult to fine-tune the process to give reliable answers to the most interesting questions, such as whether kids learned more in the 2012 PISA than the 2009 PISA, or whether Poland is really doing better than, say, the Czech Republic.

A big question is how motivated the test-takers are. PISA tests aren’t supposed to have any consequences for the students, so lazy ones might be inclined to bubble in their answers after they get bored or tired. Perhaps some countries do a more effective job than others of motivating test-takers to work the full two hours for national pride? Or are their students just more cooperative? Ripley mentions that Finns, Koreans, and Poles were exceptionally dutiful about filling in their PISA demographic questionnaires, so that may offer a clue to their high scores.

At the continental level, PISA (and the competing TIMSS test) show the same general broad patterns as seen in countless tests in the US: On average, Northeast Asians are out front, then Europeans, then Latin Americans, then blacks. So the PISA results have some prima facie validity. Of course, it’s discouraging to learn that PISA’s global results are largely the American bell curve writ large, so nobody notices.

Instead, what everybody wants to know about are World Cup-style matchups: Did France beat Germany? Did Japan outscore Taiwan? Yet those are precisely the kind of comparisons where the margin for error may well overwhelm the reliability of the results.

Ripley simply accepts the 2009 PISA scores at face value and focuses on an American exchange student in each of three countries: up and comer Poland, South Korea (which rather than being an “educational superpower” appears to be a fairly standard northeast Asian culture), and Finland. That far northern land has consistently done the best of the white countries on the PISA. (It also done well on the competing TIMSS test.)

First, a few general observations on being an exchange student. While it can sound exciting to bored teens, it can turn out to be a depressing experience. Countries that score well on the PISA tend to be fairly far north and thus have limited daylight during the school year. Moreover, kids who want to get away from their own high school’s social grind sometimes find that no matter where you go, there you are. The lonely Oklahoma girl who went to a small town in Finland found winter there to be darker, colder, and more introverted than back home. She almost wound up being sent home for depression.

Moreover, none of Ripley’s three American visitors spoke the local language, so it’s not clear how much they learned at school. And their hazy perceptions of the schooling they sat through are less than definitive. Still, their recollections are better than listening to adults theorize.

In general, two semesters as an exchange student seem about twice as long as is useful. Diminishing returns set in. Eric, the Minnesota lad who went to South Korea, grew bored and went home early.

You can’t blame him. Koreans score high, both in South Korea and in the US, but their educational system is absurd. Here is Ripley’s believe-it-or-not tale of Eric’s first-ever class period in Korea:

A third of the class was asleep. Not nodding off, but flat-out, no apology sleeping with their heads down on the desks. One girl actually had her head on a special pillow that slipped over her forearm. This was pre-meditated napping.