Cultural Caviar

The Replication Crisis and the Repetition Crisis

February 17, 2016

Multiple Pages
The Replication Crisis and the Repetition Crisis

With data becoming ever more abundant, this should be the golden age of the social sciences. And yet they seem to be suffering two mirror-image nervous breakdowns—the Replication Crisis and the Repetition Crisis.

Outright made-up-data fraud is hardly unknown in academia, but the double disasters have more to do with shortcomings in how contemporary researchers analyze relatively honest data. I suspect that the systemic failures stem more from researchers being allowed both too many and too few of that evocative (if actually rather dry) technical term: “degrees of freedom.”

One cause of the Replication Crisis has been that analysts grant themselves excessive post hoc liberties to crunch the numbers however many ways it takes to find something—anything—that is “statistically significant” (which isn’t the same as actually significant) and thus qualifies as a paper for publish-or-perish purposes. Hence, social scientists seem to be coming up with a surplus of implausible junk science findings on trivial topics, such as “priming” (the contemporary version of subliminal advertising), which then routinely fail to replicate.

In contrast, in what I’ll dub the Repetition Crisis (a.k.a. the Explanation Crisis), academics hamstring the interest and usefulness of their findings by ruling out ahead of time any explanatory factors other than the same tiny number of politically correct concepts that were exhausted decades ago.

“In 2016, blaming white privilege for everything you don’t like isn’t quite as lame as blaming the Bavarian Illuminati, but the gap is closing.”

Why a Repetition Crisis? Dissident social psychologist Jonathan Haidt of NYU’s Stern School of Business, author of The Righteous Mind, pointed out in a freewheeling interview with John Leo how the ever-growing list of sacred cows in American life restricts what social scientists can allow themselves to discover about important issues:

For many years now, there have been six sacred groups. You know, the big three are African-Americans, women and LGBT. That’s where most of the action is. Then there are three other groups: Latinos, Native Americans…and people with disabilities. So those are the six that have been there for a while. But now we have a seventh—Muslims.

One could argue that there are more sacred groups than seven, but Haidt’s next point was illuminating:

Something like 70 or 75 percent of America is now in a protected group. This is a disaster for social science because social science is really hard to begin with. And now you have to try to explain social problems without saying anything that casts any blame on any member of a protected group. And not just moral blame, but causal blame. None of these groups can have done anything that led to their victimization or marginalization.

For example, in discussing crime or poverty, social scientists are allowed to imply that the dirt that white people live upon is inherently magic while the dirt under black people is obviously tragic. But anything smarter and more interesting could get them furiously denounced by angry know-nothing students (or Watsoned out of their jobs if they lack tenure). So it’s safest just to blame everything and anything on white people.

Still, as the generations roll by, that’s increasingly sounding like a senile conspiracy theory. In 2016, blaming white privilege for everything you don’t like isn’t quite as lame as blaming the Bavarian Illuminati, but the gap is closing.

As the range of acceptable insights narrows, boredom stalks the social sciences.

Haidt notes:

Anthropology and sociology are the worst—those fields seem to be really hostile and rejecting toward people who aren’t devoted to social justice.

Today, for example, it seems astonishing that 60 years ago cultural anthropologists like Margaret Mead could be celebrities. The educated public now assumes that cultural anthropologists are pedantic and petulant, best avoided.

It’s not surprising, therefore, that many social scientists try to sidestep the Repetition Crisis by avoiding important issues in favor of marketing-research-like problems, which in turn worsens the Replication Crisis. (The central distinction between science and marketing research is that the latter doesn’t strive to discover permanent truths: That, say, Bill Cosby was good at advertising Jell-O Pudding Pops in 1979 is good enough for marketing research. If you want to know whether to hire Cosby in 2016, marketing researchers would be happy to take your money.)

One cause of the Replication Crisis is the social-science version of the Hollywood excuse “We’ll fix it in post.” As postproduction computer-generated imagery has gotten cheaper, movie directors have become more likely to rationalize on-set flaws in dialogue, acting, or their own direction with the reassurance that the scene can always be salvaged in postproduction by computer wizardry.

Similarly, Malcolm Gladwell-ish experiments can be often rescued after the fact by comparing multiple effects across subdivisions of the sample. Because you need to achieve a single result that would happen only 5 percent of the time by chance, if you can crunch your data twenty different ways, you have a 50-50 shot at statistical significance.

One way to think of the Replication and Repetition Crises is as emanating from opposite abuses of degrees of freedom. That cool-sounding phrase from early-20th-century statistics has been adopted over the years by mechanical engineering, rocket science, and robotics, although its statistical definition—“the number of values in the final calculation of a statistic that are free to vary”—remains notoriously frustrating for statistics instructors to get across verbally.

The term “degrees of freedom” was popularized by Ronald A. Fisher in the 1920s based on a 1908 paper published under the pseudonym “Student” by a quality-control expert at the Guinness brewery in Dublin. William Sealy Gosset was among the first to think rigorously about how much a statistical analyst’s confidence in his own conclusions ought to be reduced by the limited sample sizes he was forced to work with.

An influential 2011 paper on the Replication Crisis by Joseph P. Simmons, Leif D. Nelson, and Uri Simonsohn offered the term “researcher degrees of freedom” as a critique of the growing ability of researchers to slice and dice their way to statistically significant but temporary or even nonexistent correlations:

[I]t is unacceptably easy to publish “statistically significant” evidence consistent with any hypothesis.

The culprit is a construct we refer to as researcher degrees of freedom. In the course of collecting and analyzing data, researchers have many decisions to make: Should more data be collected? Should some observations be excluded? Which conditions should be combined and which ones compared? Which control variables should be considered? Should specific measures be combined or transformed or both?

It is rare, and sometimes impractical, for researchers to make all these decisions beforehand. Rather, it is common (and accepted practice) for researchers to explore various analytic alternatives, to search for a combination that yields “statistical significance,” and to then report only what “worked.” The problem, of course, is that the likelihood of at least one (of many) analyses producing a falsely positive finding at the 5% level is necessarily greater than 5%.

This term, “researcher degrees of freedom,” is even more useful if we recognize that just as analysts can overfit models that therefore won’t be replicable, they can also underfit by not being allowed adequate intellectual degrees of freedom to offer “controversial” explanations, driving them into endless repetitions of aging mantras about racism and sexism. The issue for Student was that data were expensive while potential explanatory factors were cheap. Today, the mirror image often reigns: Data are readily available, but honest explanatory factors can cost you your job.

Too many researcher degrees of freedom permit trickery; but too few cause stupidity.

Daily updates with TM’s latest


The opinions of our commenters do not necessarily represent the opinions of Taki's Magazine or its contributors.