Innovation in all fields of research and invention has entered a uniform, precipitous decline since at least 1945

A recent Nature article sheds new light on our decaying scientific institutions.

I’ve been silent for a few days, because the latest volume of the journal I edit is about to go to press, and this means I must spend hours imposing consistency and accuracy upon unbelievably tedious minutiae in footnotes that nobody will ever read or care about. There are assistants who should in theory do this work for me, but I have learned that their involvement is to be minimised, as they’re just as likely to introduce more problems as they are to solve them, and even when they don’t mess up, they never return pristine edited files. Instead, after weeks, they proudly forward heavily annotated PDFs, which I must then painstakingly correct regardless. In short, academic publishing is a plague on the land, and as long as I’m grumpy about it, I might as well make it into a post.

Last summer, we reviewed research showing that, as scientific fields grow, they become more conformist, and advances in knowledge occur more lethargically. This is a problem, both because academia is now bloated with more researchers than ever before, and their ranks are constantly growing; and also because anything seen as new, exciting or promising, is immediately flooded with money and countless new researchers, whose presence will only slow progress and ensure greater conformity. The pandemic era has been a perfect object lesson in how this happens.

A new paper in Nature, finding that Papers and patents are becoming less disruptive over time, confirms and expands upon this basic picture.

The authors look at citation networks among 25 million papers published between 1945 and 2010 indexed at the Web of Science. Basically, papers cite other papers which cite other papers, and by studying these patterns of citation over time, you can work out roughly what is happening in a discipline. If a field is utterly changed by new, foundational discoveries, a lot of older work will never be cited again. Papers in fields experiencing stagnation or steady, cumulative advancement, on the other hand, will continue to cite increasingly old research, as work from many generations ago remains relevant.

Via a statistical measure known as the “CD index” (for “consolidating” or “disruptive”), the authors can (very roughly) measure the foundational nature of any given paper, by looking at how often later citations of that paper also cite earlier work.¹ When the CD index of papers in different scientific fields is plotted over time, we see a pervasive collapse in innovative, foundational work across all fields, converging towards the same baseline.

CD5 in the y-axis is the consolidating or disruptive nature of a paper measured five years after publication.

Don’t like bespoke statistical measures like the CD index? No problem. The authors also look at trends in paper vocabulary. New, foundational papers and patents are obviously more likely to introduce new terminology and to use current words in novel combinations, whereas a world in which progress has stagnated would see publications deploying increasingly uniform language.

This measure, too, reveals a general, steady collapse in innovation across all fields:

Equally interesting are specific vocabulary trends. Whereas words like “produce” and “make” dominated research in the 1950s and patents in the 1980s, recent work is more likely to talk of improving, enhancing, or including. It is depressing even to type this.

What is more, the authors find that, despite the general decline in innovation, there has been “remarkable stability in the absolute number of highly disruptive works” in all major fields since 1945:

Papers and patents, broken down by their CD index ranges. There has been a massive increase in research and patents that score low in “disruptiveness” (the blue lines in both charts above), while highly innovative work has remained numerically constant despite massive increases in production.

This confirms my general suspicion, that postwar expansions to science (and academia more broadly) have happened via increases to the number of less prestigious schools, less talented professors and less intelligent students. The whole enterprise has been inflated at the bottom, in other words, and not at the top, such that we’re wasting huge amounts of money for very little added advantage.

The authors appear to be uncomfortable about the implications of their work. They protest that “the declining rates of disruptive activity are unlikely to be caused by the diminishing quality of science and technology,” because high-quality work in the top journals is also subject to the same decline. This is, when you think about it, a pretty bizarre argument. If publications in top journals are also becoming less innovative, that simply means that the decay afflicts all levels of every field. Evidence for this is pervasive, and we don’t even have to look beyond the present paper. The authors themselves at one point note that “The gap between the year of discovery and the awarding of a Nobel Prize has … increased,” as the innovative nature of new work has grown less obvious; and people within specific fields more and more face what the authors describe as a “knowledge burden,” because the scarcity of revolutionary work requires them to maintain currency with a growing number of older and older studies going back decades.

No, the authors say, what’s actually happening here, is scientists are becoming overspecialised, leading them to publish narrower, less interesting stuff:

In particular, the growth in publishing and patenting may lead scientists and inventors to focus on narrower slices of previous work … Using three proxies, we document a decline in the use of previous knowledge among scientists and inventors. First, we see a decline in the diversity of work cited, indicating that contemporary science and technology are engaging with narrower slices of existing knowledge. … Second, we see an increase in self-citation … which is consistent with scientists and inventors relying more on highly familiar knowledge. Third, the mean age of work cited … is increasing … All three indicators point to a consistent story: a narrower scope of existing knowledge is informing contemporary discovery and invention.

This is like telling a wounded man that he’s not dying, he’s just bleeding out. A narrow research focus is above all a careerist tactic. Self-citation, a lack of currency with new publications, and the tendency to cite the same stuff over and over again, are all just symptoms of dimmer people publishing too much.

Our scientific institutions rest on reputations they earned generations ago, and it’s going to take a long time for the wider culture to internalise the fact that the guys who discovered antibiotics and deduced the structure of DNA aren’t around anymore. Science has entered a new, careerist era, one in which it will grow ever more conformist, erratic and unreliable. There’s nothing unusual about this: In the premodern era, doctors were routinely mocked as worthless charlatans, and all areas of scientific inquiry were larded with total lunacy well through the eighteenth century. We desperately need to increase our scepticism of and our pessimism about The Science. If we can’t, mass house arrests, mandated facial coverings, and criminal vaccination campaigns will pale in comparison to the evils yet to come.

1. From the paper:

The intuition is that if a paper or patent is disruptive, the subsequent work that cites it is less likely to also cite its predecessors; for future researchers, the ideas that went into its production are less relevant (for example, Pauling’s triple helix). If a paper or patent is consolidating, subsequent work that cites it is also more likely to cite its predecessors; for future researchers, the knowledge upon which the work builds is still (and perhaps more) relevant (for example, the theorems Kohn and Sham used). The CD index ranges from −1 (consolidating) to 1 (disruptive). We measure the CD index five years after the year of each paper’s publication.

A paper would have a CD score of 1, if later papers cited it, but not any of the previous work. It would have a score of -1, if later papers cited it as well as all the previous work. A score of zero indicates, roughly, that later citing papers are about as likely as not to also cite earlier work.