Csaba Szabo is one of the most cited scientists in the world. Here’s a quote from his Wikipedia page¹:

According 2022 Elsevier Worldwide Citation Metrics Index, Dr. Szabo’s citations place him into the top 1% of the top 2% of all scientists worldwide, at the overall spot of 383 out of those 100,000 scientists who made this list.

Szabo is a scientist. He’s a funny scientist, too. This book blows the lid off of the biomedicine science field; the subtitle of the book says everything about his subject.

Is a claim reliable if the results can’t be reproduced? No, it’s not reliable. Is this often the case in biomedicine?

In 2011, scientists at the German pharmaceutical giant Bayer made a splash with news that they could not replicate 75 to 80 percent of the sixty-seven preclinical academic publications that they took on. In 2012, the hematology and oncology department of the American biotechnology firm Amgen selected fifty-three cancer research papers that were published in leading scientific journals and tried to reproduce them in their laboratory. In 89 percent of the cases, they were unable to do so. Several systematic reproducibility projects have been launched subsequently. The “Reproducibility Project: Cancer Biology” selected key experiments from fifty-three recent high-impact cancer biology papers. Although the replicators used the same materials and methods as the original paper, 66 percent of the published work could not be replicated. Several other replication projects are currently ongoing, and as you will see later in this book, the results aren’t looking too good either. If you think it’s bad enough that scientists can’t reproduce other scientists’ data, then consider this. It turns out that most scientists can’t even reproduce their own data. In 2016, the prestigious scientific journal Nature published the results of its anonymous survey. More than 1,500 scientists replied, and in the fields of biology and medicine, 65 to 75 percent of the respondents said they were unable to reproduce data published by others and . . . 50 to 60 percent said they had trouble reproducing their own previous research findings. The implications are scary: it looks like the published body of biomedical literature is unreliable. I attempt to get to the bottom of this matter in this book.

The best part of the book is that Szabo shows the rat race of the entire industry. When young and ambitious scientists start out, they quickly learn that they must publish papers to make it in the industry; in order to get funds, they must publish more. They have to chase a PhD; they should write their research in a way that’s appealing to reviewers who rarely get thoroughly test claims in papers; they will try to become professors; they have to pay publishers a lot of money to have their research published.

Disruptive content or not, the scientific publishing industry is a very big business. The entire scientific publishing industry is estimated to be a $19 billion business globally and annually, with about half of the business held between five companies—Elsevier, John Wiley & Sons, Taylor & Francis, Springer Nature, and SAGE—who together publish about ten thousand journals.

By the way, on those parasitic companies:

The entire scientific publishing industry is estimated to be a $19 billion business globally and annually, with about half of the business held between five companies—Elsevier, John Wiley & Sons, Taylor & Francis, Springer Nature, and SAGE—who together publish about ten thousand journals. Biomedical science is estimated to be at least half of the above figures, with double-digit profit margins. The massive pro bono work that scientists provide to this industry should be considered in that context. One of the greatest revolutions in the field has been the introduction of “open access” journals, which has been driven in large part by the switch to online publishing. In the old system, legacy journals were published on paper, and university libraries would subscribe to them; these subscription fees made up the majority of the publishers’ income. Articles published in these journals were accessible for scientists working at these institutions—but for nobody else. (Many investigators also personally subscribe to Science or Nature, which publish scientific news articles in addition to regular scientific content.) The publication cost of the open access journals is paid by the authors themselves. But, in return, everybody can read them, even lay audiences: no library and no subscription is needed. The hope was that this new publishing system would be more transparent, more cost-effective, and everybody would benefit. Over time, however, the open access fees kept creeping up, and the open access model became a big moneymaker for publishers. The average open access fee is currently a few thousand dollars per paper, but some of the top journals have begun to charge $10,000 or more. Many new publishers have jumped onto this gravy train after seeing the kind of profits that can be made in this area. After all, the new model didn’t need a printing press, and it didn’t need a distribution network; the only “product” of this new kind of publisher was a PDF file made accessible for everyone to download. Many publishers even force the authors to do the professional-looking formatting of this file. And—as stated earlier—the refereeing of manuscripts is principally done by scientists who work for free. This development was certainly a factor in the creation of many new low-quality open access journals where the search for profit takes precedence over scientific quality.

The fact that so many parts of the scientific-research entity is commoditised is sheer capitalism. It’s not good for science, which just falls to the wayside when profit is involved. PhD students aren’t exactly having an easy time to begin with.

According to the 2019 survey conducted by the prestigious journal Nature, 37 percent are very satisfied, and 36 percent are somewhat satisfied with the decision to pursue a PhD, and more than 70 percent say that the program meets their expectations. Shockingly, about half of the responders receive less than one hour of one-on-one time with their supervisors each week. This is one hour per week, not per day. The top concern expressed by PhD students includes their uncertainty about their career prospects (79 percent). There were also many concerns expressed about the overall low success rates of grant applications and the pressure to publish. About 36 percent had reported that they have sought help for anxiety or depression related to their PhD. In most institutions, the acceptance of at least one peer-reviewed publication, preferably with the candidate being the first author, in a reasonably well-respected scientific journal is the expectation.

For the sake of argument, let’s say that the postdoc makes it out of the postdoc phase relatively unscathed and is offered a junior faculty position. The steps are typically instructor, assistant professor, associate professor, and full professor […]. The names of these positions vary from country to country, but the principle is the same: one has to satisfy various expectations—as measured in dollars and cents (i.e., the amount of extramural grant support)—and progress through the promotions committees to reach the next level.

So, students are inundated with work and are constantly pressured to write and get their research published to have a shot at ‘making it’ in the biomedicine field. Naturally, this is not only restricted to the biomedicine field. Just look at Piero Anversa² as an example of fraud. Check out the site Retraction Watch³ that’s run by a couple of hard-hitting scientists^{4 5}.

This book is great in providing a lot of ‘oh my fucking god’ moments. The area should be science. It’s not.

But here comes the real kicker: with all this complicated process, with all these arcane rules and regulations, and with all the time and money spent on grant review, an astonishing degree of randomness is baked into the process. The NIH grant process, after all, was developed in the 1970s, when the overall number of grant applications was a fraction of what it is today and the funding success rates were as high as 50 percent. The same process that may work well with a high granting success rate can fail miserably at the current 80 to 90 percent rejection rates. (And most of us sitting around the table know about it and never mention it—we are in polite circles, as I mentioned.) This randomness has been studied using proper statistical methods. It has been proven, time and again—not only for the NIH grant system but also for a multitude of grant systems in North America and Europe—that the grant reviewing process has an unacceptably high level of randomness and unpredictability. Famously, in a study published in the prestigious journal PNAS led by Elizabeth Pier at the University of Wisconsin, after replicating all relevant aspects of the NIH peer review process (including the crowded rooms and probably the undrinkable coffee as well), the “inter-rater reliability for grant application reviews” was determined by two different statistical methods to be . . . wait for it . . . zero. Two randomly selected ratings for the same application were, on average, just as similar to each other as were two randomly selected ratings for different applications. Different reviewers of the same application, in effect, were not in agreement on anything: not on the strengths, not on the weaknesses. Other studies found strikingly low nonzero numbers ranging from 0.15 to 0.3. Any agreement number below 0.6 can be considered terrible; the desired number would be above 0.75. Let me rephrase the conclusion: different referees will have wildly different views of any proposal. Statistically speaking, the applicant will be funded if luck strikes and all three randomly selected reviewers happen to like a particular proposal. I have asked Pier if her study has received any official follow-up or discussion from the NIH or any other agency. The answer was “no, not really.” The investigators who conducted the landmark PNAS paper on the randomness of the NIH grant review process have drawn their own conclusions. Pier now works at a nonprofit institution that supports U.S. public schools. The center where the research originated is no longer in existence, and the researchers involved have moved on to other institutions, retired, or, sadly, passed away. In my own experience, there may be a somewhat better level of agreement at the top. Occasionally, one sees a top project, by a top investigator, looking at an exciting question. Some grants seem to jump off the page. In cases like this, reviewers tend to agree more.

These are just a few examples from the book. Don’t even get me started on paper-submission farms; AI; the extreme loss of money (the author considers 30-60 billion USD being wasted each year and supports this with science, surprise); the fact that reproduction is seen more as a threat than mandatory science; the so-called review process in publishing…

There are heroes around. These persons are often anonymous scientists.

Numerous individuals or perhaps groups of individuals (data sleuths or data detectives) who work on these issues use pseudonyms (e.g., Clare Francis, Smut Clyde, Aneurus Inconstants, Artemisia Stricta, etc.). They spend significant amounts of time exposing bad science—often working closely with PubPeer or ForBetterScience. Those who work at the cutting edge, discovering and exposing hitherto undetected problems, often remain anonymous. Anonymity protects them from retaliation, but at the same time it can diminish their impact: many journals and journal editors dismiss and do not investigate accusations of fraud that come from anonymous tips. Other data detectives use their own name. Perhaps the most prominent is Elisabeth Bik, who maintains a blog and has detected countless problematic papers over the years—including whole clusters of paper mill products—resulting in about one thousand retractions and one thousand corrections.10 Another one is Nick Wise, whose keen eye has resulted in more than eight hundred retractions.11 Recently, Smut Clyde’s identity was revealed through an article he published on the subject of—what else?—paper mills. His real name is David Bimler, and he is a psychologist formerly with Massey University in New Zealand. He has single-handedly taken down several thousand paper mill products and other fraudulent and questionable publications. Another prominent blogger and activist is Dorothy Bishop. Her scientific background is in experimental psychology, but her blog covers a wide range of topics—from paper mills to the emerging practice of fake/fraudulent (joint) academic affiliations—that apply to all areas of biomedical science. I also recommend the blog of a second experimental psychologist, Adam Mastroianni, who covers a wide range of topics around research, including the futility of the peer review process, various problems related to research grant applications, and how the strong-link/weak-link problem hinders scientific progress.15 There are also several young vloggers on Youtube who are focusing on various aspects of scientific misconduct and fraud. One example of these “young turks” is Pete Judo, of Warwick University in the UK. There are some rare instances of support and occasional public recognition for prominent data detectives. In 2021 Elisabeth Bik received the John Maddox Prize for “courageously advancing public discourse with sound science.” Oransky and Bik are often featured in news articles that point the public’s attention toward scientific misconduct and the reproducibility crisis. But the proactive aspects of scientific fraud detection (i.e., when the issue is not raised by an institutional whistleblower and the system is forced to react) are primarily based on the initiatives and efforts of concerned and dedicated individuals, who often do this on a shoestring budget as a supplemental activity in addition to their day jobs—with potential defamation lawsuits hanging over their heads.

In total, the book is wonderful. It’s written by a scientist who wants to rid science from gunk, and his efforts are affable and admirable. This piece of work should be read by anybody who’s even remotely interested in science and wants to know how corruption works, see what happens when science is supposed to work under the guise of capitalism.