Skip to main content

Essay Contest: Physics in 2116

Accompanying Frank Wilczek’s article about what physics will look like in 100 years is an opportunity for our readers to submit their own predictions for the chance to win $7500. Find out about our Physics in 2116 contest online.

New York Times columnist seeks scientists’ views on irreproducibility

A Times blurb summarizes, “Are some scientific fields, like physics, less prone to faulty research?”

John P.  A. Ioannidis’s 2005 PLOS Medicine paper “Why most published research findings are false” continues to resound. On 31 January, for example, the President’s Council of Advisors on Science and Technology examined the issue, often called irreproducibility. Nature maintains a special archive on it.

Now New York Times science columnist George Johnson is asking scientists to comment. He observes that “physical science—everything from geology and climatology to cosmology and particle physics” has not “received the same kind of scrutiny” as other fields. He wonders whether irreproducibility is less common in physical science.

The introduction to Nature’s archive begins without excluding physical science: “No research paper can ever be considered to be the final word, and the replication and corroboration of research results is key to the scientific process.” But the second sentence narrows the focus to the life sciences: “In studying complex entities, especially animals and human beings, the complexity of the system and of the techniques can all too easily lead to results that seem robust in the lab, and valid to editors and referees of journals, but which do not stand the test of further studies.”

Nature’s 24 April 2013 editorial “Announcement: Reducing our irreproducibility” employed that narrowed focus in promising, “From next month, Nature and the Nature research journals will introduce editorial measures to address the problem by improving the consistency and quality of reporting in life-sciences articles.” The promised measures included increased attention to experimental methods, provision of unlimited space for descriptions of methods and of experimental design, encouragement of the posting of data, and additional rigor in assessing statistical practices.

So what about the physical sciences in all of this? Johnson discussed that with Ioannidis, who holds several professorial titles at the Stanford University School of Medicine.

“Physical sciences have a stronger tradition of some solid practices that improve reproducibility,” [Ioannidis said] in an email. Collaborative research, for example, is customary in physics, including large consortiums of experimenters like the teams that announced the discovery of the Higgs particle. “This certainly increases the transparency, reliability and cross-checking of proposed research findings,” he wrote.
He also mentioned more stringent statistical standards in particle physics—like the five sigma measure I mentioned in my second column—as well as sociological factors: “There seems to be a higher community standard for ‘shaming’ reputations if people step out and make claims that are subsequently refuted.” Cold fusion was a notorious example. He also saw less of an aversion to publishing negative experimental results—that is, failed replications.
Another how constrained a field is in generating plausible hypotheses to test. Almost anything might be suspected of causing cancer, but physicists are unlikely to propose conjectures that violate quantum mechanics or general relativity.

Johnson ended the column by asking scientists to comment:

Dr. Ioannidis said he was struck by an “arrogant dismissal” by some physical scientists of the suggestion that their field might be anything less than pristine. We won’t really know, he said, unless there are empirical studies like the recent ones in medical science.
“I have no doubt that false positives occur in all of these fields,” he concluded, “and occasionally they may be a major problem.”
I’ll be looking further into this matter for a future column and would welcome comments from scientists about the situation in their own domain.

Johnson can be contacted via his page at the Times.


Steven T. Corneliussen, a media analyst for the American Institute of Physics, monitors three national newspapers, the weeklies Nature and Science, and occasionally other publications. He has published op-eds in the Washington Post and other newspapers, has written for NASA's history program, and is a science writer at a particle-accelerator laboratory.

Submit comment
Comment moderation successfully completed
ec53185b92e297f26a707ada48e7a20c ptol.magazine_postzxybnytfddd