Why most published research results can be right
John Ioannidis of Stanford University is considered a pioneer in the field of meta-research. Now the researcher, who is often referred to as "the conscience of science", has founded a new Meta-Research Innovation Center in Berlin. During his first visit as Einstein BIH Visiting Fellow of Stiftung Charité, he told us why it is good for science to have a guilty conscience and why Berlin is the ideal location for his new institute:
Over ten years ago you published your famous article “Why most published research findings are false”. At that time, the field of meta-analysis wasn’t completely new, but through the dynamics generated by your article, you seem to have harnessed considerable energy in support of your initiative to improve research practice. How would you assess the changes since then?
There have been significant improvements in the way that we research and what we can do with our research. This was made possible through several interventions, ideas and concepts and should be credited to the whole scientific community, not individual people. It’s an evolutionary process that will hopefully lead to our being able to better apply our scientific instruments. Every week, there are new developments with regard to what methods we use, how we recognize and rectify problems in the methodology, how we publish and evaluate research, and according to what criteria we honor researchers. I'm curious to see what next week will bring.
Can you give us a couple of examples of what has changed?
One area in which impressive changes are taking place is data sharing. This development is more pronounced in some disciplines than in others. For example, in genetics, physics and even some social sciences like psychology, scientists are particularly ready to share their data with one another. In other disciplines, such as nutritional epidemiology, people are considerably more reticent in this regard. In the past, there wasn’t even any guarantee that data would be made available at all; today, we argue that the data should be made available – in some fields, even before it’s been analyzed for the first time. Particularly in the heavily quantitatively oriented specialist fields, the so-called “omics”, a single research group only has very limited possibilities of drawing meaningful conclusions from the pieces of data that they collect. However, if all research groups fed their results into a steadily growing database, the scientific community could analyze the entire body of data considerably more efficiently. There have also been massive changes in the process of registering studies. Only ten years ago, registration was rarely used; today, it’s a widespread practice in the case of randomized studies. This gives us a better idea of which results are potentially available compared to the visible portion that’s been published. Also, many disciplines have realized that both their statistical methods and their precautions against bias are inadequate. Until a short time ago, the study sizes and case numbers from many animal experiments, laboratory analyses and clinical studies were far too low to be conclusive and were therefore susceptible to producing false-positive or false-negative results. Scientists from all over the world are unanimous that extensive, carefully conducted and documented studies can be reproduced better.
Funding program
Einstein BIH Visiting Fellows
Funding period
2019 – 2021
Project title
Meta-Research Innovation Center at Berlin (METRIC-B): Identifying and maximizing the use of best scientific practice in biomedicine and beyond
Research areas
Quality assurance, meta-research
Institution
Berlin Institute of Health (BIH)
Since 2013
Co-director of the Meta-Research Innovation Center at Stanford (METRICS)
Since 2010
C.F. Rehnborg Chair for Disease Prevention, Professor of Medicine, Health Research and Policy, Biomedical Data Science, and Statistics at Stanford University, California, USA
2005
Publication of the article "Why most published research findings are false" (PLoS Medicine)
When are you going to write an article entitled “Why most published research results are right”?
There’s no reason why most published articles can’t be right. That article could already be written for many fields. Some of the fields with the greatest credibility problem, however, belong to the most promising research areas. It’s those fields about which we have little knowledge up to now, but where any discoveries are hugely groundbreaking, that need motivated researchers to leave the beaten track and venture into new territory. Here, the failure rate is probably very high, but we need this kind of research too. It may seem paradoxical to be simultaneously making the case for more credible science on the one hand, and for extremely high-risk research with a very high failure rate on the other – but there’s also a little kamikaze pilot inside me (laughs) and we need such kamikaze fliers: not to fall into the abyss, but to try things out and also to fail too. Nevertheless, even these fields have some room for improvement.
You’ve also said in an article that that one of the problems underlying the fact that researchers only publish sensational, positive research results is the funding landscape. How can we motivate researchers to report honestly on failures?
The reward system needs to be adapted to the expectations that we have of science. Research is the best chance that we humans have of leading longer and better lives. So, we need to find ways of rewarding the quality of the preparation, the methodology and the design, as well as the community spirit, of researchers. If it involves high-risk research, the returns per study may amount to less, but even at failure rates of 99.9 per cent, the 0.01 per cent that might lead us to a therapy for Alzheimer’s is worth every cent.
So, we must protect researchers who take considerable risks yet thereby meet high standards of quality and do excellent preparation work. If we don’t do this, in the end we’re rewarding those who present their research results only positively distorted. And that’s already happening to a very great extent, and that’s very dangerous.
You commute between continents as well as between research areas: There is hardly a discipline in the life sciences with which you have not yet come into contact...
Of course, it wasn’t my plan to have no focus. In the course of my career, however, I’ve tried to get to grips with as many new ideas and fields as possible. The common denominator is that I’ve always concerned myself more with methodology, study design, statistics and bias than with their discipline-specific application. In this respect, there are often many commonalities between apparently different research fields. Some patterns in using and also abusing certain methodologies can be observed in both the neurosciences and the economic sciences. I find it exciting to learn approaches and methods that have proved helpful in a given area and to see whether they could also be applied in other specialist fields. Naturally, there are always limitations. Nevertheless, I think that increasing specialization leaves little room for mavericks, and as a rule they’re even punished when applying for funding. But there’s a lot we could learn from interdisciplinarity.
To a certain extent, you embody the ancient concept of the Greek philosopher mathematician scientist in the era of meta-analysis…
A comparison to the ancient Greek visionary thinkers is setting the benchmark rather high! (laughs) But of course, as a Greek it is practical that, for example, most of the academic vocabulary is of Greek origin. This makes you feel very much at home with the scientific literature, even when you haven’t got a clue about the subject otherwise.
It also creates a certain continuity between ancient philosophy and modern science – although scientific methods in their modern form emerged in the 16th century and can’t necessarily be traced back to Plato or Aristotle. Nonetheless, they were already concerned with many of the fundamental ideas, challenges and concepts that we’re still debating today.
You’re now opening the Meta-Research Innovation Center Berlin (METRIC-B). What are your plans for the METRIC-B and your collaboration with the QUEST Center of the Berlin Institute of Health?
METRIC-B is intended to be a counterpart of the Meta-Research Innovation Center at Stanford – METRICS, where the focus is on researching research itself. Our aim is to improve research practice and to thus enhance the quality and credibility of science. Our work is characterized by interdisciplinarity and the ability to learn from other disciplines and to combine them together. Berlin is an exciting environment for the new Meta-Research Center, as many research groups here are already sensitized to the issue. One and a half years ago, the QUEST Center was founded at the Berlin Institute of Health in order to change the scientific culture in biomedicine in Berlin: the agenda of QUEST thus has many points of intersection with the aims of METRICS and METRIC-B. Many scientists I know in Berlin, Germany and Europe would like to take part in our projects, so I want to create a hub for networking and an arena for different initiatives. Being on the spot in Berlin offers me possibilities that I wouldn’t have if I was located on the other side of the world. In particular, it allows me to meet interesting people and to network with others beyond the boundaries of individual research fields.
In Germany, we often look admiringly at the United States with its many elite universities and Nobel Prize winners. In your opinion, what role do Germany and, in particular, Berlin play with regard to the biomedical research community?
Berlin is the ideal location for starting our quality offensive in biomedicine. There is outstanding research in the USA and I’m very happy to be working at Stanford University – the research environment there is very special. Nevertheless, equally impressive research is being done in Europe, particularly Berlin – and Europe has great potential to achieve still more. Not too long ago, Berlin was regarded as the capital of the natural sciences and physics, mathematics and academic research. A lot depends on how much effort a society makes to take on a leading role in science. Europe has a strong scientific tradition and today Germany is the epicenter of Europe. Of course, no individual can make this happen by waving a magic wand – it calls for social engagement. I’m very proud to come to Berlin as an Einstein BIH Visiting Fellow. I think the program is a brilliant idea. For me, it’s a unique opportunity to create something in a place that feels almost like home to me. This makes me even more enthusiastic – and enthusiasm is one of the main ingredients for initiating a big project. So I’m eager to see what we can achieve together here over the next years.
March 2019 / MM