You can’t google a sneeze or a pimple without stumbling on numerous articles and blog posts containing phrases like “scientists agree” and “research shows”. Unfortunately, this kind of article rarely includes additional phrases such as “cluster sampling technique” or “p value of under 0.01”, which are some of the first things any real scientist will look for.
Physicians, scientists and individual studies sometimes just don’t agree with each other. Data measured using different techniques, or in different populations, sometimes show divergent results, and the real world and human body are too complex to make it possible to control for every single factor. This is unlikely to ever change, but the scientific community has, over the years, structured medical and other research in such a way that these differences can mostly be resolved or at least interpreted correctly, and errors kept to a minimum.
It would seem that the primary goal of all medical research should be an answer to the question “did the patient get better or worse?” Unfortunately, it is rarely possible to give a completely clear-cut answer to such a question.
A research report might contain a phrase such as “a significant improvement of 2% was observed”. How important a difference of 2% might be depends on the circumstances, but it doesn’t seem very significant if it refers to something like blood pressure or body mass. This is, however, not at all what statistical significance means: it refers not to how dramatically one outcome differs from another, but to how likely it is that the difference is not just the result of random forces, but of the particular effect being studied.
A large part of medical research is concerned not with anatomy or molecular biology, but with statistics. This is the only way of knowing not only what conclusions to draw from data, but how certain of those conclusions we are allowed to be.
Very High Cost
When someone consults a doctor, they want to be sure that the letters “M.D.” after his name mean that they can trust his training and advice. Similarly, a new medicine can’t be put on the market until both its safety and effectiveness have been proven to a very high degree of statistical significance. This process takes years and costs hundreds of millions, and often fails partway through, leaving the manufacturer with nothing but loss.
Since people’s lives and health are at stake, similar degrees of certainty are expected for any kind of medical research. This often implies dozens if not hundreds of volunteer or paid participants, often for a period of more than a year, often with periodic blood tests or other kinds of monitoring.
All of this costs money, even if the test subjects are rats or monkeys. One potentially important implication of the high cost of medical research is this: treatments that aren’t commercially valuable do not receive the same level of scientific attention as those that can turn a profit. For a company that may sell thousands of doses of a life-saving drug in the future, research is an investment. When it comes to holistic or alternative treatments, the amount of money to be made is much less, and no such research spending can expected to yield a monetary return at any point in the future. In the United States, less than a quarter of medical research is federally funded, and this budget is spread over a very large number of avenues of inquiry.
High Level of Rigor
The precision of any kind of scientific research requires heavily on the system known as peer review. Essentially, any theory or research is reviewed by knowledgeable individuals before it can even be published, and is understood to be open to criticism thereafter. Numerous papers have been retracted by their authors after others pointed out errors in them.
If a physicist publicizes new data or speculation, usually no more than a few hundred people in the world will be able to understand the field well enough to comment on it intelligently. In the medical field, however, thousands upon thousands of educated professionals can be expected to scrutinize any study’s experimental design, statistical methods, conclusions as compared to that of previous research and conceptual arguments. A fraudulent (it happens!) or carelessly performed study is almost certain to be exposed, and if the error is sufficiently great, the author’s career can be expected to suffer.
Bias Towards the Conventional
It’s a basic tenet of any kind of scientific research that what most informed people believe to be true, and what has worked well in practice over the years, is accepted as fact by default. A new or contrarian view requires a much higher standard of proof before being accepted.
Another way to interpret this is as Occam’s razor: the simplest explanation, or the one requiring the fewest unproven assumptions, is probably the correct one. Any new theory that presumes that basic natural laws aren’t what they’re thought to be, or that a causal connection that’s fairly obvious doesn’t exist, is conceptually less likely to be true and therefore needs to earn its stripes with masses of data before it can be taken seriously. This may mean that new discoveries can take a long time before being accepted, but it also prevents proven conventional knowledge from being discarded without sufficient evidence.
Magazine journalists and blog writers are often pressed by a deadline, lack scientific training or are influenced by their own opinions. When it comes to any second-hand claim that “science” is sure of this or that, the only way to verify it is to go to the primary sources and try to make sense of them. In the case of medical research, even some doctors don’t make much of an effort to keep abreast of the facts, so it may be necessary to talk to a specialist or even two.