Coffee: Elixer of Life or Poison? - What's the deal with all these conflicting studies?

It seems like every day brings another study about something that will either help you live to 100 or kill you before you wake up tomorrow. The irony is that often the very thing that is supposed to deliver immortality on Monday becomes the agent of your early demise on Tuesday. Just this week a study was reported that linked the consumption of 4 or more cups of coffee per day and early death (Association of Coffee Consumption With All-Cause and Cardiovascular Disease Mortality) while another study 2 days later showed that coffee was able to improve a common condition known as fatty liver (High coffee intake is associated with lower grade nonalcoholic fatty liver disease). Why is it that there are so many conflicting reports? It makes you wonder if they are just playing games and if anyone really knows what they are talking about. While that is not true at all, the way in which these things are reported by the mass media sure makes the average person just want to throw up their hands and stop listening. In this article I will attempt to explain why this happens, why we shouldn't throw our hands up, and what we can do to sort out the real facts from the media hype.

There are several reasons for the confusion.

• Comparing Apples and Oranges - Studies that seem to be looking at the same question may actually be looking at different things. The two coffee studies mentioned above are not actually examining the same question. One looked at lifespan and the other examined coffee's effect on a condition known as fatty liver. They really do not contradict each other since its possible that coffee could help someone's fatty liver but still cause an earlier death by negatively affecting some other organ system.
• Studies are incorrectly interpreted - News reporters are not scientists and most of them with few exceptions have little or no scientific training. This often results in a situation where the reporter doesn't understand the story they are covering and they misinterpret the outcome of the study. In the example above several news reporters proclaimed that these two coffee studies contradicted each other and as I already pointed out that is not true.
• Not all studies are equal - There are good studies and bad studies. Good studies require thoughtful design, take a lot of time, a lot of people, and a lot of money. Bad studies are much easier to do. Learning how to tell a good study from a bad one really isn't that difficult but reporters either don't understand the difference or don't take the time to examine this issue.  I'll spend some time below going over the basic questions you should ask and the tools you can use to determine if the results of the latest study are valid before you give up your favorite beverage.
• Science doesn't progress in a straight line - Even well done studies can't eliminate all confounding variables. We can't entirely eliminate genetic and environmental differences between two groups of test subjects that may throw off our results. We can try to compensate for those things but its not always possible to anticipate everything. For this reason any given study doesn't mean a lot and needs to be followed by additional studies done by other researchers. Retrospective studies need to be followed by double blind placebo controlled trials. Small pilot studies need to be followed by larger trials that may better eliminate "noise" in the data. This is the way science proceeds. One small study may show a benefit to a particular food, drug, or activity. A second larger study may not confirm that. A third study that is better designed and controlled may be done to clarify the conflicting results and this study may confirm or refute the original study.  Sometimes a "Meta-analysis" will be done to combine the results of a group of studies in an attempt to gain the power of a much larger trial. The process proceeds until there is sufficient evidence to come to a conclusion with some reasonable level of certainty. This is the way scientific investigations proceed. It is not confusing to scientists who do this every day but it can be bewildering to the lay person, especially when the information is coming from a reporter who also doesn't understand how the scientific method works.

Science and medicine progress in a very deliberate fashion to find answers to questions we all want. They do this by using something called the scientific method. Its a method by which we ask a question, propose a possible explanation, and then devise experiments to prove or disprove our theory. As evidence mounts from those experiments the theory is either discarded or accepted depending on what the evidence shows. Its important to use the scientific method because it is the best and only way we have of finding answers that are not influenced by human biases, preconception, mysticism, and superstition. It is the best way we have to find out the truth about the word around us.

Along the way as we conduct studies to answer our questions we may find that these studies do not always give us the same answer. What we are witnessing in all the reports we hear is science in action. Scientists design studies in an attempt to answer questions we have but good studies are difficult to do. The investigator must design a study that removes as many conflicting variables as possible so that the only variable left is the one we are trying to examine. If you want to study the effect of coffee on life span the perfect study would look at thousands of identical twins. We would randomly assign one twin to drink coffee and the other to drink a placebo that looked smelled and tasted like coffee but wasn't coffee. Neither the investigators nor the participants would know who was drinking the placebo and who was drinking the real coffee during the trial period. Both groups of people would have identical lives and identical diets. Then they would follow these people over many decades to see what diseases they developed and at what age they died. Finally when the study was done the code would be broken and a analysis would be done to see if there was a statistically significant difference in the rate of disease and the life span of the two groups. By imposing all these limitations we can be sure that the only difference between the two groups is the consumption of coffee. In all other ways they would be nearly identical.

As you probably already figured out, doing a study that met all those criteria would be close to impossible. At the very least it would be very difficult, time consuming, and expensive. Its certainly wouldn't be easy to find a large number of identical twins who would agree to a lifelong study that imposed these restrictions.

So how do we ever set up a study that will answer our question? While its nearly impossible to put together a perfect study we can design a good study. The best way to do this is to design a large double blinded randomized placebo controlled study. Ideally in this type of study thousands of participants are enrolled and randomly assigned to one of two groups. One group is given a placebo and the other group gets the food or drug which we are trying to study. We use large numbers of people and randomly assign them to groups in order to increase the probability of getting two groups that are as similar as possible. If we allowed the doctors or the patients themselves to choose the group they wanted to be in then we may introduce unwanted variables into the study. If for example we were studying the effect of multivitamins on health and we allowed the participants to choose which group they wanted to join we might find that people who were more health conscious went into the vitamin group rather than the placebo group and this group might have other healthy habits like better diets and exercise routines which would artificially skew the results.

When we say a study is double blinded what we mean is that neither the doctors nor the subjects know who is getting what during the study. This is important because physicians and patients might report different responses to treatment if they think they are on the placebo or the real thing. Blinding the doctors and participants reduces the risk of biases being introduced into the results.

Small studies, studies that are not double blinded, or in which the participants were not randomly assigned should be  viewed with some skepticism and should not be used  for major life decisions. Although news program often fail to give these sort of details about the studies they report on its not hard to find them on the internet. I'll show you how I usually do this.

In the cases above I Googled the key words "coffee fatty liver" for one study and "coffee life span" for the other and clicked the News tab at the top since it had just been in the news. From this I was able to learn some basic details and the names of the journals the studies were in. Once I had this information I went to www.scholar.google.com and put in the name of the journal along with the same search terms I used above. That brought up the links to the actual articles. Its not necessary to read the entire article. Just read the few paragraphs in the abstract that have the methods and the conclusions and you will learn most of what you need to know about how the study was done and what the results were.

So far we have discussed double blind placebo controlled trials but many of the studies quoted in the news are not placebo controlled at all. Most of them are what we call "retrospective" or "population" studies. These studies are generally easier and cheaper to do but are of limited usefulness because we don't really control a variable and then observe the outcome. We simply ask a bunch of people what they eat for example and then ask them about their health problems. The study above linking coffee intake to an early demise was this sort of study. They asked people how much coffee they drank and then looked at the age when they died. People who drank more than 4 cups a day died at earlier ages. Media reporters then took this correlation ( more coffee linked to earlier deaths) and assumed causation ( more coffee was causing earlier deaths). While its tempting, its always dangerous and wrong to assign causation when all you have is a study showing correlation. There are other possible explanations for the outcome in a study like this. Its possible that coffee drinkers have a higher rate of other unhealthy habits and its those habits that increase the risk of premature death. For example coffee drinkers are more likely to be smokers and that may have lead to the early deaths rather than the coffee itself. Or perhaps patients with certain serious health problems drink more coffee in which case it would be the deadly disease that causes the increased coffee consumption and not the other way around. Its also possible that the correlation may have been a fluke and may not show up in other studies. ( For more on this see previous blog entry "What do Pirates and Global Warming have to do with Autism and Labor Induction?")

To summarize quickly:

• When two studies seem to give conflicting results ask yourself if they are both looking at the same question.
• Most media quoted studies are retrospective or population based studies that only show a link or correlation and should be viewed with a skeptical eye. If the reporter says a study showed "a link between" A and B this is most likely the case. Try to remember all the shortcomings associated with this sort of study and resist the temptation to assign cause and effect.
• Double blinded placebo trials are the gold standard when it comes to studies, but bigger is always better. Studies on less than 100 people are small and should not be used alone for important decisions. Studies with thousands or tens of thousands of patients give much more reliable results.
• No single study regardless of the type can provide a reliable answer to important questions. We should always wait to see if other researchers can reproduce the same results.
• You don't need to do your own research on every study you hear about in the news. None of us have the time for that, but If the results of a study are important to you do your homework. Don't rely on the media reports since these are transmitted mostly by reporters who are not trained in the sciences and often misrepresent or don't understand the material they are presenting. Use PubMed or Google Scholar to find articles of interest.
  
• There's no need to stop drinking coffee.