StATS: I hate bad research examples (April 23, 2008).

This page is moving to a new website.

Someone wrote in asking if I know of any good examples of research studies that illustrate problems of making false generalizations. I had to mention my book, of course,

• Stats: Statistical Evidence in Medical Trials (January 27, 2005)

which has lots of commentary of actual publications, most of which are open source and freely available on the web. I am way overdue with providing answers to the exercises, by the way. It has been very busy for me for quite a while—pretty much for the entire century so far.

For what it’s worth, I do have a pedagogical bone to pick. I believe it is not a good idea to find a “bad” publication and tear it apart. It encourages people to view research publications in a dichotomy (good paper/bad paper) when there are very few articles that are so bad that they are not worth reading. Similarly, there are very few articles that meet a standard of perfection that some people seem to apply. I also rebel against the “checklist” mentality that seems to be rampant in Evidence Based Medicine. This belief that if you have blinding, randomization, and intention to treat, you are home free is a dangerously simplistic approach. Although I hate the word, I believe you have to take a holistic approach to a research publication. Take a look at the paper as a whole with all its bright points and all its flaws. Do the bright points sufficiently outweigh the flaws to make the research sufficiently persuasive?

So I would encourage anyone wanting to teach about how to interpret the literature not to take a paper that has led to false generalizations. Just take a real paper and discuss its strengths and weaknesses as a whole.

This page was written by Steve Simon while working at Children's Mercy Hospital. Although I do not hold the copyright for this material, I am reproducing it here as a service, as it is no longer available on the Children's Mercy Hospital website. Need more information? I have a page with general help resources. You can also browse for pages similar to this one at Category: Teaching resources.