P.Mean: Good examples of bad studies (created 2008-09-26).

This page is moving to a new website.

Does any one have a good example of a fairly flawed therapy article for a course that I teach in EBM. Seems most of the articles I find aren't too badly designed. Students always want to see some bad articles to critique.

Are you and I reading the same journals?

Sorry, that was a cynical comment. I'm not much of cynic actually, and if anything I would discourage this activity.

I've always disliked the concept of finding an article that you can trash. I think it creates a sense of cynicism in students and also develops the false impression that critical appraisal means literally being critical of everything you see. It also encourage black and white thinking while most studies are on a continuum of the persuasiveness of the evidence.

I've heard the comment made more than once along the lines of read the methods section first and if you see something wrong then stop reading the article. I would disagree with this advice. I would say that you still read the methods section first, but only to assess the degree of persuasiveness that this article is likely to have. Then look at the results. Finally, see if there is any corroborating evidence (plausible mechanism, replication of an early result, strong effect, dose-response pattern, specificity of effect, and so forth).

Conflicts of interest factor in here also. If the evidence is highly objective, then it doesn't matter if there was pharmaceutical money involved. But a borderline finding becomes that much less persuasive when there is such a conflict.

I do mention a few bad studies in my book (Statistical Evidence in Medical Trials). For example, there is the famous study by Linus Pauling about vitamin C that prospectively evaluated patients with terminal cancer on vitamin C but used for a control group patients selected from a death certificate database.

But those examples are to illustrate a specific lesson. I would be uncomfortable asking students to find the flaw in these studies. You should critique studies that are representative of what students will encounter in the real world. If those articles are mostly well designed, then so be it..

Creative Commons License This work is licensed under a Creative Commons Attribution 3.0 United States License. This page was written by Steve Simon and was last modified on 2010-04-01. Need more information? I have a page with general help resources. You can also browse for pages similar to this one at Category: Teaching resources.