More on the weaknesses of randomized trials (created 2005-02-14)

This page is moving to a new website.

I've written about the randomized trial and how it can sometimes oversimplify the research question and how it often raises difficult ethical questions. Complementary and Alternative Medicine (CAM) practices raise interesting questions about the randomized trial. Here are a few weblog entries on the topic:

• Evidence Based Complementary and Alternative Medicine (June 15, 2004)
• So you want to volunteer for a research study? (August 4, 2004)
• Is the randomized trial the gold standard for research? (September 23, 2004)
• Alternatives to equipoise (November 24, 2004)
• The ethics of randomization (January 14, 2005)
• More thoughts on equipoise (January 17, 2005)
• Ethical principles for Complementary and Alternative Medicine Research (January 24, 2005)

and I am writing about this in the first chapter of my book on Statistical Evidence.

Two new web resources seem to confirm this. The first:

• http://www.cebm.utoronto.ca/syllabi/gs/intro.htm

discusses the problems with applying EBM to surgery outcomes. I have several citations about this in my bibliography, but this page raises an additional issue. Surgery is a highly individualized intervention with wide practices in variation. Unless you can quantify this practice variation, randomized trials will always have an unacceptable amount of noise in them. Also, since there is a difficult learning curve, it is unrealistic to expect a surgeon to offer both a new and a standard surgery with equal skill in a randomized trial.

The second resource discusses the application of Evidence Based Medicine to CAM.

• http://www.cebm.utoronto.ca/syllabi/comp/intro.htm

This article also mentions the large practice variation in CAM and cites the variety of ways that accupuncture is performed as an example. The author also suggests that a formal decision analysis might work to the benefit of CAM.

CM therapies have long been prey to the view that no decision is worth taking until the evidence crosses a certain threshold (the level of which is never made explicit) and at which point the evidence becomes "strong." Decision analysis, conversely, allows an explicit estimate of how much evidence is enough evidence.

In the example that he cites, if the cost of untreated disease is 20 times worse than the cost of the treatment, then a threshold for using the treatment is 5% or greater. This might be an excellent example of how to apply the NNT in a practical sense.