[The Monthly Mean] January 2012 -- Some quasi-experimental alternatives to randomization
You are viewing an early draft of the Monthly Mean newsletter for January 2012. I hope to send this newsletter out by January 31.
The Monthly Mean is a newsletter with articles about Statistics with occasional forays into research ethics and evidence based medicine. I try to keep the articles non-technical, as far as that is possible in Statistics. The newsletter also includes links to interesting articles and websites. There is a very bad joke in every newsletter as well as a bit of personal news about me and my family.
Welcome to the Monthly Mean newsletter for January 2012. If you are having trouble reading this newsletter in your email system, please go to www.pmean.com/news/201201.html. If you are not yet subscribed to this newsletter, you can sign on at www.pmean.com/news. If you no longer wish to receive this newsletter, there is a link to unsubscribe at the bottom of this email. Here's a list of topics.
--> Some quasi-experimental alternatives to randomization
--> TBA
--> TBA
--> Monthly Mean Article (peer reviewed): What does randomisation achieve?
--> Monthly Mean Article (popular press): TBA
--> Monthly Mean Book: TBAs
--> Monthly Mean Definition: TBA
--> Monthly Mean Quote: "Of course, from the quasi-experimental perspective...
--> Monthly Mean Video: TBA
--> Monthly Mean Website: TBA
--> Nick News: TBA
--> Very bad joke: TBA...
--> Tell me what you think.
--> Join me on Facebook, LinkedIn and Twitter
--> Some quasi-experimental alternatives to randomization. There are many situations where you can't use randomization. When you have absolutely no control over the intervention at all, you use what is typically called an observational study, such as a cohort, case-control, cross-sectional, or historical control study. There's a gray area, however, between observational studies and randomizaed studies where you have some level of control over the intervention, but not enough control to randomize. These settings occur quite often in quality improvement studies, and the class of designs used are referred to as quasi-experimental designs.
Below is a schematic graph of a randomized experiment. You take a group of 100 patients and randomly assign 50 of them to the treatment and 50 to the control. The red letter C represents the mean of the 50 control subjects and the green letter T represents the mean of the 50 treatment subjects. The means are identical (within sampling error, of course) prior to the intervention ("Pre" on the x axis). After the intervention ("Post" on the x axis), the means differ. The size of this difference in means is the estimated effect of the treatment or intervention.
Now, if you randomize, you really don't need to take a measurement at baseline.
Randomization assures comparability, so any difference seen must be due to the intervention. Now getting a baseline measurement is always a good idea, even in a randomized experiment because it offers an important quality check of the success of the randomization, and having a baseline can improve precision. For non-randomized experiments, of course, you need the baseline, because imbalances at baseline can produce an atefactual response
or can mask a true response
In many settings, you control when the intervention occurs, but you can't control it at a fine enough level to allow randomization. For example, you make a change in training or education practices at a hospital. You can't randomly assign half of the health care professionals to the training, because they work in such close proximity that the changes in the trained group will rub off on the control group. Or you change the environment, such as the location of the desk where patients are screened in an emergency room. You can't randomly shift the desk back to its old spot for half of the arriving patients. So the intervention has to be applied in an all or nothing fashion.
Now you could still randomize if you had ten hospitals, and you randomly select five for your intervention and five for your control group. But it's hard enough to get a study going at a single hospital, and trying to coordinate at multiple sites is a problem. So what you end up with in many cases is a pre/post design with no concurrent control group.
This is a very weak research design, but it is commonly used. If the change you see is very large, if the effect has a solid and scientifically plausible explanation, and if the result is consistent with other studies, then even a weak design like this can have credibility.
There are some good alternatives, however, to a pre/post design with no concurrent control group. A very simple alternative is called the withdrawal design or the ABA design. You make an assessment at T0 while still under the control conditions. Then you switch to the treatment and make an assessment at T1. Then you withdraw the treatment (switch back to the control condition) and make an assessment at T2.
The graph above shows a positive effect in a withdrawal design. You see an improvement when you intervene and that improvement disappears when you withdraw that intervention.
In contrast, this figure shows a negative effect in a withdrawal design. Something else was going on at the time that the intervention occured because when the intervention was withdrawn, the effect persisted.
A simple example of the withdrawal design is described in the book, The Lucifer Effect: Understaning How Good People Turn Evil, by Philip Zimbardo. He described an experiment (not his infamous prison experiment) that used a withdrawal design. A researcher wanted to show how anonymity increases the tendency to engage in violent and aggressive actions. He set up an experiment with school children at a Halloween Party. They had a choice between several games some of which were cooperative and some of which were aggressive and competitive. The children were assessed at the start of the party, before they had donned their costumes. After they had donned their costumes (which were designed to provide them with a high degree of anonymity), they were assessed again. The degree of engagement in aggressive and competitive activities increased when the children had greater anonymity. Now you might think of an alternative explanation for this finding, because maybe the children changed just as the party went on. To discredit this alternative explanation, they then asked the children to remove their costumes and assessed them a third time. The aggression and competition dropped to the level prior to donning costumes.
The withdrawal design can be extended to have a re-introduction of the intervention after withdrawal (an ABAB design) or an introduction of a second intervention after withdrawal (an ABAC design). These designs are also very common in single subject research designs.
The withdrawal design is a very simple example of using more than one control group in a research design to overcome some of the weaknesses caused by inability to randomize.
Another design used in this setting is the interrupted time series. As mentioned above a trend in a pre/post design with no concurrent control group, baseline imbalances could produce an artefactual positve difference or could mask a true shift in the outcome. If you collect a time series including several data points prior to the intervention and several after the intervention, then a pattern like the one shown above.
Did you like this article? Visit http://www.pmean.com/category/QualityControl.html for related links and pages.
--> TBA [[Insert content here]]
Did you like this article? Visit http://www.pmean.com/category/[[insert category page]].html for related links and pages.
--> TBA [[Insert content here]]
Did you like this article? Visit http://www.pmean.com/category/[[insert category page]].html for related links and pages.--> Monthly Mean Article (peer reviewed): Adam La Caze, Benjamin Djulbegovic, Stephen Senn. What does randomisation achieve? Evidence Based Medicine. 2012;17(1):1 -2. Excerpt: "What are the benefits of random allocation in clinical studies? John Worrall, a philosopher of science, recently questioned whether evidence-based medicine's advice to base therapeutic decisions on the results of randomised controlled trials (RCTs) could be justified.1 2 Here we provide a response to Worrall and others who challenge the epistemological value of RCTs." [Accessed on January 25, 2012]. http://ebm.bmj.com/content/17/1/1.short.
Did you like this article? Visit http://www.pmean.com/category/RandomizationInResearch.html for related links and pages.
--> Monthly Mean Article (popular press): TBA.
Did you like this article? Visit http://www.pmean.com/category/[[insert category page]].html for related links and pages.
Did you like this book? Visit http://www.pmean.com/category/[[insert category page]].html for related links and pages.
--> Monthly Mean Definition: TBA [[Insert content here]]
Did you like this article? Visit http://www.pmean.com/category/[[insert category page]].html for related links and pages.
--> Monthly Mean Quote: "Of course, from the quasi-experimental perspective, just as from that of physical science methodology, it is obvious that moving out into the real world increases the number of plausible rival hypotheses. Experiments move to quasi-experiemtns and on into queasy experiments, all too easily." Donald T. Campbell, in Methodology and Epistemology for Social Science: Selected Papers, page 322.
Did you like this video? Visit http://www.pmean.com/category/[[insert category page]].html for related links and pages.
Did you like this website? Visit http://www.pmean.com/category/[[insert category page]].html for related links and pages.
--> Tell me what you think. How did you like this newsletter? Give me some feedback by responding to this email. Unlike most newsletters where your reply goes to the bottomless bit bucket, a reply to this newsletter goes back to my main email account. Comment on anything you like but I am especially interested in answers to the following three
questions.
--> What was the most important thing that you learned in this newsletter?
--> What was the one thing that you found confusing or difficult to follow?
--> What other topics would you like to see covered in a future newsletter?
I received feedback from [[insert number]] people. One person liked the discussion of the minimum clinically important difference, the linear regression explanation, and the discussion of two different Chisquare tests from the same data set. Another person liked the Statistics take-off on Joni Mitchel's "Both Sides Now" and reposted the link on her Facebook page. That's the closest I will ever come to promoting a viral sensation on the Internet.
--> Join me on Facebook, LinkedIn, and Twitter. I'm just getting started with social media. My Facebook page is www.facebook.com/pmean, my page on LinkedIn is www.linkedin.com/in/pmean, and my Twitter feed name is @profmean. If you'd like to be a Facebook friend, LinkedIn connection (my email is mail (at) pmean (dot) com), or tweet follower, I'd love to add you. If you have suggestions on how I could use these social media better, please let me know.
What now?
Sign up for the Monthly Mean newsletter
Review the archive of Monthly Mean newsletters
This work is licensed under a
Creative
Commons Attribution 3.0 United States License.