StATS: Oops! I found statistical significance in my pilot study (September 18, 2006).
Dear Professor Mean, I ran a pilot study to help me estimate the sample size for a
larger study. There were 15 patients in each arm of the pilot, and when I analyzed the pilot
data for efficacy in the primary outcome, I was shocked to find out that the results were
already statistically significant. Do I still need to run the full study?
Before you do anything else, go out and buy a lottery ticket. You obviously have a Midas
touch.
The first thing you should do is to go back and look at the original protocol. Did you
include a sentence like
"If the results of the pilot study achieve statistical significance, we will..."
I didn't think so. If you had that level of foresight, you wouldn't be writing to me. So
what you are looking at is a post hoc modification of the original research protocol.
Post hoc modifications of research protocols happen all the time. Stephen Senn has a cute
definition:
"Medical statistician: one who will not accept that Columbus discovered America
... because he said he was looking for India in the trial plan."
Still, it is worth noting that post hoc modifications will dilute the credibility of your
research finding. Is your research finding so robust that it can withstand such dilution?
If you answer yes, then you have to put yourself conceptually in a time machine and ask
the question "If I was smart enough during the writing of the protocol to consider the
possibility of stopping the study after the pilot, how would I have written the protocol?"
You'd probably include some sort of adjustment to the p-value to reflect the fact that you
have potentially multiple peeks at the data. Analyze the data under the new protocol
procedures.
If you answer no, then pretend that you never looked at statistical significance in the
pilot study. Proceed as originally planned with the full study. The extra work is intended
not to offer proof of what you already know, but rather to ensure that your results will be
treated seriously when you do publish.
Another possibility is to treat the pilot study as a preliminary finding in need of
replication and then replicate the pilot study with an additional 15 patients in each arm.
More information about Early Stopping in Clinical Trials.
Other pages on this website:
Books:
-
Group Sequential Methods with Applications to Clinical Trails. Christopher Jennison,
Bruce W. Turnbull (2000) Boca Raton, Florida: Chapman & Hall/CRC.
-
Biopharmaceutical Sequential Statistical Applications. Karl E. Peace (1992) New York:
Marcel Dekker, Inc.
-
Sequential Methods in Statistics. Barrie G. Wetherill, Kevin D. Glazebrook (1986) New
York: Chapman and Hall.
Research articles:
-
Monitoring randomised controlled trials. Parkinson's disease trial illustrates the
dangers of stopping early. K. R. Abrams. British Medical Journal 1998: 316(7139);
1183-4.
[Medline] [Full text]
[PDF]
-
Lessons learned from a clinical trial. P. W. Armstrong, L. K. Newby, C. B. Granger,
K. L. Lee, R. J. Simes, F. Van de Werf, H. D. White, R. M. Califf. Circulation 2004:
110(23); 3610-4.
[Medline] [Full text]
[PDF]
-
Responsibilities of sponsors are limited in premature discontinuation of trials.
Richard Ashcroft. BMJ 2001: 323(7303); 53-.
[Full text]
-
Selegiline, or the problem of early termination of clinical trials. The clinical
questions are not well answered, and probably never will be. M. M. Breteler. British
Medical Journal 1998: 316(7139); 1182-3.
[Medline] [Full text]
[PDF]
-
Tamoxifen for the prevention of breast cancer. Important questions remain unanswered, and
existing trials should continue. P. Bruzzi. British Medical Journal 1998: 316(7139);
1181-2.
[Medline] [Full text]
[PDF]
-
Should all trials have a data safety and monitoring committee? J. A. Cairns, A.
Hallstrom, P. Held. Am Heart J 2001: 141(1); 156-63.
[Medline]
-
The Ethics of Early Stopping Rules: Who Is Protecting Whom? Stephen A. Cannistra.
Journal of Clinical Oncology 2004: 22(9); 1542-1545.
[PDF]
-
When and how to stop a clinical trial: invited remarks. T. C. Chalmers. Clin
Pharmacol Ther 1979: 25(5 Pt 2); 649-50.
[Medline]
-
Optimal three-stage designs for phase II cancer clinical trials. T. T. Chen. Stat Med
1997: 16(23); 2701-11.
[Medline]
-
Societal responsibilities of clinical trial sponsors. Stephen Evans, Stuart Pocock.
BMJ 2001: 322(7286); 569-570.
[Medline] [Full text]
[PDF]
-
Safeguarding patients in clinical trials with high mortality rates. Bradley D.
Freeman, Robert L. Danner, Steven M. Banks, Charles Natanson. Am J Respir Crit Care Med
2001: 164(2); 190-192.
[Medline] [Full text]
[PDF]
-
Issues in data monitoring and interim analysis of trials. AM Grant, DG Altman, AB
Babiker, MK Campbell, FJ Clemens, JH Darbyshire, DR Elbourne, SK McLeer, MKB Parmar, SJ
Pocock, DJ Spiegelhalter, MR Sydes, AF Walker, SA Wallace. Health Technolgoy Assessment
2005: 9(7);
[Medline] [Abstract]
[PDF]
-
Breast cancer trial stopped early [news]. D. Josefson. British Medical Journal 1998:
316(7139); 1185.
[Medline] [Full text]
-
Sample size and interim analysis issues for dose selection. K. L. Lee. Am Heart J
2000: 139(4); S161-5.
[Medline]
-
Premature discontinuation of clinical trial for reasons not related to efficacy, safety,
or feasibility Commentary: Early discontinuation violates Helsinki principles. Michel
Lievre, Joel Menard, Eric Bruckert, Joel Cogneau, Francois Delahaye, Philippe Giral, Eran
Leitersdorf, Gerald Luc, Luis Masana, Philippe Moulin, Philippe Passa, Denis Pouchain,
Gerard Siest, K Boyd. BMJ 2001: 322(7286); 603-606.
[Full text]
[PDF]
-
Monitoring clinical trials--interim data should be publicly available. R. J. Lilford,
D. Braunholtz, S. Edwards, A. Stevens. British Medical Journal 2001: 323(7310); 441-2.
[Medline] [Full text]
[PDF]
-
Group sequential designs for monitoring survival probabilities. D. Y. Lin, L. Shen,
Z. Ying, N. E. Breslow. Biometrics 1996: 52(3); 1033-41.
-
Interim analyses of data as they accumulate in laboratory experimentation. J.
Ludbrook. BMC Med Res Methodol 2003: 3(1); 15.
[Medline] [Abstract]
[Full text]
[PDF]
-
Upper Extremity Deep Venous Thrombosis in Cancer Patients with Venous Access Devices-
Prophylaxis with a Low Molecular Weight Heparin (Fragmin). M Monreal, A Alastrue, M Rull,
X Mira, J Muxart, R Rosell, A Abad. Thromb Haemost 1996: 75(2); 251-253.
[Medline]
-
Monitoring and ensuring safety during clinical research. M. A. Morse, R. M. Califf,
J. Sugarman. Jama 2001: 285(9); 1201-5.
[Medline] [Abstract]
[Full text]
[PDF]
-
Patients at high risk of death after lung-volume-reduction surgery. National
Emphysema Treatment Trial Research Group. N Engl J Med 2001: 345(15); 1075-83.
[Medline] [Abstract]
[PDF]
-
Ethical dilemmas in continuing a zidovudine trial after early termination of similar
trials. MS Simberkoff, PM Hartigan, JD Hamilton, D Deykin, M Gail, JG Bartlett, P
Feorino, R Redfield, R Roberts, D Collins, et al. Control Clin Trials 1993: 14(1); 6-18.
-
Data safety and monitoring boards. Arthur S Slutsky, James V Lavery. NEJM 2004:
350(11); 1143-1147.
-
Vasectomy by ligation and excision, with or without fascial interposition: a randomized
controlled trial [ISRCTN77781689]. D Sokal, B Irsula, M Hays, M Chen-Mok, MA Barone. BMC
Med 2004: 2(1); 6.
[Medline] [Abstract]
[Full text]
[PDF]
-
Design and analysis of prostate cancer trials. R Sylvester. Acta Urologica Belgica
1994: 62(1); 23-29.
Web pages:
-
Human Subjects Case
2: Early Termination of Clinical Trials. Michael Kalichman, Susan Eastwood.
Accessed on 2003-06-20. ethics.ucsd.edu/workshops/CRW/cases/case2.htm
-
Flexible Clinical
Trial Design [pdf]. Cyrus Mehta, Presented at the Adaptive Design For Clinical
Trials: Statistical Methodologies and Applications Conference, March 4, 5, 2004 -
Philadelphia, PA. Accessed on 2004-03-25. www.cytel.com/Papers/flexible_ct_design_04.pdf
-
Design and Interim Monitoring of
Flexible Clinical Trials [pdf]. Cyrus R. Mehta, Presented at Massachusetts
Biotechnology March 9, 2004 - Boston, MA. Accessed on 2004-03-25. www.cytel.com/Papers/MBC_2004.pdf
-
Adaptive vs Group Sequential
Designs: Some Trade-Offs [pdf]. Cyrus R. Mehta, Anastasios A. Tsiatis, Cytel
Statistical Software. Accessed on 2003-07-15. www.cytel.com/papers/ENARTalk.pdf
-
Reporting Adverse Event/Safety Issues. Penn State College of Medicine. Accessed
on 2004-01-28. www.hmc.psu.edu/hmc-irb/Forms%20&%20Instructions/Ongoing%20Studies/OngoingStudies.htm
-
FDA Guidance on Clinical
Trial Data Monitoring Committees (DMCs). U.S. Food and Drug Administration,
Transcript of an open public meeting, November 27, 2001. Accessed on 2003-05-12. www.fda.gov/cber/minutes/clinmon1101p1.htm
-
Guidance for Clinical Trial
Sponsors. On the Establishment and Operation of Clinical Trial Data Monitoring Committees.
U.S. Food and Drug Administration. Accessed on 2003-05-12. www.fda.gov/cber/gdlns/clindatmon.htm
-
Study of
Asthma-Drug Halted. U.S. Food and Drug Administration. Accessed on 2003-01-29.
www.fda.gov/bbs/topics/ANSWERS/2003/ANS01192.html
-
Further Guidance on a Data and Safety Monitoring for Phase I and Phase II Trials.
U.S. National Institutes of Health. Accessed on 2003-05-12.
grants1.nih.gov/grants/guide/notice-files/NOT-OD-00-038.html
-
NIH Policy
for Data and Safety Monitoring. U.S. National Institutes of Health. Accessed on
2003-05-12. grants1.nih.gov/grants/guide/notice-files/not98-084.html
-
Data And Safety Monitoring Plans. UCSF School of Medicine. Accessed on
2003-01-21. www.som.ucsf.edu/som/research/clinical_research/resources/dsmp.asp
This page was written by
Steve Simon while working at Children's Mercy Hospital. Although I do not hold the copyright for this material, I am reproducing it here as a service, as it is no longer available on the Children's Mercy Hospital website. Need more
information? I have a page with general help
resources. You can also browse for pages similar to this one at Category: Early stopping.