StATS: Early detection of accrual problems in clinical trials (created 2006-06-30)
The most common reason why clinical trials fail is that they fall well below their goals for patient accrual. Institutional Review Boards (IRBs) are charged with the continual monitoring of clinical trials and they need to identify when these trials encounter problems with accrual. When do they "jump the shark" so to speak?
Monitoring accrual rates in a clinical trial is an important task, but there are no good tools to identify when the accrual rates decline to a point that threaten the ability of the research to ultimately produce scientifically valid and useful results. I tell the story of a researcher who gets a six year, ten million dollar NIH grant and after the project is over, prepares a report that says
"This is a novel surgical approach and we are 95% confident that the survival rate is somewhere between 3% and 98%."
Clearly, it is in the best interest of the researcher to identify such a study early and to avoid throwing more money down a rat hole. The IRB should be concerned about such a study as well, because when a study ends up producing such inconclusive results, you are exposing subjects to excess risk without any compensating benefit to the scientific and medical community. It would be unethical to allow a trial to continue if there is strong evidence that it will ultimately produce a confidence interval of 3% to 98%.
This may seem like a rather extreme example, but it is not too far from the truth. Too many well intentioned researchers end up falling terribly short on their proposed sample sizes and end up with results that are so imprecise that they are effectively uninterpretable.
The research community and especially IRBs need easy to use tools based on solid quantitative models that can evaluate accrual rates on a regular basis. As part of the continuing review of research, IRBs must examine whether a slow accrual rate threatens the integrity of a research study.
I am seeking research grant funding to develop some simple web based tools the will encourage researchers to think carefully about accrual rates at the start of the study and set thoughtful accrual goals. These same tools would then evaluate accrual rates on a regular basis and provide early detection of accrual problems, so that the researcher and the IRB can take corrective actions or shut the study down.
Inadequate accrual of subjects is not the only issue that IRBs face at continuing review, of course, but it is an important one. I have selected accrual as an area to research because if I can come up with a good model that identifies quickly when patients are accruing too slowly, if would not be hard to adapt those methods to identify situations where patients are dropping out at an alarming rate. An even more important generalization is identifying when adverse events are occuring at a rate that is unacceptably high. All of these areas require analysis of rates, so a good method for one should be readily adapted to the other areas. I wanted to start with accrual because it is simple. You don't worry about differential accrual rates (except possibly in mutli-center trials), and accrual rates don't need to be adjusted for the number of patients currently enrolled in the trial.
I have written extensively about accrual problems (and closely related issues like reporting of adverse events) on my weblog, and I have talked to several people already and have some excellent ideas for a good collaborative research grant. I am interested in getting additional help, especially from members of IRBs. If you are reading this and you want to help or if you want to be kept informed of my progress, send me an email at ssimon at cmh dot edu. The mail link at the top of this page is temporarily broken (sorry!).
The title of the grant I am working on is "Early detection of accrual problems in clinical trials." If you want to help out, I am thinking of applying for a small local grant through the Kansas City Area Life Sciences Institute and then developing a broader proposal for an R03 grant to NIH. I have never been a principal investigator for a grant before, so anyone who is a "seasoned pro" would be especially appreciated.
Here are some areas you can help with.
1. I need some references for my literature review. I'd like something published in the peer reviewed literature that points out that (a) continuing review is difficult, (b) continuing review is not being done well, (c) continuing review relies on qualitative rather than quantitative assessments. I'd also like something to back up my statement that the most common reason that clinical trials fail is inadequate accrual of patients. Here's a quote I found, but it is not backed up with a reference or any hard data:
2. I need some interesting data sets showing accrual patterns in real clinical trials. These can be masked or coded in whatever way you think is best. I am especially interested in studies that "jumped the shark" at a certain point because I want to see how early various approaches would have provided warnings of inadequate accrual. If you provide any data sets, I need to exact date that each individual was recruited into the study. These data can be coded if needed to preserve confidentiality. You can also code the dates, as long as I know exactly how many days from the start of the study each patient was recruited.
3. I need some people who would volunteer to look at various graphical and statistical analyses of accrual rates and comment on them.
4. I need some people who would review my grant proposals and offer constructive suggestions.
5. If you have interesting anecdotes to share about your experience with continuing review, that might help me develop a better perspective on the problem.
6. Moral support is always important. If you think this is a great idea for a research project, send me an email encouraging me to work on this.
7. If you can help in other ways, feel free to let me know.
Another area I am looking for help in is in web based computing. I want to develop a system that can be accessed as easily as a web page, which can offer strong level of security and confidentiality, and which can interface with open source programs like R and WinBUGS, or possibly with commercial programs like MatLab. If you know of anyone with familiarity in those areas, send them my way.
It's bit early to talk about computing tools, but I thinking about making anything produced by the grant open source and available at no cost. IRBs don't have the really big budgets.
Update: July 7, 2006. I submitted a letter of intent to the Kansas City Area Life Sciences Institute describing a grant that Byron Gajewski and I plan to submit. The text of this letter is taken largely from the above material, but I am reproducing it here as it states things slightly differently (and quite succinctly).
Friday, July 7, 2006
Dear Dr. Gary:
On behalf of Dr. Byron Gajewski and myself, I am sending you this letter to declare our intent to apply for funding from the 2006 KCALSI Research Development Grants. I am a Research Biostatistician and will serve as the Principal Investigator for this grant. I work in the Office of Medical Research at Children's Mercy Hospital & Clinics, which will serve as the lead stakeholder institution.
My Co-PI, Dr. Byron Gajewski, is a Biostatistician and Assistant Professor in the School of Nursing at University of Kansas Medical Center. Please note his attached letter of support.
The title of the grant we intend to submit is "Early detection of accrual problems in clinical trials." The most common reason why clinical trials fail is that they fall well below their goals for patient accrual. The net result is a grossly inadequate sample size and results that are so imprecise that they are effectively uninterpretable.
We want funding to build some prototype statistical models for monitoring accrual rates in a clinical trial. We propose that statistical process control charts and Bayesian predictive distributions will offer a flexible and useful set of planning and monitoring tools and will allow researchers and IRBs to replace ad hoc and subjective assessments of accrual rates with a mathematically rigorous and scientifically valid approach. We also plan to develop a prototype web server that will run open source programs from R and WinBUGS.
The initial work on the statistical models and web server will produce a "proof of concept" that will enhance our ability to seek funding from NIH to develop a fully polished system ready to market to researchers and to IRBs.
We believe that this grant fits well with the KCALSI goal of developing cross cutting enabling tools in information technology. We are hopeful that we will receive support that will allow us to start work on this exciting project.
Sincerely,
Stephen D. Simon, Ph.D.
One of the reasons that agencies ask for letters of intent like this is so they can plan for the appropriate expertise on the review panel. So I took some pains to emphasize those features of the grant that would require specialized knowledge.
This page was written by Steve Simon while working at Children's Mercy Hospital. Although I do not hold the copyright for this material, I am reproducing it here as a service, as it is no longer available on the Children's Mercy Hospital website. Need more information? I have a page with general help resources. You can also browse for pages similar to this one at Category: Accrual problems in clinical trials.