P.Mean >> Category >> Exclusions in research studies (created 2007-06-18).

These pages discuss the problems with generalizability that occur when researchers include important segments of the population from their research or when research subjects refuse to participate. Articles are arranged by date with the most recent entries at the top. You can find outside resources at the bottom of this page. Other entries about exclusions in research studies can be found in the exclusions in research page at the StATS website.

2010

  1. P.Mean: Withdrawing from a study and taking your data with you (created 2010-05-15). Someone asked me what the phrase "you can withdraw from the study at any time" really means. Can a research subject withdraw and take their data with them (that is, ask that their data be expunged from the database)? What if they raise the objection after the data analysis is done, because they don't like the results of the study. Can they ask for their data to be expunged then? What if they raise the objection after the data is published?

    2009
     
  2. P.Mean: Sneaking ineligible patients into a clinical trial (created 2009-10-30). There was an interesting article in the New York Times (Chen PW. Bending the Rules of Clinical Trials) that described a terminal cancer patient and the doctor's goal to get them access to a new experimental drug, even though the patient was not eligible for the clinical trial that was studying this drug. It's a difficult situation for doctors. Do you do what's best for the patient in front of you, knowing that the data collected from this patient might corrupt the findings of the clinical trial?

Outside resources:

  1. Mark Zimmerman, Jill I. Mattia, Michael A. Posternak. Are Subjects in Pharmacological Treatment Trials of Depression Representative of Patients in Routine Clinical Practice?. Am J Psychiatry. 2002;159(3):469-473. Abstract: "OBJECTIVE: The methods used to evaluate the efficacy of antidepressants differ from treatment for depression in routine clinical practice. The rigorous inclusion/exclusion criteria used to select subjects for participation in efficacy studies potentially limit the generalizability of these trials' results. It is unknown how much impact these criteria have on the representativeness of subjects in efficacy trials. This study estimated the proportion of depressed patients treated in routine clinical practice who would meet standard inclusion/exclusion criteria for an efficacy trial. METHOD: A total of 803 individuals, aged 16-65 years, who were seen at intake at an outpatient practice underwent a thorough diagnostic evaluation, including the administration of semistructured diagnostic interviews; 346 patients had current major depression. Common inclusion/exclusion criteria used in efficacy studies of antidepressants were applied to the depressed patients to determine how many would have qualified for an efficacy trial. RESULTS: Approximately one-sixth of the 346 depressed patients would have been excluded from an efficacy trial because they had a bipolar or psychotic subtype of depression. The presence of a comorbid anxiety or substance use disorder, insufficient severity of depressive symptoms, or current suicidal ideation would have excluded 86.0% (N=252) of the remaining 293 outpatients with nonpsychotic unipolar major depressive disorder from an antidepressant efficacy trial. CONCLUSIONS: Subjects treated in antidepressant trials represent a minority of patients treated for major depression in routine clinical practice. These results show that antidepressant efficacy trials tend to evaluate a subset of depressed individuals with a specific clinical profile." [Accessed December 9, 2009]. Available at: http://ajp.psychiatryonline.org/cgi/content/abstract/159/3/469.
  2. C Bartlett, L Doyal, S Ebrahim, et al. The causes and effects of socio-demographic exclusions from clinical trials. Excerpt: "The exclusion from trials of people likely to be in need of or to benefit from an intervention could compromise the trials’ generalisability. We investigated the exclusion of women, older people and minority ethnic groups, focusing on two drug exemplars, statins and non-steroidal anti-inflammatory drugs (NSAIDs)." [Accessed May 19, 2010]. Available at: http://www.hta.ac.uk/execsumm/summ938.shtml.
  3. Harriette G. C. Van Spall, Andrew Toren, Alex Kiss, Robert A. Fowler. Eligibility Criteria of Randomized Controlled Trials Published in High-Impact General Medical Journals: A Systematic Sampling Review. JAMA. 2007;297(11):1233-1240. Abstract: "Context: Selective eligibility criteria of randomized controlled trials (RCTs) are vital to trial feasibility and internal validity. However, the exclusion of certain patient populations may lead to impaired generalizability of results. Objective: To determine the nature and extent of exclusion criteria among RCTs published in major medical journals and the contribution of exclusion criteria to the representation of certain patient populations. Data Sources and Study Selection: The MEDLINE database was searched for RCTs published between 1994 and 2006 in certain general medical journals with a high impact factor. Of 4827 articles, 283 were selected using a series technique. Data Extraction: Trial characteristics and the details regarding exclusions were extracted independently. All exclusion criteria were graded independently and in duplicate as either strongly justified, potentially justified, or poorly justified according to previously developed and pilot-tested guidelines. Data Synthesis: Common medical conditions formed the basis for exclusion in 81.3% of trials. Patients were excluded due to age in 72.1% of all trials (60.1% in pediatric populations and 38.5% in older adults). Individuals receiving commonly prescribed medications were excluded in 54.1% of trials. Conditions related to female sex were grounds for exclusion in 39.2% of trials. Of all exclusion criteria, only 47.2% were graded as strongly justified in the context of the specific RCT. Exclusion criteria were not reported in 12.0% of trials. Multivariable analyses revealed independent associations between the total number of exclusion criteria and drug intervention trials (risk ratio, 1.35; 95% confidence interval, 1.11-1.65; P = .003) and between the total number of exclusion criteria and multicenter trials (risk ratio, 1.26; 95% confidence interval, 1.06-1.52; P = .009). Industry-sponsored trials were more likely to exclude individuals due to concomitant medication use, medical comorbidities, and age. Drug intervention trials were more likely to exclude individuals due to concomitant medication use, medical comorbidities, female sex, and socioeconomic status. Among such trials, justification for exclusions related to concomitant medication use and comorbidities were more likely to be poorly justified. Conclusions: The RCTs published in major medical journals do not always clearly report exclusion criteria. Women, children, the elderly, and those with common medical conditions are frequently excluded from RCTs. Trials with multiple centers and those involving drug interventions are most likely to have extensive exclusions. Such exclusions may impair the generalizability of RCT results. These findings highlight a need for careful consideration and transparent reporting and justification of exclusion criteria in clinical trials." [Accessed January 15, 2010]. Available at: http://jama.ama-assn.org/cgi/content/abstract/297/11/1233.
  4. Richard Feinman. Intention-to-treat. What is the question?. Nutrition & Metabolism. 2009;6(1):1. Abstract: "It has become commonplace for Randomized Controlled Trials (RCTs) to be analyzed according to Intention-to-Treat (ITT) principles in which data from all subjects are used regardless of the subjects' adherence to protocol. While ITT analyses can provide useful information in some cases, they do not answer the question that motivates many RCTs, namely, whether the treatments differ in efficacy. ITT tends to reduce information by combining two questions, whether the intervention is effective and whether, as implemented, it has good compliance. Because these questions may be separate there is a risk of misuse. Two examples are presented that demonstrate this potential for abuse: a study on the effectiveness of vitamin E in reducing cardiovascular risk and comparisons of low fat and low carbohydrate diets. In the first case, a treatment that is demonstrably effective is described as without merit. In the second, ITT describes as the same, two diets that actually have different outcomes. These misuses of ITT are not atypical and are not technical problems in statistics but have real consequences for scientific principles and health recommendations. ITT analyses may answer the question of what happens when treatments are recommended but are inappropriate where separate information on adherence and performance is available. It is proposed that results of RCTs, or any experimental study, be reported, not in terms of the analyses that were performed, but rather in terms of the questions that the analyses can answer properly." [Accessed February 24, 2009]. Available at: http://www.nutritionandmetabolism.com/content/6/1/1.
  5. MediciGlobal. L2FU - Lost to Follow Up. Excerpt: "Patient drop outs in a clinical trial costs your company money. It can cost you the integrity of your study too! If it's important to recover patients lost from your clinical trial, you've come to the right place. Here, you'll read how L2FU's services can help you and how to begin finding patients today!" [Accessed July 26, 2010]. Available at: http://www.l2fu.com.
  6. Chronic Disease Prevention and Control Research Center at Baylor College of Medicine. Major Deficiencies in the Design and Funding of Clinical Trials: A Report to the Nation Improving on How Human Studies Are Conducted. Excerpt: "Clinical trials are a critical resource for the discovery of new, life-saving drugs and for developing better prevention and diagnostic screening methods. Today’s most effective prevention and treatment modalities are based on previous clinical trial results. But while the need for clinical research is undisputed, how clinical trials are now conducted remains problematic. Increasing research finds major deficiencies in the way clinical trials are designed, carried out and funded in the U.S. with serious implications for the outcomes of medical research studies. Of key significance for the future of scientific innovation is the exclusion or underrepresentation of women, older people, minorities, disabled persons, and rural populations in the vast majority of the research studies conducted in the U.S. Without adequate representation of all patient populations, researchers cannot learn about potential differences among groups and cannot ensure the generalization of results." [Accessed December 9, 2009]. Available at: http://www.bcm.edu/edict/PDF/EDICT_Project_White_Paper.pdf.
  7. Cora Craig, Christine Cameron, Joe Griffiths, et al. Non-response bias in physical activity trend estimates. BMC Public Health. 2009;9(1):425. Abstract: "BACKGROUND: Increases in reported leisure time physical activity (PA) and obesity have been observed in several countries. One hypothesis for these apparently contradictory trends is differential bias in estimates over time. The purpose of this short report is to examine the potential impact of changes in response rates over time on the prevalence of adequate PA in Canadian adults. METHODS: Participants were recruited in representative national telephone surveys of PA from 1995-2007. Differences in PA prevalence estimates between participants and those hard to reach were assessed using Student's t tests adjusted for multiple comparisons. RESULTS: The number of telephone calls required to reach and speak with someone in the household increased over time, as did the percentage of selected participants who initially refused during the first interview attempt. A higher prevalence of adequate PA was observed with 5-9 attempts to reach anyone in the household in 1999-2002, but this was not significant after adjustment for multiple comparisons. CONCLUSIONS: No significant impact on PA trend estimates was observed due to differential non response rates. It is important for health policy makers to understand potential biases and how these may affect secular trends in all aspects of the energy balance equation." [Accessed November 29, 2009]. Available at: http://www.biomedcentral.com/1471-2458/9/425.
  8. Tara Parker-Poe. Older Cancer Patients Often Excluded From Research. Excerpt: "The majority of people diagnosed with cancer are over 65, but most major cancer studies exclude them, leaving a wide gap in knowledge about how best to treat older patients. The knowledge gap about older cancer patients was highlighted recently by researchers from Barcelona who studied the role that age played in the prognosis of 224 cancer patients." [Accessed December 9, 2009]. Available at: http://well.blogs.nytimes.com/2008/11/17/older-cancer-patients-excluded-from-research/.
  9. Irving Kuo. Randomization May Not be Valid in Tests of Psychotherapy vs Medications. Description: In a study comparing various combinations of medication and/or cognitive behavioral therapy for treating depression, only 1% of all patients surveyed found all seven arms of the study acceptable. This leads to serious problems with volunteer bias. [Accessed December 9, 2009]. Available at: http://www.medscape.com/viewarticle/564001.
  10. Nicole Huang, Shu-Fang Shih, Hsing-Yi Chang, Yiing-Jenq Chou. Record linkage research and informed consent: who consents?. BMC Health Services Research. 2007;7(1):18. Abstract: "BACKGROUND: Linking computerized health insurance records with routinely collected survey data is becoming increasingly popular in health services research. However, if consent is not universal, the requirement of written informed consent may introduce a number of research biases. The participants of a national health survey in Taiwan were asked to have their questionnaire results linked to their national health insurance records. This study compares those who consented with those who refused. METHODS: A national representative sample (n = 14,611 adults) of the general adult population aged 20 years or older who participated in the Taiwan National Health Interview Survey (NHIS) and who provided complete survey information were used in this study. At the end of the survey, the respondents were asked if they would give permission to access their National Health Insurance records. Information given by the interviewees in the survey was used to analyze who was more likely to consent to linkage and who wasn't. RESULTS: Of the 14,611 NHIS participants, 12,911 (88%) gave consent, and 1,700 (12%) denied consent. The elderly, the illiterate, those with a lower income, and the suburban area residents were significantly more likely to deny consent. The aborigines were significantly less likely to refuse. No discrepancy in gender and self-reported health was found between individuals who consented and those who refused. CONCLUSION: This study is the first population-based study in assessing the consent pattern in a general Asian population. Consistent with people in Western societies, in Taiwan, a typical Asian society, a high percentage of adults gave consent for their health insurance records and questionnaire results to be linked. Consenters differed significantly from non-consenters in important aspects such as age, ethnicity, and educational background. Consequently, having a high consent rate (88%) may not fully eliminate the possibility of selection bias. Researchers should take this source of bias into consideration in their study design and investigate any potential impact of this source of bias on their results. [Accessed December 9, 2009]. Available at: http://www.biomedcentral.com/1472-6963/7/18.
  11. G S May, D L DeMets, L M Friedman, C Furberg, E Passamani. The randomized clinical trial: bias in analysis. Circulation. 1981;64(4):669-673. Abstract: "The realization that bias in patient selection may influence the results of clinical studies has helped to establish the randomized controlled clinical trial in medical research. However, bias can be equally important at other stages of a trial, especially at the time of analysis. Withdrawing patients from consideration in the analysis because of ineligibility on account of study entry criteria, lack of compliance to the protocol, or data of poor quality may be a source of systematic error. Examples to illustrate the possible consequences are taken from trials in the cardiovascular field. We recommended that reported study results should include outcome data from all subjects randomized in the group to which they were originally assigned." [Accessed December 9, 2009]. Available at: http://circ.ahajournals.org/cgi/content/abstract/64/4/669.
  12. Fred Andersen, Torgeir Engstad, Bjorn Straume, et al. Recruitment methods in Alzheimer's disease research: general practice versus population based screening by mail. BMC Medical Research Methodology. 2010;10(1):35. Abstract: "BACKGROUND: In Alzheimer's disease (AD) research patients are usually recruited from clinical practice, memory clinics or nursing homes. Lack of standardised inclusion and diagnostic criteria is a major concern in current AD studies. The aim of the study was to explore whether patient characteristics differ between study samples recruited from general practice and from a population based screening by mail within the same geographic areas in rural Northern Norway. METHODS: An interventional study in nine municipalities with 70000 inhabitants was designed. Patients were recruited from general practice or by population based screening of cognitive function by mail. We sent a questionnaire to 11807 individuals [greater than or equal to] 65 years of age of whom 3767 responded. Among these, 438 individuals whose answers raised a suspicion of cognitive impairment were invited to extended cognitive testing and a clinical examination. Descriptive statistics, chi-square, independent sample t-test and analyses of covariance adjusted for possible confounders were used. RESULTS: The final study samples included 100 patients recruited by screening and 87 from general practice. Screening through mail recruited younger and more self-reliant male patients with a higher MMSE sum score, whereas older women with more severe cognitive impairment were recruited from general practice. Adjustment for age did not alter the statistically significant differences of cognitive function, self-reliance and gender distribution between patients recruited by screening and from general practice. CONCLUSIONS: Different recruitment procedures of individuals with cognitive impairment provided study samples with different demographic characteristics. Initial cognitive screening by mail, preceding extended cognitive testing and clinical examination may be a suitable recruitment strategy in studies of early stage AD. Registration: ClinicalTrial.gov Identifier: NCT00443014" [Accessed May 6, 2010]. Available at: http://www.biomedcentral.com/1471-2288/10/35.

Creative Commons License All of the material above this paragraph is licensed under a Creative Commons Attribution 3.0 United States License. This page was written by Steve Simon and was last modified on 2010-07-26. The material below this paragraph links to my old website, StATS. Although I wrote all of the material listed below, my ex-employer, Children's Mercy Hospital, has claimed copyright ownership of this material. The brief excerpts shown here are included under the fair use provisions of U.S. Copyright laws.

2006

  1. Stats: Characterizing reasons for refusal (January 17, 2006). A participant in the IRB Forum raised a question about a research study where people were asked to participate, and when some of them said "no" they were then asked why they did not want to participate. The  reasons were tallied across all of the refusals, and percentages were published as part of the full research study. Since the people had already said "no" to participating in the research study, did that also imply that they did not want to share information about their reasons for not participating?

    2004
     
  2. Stats: Excluding placebo responders (June 25, 2004). I've always been fascinated by the placebo effect and the ethical issues associated with use of placebos in research. A correspondent in the IRBForum email discussion group asked about the recent efforts of drug companies to identify patients who are likely to show a placebo effect and then exclude them from randomized trials.
  3. Stats: Selection bias (August 24, 2004). The CarTalk radio show has an interesting puzzle every week and often these puzzles involve mathematics. These puzzles can sometimes help you understand complex mathematical concepts that are important in Statistics. In the summer, they re-use puzzlers from earlier years, and just last week, they re-used one of my favorites. A "nameless mathematician" during World War II was asked to help with a military problem. A lot of bombers were not returning from their missions, so the Royal Air Force wanted to put armor on the bombers. But where to put it?
  4. Stats: So you want to volunteer for a research study? (August 4, 2004). Here's a draft of a speech that I am planning to give on August 5, 2004 for the Bluejacket Toastmasters humorous speech competition. So you want to volunteer for a research study? Good for you! Mister Contestmaster, fellow Toastmasters, and Guests. I work as a statistician at Children's Mercy Hospital. So when you volunteer for a research study, you provide the data that gives me job security.

What now?

Browse other categories at this site

Browse through the most recent entries

Get help

Creative Commons License This work is licensed under a Creative Commons Attribution 3.0 United States License. This page was written by Steve Simon and was last modified on 2010-07-26.