P.Mean >> Category >> Small sample size issues (created 2007-09-11).

These pages outline some of the practical issues and ethical concerns with small sample sizes. Also see Category: Confidence intervals, Category: Sample size justification. Other entries about small sample size issues can be found in the small sample size issues page at the StATS website.

2008

1. P.Mean: T-test with 3 treatment values and 2 controls (created 2008-10-14). I received a question about how to run a t-test when one group has 3 observations and the other group has 2 observations? It turns out that you use the same formula/program that you would use with 30 observations in one group and 20 observations in the other group. There are two things, however, that you need to watch out for.
2. P.Mean: Survey results from nine out of thirty six employees (created 2008-07-21). Hi, hope you can help a struggling grad student in health promotion and education. If I administer a questionnaire to 9 out of 36 staff members, are the results statistically significant or is the survey respondents number too small? This is a needs assessment questionnaire-what the staff feels they need from an educational standpoint. Or am I floating off course and hopeless? Thank you for your time and help!

Outside resources:

• J M. Bland, D. G Altman. Analysis of continuous data from small samples. BMJ. 2009;338(apr06 1):a3166-a3166. Excerpt: "Studies with small numbers of measurements are rare in the modern BMJ, but they used to be common and remain plentiful in specialist clinical journals. Their analysis is often more problematic than that for large samples." [Accessed January 26, 2010]. Available at: http://www.bmj.com/cgi/content/extract/338/apr06_1/a3166.
• Gordon H. Guyatt, Edward J. Mills, Diana Elbourne. In the Era of Systematic Reviews, Does the Size of an Individual Trial Still Matter. PLoS Medicine Vol. 5, No. 1, e4 doi:10.1371/journal.pmed.0050004. [Full text] [PDF]. Description: Gordon Guyatt and Edward Mills argue that a requirement that all trials have a sample size justification has prevented a large number of research studies from starting. These studies, even though they each individually would fail to have appropriate power and precision, would contribute to a systematic overview that would be able to produce definitive results. Diana Elbourne argues that if a small negative trial stifles the production of further trials, then the systematic overview will not get a sufficient number of small trials. I think that both authors miss the point. I have argued that a systematic overview is like a multi-center trial where each center gets to use its own protocol and where the centers have the option of not reporting their data. There are no meta-analytic tools that can patch up a large number of small inadequately powered trials. A better solution is to encourage more collaborative multi-center trials rather than a patchwork of small single center trials.
• Committee on Strategies for Small-Number-Participant Clinical Research Trials, Board on Health Sciences Policy. Small Clinical Trials: Issues and Challenges. Washington, D.C.: The National Academies Press; 2001. Abstract: "Scientific research has a long history of using well-established, well documented, and validated methods for the design, conduct, and analysis of clinical trials. A study design that is considered appropriate includes sufficient sample size (n) and statistical power and proper control of bias to allow a meaningful interpretation of the results. Whenever feasible, clinical trials should be designed and performed so that they have adequate statistical power. However, when the clinical context does not provide a sufficient number of research participants for a trial with adequate statistical power but the research question has great clinical significance, research can still proceed under certain conditions. Small clinical trials might be warranted for the study of rare diseases, unique study populations (e.g., astronauts), individually tailored therapies, in environments that are isolated, in emergency situations, and in instances of public health urgency. Properly designed trials with small sample sizes may provide substantial evidence of efficacy and are especially appropriate in particular situations. However, the conclusions derived from such studies may require careful consideration of the assumptions and inferences, given the small number of paticipants. Bearing in mind the statistical power, precision, and validity limitations of trials with small sample sizes, there are innovative design and analysis approaches that can improve the quality of such trials. A number of trial designs especially lend themselves to use in studies with small sample sizes, including one subject (n-of-1) designs, sequential designs, “within-subject” designs, decision analysis-based designs, ranking and selection designs, adaptive designs, and risk-based allocation designs. Data analysis for trials with small numbers of participants in particular must be focused. In general, certain types of analyses are more amenable to studies with small numbers of participants, including sequential analysis, hierarchical analysis, Bayesian analysis, decision analysis, statistical prediction, meta-analysis, and risk-based allocation. Because of the constraints of conducting research with small sample sizes, the committee makes recommendations in several areas: defining the research question, tailoring the study design by giving careful consideration to alternative methods, clarifying sample characteristics and methods for the reporting of results of clinical trials with small sample sizes, performing corroborative analyses to evaluate the consistency and robustness of the results of clinical trials with small sample sizes, and exercising caution in the interpretation of the results before attempting to extrapolate or generalize the findings of clinical trials with small sample sizes. The committee also recommends that more research be conducted on the development and evaluation of alternative experimental designs and analysis methods for trials with small sample sizes. Available at: http://www.nap.edu/catalog.php?record_id=10078.

All of the material above this paragraph is licensed under a Creative Commons Attribution 3.0 United States License. This page was written by Steve Simon and was last modified on 2010-05-26. The material below this paragraph links to my old website, StATS. Although I wrote all of the material listed below, my ex-employer, Children's Mercy Hospital, has claimed copyright ownership of this material. The brief excerpts shown here are included under the fair use provisions of U.S. Copyright laws.

2008

Stats: Can I run a quantitative analysis on this data? (June 17, 2008). I get lots of questions about how small a sample size can be before you can't perform a quantitative analysis and instead are forced to summarize the data in a qualitative fashion. The most recent question involved looking at infants with feeding disorders. There were 29 of these infants, but a subgroup of 5 had disorders so severe that they still required a feeding tube at 3 years of age. The researcher wanted to compare this group of 5 to the remaining 24.

Stats: Cohen's Kappa with small cell sizes (April 26, 2007). Someone on Edstat-L wrote in asking about using Cohen' Kappa with a small sample size in some of the cells.

Stats: Perfect isn't quite good enough (December 12, 2006). Someone wanted me to double check their calculations for Fisher's Exact test. If the control group, 3 out of 10 patients experienced an unfortunate outcome. In the treatment group none did (out of 6). You would think that a perfect result in the treatment group would be compelling, but the one-sided p-value for Fisher's Exact test is 0.21.

Stats: Small sample size, yet again (March 29, 2006). Dear Professor Mean, Is there any statistical test/method that will allow you to make statistically significant conclusions from a sample of nine? Someone was trying to tell me that if you use a nonparametric test, you can make get statistical significance, even with a very small sample size.

Stats: When one group only has a single observation (May 24, 2005). Someone asked me about a lab study comparing expression levels for two groups of patients. The first group has two copies of a gene and the second group has three copies of the gene, thanks to a chromosomal duplication. That sounds easy enough to do. You could use a t-test in SPSS. Actually, I prefer to use the general linear model, which provides exactly the same test, but the output looks nicer and it allows you to easily incorporate more complex research designs. The kicker in this analysis, though, is that there is only one patient in the second group. This person asked if he could perform a t-test in SPSS.

Stats: Small sample size (October 11, 2001). Dear Professor Mean, Are there problems with a very small sample? Can the t-test be used with a sample of just three subjects? -- Anxious Abdelwahab

Stats: All or nothing (August 18, 1999). Dear Professor Mean, I would like to know the minimum number of patients needed in order to achieve statistical significance. I am assuming a perfect research situation where all of the patients who got a treatment lived and all the patients who got the placebo died. What would the proper sample size for an all or nothing response be?-- Hesitant Harrison

What now?

Browse other categories at this site

Browse through the most recent entries

Get help