P.Mean: Post hoc power persists because peer-reviewers demand it (created 2012-01-04).

News: Sign up for "The Monthly Mean," the newsletter that dares to call itself average, www.pmean.com/news.

I was in the middle of writing a grant looking at best research practices and wanted to give an example of when best practices weren't being followed. The easiest example to find was the use of post hoc power calculations. There's been at least two decades of criticism of this practice and yet it still occurs. The example I found, however, has an interesting twist to the tale.

The reference in question is

Masese LN, Graham SM, Gitau R, Peshu N, Jaoko W, Ndinya-Achola JO, Mandaliya K, Richardson BA, Overbaugh J, McClelland RS (2011). A prospective study of vaginal trichomoniasis and HIV-1 shedding in women on antiretroviral therapy. BMC Infectious Diseases; 11(1): 307. Available at: http://www.biomedcentral.com/1471-2334/11/307

This paper has the words "post hoc power" in the text, so it was pretty easy to find.

This is an open source publication, but it is also interesting because the peer-review process is open for review. One of my collaborators on the grant took the time to review this history and found that the post hoc power calculation was not in the original publication but that it was put in at the suggestion of the peer reviewers.

Click on the link called Pre-publication history (or click here) and see this for yourself.

The manuscript, as originally submitted does not mention the words "post hoc power." The first referee lists, as her second major compulsory revision

The authors followed a pre-specified alternative analysis plan but no power calculations for this have been presented. The paper should include a post hoc power calculation (if no a priori calculation was carried out) to indicate what differences this sample size of 31 was able to detect with reasonable power.

Sure enough, in the second submission, the manuscript has the following text

A post hoc power calculation shows that with a sample size of 31, we had 80% power to detect a 5-fold increase in genital HIV-1 RNA with T. vaginalis infection.

In the author's comment to the first set of reviews, was this statement:

Following the reviewers' suggestion, we have included the post hoc power calculations in the discussion.

This statement was tweaked a bit in the next iteration (the word "detection" was added).

A post hoc power calculation shows that with a sample size of 31, we had 80% power to detect a 5-fold increase in genital HIV-1 RNA detection with T. vaginalis infection.

and this is what appears in the official publication.

It's interesting that this has occured, and it would be a fascinating study to look at all papers like this with an open process for peer review. You could get an answer to the question: How often is a post hoc power calculation included at the insistence of the peer-reviewers? Just search for papers with "post hoc power" in the final version and see how often it was not mentioned in the original submission.

Creative Commons License This page was written by Steve Simon and is licensed under the Creative Commons Attribution 3.0 United States License. Need more information? I have a page with general help resources. You can also browse for pages similar to this one at Post Hoc Power.