Drs. Bacchetti and Leung comment on both the published article by Norris et al. , and my accompanying editorial. They specifically argue that a priori sample size calculations are “in fact undesirable,” and argue that the article by Norris et al. “serves to illustrate the drawbacks of sample size calculation as a data analysis tool.” I strongly disagree with the former statement, but appreciate their arguments regarding the latter. I think that two sources of disagreement involve our differing perspectives and our terminology.
Regarding perspective: I am not a statistician, but I have personal experience in clinical trial design and management. More importantly, I've also had the opportunity to edit nearly 9,000 manuscripts in the last 6 yr, many of which illustrate the drawbacks of NOT performing a priori sample size calculations. In my personal view, these calculations are a critical part of the planning of any clinical trial (and, in fact, many laboratory studies). Every week, we receive papers describing studies in which no advance consideration was given to how many patients need to be enrolled. The result is often woefully inadequate group sizes, and a meaningless conclusion of “no difference” between treatment groups. This frequently means that a great deal of work (and sometimes money) was wasted. There is also an ethical component to this matter. Exposing patients to the risks of a trial that cannot result in any meaningful conclusions (due to inadequate planning) is obviously inappropriate. Conversely, exposing excessive numbers of patients to these same risks is ethically questionable.
I want to emphasize that these issues are not abstract. They are an every day matter in editorial offices such as ours. My comments about sample size calculations are not, fundamentally, based on statistical considerations, but rather represent an effort to encourage (force?) investigators to do a better job of planning their work. I remain convinced of the value of careful advance planning when designing a clinical trial. So do nearly all other experts in trial design and all funding agencies. (Imagine NIH giving an investigator $10,000,000 for a clinical trial without insisting on some careful thought about how big the trial needs to be.) This planning must include some consideration of what exactly is being studied (i.e. , the formulation of a clear, unambiguous hypothesis based on a limited number of well defined primary outcomes), how many patients need to be enrolled, and some assessment of how the data are to be evaluated.
On the other hand, I believe it's important that the reader not confuse a priori sample size calculations (used as a critical component of trial design) and post hoc power calculations issues (used after completion of a study to assess the statistical strength of various conclusions, particularly conclusions of “no difference”). Drs. Bacchetti and Leung clearly point out the problems and limitations of post hoc calculations and the advantages of using confidence intervals. Their points are well taken and authors are advised to consider them seriously.
I would like to thank William Clarke, Ph.D. (Professor of Biostatistics, College of Public Health, The University of Iowa, Iowa City, Iowa) for his assistance in composing this reply.