Every few weeks, we receive an article based on some form of survey. Unfortunately, most of those submissions are fatally flawed, usually because the survey was performed in a fashion that calls any conclusions into question. The survey in this month's issue, from the American Society of Anesthesiologists Committee on Transfusion Medicine, is likely to be of great interest to many anesthesiologists, but there are problems with its design and conduct. I asked Dr. Burmeister, who has long experience in the design and conduct of surveys, to provide a basic introduction to survey design, from the perspective of a professional survey expert. I would strongly urge anyone considering carrying out a survey in the future to read his comments very closely. Surveys can be a very valuable way of collecting important information; however, like all good experiments, they are rarely as easy or straightforward to perform as they might seem.
THE results of sample surveys are important in our personal and professional lives. Gallup polls and results from various news agency polls tell us what is important politically. Nielsen studies determine what television shows are presented on national networks. Similarly, results of samples drawn from professional organizations help formulate and evaluate recommendations important to the practice of medical specialties. An example of such a study is included in this issue of the Journal. 1It is essential that such studies be properly conducted. If they are not, serious consequences could result.
Perhaps the most important necessity for a valid study is the existence of a complete sampling frame, which is a listing of all individuals constituting the population of interest. The identification of the sampling frame may be difficult, even when the population of interest is a professional organization. The list of members is often dynamic, including those who may no longer be active and excluding the very recent additions to membership. If a simple random sample is to be selected from a membership list, as done in the Nuttall et al. study, all reasonable efforts should be made to ensure that the list is current. If not, the results of the survey are potentially biased, assuming former members on the list and new subjects not included are different in characteristics and opinions from those included in the population of interest.
Another aspect of sample surveys that rivals sampling frames in importance is the survey instrument. Questions must be of established reliability and validity. Occasionally, well-established questionnaires can be used. It is more likely, however, that at least some questions may be newly developed for the proposed study. In such cases, it is advisable to seek consultation from experienced survey personnel and to complete pilot studies to evaluate the survey instrument.
This Editorial View accompanies the following article: Nuttall GA, Stehling LC, Beighley CM, Faust RJ: Current transfusion practices of members of the American Society of Anesthesiologists: A survey. Anesthesiology 2003; 99:1433-43
Adequate sample size is another requirement for a successful sample survey. There is no easy determination of adequate sample size; for example, sample size is not determined by selecting a specific percentage of the population. Instead, it depends on the desired precision of characteristics to be estimated and the confidence level assigned to achieving the specified precision. Most sample size determinations, as done by Nuttall et al. , are based on precision by specifying the half-width of a confidence interval, or margin of error. If hypotheses are to be tested, the necessary sample size is increased because statistical power is an additional consideration. It should be noted that Nuttall et al. did not actually include power in their sample size computation. If sample designs less efficient than simple random sampling are used, the sample size is also increased.
Simple random sampling may not be the most efficient of the possible sampling plans. For example, if the characteristics being estimated vary by age, experience, gender, subspecialty, and so forth, it may be advantageous to stratify the population of interest and select simple random samples within each stratum. Doing so could improve the precision of the estimates, or reduce the necessary sample size for a specified level of precision. On the other hand, because of the lack of an up-to-date sample frame, it may be necessary to cluster the population and to select samples of the clusters, rather than a simple random sample of individuals. Clusters are often geographic in nature, such as state or local professional organizations. Individuals comprising a given cluster are often similar, which decreases precision and increases the necessary sample size.
The use of alternative sampling plans not only affects the sample size, but it also affects the estimation of precision and alters the width of confidence intervals. Effective stratification will increase the precision of estimates; however, the use of cluster sampling almost always decreases the precision. The overall effect on the precision of complex sample designs, using both stratification and cluster sampling and, perhaps, other types of sampling, is difficult to predict. Therefore, it is essential to analyze the collected data in a manner consistent with the sample design. This often requires the use of challenging computer programs such as SUDAAN (Survey Data Analysis). However, the assumption of sample random sampling when, in fact, a more complex sample design was used can lead to misleading results.
Misleading results are also a consequence of nonresponse. Nonresponse bias is equal to the proportion of nonresponse multiplied by the difference of the responders and nonresponders; consequently, there is no absolutely acceptable level of response. Increasing the initial sample size to accommodate a relatively low expected response rate, as done by Nuttall et al. , does not eliminate nonresponse bias. A better use of resources would be to decrease the initial sample size and increase efforts to contact initial nonresponders.
Because it is nearly impossible to eliminate nonresponse bias when studying human populations, it is essential to describe the potential nonresponse bias and minimize the fraction of nonresponders. Description of the potential nonresponse bias is not always possible; however, demographic characteristics of the entire population may be available from membership files, Bureau of the Census data, or other sources. If these data are available, then characteristics of the sample can be compared to those of the population. Of course, even nearly identical demographic characteristics would not rule out potential nonresponse bias, as it is likely that the demographic characteristics would only be moderately correlated, at best, with the characteristics of interest.
Therefore, strategies to reduce the proportion of nonresponders are highly recommended. At minimum, a sampling of the nonresponders should be attempted. It would be naïve to hope that doing so would eliminate nonresponse bias; after all, nonresponders have their reasons for nonparticipation. A simple second contact will result in only some increase in participation; however, those who respond initially and after second contact can be compared to gain some insight relative to potential nonresponse bias.
As noted above, it is preferable to use strategies to reduce the initial level of nonresponse. Such strategies might include endorsements from community leaders or leadership boards, use of short questionnaires, incentives, and so forth. However, it must be concluded that such strategies will not eliminate nonresponse bias. The best we can hope for is a reasonable reduction in such bias.
In summary, the survey is a very important research tool that is challenging to do well. It certainly has inherent limitations, but it also has the potential to make important contributions. The purpose of this summary of challenges is to serve as a reminder to readers of the Journal and potential authors of the many considerations necessary to complete good survey research.
