Development of a Measure of Patient Satisfaction with Monitored Anesthesia Care : The Iowa Satisfaction with Anesthesia Scale

Each question is a statement designed to assess how patients felt during MAC. Patients specify how strongly they agree or disagree with each statement. Questions included in the scale should comprehensively cover the topic of interest: patient satisfaction with MAC. [2,3] This is called “content validity.” Content validity is a subjective but systematic evaluation of how well the content of the questionnaire represents the measurement task at issue. The authors wrote the initial set of questions, with advice from other anesthesiologists, certified registered nurse anesthetists, anesthesiology residents, student nurse anesthetists, experts in patient satisfaction questionnaire development, surgeons, operating room (OR) nurses, postanesthesia care unit (PACU) nurses, and health care administrators. In addition, a search was done of the medical literature for other validated satisfaction surveys pertaining to anesthesia.

We wrote and printed the questions (Table 1) in a manner to ensure clarity, enhance our response rate, and help respondents form opinions about each statement. Each question expresses one idea [4](e.g., no question contains an “and”). No question contains a negative word (e.g., not or neither). All questions avoid colloquialisms, expressions, and jargon. The questionnaire was printed on one side of quality paper; pages were stapled together; each question was reproduced on a single page; and an 18-point font was used. The ISAS instructions and questions were written at a fourth grade reading level, as assessed by the Flesch-Kincaid method (Microsoft Word[registered sign] 7.0).

Patients respond to each statement in Table 1using the vertical response column shown in Table 2. For “positive” questions, patient agreement implies satisfaction with anesthesia. For “negative” questions, the opposite is true. For example, “I felt good” and “I hurt” are positively and negatively worded questions, respectively. Before scoring is done, scores for “negative” statements are reversed. Then numbers are assigned:-3 = disagree very much, -2 = disagree moderately, -1 = disagree slightly, 1 = agree slightly, 2 = agree moderately, and 3 = agree very much. A totally satisfied patient would score +3 on all questions (after responses to “negative” questions have been reversed). [4]

The University of Iowa Human Subjects Review Committee approved the protocols to be described. The authors invited patients to participate in the study. The investigator was wearing a white medical coat. Their identification badge with photograph was clearly displayed. The investigator walked up to patient and said the following:“(Mr. or Ms.) X? I am (Dr. Dexter, Dr. Wright, or Mr. Aker) in the Department of Anesthesia. We want to learn what we can do to make our patients feel better during their anesthetic. Would you mind taking 5 minutes to fill out a survey?” If the patient agreed to participate, the investigator handed to them a questionnaire and a pen. Patient information and instructions (Table 3) were printed using a 22-point font. The investigator then walked at least several yards away from the patient. Some patients (e.g., after hand surgery) had trouble folding the questionnaire and sealing an envelope. Thus, we had patients hand the questionnaire back to the investigator. Investigators never asked patients about missing responses or refusals to participate.

Patients were invited to participate whenever an investigator did not have patient-care responsibilities. Patients were contacted at least 15 min after admission to a phase II PACU. Adult (aged >or= to 18 yr) patients undergoing MAC were studied. We excluded patients who did not speak English, were cognitively impaired, or were prisoners.

We pretested the 18 questions on the left-hand side of Table 1. Then, testing of the ISAS (right-hand side of Table 1) was started using new patients (i.e., studies to be described were done by inviting patients to participate after we had designed the ISAS).

Questions in the ISAS should all be measuring some aspect of patient satisfaction. This is referred to as “internal consistency.” Internal consistency is a property of the questions.** Responses to each question should be consistent (i.e., correlated) in their assessment of a patient's satisfaction with their anesthetic. For scales such as the ISAS, the usual measure of internal consistency is Cronbach's alpha, which we calculated. [4]

"Convergent validity" refers to the extent to which the ISAS scale correlates positively with other measures of patient satisfaction. We compared patients' scores with those predicted by an observer. Most of our MAC patients are admitted directly to a phase II PACU from an OR. After patients were transferred, we asked the patient's anesthesia provider to complete the ISAS, predicting their patient's responses. During the study period, we approached, in the PACU, all such anesthesia providers. We then invited the patient to complete the ISAS.

We also used another method to test for convergent validity. A patient's overall score for the ISAS equals the mean of their responses to each question. One of the questions in the ISAS is “I was satisfied with my anesthetic care.” Occasionally, this question is used alone to measure patient satisfaction with anesthesia. We compared all patients' responses to the question “I was satisfied with my anesthetic care” to their mean scores for the other 10 questions using the nonparametric correlation coefficient Kendall's tau.

Reliability refers to the extent to which a scale produces consistent results if repeated measurements are made. Patients' responses to the ISAS should not change rapidly over time (i.e., there should be “test-retest reliability”). Patients filled out the ISAS at least 15 min after admission to a phase II PACU. To assess reliability, we had some of these patients fill out a new copy of the ISAS at least 1 h after completion of the original questionnaire.

Patients filled out the ISAS at least 15 min after admission to a phase II PACU. We gave some of these patients a new copy of the questionnaire and a stamped self-addressed envelope. Patients were asked to complete and mail the repeat questionnaire the next morning.

Patients were invited to complete the ISAS. Then, the investigator walked at least several yards away from the patient. For some patients, the investigator remained within sight of the patient and recorded, to the nearest half min, the time until the patient closed the questionnaire and did not reopen it.

The authors are all practicing anesthesia providers. Their routine practice is to do a postoperative interview in the phase II PACU before they discharge their ambulatory patients. During the period of questionnaire development, the authors continued their routine practice of using this interview to ask their patients whether there were any aspects of their MAC with which they were satisfied or not satisfied. The initial set of questions (left-hand side of Table 1) included all such topics suggested by the patients. The other anesthesia providers who reviewed the questions were asked to do the same. All agreed that the initial set of questions included all such topics of concern reported by their patients. Thus, we believe the questionnaire has content validity.

We pretested the initial set of questions on the left-hand side of Table 1. Of the 65 patients invited to participate, 61 agreed. No patient agreed with the statement “I felt alone.” Thus, this question was deleted. Greater than 90% of patients responded to the following four questions by agreeing or disagreeing very much:“I threw up or felt like throwing up;”“I liked my anesthetist;”“No one was helping me;” and “No one cared about me.” We deleted the latter three. However, we retained “I threw up or felt like throwing up” with the expectation that it would be important when the ISAS is applied to general anesthesia patients.

When calculating Cronbach's alpha, patients must have responded to all questions. Thus, questionnaires with any missing responses were not used. Seven of the completed questionnaires had missing pages and were deleted. Four of the patients did not respond to all questions. One response was not legible. Thus, 61 - 7 - 4 - 1 = 49 questionnaires were analyzed. For each question, we calculated the alpha that would be obtained if that question were not included in the ISAS. Deleting questions that give responses that are not correlated with other questions' responses will increase alpha. Three questions were deleted because deleting them increased alpha:“I was scared;”“I felt rushed;” and “I felt angry.”

The final 11 questions, given on the right-hand side of Table 1, resulted in a Cronbach's alpha = 0.78 (N = 49). The set of 11 questions contains five positively and six negatively worded questions. This convenient mix of positively and negatively worded questions gave us the opportunity to rearrange the order of the questions, to alternate negatively and positively worded questions (Table 1).*** Thus, the right-hand side of Table 1shows the ISAS. Because we changed the questions and their order, all testing of the ISAS (to be described) was done using new patients.

All 32 anesthesia providers and 86 of 94 patients invited to participate in testing of the ISAS agreed to do so. Five of the patients who refused to participate offered, without being asked, the explanation that after their eye surgery they could not read the questionnaire. All responses were legible. One of the patients did not respond to question five (Table 1). Three of the patients did not answer question seven. One of the patients did not answer the last two questions. One of the anesthesia providers did not respond to the question “I threw up or felt like throwing up.” Results from the remaining 86 - 6 = 80 patients were analyzed. Demographics of the patients included in the analysis are given in Table 4.

A patient's overall score on the ISAS equals the mean of their responses to each question. Before scoring is done, scores for “negative” statements are reversed. Scores of -3 and +3 correspond to “disagree very much” and “agree very much,” respectively. The mean +/- SD of patients' overall scores on the -3 to 3 scale was 2.1 +/- 0.87 (N = 80). No patients had a score of -3. Twelve of the 80 patients or 15% had a score of +3. In contrast, in response to the question “I was satisfied with my anesthetic care,” 77 of the 80 patients or 96% responded as “agree moderately” or “agree very much.”

The time to complete the questionnaire was 4.6 +/- 2.1 min (median, 5.0 min; lower and upper quartiles, 3.0 and 5.2 min, respectively; N = 17). The patient who took the longest time, 10 min, spoke to nurses while they completed the ISAS.

Cronbach's alpha equaled 0.80 (N = 80). When responses to questions are not correlated, Cronbach's alpha equals zero. The maximum value for Cronbach's alpha is one. There is no consensus for what is an unacceptably low value for Cronbach's alpha. However, the value should be at least 0.6 or 0.7. [2,4]

Adding duplicates of a question to a questionnaire increases the questionnaire's alpha. We were concerned that the ISAS included two similar questions:“I felt pain” and “I felt pain during surgery.” We compared patients' responses to these questions in the same questionnaire to assure that each question measures a different aspect of patient satisfaction. Thirty-four of the 80 patients or 42% gave different responses to these two questions. Of these 34 patients, 15 or 44% gave responses that differed by two or more units.

We compared patients' overall scores to those predicted by their anesthesia provider. Scores were positively correlated (r²= 0.23, two-sided P < 0.01; N = 32). There is no consensus for what is an unacceptably low value for the Pearson correlation coefficient r when used to assess convergent validity. However, in the behavioral sciences, values of r²= 0.09 and r²= 0.25 are considered to be medium and large, respectively. [5] Overall, the difference in scores (anesthesia provider-patient) was small (0.3 +/- 0.9, N = 32). However, the magnitude of the difference between anesthesia provider and their patient's scores was, generally, greater among less-satisfied patients, as reported previously by Cohen et al., [8] The positive (anesthesia provider-patient) difference was accounted for predominantly by answers to the questions “I felt pain” and “I felt pain during surgery.” For only these two questions were median differences between anesthesia provider and their patient's answers not equal to zero. Anesthesia providers underestimated their patients' perception of their pain.

We compared all patients' responses to the question “I was satisfied with my anesthetic care” to their mean scores for the other 10 questions. These two values were correlated (Kendall's tau =+0.41, P <0.0001, N = 80).

Patients' comments suggested that the ISAS has content validity. When patients handed the ISAS back to us, they often told us (without our prompting) about some aspect of their care. No patient (N = 86) ever mentioned that there was some aspect of their anesthetic that our ISAS did not mention.

Overall scores on the initial and repeat questionnaires were positively correlated (r²= 0.74; two-sided P < 0.01; N = 9). The differences between scores (initial-repeat) were small (-0.1 +/- 0.4, N = 9). The magnitudes of the differences in scores are not correlated with the initial scores. Responses to the initial and repeat question “I was satisfied with my anesthetic care” were not correlated (Kendall's tau =-0.19; 95% confidence interval -0.41 to 0.04; N = 9).

Among the 22 patients who agreed to participate in the mail study, 14 or 64% returned the repeat questionnaire. Among the 14 patients returning the questionnaire, three patients did not answer all of the questions. Thus, results were analyzed for 11 of 22 or 50% of the patients. Patients completed the questionnaires within 4.4 +/- 1.7 days after surgery. Overall scores on the initial and repeat questionnaires were positively correlated (r²= 0.76; two-sided P < 0.001; N = 11). The differences between scores (initial-repeat) were small (-0.1 +/- 0.4; N = 11).

"Acquiescence" refers to the tendency for patients to agree with statements independent of the content of the statement. [1] Acquiescence increases the correlation between similarly (i.e., positively or negatively) worded questions, regardless of content. Thus, acquiescence increases the internal consistency of scales in which all questions are worded positively or negatively. We decreased the likelihood of acquiescence bias by including positively and negatively worded questions in the ISAS. We may have also reduced the internal consistency of the ISAS. Yet, our internal consistency (Cronbach's alpha = 0.80) exceeds the lowest acceptable values (0.6 to 0.7). [2,4]

For the ISAS to have content validity, the questions must be representative of patient satisfaction with anesthesia. First, the authors sought patient responses from other anesthesia providers, as well as health professionals in other perioperative fields. During the period of questionnaire development, the authors and other anesthesia providers asked their patients whether there were any aspects of their anesthetic with which they were not satisfied. The initial set of questions comprehensively included all such topics of concern to the patients. Second, no patient (N = 140) ever mentioned (without our prompting) that there was some aspect of their anesthetic that our ISAS did not mention.

The ISAS correlates positively with two other measures of patient satisfaction (i.e., has convergent validity). For example, we compared patients' responses to the question “I was satisfied with my anesthetic care” with their mean scores for the other 10 questions. These two values were correlated (Kendall's tau =+0.41). Thus, the ISAS measures what patients consider to be related to satisfaction with anesthesia.

We also considered testing for predictive validity. [2] By this we mean that the ISAS should predict future decisions with respect to anesthesia. For example, patients who were not satisfied may choose to not undergo the same surgical procedure again. Oocyte retrieval patients (Table 4) seemed to be a good group of patients to study. However, medical and financial factors probably have an overwhelming effect on couples' decisions to proceed with additional infertility procedures after failed attempts at pregnancy.

Our test-retest reliability was acceptable (r²= 0.74 1 h; r²= 0.76 mail). The high correlation between test-retest scores in the mail study shows that results obtained in the phase II PACU are the same as those obtained after discharge. The test-retest method of assessing reliability is sensitive to the time interval between testing. The longer the time interval, the lower is the reliability. The correlation may have been poorer had patients completed the questionnaire weeks or months after surgery. Yet, our response rate would probably have also been lower. During our first test-retest study, the time interval was 1 h, which seems brief. However, our patients leave the hospital sufficiently quickly after surgery that a 1-h interval was the longest that was feasible. Yet, 1 h was sufficiently long for responses to the initial and repeat question “I was satisfied with my anesthetic care” to not be correlated. [1]**** Simply asking this one question would be an unacceptable way to measure patient satisfaction with anesthesia. This finding was expected because single questions are usually found to produce responses that are not reliable over time. [4]

We designed the ISAS to measure patient satisfaction with their MAC itself, not other aspects of their perioperative experience. As we designed the questionnaire, we considered a hypothetical clinical trial. Patients undergoing MAC are randomized to receive one of two different anesthetic agents (e.g., alfentanil versus propofol). The ISAS would then be used to compare patient satisfaction with the different drugs. This limitation has two consequences, as considered in the following paragraphs.

Process questions were not included because some anesthetic drugs have amnestic properties. For example, patients were not asked whether their anesthesia provider spoke directly to them during the anesthetic. [7] In the hypothetical trial, patients receiving propofol might “disagree very much” that this process occurred. Thus, had such a positive process question been included, amnestic agents would have decreased patients' scores yet not changed their satisfaction. Further, patients are generally satisfied with being amnestic for their surgery. Among patients receiving amnestic agents, responses to positive process questions would not have been correlated to responses to other questions. Thus, had process questions been included, the satisfaction scale may have had a poor internal consistency.

Questions were not included in the ISAS that addressed the perioperative experience other than the anesthetic itself. For example, patients were not asked whether they felt physically exposed in the corridors. [8] Such issues are important to patients. [8] However, our postoperative interviews with patients showed that they do not consider such issues to be aspects of the anesthetic itself. More specifically, Gerteis et al. identified seven “dimensions” of inpatient care from the patient's perspective. [8] We found that two of these dimensions overlap with patient's satisfaction with their anesthetic:“physical comfort” and “emotional support and alleviation of fear and anxiety.” Several questions included in the ISAS (Table 1) are representative of these two dimensions. A third dimension is “respect for patients values, preferences, and expressed needs.” This dimension includes treating patients kindly. The initial set of questions (Table 1, left column) included several in this category, including “I felt alone,”“No one was helping me,”“No one cared about me,” and “I liked my anesthetist.” Our results show that this dimension of care contributes little to distinguishing patient's satisfaction with their anesthetic. More than 90% of patients disagreed very much with each of the first three questions and agreed very much with the fourth question. This result is logical because in the OR during MAC the anesthesia provider remains next to the patient and devotes his or her attention to the patient. A fourth dimension is “information on clinical status, progress, and prognosis.”[8] This dimension includes mostly process questions, and so was not included as considered in the preceding paragraph. The other three dimensions of patient-centered care are “coordination and integration of clinical care,”“involvement of family and friends,” and “transition and continuity. [8]“None of these dimensions was expressed by the patients during the postoperative interviews to be elements of satisfaction with anesthesia. This result makes sense.” Coordination of clinical care is clear in the OR with an anesthesia provider sitting next to the patient. Patients understand that family and friends rarely have a role inside the OR. Further, patients seem to consider their transition to home in the phase II PACU to occur after their anesthetic has ended.

We compared patients' scores on the ISAS to those predicted by observers (convergent validity), to make sure the ISAS measures what happens during the anesthetic. For convenience, we used each patient's anesthesia provider as the observer because they are present continuously during the anesthetic. A patient's score on a satisfaction with anesthesia scale may not reflect what happens during their anesthetic. We consider three examples. First, the concept of postoperatively measuring satisfaction with anesthesia conceptually raises an ethical issue. Patients could be dissatisfied intraoperatively (e.g., from pain) and yet satisfied postoperatively (from amnesia). Second, altered states of consciousness from anesthetic drugs may alter patient's perception of their condition during the intraoperative period. Patients' responses on the ISAS could reflect hallucinations rather than care during the anesthetic. Third, anesthetic drugs may alter patients' perception of time. Patients' responses on the ISAS may reflect the preoperative and/or postoperative periods rather than the period during which the anesthesia provider was caring for the patient. The presence of convergent validity shows that these three scenarios are not a problem. The ISAS appears to measure what happens during the anesthetic.

We designed the ISAS based on patients undergoing ambulatory MAC. Interviewing patients at home after surgery may have had advantages, since patients' satisfaction with anesthesia could have been measured several days after surgery. This would have been advantageous because our goal was to measure patients' long-term satisfaction with anesthesia. However, many of our patients come to the University of Iowa from large distances, making home interviews prohibitively expensive for routine use of the ISAS.

We did not consider telephone interviews to be feasible. First, telephone interviews increase the problem of confidentiality. Our patients may perceive that their responses are not confidential, decreasing response rates. [12] Second, some of our patients do not have phones, and so could not be interviewed. Third, we may not be able to contact some of the patients. Trained telephone interviewers make at least five attempts to reach a respondent. [12] We were concerned that some users of the ISAS may not be this diligent in their patient follow-up. Fourth, and most importantly, interviewers would be speaking the questions to the patients. Development of the ISAS with professional interviewers would have been expensive. Also, the ISAS would have then been of limited routine use. Most health care professionals (e.g., surgical nurses) are not trained interviewers. Untrained interviewers are generally poor at handling patients' inquiries about questions and are rated as unsatisfactory when their interviews are tape recorded. [9] To minimize the potential for interviewer bias, we made the ISAS a self-administered questionnaire.

Our Department of Anesthesia has experience with mailing brief postoperative questionnaires to our patients. When a stamped envelope was included with the questionnaire, the response rate was 57%. [10] In our mail study, the response rate was 64%. Although this response rate is high for a mail survey, [2] it is too low for the ISAS to be useful for many clinical trials. [11] Only those patients sufficiently motivated to complete and return the questionnaire would do so. Patients returning the questionnaire might not be representative of all patients undergoing anesthesia. [12] Further, with a self-administered questionnaire completed at home, the patient may not be the person answering the questions.

For these reasons, we had patients complete the ISAS in the hospital. Length of time that elapses between the end of surgery and completion of the ISAS may change patient responses. We wanted as much time to pass as was possible. We also wanted the patients to be awake and have their reading glasses. Thus, we asked patients to complete the questionnaire when they were in the phase II PACU.

We do not know whether completion of the ISAS in the hospital biased patients against full disclosure of poor satisfaction. To decrease the chance that patients would try to please the provider with their responses, the investigator inviting each patient to participate in our study was never one of the patient's anesthesia providers. In addition, after the investigator handed the ISAS to a patient, he walked at least several yards away from the patient. We followed this protocol to assure patients that their responses were confidential.

The response choices (Table 2) are bipolar and symmetrical, as is usual for a summated rating scale. Response choices are displayed in a single, vertical response column printed below each statement for clarity. [2,3] We did not list the responses horizontally to save space because this method has been shown to increase the chance that a patient marks a response other than the one they intended. [3]

There are six response choices for each question (Table 2). We did not consider using yes-or-no questions because restricting measurement to two levels decreases precision of the satisfaction scale. We chose to use these six response choices based on psychometric studies. [4] Differences between these six successive response choices are equally spaced. [4] Thus, patients' responses to each statement can be analyzed quantitatively. The numbers assigned to each of the six response choices (-3, -2, -1, 1, 2, or 3) can be interpreted as numbers, not just categories. [4]

A patient's overall score for the ISAS equals the mean of their responses to each question (i.e., the ISAS is a summated rating scale [4]). A concern about using such scales is whether responses to each question should be given equal weight. [3] For example, perhaps patients' responses to “I felt safe” are less important than responses to “I felt pain.” Usually summated rating scales are simple sums of scores. [3] We had no a priori reason to assign greater weight to any one question. Thus, we gave equal weight to each question.

Our study's design introduces the statistical problem of how to handle questions that participating patients did not answer (i.e., “item nonresponse”). Item nonresponse rates are often high for questions dealing with private or sensitive issues. [2] The lack of such questions in the ISAS may explain why our item nonresponse rate was low [2](4% for one question, 1% for eight questions, and 0% for the other two questions). We chose to handle item nonresponse by deleting entirely responses from the 9 + 140 = 6% of patients who did not complete the questionnaire. This approach was practical because the item nonresponse rate was below 10%. [2,11] Patients who did not answer all of the questions were unlikely to have been different than patients who answered all questions. Thus, the calculated mean and standard deviation of all patient's scores were probably not affected by deletion of these patients. Even if these patients' satisfaction scores were somehow different, the fact that so few questionnaires were deleted decreases the likelihood of an important bias in the mean and SD. Had we observed a higher rate of item nonresponse, then we would have used the next most sophisticated approach to scoring the ISAS. Responses to a question that a subject skips would have been set equal to the median response computed from the other respondents or perhaps a demographically “matched” subset of patients.

We have developed a reliable and valid measure of patient satisfaction with MAC. We chose to focus on MAC for two reasons. First, during MAC, maximizing patient satisfaction is one of the major goals. Learning how to measure the benefit that anesthesia providers contribute to patient satisfaction is particularly important. Second, developing a satisfaction scale for MAC may be an easier task than simultaneously developing a satisfaction scale for all types of anesthesia. For example, by studying patients undergoing MAC we could establish convergent validity of the ISAS. Future research will show whether the ISAS is a reliable and valid measure of patient satisfaction with other types of anesthesia and groups of patients.

*Johnstone RE: Value-based anesthesia care: maturing beyond cost-containment. ASA Newsletter 60 (10): 7–8, 1996.

**Cronbach's alpha involves comparison of the variance of the sum of all responses (total score) with the variances of the responses to individual questions. Mathematically, when responses to questions are not correlated, the variance of the total score will be equal to the sum of variances for each question that comprises the total score. As the responses to questions become more and more correlated, the variance of the total score will increase. [1]

***Some patients may always mark the first response to a question. By varying the direction of the questions, we tried to minimize this order bias. [4] Such bias is of greater concern with telephone interviews than with self-administered questionnaires. [12] Yet, we took the conservative approach of including questions written in both directions.

****Combining all test-retest data, tau =-0.07, P = 0.35, N = 31. In contrast, the most “physical” questions in the ISAS (numbers 1, 3, 5, 9, and 11) have significant positive test-retest reliability (all tau > 0.53, P < 0.001, N = 31).