Development and Validation of the Questionnaire of Satisfaction with Perioperative Anesthetic Care for General and Regional Anesthesia in Taiwanese Patients

• Few measures have been validated to assess patient satisfaction after anesthesia.

• In more than 1,000 patients receiving general anesthesia in Taiwan, a questionnaire instrument was developed and validated.

THE development of a valid and reliable instrument for measuring patient satisfaction with anesthetic care is a complicated and time-consuming process.1–5A few questionnaires or instruments related to patient satisfaction with perioperative anesthetic care were developed by using a rigorous and standardized psychometric process that included the inputs of viewpoint from patients, multiitems and/or multidimensionality generation, and assessment of reliability and validity of the instruments.6–13 

Many shortcomings did exist in these reports. For instance, most of these reports6–10used exploratory factor analysis (EFA), which is a technique for data reduction or hypothesis generation used to uncover the underlying factor structure of a relatively large set of variables. Thus, EFA explores the data and provides researchers with information about how many factors are needed to best represent the data.

More recently, psychometricians and sociometricians, in addition to using EFA for data reduction and dimension determination, have tended to use a more sophisticated statistical approach called confirmatory factor analysis (CFA). CFA is hypothesis-testing approach that is based on a strong theoretic and/or empiric foundation14that is used to confirm or reject the postulated model. Meanwhile, CFA can also be used to examine whether the items and the underlying constructs of the model could react the same for different groups, such as men and women or patients receiving general anesthesia (GA) or regional anesthesia (RA). For the previous questionnaires developed for the evaluation of anesthesia-related perioperative patient satisfaction, none of these studies used CFA to perform this kind of confirmatory test and validation of the conceptual framework of the measurement models.

Four reports6,7,9,11included not only GA patients to evaluate perioperative patient satisfaction. However, no cross-validation procedure was performed to elucidate whether validity extension did exist in these four questionnaires for the assessment of perioperative patient satisfaction with GA versus  other types of anesthesia.

The current study describes the use of EFA, CFA, and cross-validation techniques to validate a pilot questionnaire developed in a previous study15to generate a high-quality psychometrically developed and conceptual model estimated questionnaire called the Patient Satisfaction with Perioperative Anesthetic Care questionnaire (PSPACq). This questionnaire assesses patient satisfaction by focusing on perioperative anesthetic care in Taiwanese patients.

The protocol was approved by the Institutional Review Board (Chia-Yi, Taiwan) of Chia-Yi Christian hospital before commencement of the study. Informed consent was obtained from all patients who participated. Our hospital is the largest and is a functional Christian general and teaching hospital in southern Taiwan. A wide range of surgical procedures could be performed in our hospital. The number of hospital beds is 997. The average number of anesthetics per year is more than 18,000.

The criteria for patient inclusion were as follows: age older than 18 years, consent to participate in the study, ability to read and understand Chinese, ability to speak mandarin Chinese or Taiwanese, ability to complete a Chinese questionnaire within 2 days after surgery (not including the patients who participated in a face-to-face interview), and elective procedures (except obstetric) under GA, RA, or monitored anesthesia care. Any patient who participated in one part of the study was excluded from any other part of the study. The exclusion criteria were as follows: patients unable to communicate in the face-to-face interview, those incapable of completing the questionnaire by themselves, patients expected to stay in the intensive care unit postoperatively, those who were sedated or cognitively impaired, and patients receiving combined GA and RA during surgery.

The process for developing our pilot questionnaire was described in a previous report.15We summarized the process and added more detailed description, as follows: item generation began with a comprehensive literature review of a panel of experts comprising six anesthesiologists, four nurse anesthetists, two sociologists, and one statistician. The professionals were also involved in the ongoing revisions in every step of the questionnaire construction.

We searched the EMBASE, PubMed, MEDLINE–Ovid, Cochrane Library, CINHAL, and PsychINFO databases with the following keywords: patient satisfaction, perioperative, questionnaire, psychometric, quality, anesthesia (including anesthesia), anesthesia care, patient perception, and outcomes measures. The search was restricted to English-language publications only.

Four related reports,7–9,12developed with rigorous psychometric methods, were selected as our main guidelines to develop our new questionnaire. We incorporated the dimensions and items inferred from these four reports into our interview guide plus the items we considered to be important in perioperative patient satisfaction with anesthetic care in Taiwanese patients. Our interview guide lists 78 items evolved from five dimensions (i.e. , information, discomfort and needs, provider–patient relationships, waiting period, and fear and concern).

Before discharge from the hospital, appointments were made with patients who received GA, RA, or monitored anesthesia care for their surgical procedures and consented to attend the semistructured interviews in our preanesthesia consultation clinic postoperatively. All the stage 1 patients were chosen using convenience sampling to obtain an adequate number of GA, RA, and monitored anesthesia care patients. The interviews were conducted by a trained interviewer (W.C.M.), usually on the days when the patients returned to our hospital for postoperative follow-up. The interviews were held in a quiet room in the preanesthesia consultation clinic and divided into two parts. (1) Patients were asked to freely talk about their feelings regarding satisfaction with perioperative anesthetic care, so that we could identify possible items to be included in the pilot questionnaire. (2) This part of the interview was performed on the list in the interview guide that consisted of 78 items written in Chinese. We used an open-ended question at the end of the interview to elicit remarks on missing items, wording, content, and any supplemental comments.

An interview was continued until no new ideas emerged. All interviews were video recorded and transcribed. The results of each interview were analyzed by three members (F.M.H., C.M.C., W.C.M.) of our expert group who were skilled in textual analysis. Items were regenerated and revised by the expert group to ensure that all the important elements were included; then, the first version of the pilot questionnaire was constructed. The wording of the questionnaire was deliberately designed by using simple, concise, and nondirective Chinese to minimize measurement biases created from confounding variables, such as social desirability, prolonged completion time, use of proxy, or low response rate.16 

We also performed pretest and determination of the content validity coefficient (V value) and homogeneity reliability coefficient (H value)17,18of each item and the questionnaire as a whole to develop our final version of the pilot questionnaire. Aiken17,18offered a set of procedures for computing and determining the V and H coefficients to objectively evaluate the statistical nature of the reliability and validity of the content of the developed instruments. The techniques used to calculate the V and H coefficients were described in a previous report.15 

This stage included the process of EFA of the pilot questionnaire for factor structure generation; therefore, the PSPACq, and the CFA for model confirmation of the PSPACq for patients receiving GA for surgery, was developed. If the measurement model of the PSPACq for the GA patients was confirmed, CFA of the group of patients who received RA for their surgery would be performed likewise to assess the feasibility of cross validity of the PSPACq from GA to RA patients.

Patients were given the self-administered anonymous questionnaire in the ward within 6–48 h after their surgical procedures under GA or RA by two investigators who were trained to ensure that the whole data collection process was performed using a standardized procedure. The investigators were wearing a uniform and had a photo-identification badge with their names clearly shown. The method of randomized sampling was used for the stage 2 and 3 patients in this study. Each investigator walked up to a patient and said the following:

“Mr or Ms ..., I am Ms ..., I am a research personnel of the department of anesthesia. I am not a member of the anesthesia team taking care of you. We would like to know what we can do to improve patient satisfaction with anesthetic care. We need your opinions to develop a questionnaire of patient satisfaction with anesthetic care which will help us to obtain information to provide a high-quality anesthesia service for patients. Your opinions are strictly confidential. Would you please take several minutes to answer our questions by filling out the questionnaire?”

If the patient agreed to participate and signed the informed consent, the investigator then handed our questionnaire and a pen to the patient and stepped several yards aside. Investigators were prohibited from prompting patients to answer missing responses or making patients participate. Answers that were ambiguous, unclear, or incomplete were considered to be missing responses.

In stage 3, we test the nomological validity of the PSPACq model. Nomological validity describes the relationships between the construct of interest and other constructs. For example, if a patient is satisfied with the services of a hospital, theoretically, the patient will have some degree of loyalty to the hospital. When a positive correlation between patient satisfaction and patient loyalty exists, there is evidence of nomological validity of patient satisfaction. In addition, if different ages of patients have varying degrees of satisfaction and, thus, the relationship of age difference affects the level of satisfaction, evidence of nomological validity of patient satisfaction with the age factor exists.

In this study, we tested the nomological validity of the PSPACq by comparing the results of the effect of the confounding variables on PSPACq scores with the results of antecedent research. The consequences of patient loyalty on PSPACq scores were also tested.

In stage 1, the final version of the pilot questionnaire was developed. Only the individual items and the questionnaire as a whole that had strongly statistically significant V and H coefficient values (P < 0.01) were included.

In stage 2, statistical analyses were conducted in several steps using the following computer programs: SPSS 18.0 (Statistical Product and Service Solutions, IBM Corporation, New York, NY) and LISREL 8.80 (Scientific Software International, Inc., Lincolnwood, IL). Three independent groups of data were collected in this stage. For group 1, the patients who underwent surgery under GA, EFA, by using principal axis factoring with promax rotation, was used to generate factor structure of the PSPACq. Factors would be identified with eigenvalues greater than 1.00. A value of 0.30 was used as a viable cut point for judging the saliency of factor loadings. For group 2, in which the patients also underwent surgery under GA, we tested the validity of factor structure derived from the results of EFA by using CFA. Finally, group 3 patients underwent surgery using RA. The PSPACq model was tested by CFA in this group; if a well-fitting model did exist, it indicated that the factor structure of the PSPACq could be applied to the patients receiving GA and those receiving RA during their surgical procedures.

The answer to each question of our 30-item questionnaire (after deletion of items 10 and 22) was rated on a five-point Likert scale, ranging from 1 (very dissatisfied) to 5 (very satisfied) for items 1–5 and 11–21; from 1 (very uncomfortable) to 5 (not uncomfortable) for items 6–9; and from 1 (very fearful) to 5 (not fearful at all) for items 23–32.

Several criteria were used in determining the overall fit of the data for PSPACq model structure. They included the Comparative Fit Index and the Non-Normed Fit Index, which must meet or exceed 0.9019; the root mean square error of approximation, in which values lower than 0.05 are indicative of good fit and those from 0.05 to lower than 0.08 are of reasonable fit20; and the standard root mean square residual, in which values lower than 0.08 indicate acceptable fit.21 

In stage 3, the effects of the potential confounding variables on each dimension and the total questionnaire scores were analyzed by using an independent t  test or one-way ANOVA with the Scheffépost hoc  strategy. The confounding variables included sociodemographic rating (i.e. , age, sex, educational level, and marital status) and clinical features (i.e. , type of anesthesia, type of surgery, American Society of Anesthesiologists class, and number of anesthesia doses received).

Much research22–25has revealed that patient satisfaction could significantly predict patient loyalty. Thus, we considered that patient loyalty could be a good external criterion to evaluate the nomological validity of the PSPACq. Data were collected from a new group of 100 patients (GA, 78; RA, 22) who completed the PSPACq and the 5-item patient loyalty scale within 48 h after their surgical procedures.

The patient loyalty scale was derived from the customer loyalty scale by Zeithaml et al .26The customer loyalty scale has five items that have excellent internal consistency, evidenced by α coefficients ranging from 0.93 to 0.94 across the four tested companies. Trasorras et al.  27also tested these five items and obtained excellent internal consistency (α= 0.91). We modified some wording in our patient loyalty scale to fit our study purpose. The items were as follows: “To say positive things about our hospital to other people,”“To recommend our hospital to acquaintances who need to see a doctor,”“To encourage your friends and relatives, if they fall ill, to visit our hospital to see a doctor,”“To consider our hospital as your first choice for seeking advice of a doctor,” and “To keep close contact with our hospital in the near future.” Each item was rated on a five-point Likert scale, ranging from 1 (strongly agree) to 5 (strongly disagree). The reliability of internal consistency of our measure was 0.74. The Pearson correlation was used to test the correlations between the loyalty score and each satisfaction dimension.

In the process of pilot questionnaire development, the study part was conducted between July 1, 2008 and January 21, 2009. A new cultural-specific dimension called “anesthesia-related sequelae” emerged from the semistructured interviews of our patients; this dimension was sufficiently distinguishable to be a separate dimension because of discrepant cultural background and a different health care system in Taiwan. A six-dimension 32-item final version of the pilot questionnaire was developed. A previous report15provides further details.

Data collection for EFA and CFA in three groups of patients was conducted between March 1, 2009 and June 28, 2010. The missing response rates for groups 1, 2, and 3 were 14%, 16%, and 12%, respectively. The average times of completion of the questionnaire in groups 1, 2, and 3 were 6 ± 3, 5 ± 3, and 5 ± 2 min, respectively. The demographic data, clinical features, and history of anesthetic care for groups 1, 2, and 3 in the validation stage are described in table 1.

Before EFA was conducted in group 1 patients (n = 320), the factorability of the correlation matrix was evaluated. The Kaiser–Meyer–Olkin Measure of Sampling Adequacy was 0.80, indicating that the factor structure was appropriate for group 1.28The Bartlett Test of Sphericity was 6104.74 (P < 0.001), allowing rejection of the hypothesis that the correlation matrix is an identity matrix and indicating an appropriate factor structure. The analysis, performed for the answers to the 32 items of the pilot questionnaire of group 1 patients, produced a seven-factor solution. Items 10 and 22 did not have enough substantial factor loadings on any of the factors. Thus, these two items were deleted and resulted in the development of the PSPACq, which consisted of 30 items. The factor loadings for the 30 remaining items are presented in table 2.

The first factor, labeled as provider–patient relationship, included items 11–17 and accounted for 19.96% of the item response variance. The second factor, labeled as information, included items 1–5 and accounted for 16.08% of the variance. The third factor, labeled as anesthesia-related sequelae, included items 29–32 and accounted for 6.67% of the variance. The fourth factor, labeled as fear, included items 24–26 and accounted for 4.77% of the variance. The fifth factor, labeled as concern, included items 23, 27, and 28 and accounted for 3.86% of the variance. The sixth factor, labeled as discomfort and needs, included items 6–9 and accounted for 2.92% of the variance. The seventh factor, labeled as waiting period, included items 18–21 and accounted for 2.38% of the variance. In combination, these factors accounted for 56.64% of total item variation. The Cronbach coefficient α values were 0.90 for provider–patient relationship, 0.87 for information, 0.87 for anesthesia-related sequelae, 0.87 for fear, 0.81 for concern, 0.58 for discomfort and needs, and 0.61 for waiting period. Although this result showed that the reliability for discomfort and needs is poor, its reliability enhanced to 0.71 in CFA analysis. This is why we did not delete this factor.

We performed CFA for testing the structure validity of the seven-factor 30-item PSPACq in a new group of patients (group 2, n = 565) who also underwent surgical procedures under GA. The results of overall fit indices showed the following: chi-square test = 1,622.52, df = 384, P < 0.001. Although the chi-square value is large, because of sensitivity to sample size, it is accordingly reported but not used to evaluate the model.29All other fit indices (i.e. , Comparative Fit Index = 0.92, Non-Normed Fit Index = 0.91, root mean square error of approximation = 0.076, and standard root mean square residual = 0.064) indicated that the model is acceptably fit. Therefore, our CFA model could likewise be accepted.

Standardized coefficients are shown in figure 1. Factor loadings of the information items ranged from 0.63 to 0.90; discomfort and needs items, from 0.49 to 0.58; provider–patient relationship items, from 0.49 to 0.97; waiting period items, from 0.45 to 0.75; fear items, from 0.70 to 0.84; concern items, from 0.47 to 0.94; and anesthesia-related sequelae items, from 0.66 to 0.89 (all significant). These results revealed that there was convergent validity for the seven- factor PSPACq in patients who receive GA.30From the viewpoint of the criteria proposed by Kline,29if the estimated correlations between factors are not excessively higher than 0.85, discriminant validity is indicated. Figure 1shows that the correlations between factors are not excessively high, indicating that the PSPACq model has discriminant validity in patients who receive GA.

In group 3 patients (n = 225), the model confirmation testing of the PSPACq in patients receiving RA during their surgical procedures proved it a well-fitting model (chi-square test = 740, df = 385, P < 0.001, Comparative Fit Index = 0.93, Non-Normed Fit Index = 0.92, root mean square error of approximation = 0.064, standard root mean square residual = 0.070). Standardized coefficients are shown in figure 2. Factor loadings of the information items ranged from 0.73 to 0.92; discomfort and needs items, from 0.53 to 0.68; provider–patient relationship items, from 0.37 to 0.94; waiting period items, from 0.24 to 0.86; fear items, from 0.82 to 0.88; concern items, from 0.48 to 0.97; and anesthesia-related sequelae items, from 0.73 to 0.87. All of the factor loadings were significant. These results revealed that there was convergent validity for the PSPACq in patients receiving RA during their surgical procedures. The correlations between factors for this model are also not excessively high (<0.85). Discriminant validity for this model held true. The α reliability coefficients for these seven factors were acceptable (information, α= 0.92; discomfort and needs, α= 0.71; provider–patient relationship, α= 0.89; waiting period, α= 0.71; fear, α= 0.88; concern, α= 0.75; anesthesia-related sequelae, α= 0.87).

The results of confounding variable analysis showed that the mean total satisfaction scores were significantly influenced by age, sex, educational level, types of anesthesia, and different surgical procedures. Older age (>45 years), male patients, primary school educational level, GA patients, and general and orthopedic surgery patients have significantly high satisfaction scores (table 3). Our results coincide somewhat with those of previous reports.6–9 

Pearson product moment correlation coefficient analysis revealed that the correlations between the loyalty score and each satisfaction dimension ranged from 0.203 to 0.461; and between the total satisfaction score, 0.548. The results were all statistically significant (table 4).

Fourteen negatively worded items were reversed so that higher scores represented higher levels of satisfaction. We followed the research performed by Auquier et al.  8by linearly transforming all dimension scores (including global satisfaction) to a 0–100 scale, with 0 indicating the worst level of satisfaction and 100 indicating the best level of satisfaction. Table 5shows the range of scores, means, and SDs of total samples of stages 2 and 3 for all dimensions. By transforming scores to a 0–100 scale, we could divide this scale into five intervals. A score lower than 20 is considered “very unsatisfied”; equal or greater than 20 and lower than 40, “unsatisfied”; equal or greater than 40 and lower than 60, “adequate”; equal or greater than 60 and lower than 80, “satisfied”; and equal or greater than 80, “very satisfied.” The range of scores, means, and standard deviance values for PSPACq dimensions is shown in table 5.

Patient satisfaction is an important indicator of health care outcome and evaluation of the quality of services in anesthesiology.1–3,30The importance and lack of standardized, valid, and reliable questionnaires to assess patient satisfaction in anesthetic care have been emphasized in many reviews.3,30–32 

Validity is a unitary concept. It refers to the degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests. Therefore, validity is the most fundamental consideration in developing and evaluating tests.31Content or face validity must be established before any theoretical testing when using CFA. Content validation is the process of assessing the extent of the content of the test representing the content domain. The process of validation involves accumulating evidence to provide a sound scientific basis for the proposed score interpretation.31,32 

We used four questionnaires (i.e. , the questionnaire of Heidegger et al. ,9the Iowa Satisfaction with Anesthesia Scale,12the Evaluation du Vecu de l'Anesthesie Generale,8and the Leiden perioperative care patient satisfaction questionnaire7) as our main guidelines. We believed that these four patient satisfaction with perioperative care questionnaires could meet the criteria necessary for good psychometric questionnaire development.16,33–36Our pilot results revealed that five dimensions of our questionnaire were compatible with these four questionnaires.

We used literature review, expert consultation, and face-to-face in-depth interviews to obtain direct input from the patients; and pretest and Aiken V and H coefficients,17,18in addition to an open-ended question at the end of the interviews, to ensure high face and content validity and reliability of the final version of our pilot questionnaire.

Traditionally, content validity is regarded as a qualitative type of validity and is evaluated by subjective logical analysis by the experts. The statistical nature of the reliability and validity of the content of the developed instruments was often neglected. For instance, two studies used different levels of relevance11and importance6to rate their items in the stage of questionnaire development.

However, the items included for questionnaire validation in these two studies were mainly based on subjective analysis by the researchers, without any objective statistical data to support them. We deleted three more items because of lower individual V values and insignificant right tail distribution (P > 0.01) in the stage of questionnaire development. Therefore, we believe that the final version of our pilot questionnaire contains evidence to support high content validity and internal consistency.

One of the primary objectives of CFA is its ability to assess the construct validity of a proposed measurement model. The term construct  is defined in a broader way, as a characteristic or concept that a test or other measurement procedure is intended to measure.31According to Hair et al. ,37construct validity is composed of four important components: face (content), convergent, discriminant, and nomological validity.

The items that are indicators of a specific construct should converge or share a high proportion of variance in common, known as convergent validity. Discriminant validity is the extent to which a construct is truly distinct from other constructs. It is beyond our scope to elaborate on the validity theory in this article. Detailed clarification can be found in other studies.31,35,38,39 

In this study, we also intended to develop a good psychometric patient satisfaction questionnaire for monitored anesthesia care patients. Therefore, we included the monitored anesthesia care patients in the stage of pilot questionnaire development. However, the number of patients who received monitored anesthesia care for their surgery is relatively small in Taiwan. We need a further study to collect enough data to determine whether this seven-factor structure or a new conceptual structure is suitable to apply to the monitored anesthesia care patients in Taiwan.

The questionnaire developed in this study was focused on Chinese patients in Taiwan. Further studies need to be performed to obtain additional scientific evidence to support the generalizability of our questionnaire, which covers the Chinese population, the non–Chinese-speaking population of Asian countries, and the English-speaking population in the Western world.

The range of time needed to complete the 30-item PSPACq by our Chinese patients was approximately 3–8 min. Although there is no well-accepted optimal timing of anesthesia satisfaction assessment, a shorter questionnaire that still maintains high validity and reliability, with simple, clear, and concise wording, will lessen the degree of burden imposed on patients who need to complete it. Further study is necessary to develop a shorter and easier-to-answer questionnaire that still has acceptable validity and reliability.

Hair et al.  38emphasized that EFA explores the data and provides the researchers with information about how many factors are needed to best represent the data. Because these factors were derived from statistical results, not from theory, they could only be named after the EFA was performed. CFA does not assign variables to factors. Instead, CFA is applied to test the extent to which the EFA-derived factor structure can represent the actual data. Therefore, CFA is a tool that enables us to either confirm or reject our preconceived conceptual model. In this study, EFA in group 1 (GA) patients revealed a 30-item seven-factor solution (i.e. , the PSPACq, with a high percentage of trace extracted and acceptable Cronbach coefficient α values). This seven-factor structure was further estimated by CFA in group 2 (GA) patients. The CFA results disclosed a well-fitting model to the samples, with significant convergent and discriminant validity (fig. 1). The cross validation by another CFA in group 3 (RA) patients using the PSPACq model also revealed well-fitting results (fig. 2).

The results of the confounding variable analysis showed that there are, to some degree, theoretically predictable relationships between each satisfaction dimension with age, sex, educational level, types of anesthesia, different type of surgery, and loyalty. The correlations of the loyalty scores with the satisfaction scores are salient. These findings are consistent with the results of previous reports; therefore, they support evidence of the nomological validity of the PSPACq.

Predictive validity was not thoroughly established in our study. To establish predictive validity in the development process of the patient satisfaction with anesthetic care instrument is practically difficult because we do not know when, in the future, patients would attend again because of becoming ill and receiving surgical and anesthetic care in our hospital. Therefore, the collection data for testing predictive validity is time-consuming and hard to control. In addition, we know that there is no generally acknowledged external criterion for evaluating predictive validity for patient satisfaction with perioperative anesthetic care. Accordingly, no predictive validity could be established in the previous published questionnaires developed for the evaluation of patient satisfaction with perioperative anesthetic care.

In conclusion, a 30-item seven-factor questionnaire (PSPACq), with Taiwanese cultural characteristics, was developed to test patient satisfaction with perioperative anesthetic care in Taiwanese patients. This questionnaire passed through the validation process of conceptual model confirmation and cross validation that is suitable for patients who received either GA or RA during their surgery.