Validation of the Non-communicating Children's Pain Checklist–Postoperative Version

Click on the links below to access all the ArticlePlus for this article.

Please note that ArticlePlus files may launch a viewer application outside of your web browser.

• https://links.lww.com/ALN/A128

DESPITE increasing research into pediatric pain in recent years, 1many children are still not appropriately medicated for postoperative pain. In 1983, Mather and Mackie 2reported 31% of their sample were not given any analgesic postoperatively. The same rate was reported by Kart et al.  3in 1996. A recent study found 51% of 48 children did not receive sufficient postoperative analgesics to keep their pain levels below their personal treatment threshold. 4 

One reason for continued inadequate management of postoperative pain could be the lack of available and validated pain assessment tools. Structured pain assessment can contribute to improved prescription and administration of analgesia for children. 5However, observational postoperative pain tools are still needed for some groups of children, such as those who cannot give self-reports. For example, pain assessment tools for preschool children have only recently appeared (e.g. , FLACC 6; Pain Observation Scale for Young Children 7; COMFORT 8).

Children with neurologic impairments are often unable to report on their pain because of their intellectual or physical limitations. However, at present, no tools have been validated for their postoperative pain. The one study that investigated use of pain tools designed for “normal” children suggests these tools may not be adequate for these special children. 9Thus, pain tools specifically designed for children with intellectual disabilities are needed.

This study investigated whether the Non-communicating Children's Pain Checklist (NCCPC), 10designed specifically for children with intellectual disabilities, could detect their postoperative pain. Previous research suggests the NCCPC is valid and reliable in the home setting 11and that caregivers’ report using the NCCPC can predict future pain behavior. 12However, pain at home may differ from postoperative pain. Thus, validation of the NCCPC for postoperative pain is needed. This should include assessment of its validity and reliability when used by adults who are not familiar with the children having pain to ensure it is clinically useful. To accomplish these goals, caregivers, researchers, and nurses observed 25 children undergoing surgery at a tertiary care children's health center. Caregivers were asked to take part in this study because their pain estimates are most closely related to children's self-reports, 13and it was felt an examination of the correlation between their ratings, and those of researchers with a different perspective and no familiarity with the children, would be a stringent test of interrater reliability.

This study was approved by the Research Ethics Board of the IWK Health Centre, a tertiary pediatric center serving Canada's three Maritime provinces. Caregivers provided informed written consent, and surgeons provided verbal agreement for their patient to participate.

Twenty-five children who required surgery during the first 2 yr of a longitudinal study of pain in nonverbal children with intellectual disabilities took part.

The NCCPC–Postoperative Version (NCCPC-PV) contains six of the seven subscales of the original version. 10Caregivers and researchers indicated how often each item listed in table 1was observed during simultaneous 10-min observations (not at all = 0, just a little = 1, fairly often = 2, very often = 3). Scores for all items were summed to create total scores. The Eating–Sleeping subscale of the original version was not included because it was believed it might lead to false-positive results. Some children were not allowed to eat, either before or after surgery, as both anesthesia 14and analgesics 15can cause sleepiness and nausea. In addition, it was believed that judgments of these items would require more than 10 min of observation, which is difficult to obtain in most clinical situations. However, analyses of the effect of adding the subscale to total scores were conducted to support this decision. A 100-mm visual analog scale (VAS) was also used to rate the intensity of children's pain during the same observations, anchored by 0 (no pain) and 100 (worst pain ever).

The Vineland Adaptive Behavior Scales, 16a semistructured interview, were administered to caregivers by a trained researcher at the time they entered the longitudinal study to assess the children's functioning. This was 8 days to 17.5 months before the day of surgery (mean, 7.6 months; SD, 5.5 months). Demographic information was also collected at that time. Once in the longitudinal study, caregivers contacted a researcher if their child was scheduled to have surgery.

A caregiver and one of three researchers, who did not know the child previously, observed each child. On the day of surgery, they observed each child for the same 10 min, approximately 30 min before surgery and 30–60 min after the child left the recovery room. They completed the NCCPC-PV and a VAS rating of the child's pain intensity for those 10 min. No training was provided to caregivers on how to complete the NCCPC-PV. They were asked to observe their child for 10 min and to indicate how often they saw each item during the 10 min. Caregivers were told not to discuss their observations or ratings with the researcher also conducting an observation. At the same time, a recovery room nurse, who was also unfamiliar with the child, completed a VAS rating of the child's pain. After the child was released from hospital, a researcher collected information regarding the surgery from the child's medical records.

Data were analyzed using SPSS 10.0.7. 17Power analyses were conducted using SamplePower. 18For all tests, α was set at 0.05, and Bonferroni corrections were made for sets of multiple tests. This conservative correction was chosen because this is a new area of study with no previous research to aid interpretation of results. Thus, avoiding false-positive results was considered essential. Because the goal was to detect clinically significant changes, prospective power estimates were based on large effect sizes as defined by Cohen 19or based on effects reported in the literature for children without impairments. They are not reported for tests that were significant. Based on the sample size, the variances of the groups and Shapiro-Wilk tests of normality, parametric tests were appropriate for all continuous data.

The same nurse did not make preoperative and postoperative VAS ratings for 14 children. Thus, a multiple regression was performed to determine whether this affected postoperative VAS pain ratings before further tests were conducted. R = 0.37 indicated having the same or a different nurse make postoperative VAS pain ratings accounted less than 14% of the variance in scores. Thus, no adjustments were made.

Pearson correlations were used to investigate relations among the VAS ratings of the three observers (caregiver, researcher, nurse). Paired t  tests were used to compare VAS ratings from before to after surgery. Power for the paired t  tests was greater than 0.80, and power for the correlations greater than 0.90.

Internal reliability was assessed using Cronbach α. Two-way mixed-effects intraclass correlation coefficients for consistency, 20which correct for chance agreement, were used to assess interrater agreement.

Validity was examined four ways. Pearson correlations were used to assess convergent validity between VAS ratings and NCCPC-PV total scores. Eleven children were needed for power of 0.82. Total NCCPC-PV scores before and after surgery were compared using paired t  tests to assess construct validity. As an additional test of construct validity, mean NCCPC-PV subscale scores from before and after surgery were compared with a repeated-measures analysis of variance. Because sphericity assumptions were not met, Wilks λ was used to test significance. 21Finally, independent t  tests were used to compare total NCCPC-PV scores of children with mild versus  moderate–severe pain. Caregivers’ VAS ratings were used to group the children based on evidence that their ratings are typically most related to children's self-report. 13,22Children who were assigned a score of 30 or greater were classed as experiencing moderate pain or greater. 23The average NCCPC-PV score of children with mild pain was 12 (SD = 11), and for those with moderate–severe pain it was 53.5 (SD = 17).

Receiver operator characteristic curves, based on the bi-negative exponential assumption, were used to determine cutoff points for caregivers and researchers that offered the best combination of sensitivity and specificity. Caregivers’ VAS scores were used to define groups of children with mild versus  moderate–severe pain for both receiver operator characteristic curves so that the cutoff value found for each observer would be relative to the same VAS score.

To determine the impact of omitting the Eat–Sleep subscale for postoperative pain, paired t  tests were used to examine differences in total NCCPC-PV scores before and after surgery with and without the subscale for both researchers and caregivers. Pearson correlations between the VAS ratings of all three observers and the NCCPC-PV total scores of caregivers’ and researchers before and after surgery were then conducted. These were compared with the correlations among VAS ratings and total scores without the subscale. Power for these tests were equivalent to those for NCCPC-PV scores without this subscale.

Children's pain scores should not vary with their personal characteristics such as gender, as there is no evidence of systematic variation of self-report ratings of pain in children based on these. Similarly, pain does not vary with type of surgery. 14Because there is no accepted classification of surgery severity, length of surgery, length of time in the recovery room, opioid administration intraoperatively, and admission to hospital were used as markers for surgery severity to determine if NCCPC-PV scores differed. Before doing this, tests were conducted to determine if analgesic administration or VAS ratings varied as a result of these factors. A categorical regression, using optimal scaling techniques, 24was used to predict analgesic administration (none, nonopioid, opioid) after surgery with length of surgery, length of time in the recovery room, opioid administration intraoperatively, and admission to hospital included as predictors. Multiple linear regressions were used to predict VAS ratings with these factors. Independent t  tests were then used to examine the effect of dichotomous demographic and surgical factors on children's total NCCPC-PV scores. Pearson correlations were used to examine the relation between continuous demographic and surgical characteristics on NCCPC-PV scores. Power exceeded 0.80 for these tests.

To explore how nurses assessed pain, a multiple regression was conducted on their postoperative VAS ratings. Because nurses’ VAS ratings appeared to be most related to researchers’ ratings and NCCPC-PV scores, researchers’ subscale scores on the NCCPC-PV were entered as predictors.

The data from an autistic toddler who was extremely frantic before surgery, because he had not eaten and was experiencing hunger pains, was omitted. The 24 remaining children ranged in age from 3.7 to 19.6 yr (mean, 11.5 yr; SD, 4.5), 25% were girls, and they had had from 0 to 18 previous surgeries (mean, 7.1; SD, 4.6). The children were able to function only at low levels: 13.1 months (SD = 7.2) for Communication, 13.3 months (SD = 16.3) for Daily Living Skills, 17.1 months (SD = 19.5) for Socialization, and 7.5 months (SD = 10.8) for Motor Skills. 16The etiology of the neurologic impairments were: dysmorphic or chromosomal syndromes (n = 9), traumatic brain injury (n = 3), neonatal complications (n = 5), extreme prematurity (n = 2), neurodegenerative syndrome (n = 1), intrauterine acquired condition (n = 1), unknown (n = 1), and information was unavailable (n = 2). Further characteristics of the children are shown in table 2.

The procedures the children underwent included dental extractions (n = 5), G-button insertions and removals (n = 3), bilateral myringotomy tube insertion (n = 2), heelcord–tendon lengthening (n = 2), other orthopedic surgery (n = 3), endoscopies and biopsies (n = 2), subcutaneous venous access device insertion (n = 2), fundoplication (n = 2), strabismus repair (n = 2), and skin graft (n = 2). Nine of the children had two procedures. Surgeries lasted from 12 min to 3 h and 45 min (mean, 74 ± 65), and length of stay in the recovery room ranged from none to 5 h and 24 min (mean, 1 h and 48 min ± 70 min). Eight children received opioids intraoperatively, and 11 were admitted to hospital. However, only 8 (32%) remained in hospital 24 h after surgery.

The NCCPC-PV was not completed by a researcher after surgery for one child. Two additional item scores were missing from another researcher after surgery. These were replaced by the median response from all children for the applicable item after surgery. VAS ratings were not made by one parent before surgery and one researcher after surgery. These were replaced by the mean score for all children for that item at that observation time (preoperative, postoperative). VAS ratings by nurses were missing for five children before surgery and four children after surgery. Analyses indicated replacement of missing data did not generate further significant results. Thus, the results presented are those conducted without replacing these values.

Caregivers’ and researchers’ VAS pain ratings were significantly higher after surgery (P = 0.001 and P < 0.001, respectively), whereas nurses’ ratings were not (table 3).

Caregivers’ VAS pain ratings were significantly related to researchers’ ratings before and after surgery (table 4). Nurses’ preoperative ratings were also significantly related to caregivers’ and researchers’ ratings before surgery, but only to caregivers’ ratings after surgery.

Cronbach α values indicated that caregivers’ had excellent internal consistency (Cronbach α= 0.91), and researchers’ internal consistency was satisfactory (Cronbach α= 0.71). The proportion of children displaying each item is shown in table 1.

Intraclass correlation coefficients for postoperative NCCPC-PV subscale scores were 0.77 for the Vocal subscale, 0.48 for Social, 0.81 for Facial, 0.61 for Activity, 0.45 for Body and Limb, and 0.63 for Physiologic Signs. Intraclass correlations for total scores were 0.82 before surgery and 0.78 after surgery. Thus, total scores showed excellent interrater reliability. 25 

Paired t  tests showed both caregiver (P = 0.003) and researcher scores (P = 0.01) were significantly greater after surgery than before (table 3). A repeated-measures analysis of variance revealed a significant effect for Time (P = 0.001) and for Subscale (P < 0.001). No other effects or interactions were significant (table 5). Thus, all subscale scores were higher after surgery than before, and scores did not differ between caregiver and researcher.

Adding the Eat–Sleep subscale to NCCPC-PV totals increased only caregivers’ presurgery scores (P = 0.05). However, the addition increased both caregivers’ (P < 0.001) and researchers’ postoperative total scores (P < 0.001). Adding this subscale also reduced all but one correlation between VAS ratings and NCCPC-PV scores before and after surgery. The correlation between researchers’ NCCPC-PV total score and caregivers’ VAS ratings after surgery did not change. Thus, adding this scale may have increased false-positive results and reduced validity.

There were significant correlations between almost all caregiver and researcher measures (table 4). One exception was the relation between researchers’ VAS pain ratings and caregivers’ NCCPC-PV scores before surgery. No correlations between nurses’ VAS ratings and the VAS ratings or NCCPC-PV scores of caregivers and researchers were significant after correction for multiple tests.

Caregivers’ and researchers’ scores for children with mild pain were lower than their scores for children with moderate pain (P = 0.001 and P = 0.03, respectively). Mean NCCPC-PV scores of children with mild pain were 7.56 (SD = 4.86) for caregivers and 9.09 (SD = 6.09) for researchers, whereas the mean score for children with moderate to severe pain was 21.50 (SD = 13.85) for caregivers and 15.31 (SD = 5.82) for researchers.

A score of 11 or greater by caregivers provided the best combination of sensitivity (0.88) and specificity (0.81). This resulted in one false-negative result and one false-positive result. A score of 11 or greater also had the best sensitivity (0.75) and specificity (0.63) for researchers, resulting in two false-negative results and six false-positive results. Thus, a score of 11 or greater was a good indication that a child was experiencing at least moderate pain.

Caregivers’ total NCCPC-PV score did not differ as a result of the child's gender, whether they had cerebral palsy, or whether they lived with their family or in a residential center. Researchers’ postoperative NCCPC-PV scores also did not differ as a result of these factors. After corrections for multiple tests, only Communication age equivalents from the Vineland Adaptive Behavior Scales 15were correlated with NCCPC-PV scores (table 6). Thus, children's scores on the NCCPC-PV were not related to their personal characteristics.

Analgesic administration could not be predicted by any surgical factors. The nonsignificant multiple R of 0.36 indicated that all factors together predicted less than 14% of the variation in analgesic administration. Similarly, all factors together predicted less than 5–9% of the variation in caregivers’, researchers’, and nurses’ postoperative VAS ratings. Similarly, neither caregivers’ nor researchers’ total NCCPC-PV scores after surgery differed as a result of whether the child received opioids intravenously during surgery or were admitted to the hospital after surgery. Correlations between NCCPC-PV scores and length of surgery and length of time in the recovery room were also nonsignificant.

A multiple regression indicated that only the Facial subscale of researchers’ NCCPC-PV significantly predicted nurses’ VAS ratings (R = 0.64; R²= 0.41;P = 0.003).

These results provide evidence that the NCCPC-PV has good psychometric properties when used with children who have severe intellectual disabilities. The NCCPC-PV was internally consistent. Interrater reliability was good for total scores and very good for some subscales, such as the Facial and Vocal subscales. Lower interrater reliability for the Social and Body and Limb subscales were probably because caregivers were more familiar with the children's abilities to interact socially and to use their body and limbs voluntarily. Caregivers perceived all items from these subscales more frequently than researchers, with the exception of “seeking comfort” and “difficult to distract.” Therefore, caregivers may have noted subtle actions or attempts at actions that researchers were unable to notice.

The NCCPC-PV total and subscale scores were significantly higher after surgery than before. It is important that there was no differences as a result of observer, suggesting familiarity with these children may not be required to assess their postoperative pain and that, with further refinement, this scale may be clinically useful for healthcare professionals. It is also important that all subscales changed, indicating all have some ability to discriminate pain caused by surgery.

Receiver operator characteristic curves indicated a score of 11 by caregivers, and researchers correctly classified 88% and 75% of the children with moderate to severe pain, respectively. Although false-positive and false-negative results were greater for researchers, the two children who were falsely classed as having mild pain had scores of 7 and 8. Of the six children who were false-positive results for researchers, all but one had scores of 17 or less. Thus, these children did not have scores that were extremely removed from their correctly classified peers within the possible NCCPC-PV range of 0 to 81. The fact that sensitivity was better than specificity for both caregiver and researcher scores is appropriate for a pain scale designed to supplement clinical judgment and to alert healthcare professionals to the possibility that a child is experiencing moderate to severe pain. Future research should examine whether training or information about the children could improve the sensitivity and specificity of unfamiliar observers.

The NCCPC-PV total scores were also significantly correlated with caregivers’ and researchers’ VAS ratings of the children's pain. Scores from each observer also correlated significantly with the VAS ratings of the other observer, adding to the evidence that familiarity with these children may not be necessary to assess their pain. However, nurses’ VAS ratings did not change significantly from before to after surgery or correlate significantly with caregivers’ and researchers’ VAS ratings or NCCPC-PV scores. One reason for this could be that completing the NCCPC-PV provided caregivers and researchers with training in what to look for and that they used this when making their VAS ratings. Because of time constraints, nurses were not given this opportunity. This coincides with previous research that indicates nurse and researcher pain ratings are more closely related when nurses use a standardized instrument rather than a global assessment. 26This might also be explained by different nurses having rated the pain before and after surgery. However, the regression conducted indicated that only 10% of the variance in nurses’ VAS pain ratings after surgery were related to whether nurses had also completed the preoperative VAS pain rating. Thus, the latter explanation is unlikely.

One other explanation explored in this study is that nurses had a priori  expectations of how the children would show pain based on their experience with children without intellectual disabilities and that these children did not show the reactions they were expecting. Previous studies indicate nurses use factors other than children's behavior to make pain judgments, such as the type of surgery, time since surgery, and time since last analgesic administration. 27Notably, these factors did not predict any pain ratings or analgesic administration in this study. There is also consistent evidence that nurses put a great deal of weight on verbal and vocal signs of pain. 27–28Children in our study could not produce verbal signs, and they showed few vocal signs. According to researchers, only 50% of children displayed any vocal behavior, and according to caregivers only 67% did so. Furthermore, both reported that only 12% of children displayed any vocal behavior “fairly often” or “very often.” Thus, the lower ratings given by these nurses may well be because they were looking for vocal reactions to pain, and few of the children showed these.

It is noteworthy that facial subscale had the highest interrater agreement and that it was also the only subscale to predict nurses’ VAS ratings. In studies of children without impairments, interrater reliability for facial expressions of pain have been relatively low. 8However, studies of adults with dementia have found that facial reaction to pain is greater in those with impairments than those without. 29,30As well, facial expression in response to noxious stimuli has been reported to be greater in children with developmental disorders than in children without. 31Thus, it could be that children with severe intellectual disabilities show more facial reaction to pain than children without impairments, or observers may be more sensitive to facial activity in people who cannot speak. Further research is needed to explore these possibilities.

Scores from the Eating–Sleeping subscale of the NCCPC-PV were not included. It would have been preferable to include these items so that comparisons of scores achieved by children for postoperative pain and pain experienced in other situations could be made. However, our analyses indicate that use of this subscale would have elevated primarily postoperative scores and reduced correlations with VAS ratings. Thus, retaining it could result in a higher false-positive rate and a less valid measure.

This study had limitations. The sample of children observed in this study was small. Thus, analyses of individual items were not possible. However, the NCCPC-PV was designed to be used with total scores. Thus, item analysis is of secondary importance if the validity of total scores are satisfactory. In addition, the fact that statistically significant results were found with such a small sample suggests these results are clinically significant. This external validity is strengthened by the fact that the results were found with a wide variety of operative procedures. In addition, intrarater reliability was not assessed. However, interrater reliability contains all sources of error in intrarater reliability, plus additional error caused by different observers. Thus, intrarater reliability would be greater than the interrater reliability we report. Nevertheless, the small sample size means cutoff scores for inferring the presence of moderate to severe pain should be taken as preliminary. Validation of any pain tool requires repeated tests of validity and reliability across samples, settings, and observers. The determination of intrarater reliability is also helpful in determining which individuals are likely to display good interrater reliability, and evaluation of this characteristic of the NCCPC should be included in future research.

In summary, these results provide the first evidence that the postoperative pain of nonverbal children who have severe intellectual disabilities can be assessed using a formal pain tool. Even more importantly, adults who were unfamiliar with these children were able to detect signs of pain using the NCCPC-PV. The results also indicated facial expressions were used by all observers and may be the most consistent cues shown by the children. Although an evaluation of ease of use in routine clinical care should be undertaken, researchers, who did not know the children, were able to use the NCCPC-PV based on only 10 min of observation. The NCCPC-PV may be useful currently to supplement clinical judgment, although exclusive use for postoperative pain assessment should await replication of these results.

The authors thank the caregivers who took part in the study; Alyson Currie (Study Coordinator, Pediatric Pain Research Laboratory, IWK Health Centre, Halifax, Nova Scotia, Canada) and Jill MacLaren (Honors Student, Department of Psychology, Dalhousie University, Halifax, Nova Scotia, Canada) for help collecting this data; and the nurses who participated and the surgeons and anesthesiologists involved in the care of the children. The commitment of everyone involved in this project to better care for children was clearly seen in their eagerness to see the project succeed.