Chapter 8: Inter-Rater Reliability

Inter-rater reliability refers to the degree of agreement between two raters who are rating the same youth. Estimates of inter-rater reliability help todescribe levels of consistency between raters, typically indexed by a correlation coefficient (in this case, Pearson’s r; LeBreton & Senter, 2008). Twointer-rater studies were conducted with the Conners 4: (1) two raters of the same rater type rated the same youth (i.e., two parents or two teachers), and(2) two raters of different rater types rated the same youth (i.e., parent/teacher, parent/self-report, or teacher/self-report).

Inter-Rater Reliability Study 1. In the first inter-rater study, dyads of parent raters (N = 68) completed the Conners 4 Parent,and teacher dyads (N = 34) completed the Conners 4 Teacher. Parent dyads were comprised of either a mother and father rating the same child (88.2%),or a mother and a non-biological father (11.8%) rating the same child. For teacher dyads, the teachers in each pair were typically providing instruction indifferent class content (e.g., math and literacy, or science and writing). Ratings were obtained from the parent and teacher dyads on two separateadministrations over a 2- to 4-week interval (14 to 30 days). In appendix F, please see Table F.3 for demographic characteristics of the youth being rated and Table F.4 for demographic characteristics of the parent and teacher raters.

Scores on the Conners 4 scales for the paired raters were correlated. The reliability coefficients are Pearson’s correlations, ranging from -1 to 1,with higher values indicating greater consistency or agreement between raters. Correlation coefficients provide us with a statistical measure of the degreeof association between two variables. Although there are several approaches to interpretation, the correlation coefficients (r) are categorized herein asfollows: absolute values lower than .20 are classified as very weak; values of .20 to .39 are considered weak; values of .40 to .59 are moderate; values of.60 to .79 are strong; and absolute values greater than or equal to .80 are very strong (Evans, 1996).

The obtained inter-rater reliability coefficients and those corrected for range variation (Bryant & Gokhale, 1972), as well as the means, medians, and SDs for each dyad are provided in Table 8.15 for the Parent inter-rater study, and Table 8.16 for the Teacher inter-rater study. Results ofthe Parent inter-rater study were indicative of high levels of consistency between raters across all scales, with slightly lower corrected correlations forAnxious Thoughts and Family Life scales (evidence that parents of the same child can still have differing perspectives, i.e., corrected r rangedfrom .58 to .93, all p< .001). In Table 8.16, the inter-rater study results for teacher dyads showed more moderate to strong associations (i.e., corrected r rangedfrom .38 to .69); this pattern of results is a reminder of the value of obtaining ratings from multiple teachers and evaluating multiple sources ofinformation when conducting a comprehensive assessment.

The consistency between raters was further evaluated by calculating the difference between scores for parent or teacher dyads in the inter-ratersamples. Table 8.17 and 8.18 present the percentage of the sample displaying differences in scores of greater than, or equal to, 10 points (i.e., 1 SD or greater), as well as the absolute mean differences (i.e., the difference between ratings from pairs of parents or teachers averaged across thesample). The results suggest high consistency between parent ratings (scores on the Conners 4 scales were less than 1 SD apart for 79.4% to 100.0%of parent dyads) and moderate to high consistency between teacher ratings (scores on the Conners 4 scales were less than 1 SD apart for 61.8% to82.4% of teacher dyads). The absolute mean and SD differences were also close to zero (i.e., on average, there was close to no difference in scoresbetween the raters, and there was similar dispersion in scores), providing further evidence of inter-rater consistency, given the high degree of agreementbetween the same type of raters.

Table 8.15. Inter-Rater Reliability Study 1: Conners 4 Parent

Note. N = 68. Reported p-values are for corrected r. Guidelines for interpreting |r|: very weak < .20, weak = .20 to .39,moderate = .40 to .59, strong = .60 to .79, very strong ≥ .80.

Table 8.16. Inter-Rater Reliability Study1: Conners 4 Teacher

Note. N = 34. Reported p-values are for corrected r. Guidelines for interpreting |r|: very weak < .20, weak = .20 to .39,moderate = .40 to .59, strong = .60 to .79, very strong ≥ .80.

Table 8.17. Difference Between Rater T-scores Study 1: Conners 4 Parent

Note. N = 68. 1 SD is equivalent to 10 T-score points.

Table 8.18. Difference Between Rater T-scores Study 1: Conners 4 Teacher

Note. N = 34. 1 SD is equivalent to 10 T-score points.

Inter-Rater Reliability Study 2. In the second inter-rater study, comparisons were made across different types of raters. As theConners 4 Parent, Teacher, and Self-Report all measure similar constructs, similarity in scores across the different types of raters would provide evidenceof the reliability of the test scores. Although some degree of similarity in ratings is expected between informants, given that they are rating the sameyouth on the same constructs, it is nonetheless expected that there be a certain degree of incongruence between their ratings, because different informantssee the youth in different contexts, and may have dissimilar perceptions of, or experiences with, the youth’s behavior.

For the Conners 4, correlation coefficients (Pearson’s r, LeBreton & Senter, 2008) were calculated between scores for the following pairs ofraters: (1) parent and teacher, (2) parent and self-report, and (3) teacher and self-report. To examine these relationships, parents, teachers, and youthprovided ratings of the same youth (N = 62; all ratings completed within a 30-day period). Refer to appendix F, Table F.4 for the demographic characteristics of the youth being rated (recall that in this studythe same youth is being rated by themselves, a parent, and a teacher), and of the parent and teacher raters.

The obtained correlation coefficients between different rater types, as well as those corrected for range restrictions (Bryant & Gokhale, 1972), areprovided in Table 8.19 (Parent/Teacher), Table 8.20, (Parent/Self-Report), and Table 8.21 (Teacher/Self-Report).

For the Parent/Teacher dyads, the majority of the corrected correlations were moderate in size (median r = .46; ranging from r = .28 to.79), with two (Depressed Mood r = .36, Anxious Thoughts r = .28) being weakly correlated, and one (DSM Conduct Disorder Symptoms) beingstrongly correlated (r = .79). This strong correspondence for the DSM Conduct Disorder Symptoms scale is likely due to infrequent occurrence, asmost raters do not report observing Conduct Disorder symptoms as they are severe behaviors. This similarity between raters in the infrequent endorsement ofbehaviors associated with Conduct Disorder is likely what accounted for the strong correlation between ratings on this scale.

The majority of corrected correlations were weak in the Parent/Self-Report dyads (median r = .30; ranging from r = .19 to .72; DSMConduct Disorder Symptoms r = .72), as well in the Teacher/Self-Report dyads (median r = .23; ranging from r = .16 to .61; DSM Conduct Disorder Symptoms r = .61). Notably, for both the Parent/Self-Report and Teacher/Self-Report dyads,Impulsivity scores were very weakly correlated (r = .16 and r = .11, respectively), as were scores for Peer Interactions (r = .12and r = .16, respectively). This low agreement between raters may be due to a variety of reasons, one of which is setting differences. Youth canreport on their own behaviors across multiple settings, whereas parents and teachers can only report on those behaviors observed in specificenvironments, such as home and school. Another reason for low agreement for Impulsivity may be the youth’s unique insight into their own impulsivebehaviors, especially in adolescence when these behaviors may be more inwardly felt than outwardly expressed, while parents and teachers might be lessaware of the effort needed to restrain such behaviors. Similarly, for Peer Interactions, youth may have different insights into their friendships andexperiences with their peers than what their parents or teachers would have access to observe.

Table 8.19. Inter-RaterReliability Study 2: Conners 4 Parent/Teacher

Note. N = 62. Reported p-values are for corrected r. Guidelines for interpreting |r|: very weak < .20, weak = .20 to .39,moderate = .40 to .59, strong = .60 to .79, very strong ≥ .80.

Table 8.20. Inter-Rater Reliability Study 2: Conners 4 Parent/Self-Report

Note. N = 62. Reported p-values are for corrected r. Guidelines for interpreting |r|: very weak < .20, weak = .20 to .39,moderate = .40 to .59, strong = .60 to .79, very strong ≥ .80.

Table 8.21. Inter-Rater Reliability Study 2:Conners 4 Teacher/Self-Report

Note. N = 62. Reported p-values are for corrected r. Guidelines for interpreting |r|: very weak< .20, weak = .20 to .39, moderate = .40 to .59, strong = .60 to .79, very strong ≥ .80.

In addition, the congruence of scores was analyzed by examining percentages of the samples with similar scores. These results are summarized in Tables8.22 to 8.24.

As seen in Table 8.22, it was a frequent occurrence in the Parent/Teacher sample for there to be congruence in scores; that is, scores of ratings of theyouth often differed by less than 1 SD (ranging from 58.1% to 75.8% of the sample, across all scales). Notably, parents provided slightly higherratings than teachers (as evidenced by the small negative mean differences across all scales). The results suggest that scores on the Conners 4 scales havemoderate consistency across parent and teacher ratings, providing supporting evidence for inter-rater agreement, but also showing the importance of acomprehensive evaluation that engages multiple raters who draw upon different perspectives and experiences with the youth.

A similar pattern was seen in the comparisons between parent and self-reported ratings, as presented in Table 8.23. Parents and youth tended to providelargely similar ratings, yielding scores that are within 1 SD of each other for 56.5% to 87.0% of this sample. Additionally, the differences betweentheir scores were modest, ranging from -0.2 to -4.6 points, with higher ratings coming from the parent raters.

Similarly, for the Teacher/Self-Report comparisons, aligned scores that are within 1 SD of each other were observed for 51.6% to 87.1% of this sample (see Table 8.24), with slightly higher scores coming from Self-Report ratings.

Overall, these findings suggest moderate differences between the ratings of parents, teachers, and youth, with parents typically yielding the highestscores of the three rater types. Parents may have a unique opportunity for insight that teachers may be lacking, such as the youth’s home-life behaviors,while teachers or youth rating themselves have more opportunity for peer comparison, providing a different perspective. These findings demonstrate, whenlooking across different types of raters, moderate inter-rater reliability for the Conners 4 scales, capturing consistency in measurement across raters, aswell as the varying perspectives of raters with differing types of relationships to the youth being rated. Although a high degree of similarity isexpected between raters, perfect agreement is improbable; instead, discrepancies between raters’ scores, as seen in this study, emphasize the importance ofconsulting multiple sources to capture unique information that can reveal nuanced differences. The raters may be drawing upon their experiences with theyouth being rated in dissimilar contexts, which may lead to slightly different responses between rater types. Results serve to highlight theimportance of examining information from multiple sources when conducting a comprehensive assessment.

Table 8.22. Difference Between Rater T-scores Study 2: Conners 4 Parent/Teacher

Note. N = 62. 1 SD is equivalent to 10 T-score points. A positive M difference indicates that the teacher's ratings are higher than the parent's ratings.

Table 8.23. Difference Between Rater T-scores Study 2:Conners 4 Parent/Self-Report

Note. N = 62. 1 SD is equivalent to 10 T-score points. A positive M difference indicates that the youth's self-report ratings are higher than the parent's ratings.

Table 8.24. Difference Between Rater T-scores Study 2:Conners 4 Teacher/Self-Report

Note. N = 62. 1 SD is equivalent to 10 T-score points. A positive M difference indicates that the youth's self-report ratings are higher than the teacher's ratings.

In addition to the differences already described regarding the Conners 4 scales, the Conners 4 Critical & Indicator Items were also evaluated fordifferences among Parent, Teacher, and Self-Report scores using the same inter-rater sample (note that the Severe Conduct Critical Items are not presentedas these items are subsumed in the DSM Conduct Disorder Symptoms scale). The frequency of endorsement of the Critical & Indicator Items was examined foreach rater group, with the expectation that varying perspectives and insight into sensitive topics may yield quite distinct patterns of responses. For theSelf-Harm Critical Items to be considered endorsed, responses had to be greater than or equal to 1, “Just a little true (Occasionally),” such thatany degree of endorsement flagged a Critical Item for inclusion in the frequency count. For the Sleep Problems Indicator, endorsement was defined asresponses greater than 2, with the exception of the appearing/feeling tired item which requires a rating of 3, “Completely true (Very often/Always)”(on Self-Report only), in accordance with how these behaviors are flagged within the Conners 4 reports (see chapter 4, Interpretation).

As seen in Table 8.25, for the Self-Harm Critical Items, there was greater endorsement on the Self-Report relative to Parent and Teacher raters. Mostnotably, the Critical Item about deliberate self-harm is endorsed more than twice as often in Self-Report ratings (19.4%) relative to Parent ratings(8.1%). These items reflect deeply sensitive and personal behaviors or attitudes, to the extent that a youth may not wish to disclose this information to aparent or teacher. The discrepancies between the reports of different rater types, specifically when inquiring about self-harm thoughts and behaviors,highlight the importance of collecting information from multiple informants.

Table 8.25. Endorsement of Critical Items &Indicator Items by Rater Type

Note. Self-Harm Critical Items are considered endorsed with an item response of ≥ 1, while Sleep Problems Indicator items are considered endorsed with an item response of ≥ 2 for trouble sleeping, an itemresponse of ≥ 2 for tiredness for Parent and Teacher, and an item response of 3 for tiredness for Self-Report.

ncG1vNJzZmilmKiwpbqNm6Ooml6YvLOxjbCgp5yfrMBvusStZqago2LEpq6Ohn%2BMZYekv6Wc0Z6qrGd8mq6zuo2Gf4xnfZa7tq3LrGacp56jsrO%2Fvm2Woaydoayurc2umKWXlqq5rXvPmqmtbF%2BYtXmryKernqqPp661sdFnn62lnA%3D%3D

 Share!

• Share on Facebook
   
• Tweet
   
• Share on Google+
   
• Post to Tumblr
   
• Pin it
   
• Submit to Reddit