THE AUTHORITARIAN
PERSONALITY
TABLE 10 (XXII)
THE AMOUNT OF AGREEMENT BETWEEN A SINGLE RATER (A) AND SEVEN OTHER RATERS IN ESTIMATING VARIABLES IN INI'AKE INI'ERVIEWS
PSYCHIATRIC CLINIC PATIENTS: MEN AND WOMEN COMBINED (N = 59)
Variable
I.
THE AMOUNT OF AGREEMENT BETWEEN A SINGLE RATER (A) AND SEVEN OTHER RATERS IN ESTIMATING VARIABLES IN INI'AKE INI'ERVIEWS
PSYCHIATRIC CLINIC PATIENTS: MEN AND WOMEN COMBINED (N = 59)
Variable
I.
Adorno-T-Authoritarian-Personality-Harper-Bros-1950
At least two judges who are not acquainted with the subjects (and in this case did not know which were high and which were low scorers)
independently rate each subject on certain characteristics. These ratings can then be compared with other data on the subjects. The problem of blind ratings has been discussed in other chapters dealing with interview scoring and projective questions. If these ratings turn out to be highly correlated with some trait of the subjects (in this case ethnocentrism) on which the judges had no information, it is very probable that relationships between the former and latter traits exist in reality and could be demonstrated by other methods. (This holds only if the rating procedure is so controlled as to pre- vent the raters from utilizing cues other than those to be experimentally tested. ) There are, however, various possible pitfalls inherent in the rating method. Such "errors of measurement" could arise, for instance, from material which did not contain sufficient information from which to judge the subject on a certain trait. Other errors might arise from the manner in which the characteristics were defined and described for the judges; from the ability of the judges to apply the instructions to the material to be judged -depending in turn on the judges' training, theoretical bias, and personality- and from other factors. Use of a rating technique, therefore, requires not only the determination of certain relationships between the ratings and other data, but also a careful consideration of various factors which could have influenced the results either in the positive or in the negative direction.
a. THE RATERS. The two primary raters were staff members of the major study and will be referred to as raters (or judges) A and B. In addition to much clinical training and experience, these two raters had a strong psycho-
? PSYCHOLOGICAL ILL HEALTH AND POTENTIAL FASCISM 925
analytic orientation. Both were familiar with all concepts, hypotheses, data,
and results of the total study. Both had had opportunity to interview high-
and low-scoring subjects, and were therefore acquainted with the behavior
I
and material usually obtained from such subjects in interview situations. Each of the judges independently rated each record on all seven cate- gories and then assigned an "over-all" intuitive estimate of "highness" or
"lowness. "
After these primary ratings had been completed, 7 independent raters
(they are referred to hereafter as the control raters, their ratings as control ratings) were used, each rating one category only. 8 Six of the control raters were clinical psychologists (of these, r was a senior clinician, 5 junior clini- cians at the level of internes, working at the Langley Porter Clinic). They were not acquainted with the concepts and data of the over-all study. This was important for reasons to be discussed below. One of the more narrowly defined categories (Onset and cause of illness are explained by subject in externalized terms) was rated by our office secretary who had had no formal training in psychology or psychiatry, but who had much intuitive psycho- logical insight and who had absorbed a great deal of the research material and hypotheses. The 7 judges varied greatly in age, training, and theoretical orientation.
b. THE RATING TECHNIQUE. The instructions for the control raters were as follows:
The material to be rated consists of "Statements of Complaint" by Langley Porter Clinic patients in a first intake interview or in the first interview with a therapist. Only the section "patient's story" or "chief complaint" was included. The inter- views are here reproduced verbatim, although a few have been slightly condensed by the writer. Each numbered paragraph refers to one case. There are 26 men, 33 women.
Each case is to be rated on one variable (or syndrome) as described in the manual. Each rater will be assigned one variable and will not know about other variables until he has completed his ratings.
The ratings are to be made in terms of presence (v) or absence (-) of the trait. A few of the variables permit of a "mixed" judgment. Assignment of such a "mixed" (M) rating should be avoided if possible. But occasionally it may have to be used. Sometimes (due to the fragmentary way in which some of these interviews are
recorded) there will be insufficient material to rate. In this case mark (o).
Each primary rater (A and B) first rated each record in terms of all seven individual categories. They knew which categories were expected to be related to high or to low ethnocentrism. They therefore tried to assign each record a "high" or "low" rating for each category. Often a record did not contain enough material to permit the rater to reach a decision on a given variable, e. g. , a given topic was not discussed, or there were few cues permit-
s W e wish to thank Dorothy Bomberg, Janet Gist, Carole Home, Virginia Patterson, Dr. Claire W. Thompson, Anne Vollmar, and Elaine Wesley Barron for the patience and care with which they carried out, on short notice, the task of doing the control ratings.
? THE AUTHORIT ARIAN PERSONALITY
ting inferences regarding character structure. In this case no rating was assigned. When there were about an equal number of cues pointing in the high and in the low directions, a "mixed" rating was assigned. The raters then went over the records a second time, trying to guess in each case whether the subject had made a high or low score on theE scale. The guesses were to be based on the decisions reached regarding the individual categories. However, no mechanical formula was set up to convert the individual ratings into "over-all" ratings. The raters arrived at the latter by a new rating process in which any or all of the categories could be used and weighted as the rater saw fit.
Two types of data were obtained from this rating material: (I) Inter- rater scoring agreement for each category and for the over-all ratings. (2) Relationship between ratings and scores on the E scale. These will now be discussed.
4. T H E RELIABILITY OF T H E MEASURES
When several persons agree considerably more than half of the time that certain subjects in a group do, others do not, possess a given trait, the chances are good that these various raters knew what they were supposed to look for, had a similar conception of the trait, understood this concept, and could clearly recognize something in the interview data to which this concept could be applied; and that personality, training, and other differences be- tween the raters influenced the ratings only to a relatively small degree.
All rating notations (high, low, presence, absence, omission, mixed) were converted into "high," "low," and "neutral" scores. E. g. , a rating of "pres- ence" on variable I-Main emphasis on subject's physical complaints-was- considered a "high" score, "absence" a "low" score; "mixed" notations and omissions were considered "neutral" scores.
Scoring reliability was then obtained by computing the percentage of times 2 raters had assigned the same scores to the same records. Whenever both raters had assigned exactly the same score (high, low, or neutral) to the same record, this was considered one agreement. When one of the raters had given either a high or a low, the other a neutral score, this was considered one-half an agreement. When one rater gave a high score, the other a low, this was counted a full disagreement. The number of agreements, divided by the total number of records rated, yielded the percentage agreement between 2 raters. There were very few instances in which both judges gave a neutral score.
Table 9(XXII) shows the percentage agreements between Raters A and B as well as the scores on which these figures are based. All of the percentage agreements, except one (category III, "lows"), are above 70, statistically higher than could have been obtained by chance (I per cent level).
Raters A and B agreed best, 9I per cent, on variable V (Subject spon-
? Over-all rating of highness or lowness on E Scale
Single Variables:
I. Main emphasis on somatic
complaints
II. Intraception
III. Ego-alienness
Low 4. . 3 High 1551 .
Low
51 82. 1 86. 4 90. 3
45 62. 1 76. 3. 71. 0
48 78. 6 83. 1 83. 9
41. 5 76. 8 70. 3 64. 5
45 75. 0 76. 3 n. 4
1 .
13 1 28. 5 91. 9
IV. Externalized theory of onset High and causes Low
13 83 4 26
Low322 1
High 15'72 1 Low 1 4 3 1
-
2 2
25 914
. 3 184
2 10 10
. 2 9 8
23
22
22 26
21. 5 20
21 24
High 14111 Low321 5
.
. . . 2
'7 2 2 2 . 2
VII. Predominant type of symptoms High 18
Low 4 - 1 4
v. Spontaneous mention o t IJI! haPPY High
childhood and family relations ! LOW 15
VI. Cues regarding character High
structure Low 3
2
25 53. 5 89. 3 90. 7
23 2
10
2 20
520 46 '71. 4 '78. 0 4 26 83. 9
TABLE 9 (XXII)
DIE AMQUNT OF AGJ! EE)IENT BETWEEII TWO RATmS IN ESTIMATING A SUBJECT' S STANDING ON DIE E SCALE FROM AN ANALYSIS OF HIS INTAKE INTERVIEW
PSYCHIATRIC CLINIC PATIENTS: MEN AND WOMEN COMBINED (N = 59; High scorers: N = 28; Low Scorers: N = 31)
No. scored High (H) No. scored Neutral (N No. scored Low (L)
Interrater Agreementa
No. of Agreements Percent Agreements
Actual
Standing (! )Rated (2)Rated H (3)Rated N (4)Rated H (5)Rated L (6)Rated L ('! )High (8)H and L (9)High (IO)H and L
onE H by by One; N by Both by One;\ L by Both by One: N or Combined or Combined
Scale Both by Other bY Other High 21 - - 5
by Other Low 2 . 23
413
1
Mean agreement on the variables High 15. 4 6. 3 1. 4 1. 4 0. 9 2. 6 22. 2 Low 4. 8 I. 7 1. 9 2. 4 14. 0 6. 3 24. 7
! ! Number of agreements for a given variable is the sum of the values, opposite that variable, in Columns 1. 3, and 5.
24 . 28
1 22. 5 19 3 25. 5
48 80. 3 81. 4 82. 3
'79. 3 79. 5 79. 7
? THE AUTHORITARIAN PERSONALITY
taneously and explicitly mentions unhappy childhood and family relation- ships). This result was to be expected because this variable is so unambiguous and requires little subjective evaluation. Besides, most subjects did not bring up this subject when first asked about their symptoms.
The next highest agreement was reached on the "over-all" guess regarding the subject's standing on ethnocentrism; then came Intraception, Types of Symptoms, and Character Traits, with agreements around So per cent.
The average agreement for all seven variables was So per cent.
There are several possible reasons why the over-all rating had so high a reliability. One reason is that the instructions prohibited "neutral" ratings in this instance. Another is that the category "over-all" highness-lowness, is a broad one, and the raters are thus given the opportunity to utilize a great variety of explicit or nonexplicit cues and impressions; that they should do this was favored by the fact that both A and B had had experience inter- viewing high and low scorers, possibly developing thereby a "feeling for" a general "high" or "low" personality factor.
Analysis of the ratings assigned by the two judges showed that rater B had a relatively greater number of omissions (meaning "I can't tell from the data given") whereas rater A had tried hard to come to a decision, even when the data offered only one subtle cue. As can be seen from Table 9(XXII), what lowered the agreements between A and B were usually instances in which one rater gave a neutral score; there were very few cases in which one gave a high, the other a low rating.
Because the ratings of A and B were so similar, and because of B's con- sistently greater number of neutrals, which lowered all reliability (and validity) figures somewhat, only rater A's ratings were compared with those made by the 7 control raters.
Table w(XXII) shows the percentage agreements between the ratings by
A and those made by the 7 control judges, each of whom rated only one category. As the control raters made no "over-all" guesses of highness or lowness on E, no agreement with A's over-all rating could be obtained. In- stead, a composite "high" or "low" score for each subject was derived from the 7 control ratings of single variables. A rating of "high" on a given variable was counted as one point, a "neutral" rating was given ? 2 point, a "low" ? rating, o points. By adding the points for each subject, scores ranging from o
to 7 were obtained. All subjects receiving such a composite score of 4 or more were then classed as "high," those with scores below 4 as "low. " The agreement between A's "over-all" rating with these composite ratings is shown near the bottom of Table w(XXII).
The composite high-low score agreed with A's over-all estimate of high- ness or lowness S5 per cent of the time. Practically the same figure was obtained when comparing A's and B's over-all guesses.
The average agreement between A and the control raters for the single
? PSYCHOLOGICAL ILL HEALTH AND POTENTIAL FASCISM 929
variables was only very slightly lower (77 per cent) than the average agree- ment between A and B (So per cent). On the whole, the reliabilities obtained were quite acceptable.
In general, those variables which had the highest reliability when the ratings of A and B were compared also had a relatively high percentage agreement in the comparison between A and the control raters, and conversely with the variables of relatively low reliability. The two categories in which this relationship was almost reversed were: categories I (Emphasis on Physical Symptoms) and II (Intraception). The reliability of the former ranked sec- ond best in the case of A and the control raters, about fifth in the case of A and B. The reliability of the latter ranked sixth with A and the controls, second best with A and B.
The relatively low agreement between A and B on category I was caused not so much by disagreements but by a relatively large proportion of "neu- tral" scores (omissions) on the part of B (see Table 9(XXII)). While the control raters and A felt able to judge the presence or absence of certain cues in the record, rater B frequently felt that the subjects' attitudes toward their symptoms were not sufficiently brought out in the interviews.
The rela"tively low agreement between the control rater and A on Intra- ception was due mainly to a larger number of disagreements. The control rater was in this case particularly dissatisfied with her ratings, feeling that she did not have a sufficient grasp of the concept of intraception nor enough acquaintance with the cues by which the trait could be recognized. In the case of A and B the concept and manifestations of intraception had been made the object of special study and played an important part in their thinking. Rater A thought that her ratings of any given case-on over-all highness- lowness and on all other variables-had been more influenced by her impres- sion of the subject's intraceptiveness than by any other cue. It is reasonable to assume that this difference in training is the cause of the difference in reli- ability between the two sets of raters. This becomes even more probable when the reliability figures are compared with the agreements between rat- ings and actual E score. Here, A's and B's ratings of presence or absence of intraception were related to high and lowE score (in the expected direction) So to S3 per cent of the time, while the control rater's judgment agreed with E only 65 per cent of the time (see Table 12 (XXII)).
The highest reliabilities (91 per cent and 95 per cent) were obtained for variable V (Patient Mentions Unhappy Childhood, etc. ). It is clear that there is not much room for disagreement here.
The lowest reliabilities were obtained for variable III (Ego-alienness). Here the agreement between A and the control rater was only 65 per cent, t)lat between A and B 70 per cent. The former figure barely meets the stand- ard of acceptable reliability of measurement. There was an unusually large discrepancy between A's and the control rater's estimate for the high scorers,
? 93?
THE AUTHORITARIAN PERSONALITY TABLE 10 (XXII)
THE AMOUNT OF AGREEMENT BETWEEN A SINGLE RATER (A) AND SEVEN OTHER RATERS IN ESTIMATING VARIABLES IN INI'AKE INI'ERVIEWS
PSYCHIATRIC CLINIC PATIENTS: MEN AND WOMEN COMBINED (N = 59)
Variable
I. Main emphasis on somatic complaints
II. Intraception
III. Ego-alienness
IV. Externalized theory of onset and causes
v. Spontaneous mention of unhappy childhood and family relations
Hby N by
4 1
1 2
4 2
One; Other
VI. CUes regarding charac- High ter structure Low
Agreement between A's over-
all H-L rating and composite
score based on ratings of 7
independent raters, each High 21 rating a single variable LOW 5
VII. Predominant type of High
22 --
Actual Standing Rated
on E Hby Scale Both
High 12 Low 4
High 18 Low 4
High 9 Low 4
Rated
Rated Rated Nby Hby One; Both L by Other
- 2 1 4
2 6 1 8
1 10 2 5
1 4 4 2
- 1 - 1
1 4 1 2
High 14 6 Low 4 4
High 26 - LOW 17 1
3 symptoms Low41-7
Averages High 16. 0
2. 9
11 5 3 2
0. 7 ? Low 5. 7 1. 9 1. 3 ?
5 4
? Rated Rated
No. of Agreements
T otal Percentage Agreement
81. 4
72. 0
65. 3
73. 7
L by L by Both N by
6 4 20 1
2 2 15 3
1 2 10 7
1 - 11 1
2 5 16 7
12 181
2 22
Percentage Agreement
18 + 4 = 22
25 + 1 = 26 83. 9
-
20 + 1 = 21 42. 5
20+1. 5=21. 5 69. 4
1 15 1
75. 0
1. 9 2. 3 15. 0 3. 0
23 27
21. 1 24. 5
50
45. 5
82. 1 84. 7 87. 1
75. 5 77. 2 78. 8
PSYCHOLOGICAL ILL HEALTH AND POTENTIAL FASCISM 93 I
PSYCHIATRIC CLINIC PATIENTS: MEN AND WOMEN COMBINED (N =59)
One Other
H and L Respectively
H and L Combined
12 + 3 = 15 38. 5
21+2. 5=23. 5 75. 8
16+4=20 43. 5 18+5. 5=23. 5
71. 4 75. 8
27 29
56 96. 4 94. 9 93. 5
48 78. 6
14+5=19 43. 5 67. 9 73. 7 20+4. 5=24. 5 79. 0
244785. 779. 7 23 74. 2
53. 6
? 932
THE AUTHORITARIAN PERSONALITY
there being only 54 per cent agreement. This figure is the result of a rather large number of disagreements between the two raters. The control rater said that she was quite unsure of her ratings because she felt that the inter- view material reflected more the physician's than the patient's attitude toward the complaints. Also, she conceived of "ego-alienness" as something unfavor- able from the mental hygiene point of view and was therefore more often willing to score "absence" of the trait than "presence. " Possibly the most important factor, which lowered both sets of reliabilities, is the relative broadness with which this variable was defined, including few behavioral criteria but leaving much to the rater's intuition.
With the exception of variable III, the ratings for all variables, as well as the over-all estimates, proved to be reliable measures.
5. RELA TIONSHIP BETWEEN RA TINGS AND ETHNOCENTRISM SCORE
The idea of the present rating method is the following: If the "blind" ratings of a given trait correlate significantly with another trait about which the raters had no information (in this case ethnocentrism), it is likely that the two traits are correlated in reality. The relationships to be described below are probably valid because we have eliminated many of the factors inherent in the rating process which could have influenced the results.
There are, however, many factors inherent in the rating method which, can still influence the "blind" ratings in such a way that statistical relation- ships of two traits are artificially raised or lowered. The possibility of arti- ficially high correlations is present particularly when several traits are rated at the same time. Here, the rating of one trait can easily influence the ratings of other traits. This factor (called halo effect) could have influenced the ratings of judges A and B. It was primarily for this reason that control raters, each of whom was trained to pay attention to and to recognize only one of
. the seven single traits, were used.
The ratings of (I) over-all highness and lowness, and (2) of the single
traits, were related to ethnocentrism in the following manner: For a given trait, we counted the number of subjects in the high E quartile who were judged to be "high" by the raters. To these were added half of the high-scor- ing subjects receiving a "neutral" rating. This sum, divided by the total num- ber of high scorers in the group, yielded the percentage of high scorers rated in the high direction. By the same procedure the percentage of low scorers rated in the low direction was obtained. The weighted average of these two percentages was then computed; this indicated the percentage of subjects related "correctly," that is, the percentage whose ratings were re- lated to E score in the expected way.
a. THE OvER-ALL RATINGs. Table I I (XXII) gives the agreements between the subject's standing on E and A's ratings. Rater A's guesses as to highness and lowness agreed with the subject's actual E score 86 per cent of the
? Variable
I. Main emphasis on somatic complaints
II. Intraception III. Ego-alienness
IV. Externalized theory of onset and causes
v. Spontaneous mention of unhappy childhood and family relations
ACtual Standing on E Scale
High Low
High
Low
High
Low
High Low
High Low
Rated Rated H L
Rated N
No. of Agreements
18-1/2 24-1/2
25 24
24-1/2 21
20-1/2 19
27 13-1/2
22-1/2 . 24:. . 112
No. o f Agreements Combined
43
49 45-1/2 39-1/2
40-1/2
47
Percentage Agreement
66. 1 'T9:lY
89. 3 77. 4
87. 5 67. 7
73. 2 61. 3
96. 4 43. 5
80. 4 7 9 . 0
T o t a l Percentage Agreement
72. 9 83. 1 77. 1 66. 9
68. 6 79. 7 84. 7 86. 4
VI. Cues regarding charac- High
13
2 23
1 23
3 26
1
7 3
2 2
0 0
ter structure
VII. Predominant type of symptoms
OV er-all A verages
Low
Hhh Low
High Low
High Low
26 50 92. 9 24 77. 4
25 51 89. 3
TABLE 11 (XXII)
THE AMOUNI' OF AGREEMENT BETWEEN RATER A' S ESTIMATE OF HIGH OR LOW ETHNOCENI'RISM, BASED ON ANALYSIS OF INTAKE INTERVIEWS, AND ETHNOCENTRISM AS MEASURED BY
THEE SCALE
PSYCHIATRIC CLINIC PATIENTS: MEN AND WOMEN COMBINED (N =59)
1567 5233
2422 5224
2323 9202
17 7
27 17
19 5
25 6
25 5
47 14 10
10
26
lJ3. 9
83. 7
09. ~
? I.
II. III. IV. V.
VI.
VII.
Main emphasis on somatic complaints
I n t r a c e p t i o n
E g o ? a l i e n n e s s
E x t e r n a l i z e d
of onset and causes
Spontaneous mention
of unhappy childhood and family relations
C u e s r e g a r d i n g character structure
Predominant type of symptoms
Means
TABLE 12 (XXII)
THE AMOUNT OF AGREEMENT BETWEEN ESTIMATES OF ETHNOCENTRISM, BASED ON RATINGS OF SINGLE VARIABLES FROM INTAKE INTERVIEWS, AND ETHNOCENTRISM AS MEASURED BY THE E SCALE
Variable Composite Ratinga Single V ariables:
Psychiatric Clinic Patients: Men And Women Combined (N =59)
Actual No. No. No. No. of Total Standing Rated Rated Rated No. of Agreements Percentage Percentage
on E Scale H L N Agreements Combined Agreement Agreement fH~i~g~h________~2~2~----~6~----*o________~2~2~. *o________~4~4~-~o________~7~1~-~o________~7~4~-~6---
Low 9 22 0 22. 0 78. 6
~H~i~g~h________~1~5~----~1~2r-----T1________~1~5~-~5________~3~8~? ~o________~5~5~. 4~------~6~4~-~0--- Low 8 22 1 22. 5 72. 6
~H~i~g~h~------~1~9~----~5~----~4~------~2~1r. ~o~------~3~8~-~5~------~7~5~-~o~------~6~5~-~3_ _ _ Low 13 17 1 17. 5 56. 5
;H~i~g~h~------~9~-----T14~----~5~------~1~1~. ? 5_ _ _ _ _ _ _ _ ~3~3~-~o~------~4~lr. ~1_ _ _ _ _ _ _ _ ~5~5~-~9_ _ _ _ Low 6 18 7 21. 5 69. 4
+H~i~g~h--------~2~1~-----i4~----*3_ _ _ _ _ _ _ _ ~2~2~. *5_ _ _ _ _ _ _ _ 24~2~-~o_ _ _ _ _ _ _ _ ~8~0~? ~4_ _ _ _ _ _ _ _ ~7~1~? ~2--- Low 7 15 9 19. 5 62. 9
~H~i~gh~-------72*6______~2~----0~-------72T6~. 0~------~37~. 5~------~9;2~. 9~------~6~3~. 6~-- Low 19 11 1 11. 5 37. 1
~H~i~gh~_ _ _ _ _ _ _ ; 1 ; 2 _ _ _ _ _ _ ~1~1~--~5~------~14~. 5~------~3~6~. 5~------~5T1~. 8~------~6~1~. 9~--
t h e o r y
Low
High Low
High Low
5 18 8 22. 0
22 6 0 22. 0 10 21 0 21. 0
17. 71 7. 7 2. 6 19. 0 9. 71 17. 4 3. 9 19. 4
43. 0
38. 4
71. 0
78. 6 72. 9 67. 7
67. 9 65. 0 62. 5
acomposite rating based on ratings by 7 independent raters on the -variables listed. A "High rating" received point, a "Neutral rating? ~ point; 4 points or more constituted a composite "High" score; everything below 4 a ? ? Low" score.
? PSYCHOLOGICAL ILL HEALTH AND POTENTIAL FASCISM 935
time. For the high scorers alone the agreement was 89 per cent, for the low scorers 84 per cent. These figures show again how closely ethnocentrism is related to personality factors, although the relationships are by no means perfect.
The remaining problem now is: How did raters A and B arrive at their rather accurate guesses about ethnocentrism from a short paragraph of inter- view material dealing almost exclusively with the subjects' complaints? Did they base their guesses mainly on the variables described in the manual, or did they inadvertently use other cues, such as type of language used and other cues that have not been made explicit but which they learned to associate with highness or lowness while interviewing subjects with known ethno- centrism scores?
The results obtained from the control raters, who had had no such previous experience with high- and low-scoring subjects, should help to decide this question.
independently rate each subject on certain characteristics. These ratings can then be compared with other data on the subjects. The problem of blind ratings has been discussed in other chapters dealing with interview scoring and projective questions. If these ratings turn out to be highly correlated with some trait of the subjects (in this case ethnocentrism) on which the judges had no information, it is very probable that relationships between the former and latter traits exist in reality and could be demonstrated by other methods. (This holds only if the rating procedure is so controlled as to pre- vent the raters from utilizing cues other than those to be experimentally tested. ) There are, however, various possible pitfalls inherent in the rating method. Such "errors of measurement" could arise, for instance, from material which did not contain sufficient information from which to judge the subject on a certain trait. Other errors might arise from the manner in which the characteristics were defined and described for the judges; from the ability of the judges to apply the instructions to the material to be judged -depending in turn on the judges' training, theoretical bias, and personality- and from other factors. Use of a rating technique, therefore, requires not only the determination of certain relationships between the ratings and other data, but also a careful consideration of various factors which could have influenced the results either in the positive or in the negative direction.
a. THE RATERS. The two primary raters were staff members of the major study and will be referred to as raters (or judges) A and B. In addition to much clinical training and experience, these two raters had a strong psycho-
? PSYCHOLOGICAL ILL HEALTH AND POTENTIAL FASCISM 925
analytic orientation. Both were familiar with all concepts, hypotheses, data,
and results of the total study. Both had had opportunity to interview high-
and low-scoring subjects, and were therefore acquainted with the behavior
I
and material usually obtained from such subjects in interview situations. Each of the judges independently rated each record on all seven cate- gories and then assigned an "over-all" intuitive estimate of "highness" or
"lowness. "
After these primary ratings had been completed, 7 independent raters
(they are referred to hereafter as the control raters, their ratings as control ratings) were used, each rating one category only. 8 Six of the control raters were clinical psychologists (of these, r was a senior clinician, 5 junior clini- cians at the level of internes, working at the Langley Porter Clinic). They were not acquainted with the concepts and data of the over-all study. This was important for reasons to be discussed below. One of the more narrowly defined categories (Onset and cause of illness are explained by subject in externalized terms) was rated by our office secretary who had had no formal training in psychology or psychiatry, but who had much intuitive psycho- logical insight and who had absorbed a great deal of the research material and hypotheses. The 7 judges varied greatly in age, training, and theoretical orientation.
b. THE RATING TECHNIQUE. The instructions for the control raters were as follows:
The material to be rated consists of "Statements of Complaint" by Langley Porter Clinic patients in a first intake interview or in the first interview with a therapist. Only the section "patient's story" or "chief complaint" was included. The inter- views are here reproduced verbatim, although a few have been slightly condensed by the writer. Each numbered paragraph refers to one case. There are 26 men, 33 women.
Each case is to be rated on one variable (or syndrome) as described in the manual. Each rater will be assigned one variable and will not know about other variables until he has completed his ratings.
The ratings are to be made in terms of presence (v) or absence (-) of the trait. A few of the variables permit of a "mixed" judgment. Assignment of such a "mixed" (M) rating should be avoided if possible. But occasionally it may have to be used. Sometimes (due to the fragmentary way in which some of these interviews are
recorded) there will be insufficient material to rate. In this case mark (o).
Each primary rater (A and B) first rated each record in terms of all seven individual categories. They knew which categories were expected to be related to high or to low ethnocentrism. They therefore tried to assign each record a "high" or "low" rating for each category. Often a record did not contain enough material to permit the rater to reach a decision on a given variable, e. g. , a given topic was not discussed, or there were few cues permit-
s W e wish to thank Dorothy Bomberg, Janet Gist, Carole Home, Virginia Patterson, Dr. Claire W. Thompson, Anne Vollmar, and Elaine Wesley Barron for the patience and care with which they carried out, on short notice, the task of doing the control ratings.
? THE AUTHORIT ARIAN PERSONALITY
ting inferences regarding character structure. In this case no rating was assigned. When there were about an equal number of cues pointing in the high and in the low directions, a "mixed" rating was assigned. The raters then went over the records a second time, trying to guess in each case whether the subject had made a high or low score on theE scale. The guesses were to be based on the decisions reached regarding the individual categories. However, no mechanical formula was set up to convert the individual ratings into "over-all" ratings. The raters arrived at the latter by a new rating process in which any or all of the categories could be used and weighted as the rater saw fit.
Two types of data were obtained from this rating material: (I) Inter- rater scoring agreement for each category and for the over-all ratings. (2) Relationship between ratings and scores on the E scale. These will now be discussed.
4. T H E RELIABILITY OF T H E MEASURES
When several persons agree considerably more than half of the time that certain subjects in a group do, others do not, possess a given trait, the chances are good that these various raters knew what they were supposed to look for, had a similar conception of the trait, understood this concept, and could clearly recognize something in the interview data to which this concept could be applied; and that personality, training, and other differences be- tween the raters influenced the ratings only to a relatively small degree.
All rating notations (high, low, presence, absence, omission, mixed) were converted into "high," "low," and "neutral" scores. E. g. , a rating of "pres- ence" on variable I-Main emphasis on subject's physical complaints-was- considered a "high" score, "absence" a "low" score; "mixed" notations and omissions were considered "neutral" scores.
Scoring reliability was then obtained by computing the percentage of times 2 raters had assigned the same scores to the same records. Whenever both raters had assigned exactly the same score (high, low, or neutral) to the same record, this was considered one agreement. When one of the raters had given either a high or a low, the other a neutral score, this was considered one-half an agreement. When one rater gave a high score, the other a low, this was counted a full disagreement. The number of agreements, divided by the total number of records rated, yielded the percentage agreement between 2 raters. There were very few instances in which both judges gave a neutral score.
Table 9(XXII) shows the percentage agreements between Raters A and B as well as the scores on which these figures are based. All of the percentage agreements, except one (category III, "lows"), are above 70, statistically higher than could have been obtained by chance (I per cent level).
Raters A and B agreed best, 9I per cent, on variable V (Subject spon-
? Over-all rating of highness or lowness on E Scale
Single Variables:
I. Main emphasis on somatic
complaints
II. Intraception
III. Ego-alienness
Low 4. . 3 High 1551 .
Low
51 82. 1 86. 4 90. 3
45 62. 1 76. 3. 71. 0
48 78. 6 83. 1 83. 9
41. 5 76. 8 70. 3 64. 5
45 75. 0 76. 3 n. 4
1 .
13 1 28. 5 91. 9
IV. Externalized theory of onset High and causes Low
13 83 4 26
Low322 1
High 15'72 1 Low 1 4 3 1
-
2 2
25 914
. 3 184
2 10 10
. 2 9 8
23
22
22 26
21. 5 20
21 24
High 14111 Low321 5
.
. . . 2
'7 2 2 2 . 2
VII. Predominant type of symptoms High 18
Low 4 - 1 4
v. Spontaneous mention o t IJI! haPPY High
childhood and family relations ! LOW 15
VI. Cues regarding character High
structure Low 3
2
25 53. 5 89. 3 90. 7
23 2
10
2 20
520 46 '71. 4 '78. 0 4 26 83. 9
TABLE 9 (XXII)
DIE AMQUNT OF AGJ! EE)IENT BETWEEII TWO RATmS IN ESTIMATING A SUBJECT' S STANDING ON DIE E SCALE FROM AN ANALYSIS OF HIS INTAKE INTERVIEW
PSYCHIATRIC CLINIC PATIENTS: MEN AND WOMEN COMBINED (N = 59; High scorers: N = 28; Low Scorers: N = 31)
No. scored High (H) No. scored Neutral (N No. scored Low (L)
Interrater Agreementa
No. of Agreements Percent Agreements
Actual
Standing (! )Rated (2)Rated H (3)Rated N (4)Rated H (5)Rated L (6)Rated L ('! )High (8)H and L (9)High (IO)H and L
onE H by by One; N by Both by One;\ L by Both by One: N or Combined or Combined
Scale Both by Other bY Other High 21 - - 5
by Other Low 2 . 23
413
1
Mean agreement on the variables High 15. 4 6. 3 1. 4 1. 4 0. 9 2. 6 22. 2 Low 4. 8 I. 7 1. 9 2. 4 14. 0 6. 3 24. 7
! ! Number of agreements for a given variable is the sum of the values, opposite that variable, in Columns 1. 3, and 5.
24 . 28
1 22. 5 19 3 25. 5
48 80. 3 81. 4 82. 3
'79. 3 79. 5 79. 7
? THE AUTHORITARIAN PERSONALITY
taneously and explicitly mentions unhappy childhood and family relation- ships). This result was to be expected because this variable is so unambiguous and requires little subjective evaluation. Besides, most subjects did not bring up this subject when first asked about their symptoms.
The next highest agreement was reached on the "over-all" guess regarding the subject's standing on ethnocentrism; then came Intraception, Types of Symptoms, and Character Traits, with agreements around So per cent.
The average agreement for all seven variables was So per cent.
There are several possible reasons why the over-all rating had so high a reliability. One reason is that the instructions prohibited "neutral" ratings in this instance. Another is that the category "over-all" highness-lowness, is a broad one, and the raters are thus given the opportunity to utilize a great variety of explicit or nonexplicit cues and impressions; that they should do this was favored by the fact that both A and B had had experience inter- viewing high and low scorers, possibly developing thereby a "feeling for" a general "high" or "low" personality factor.
Analysis of the ratings assigned by the two judges showed that rater B had a relatively greater number of omissions (meaning "I can't tell from the data given") whereas rater A had tried hard to come to a decision, even when the data offered only one subtle cue. As can be seen from Table 9(XXII), what lowered the agreements between A and B were usually instances in which one rater gave a neutral score; there were very few cases in which one gave a high, the other a low rating.
Because the ratings of A and B were so similar, and because of B's con- sistently greater number of neutrals, which lowered all reliability (and validity) figures somewhat, only rater A's ratings were compared with those made by the 7 control raters.
Table w(XXII) shows the percentage agreements between the ratings by
A and those made by the 7 control judges, each of whom rated only one category. As the control raters made no "over-all" guesses of highness or lowness on E, no agreement with A's over-all rating could be obtained. In- stead, a composite "high" or "low" score for each subject was derived from the 7 control ratings of single variables. A rating of "high" on a given variable was counted as one point, a "neutral" rating was given ? 2 point, a "low" ? rating, o points. By adding the points for each subject, scores ranging from o
to 7 were obtained. All subjects receiving such a composite score of 4 or more were then classed as "high," those with scores below 4 as "low. " The agreement between A's "over-all" rating with these composite ratings is shown near the bottom of Table w(XXII).
The composite high-low score agreed with A's over-all estimate of high- ness or lowness S5 per cent of the time. Practically the same figure was obtained when comparing A's and B's over-all guesses.
The average agreement between A and the control raters for the single
? PSYCHOLOGICAL ILL HEALTH AND POTENTIAL FASCISM 929
variables was only very slightly lower (77 per cent) than the average agree- ment between A and B (So per cent). On the whole, the reliabilities obtained were quite acceptable.
In general, those variables which had the highest reliability when the ratings of A and B were compared also had a relatively high percentage agreement in the comparison between A and the control raters, and conversely with the variables of relatively low reliability. The two categories in which this relationship was almost reversed were: categories I (Emphasis on Physical Symptoms) and II (Intraception). The reliability of the former ranked sec- ond best in the case of A and the control raters, about fifth in the case of A and B. The reliability of the latter ranked sixth with A and the controls, second best with A and B.
The relatively low agreement between A and B on category I was caused not so much by disagreements but by a relatively large proportion of "neu- tral" scores (omissions) on the part of B (see Table 9(XXII)). While the control raters and A felt able to judge the presence or absence of certain cues in the record, rater B frequently felt that the subjects' attitudes toward their symptoms were not sufficiently brought out in the interviews.
The rela"tively low agreement between the control rater and A on Intra- ception was due mainly to a larger number of disagreements. The control rater was in this case particularly dissatisfied with her ratings, feeling that she did not have a sufficient grasp of the concept of intraception nor enough acquaintance with the cues by which the trait could be recognized. In the case of A and B the concept and manifestations of intraception had been made the object of special study and played an important part in their thinking. Rater A thought that her ratings of any given case-on over-all highness- lowness and on all other variables-had been more influenced by her impres- sion of the subject's intraceptiveness than by any other cue. It is reasonable to assume that this difference in training is the cause of the difference in reli- ability between the two sets of raters. This becomes even more probable when the reliability figures are compared with the agreements between rat- ings and actual E score. Here, A's and B's ratings of presence or absence of intraception were related to high and lowE score (in the expected direction) So to S3 per cent of the time, while the control rater's judgment agreed with E only 65 per cent of the time (see Table 12 (XXII)).
The highest reliabilities (91 per cent and 95 per cent) were obtained for variable V (Patient Mentions Unhappy Childhood, etc. ). It is clear that there is not much room for disagreement here.
The lowest reliabilities were obtained for variable III (Ego-alienness). Here the agreement between A and the control rater was only 65 per cent, t)lat between A and B 70 per cent. The former figure barely meets the stand- ard of acceptable reliability of measurement. There was an unusually large discrepancy between A's and the control rater's estimate for the high scorers,
? 93?
THE AUTHORITARIAN PERSONALITY TABLE 10 (XXII)
THE AMOUNT OF AGREEMENT BETWEEN A SINGLE RATER (A) AND SEVEN OTHER RATERS IN ESTIMATING VARIABLES IN INI'AKE INI'ERVIEWS
PSYCHIATRIC CLINIC PATIENTS: MEN AND WOMEN COMBINED (N = 59)
Variable
I. Main emphasis on somatic complaints
II. Intraception
III. Ego-alienness
IV. Externalized theory of onset and causes
v. Spontaneous mention of unhappy childhood and family relations
Hby N by
4 1
1 2
4 2
One; Other
VI. CUes regarding charac- High ter structure Low
Agreement between A's over-
all H-L rating and composite
score based on ratings of 7
independent raters, each High 21 rating a single variable LOW 5
VII. Predominant type of High
22 --
Actual Standing Rated
on E Hby Scale Both
High 12 Low 4
High 18 Low 4
High 9 Low 4
Rated
Rated Rated Nby Hby One; Both L by Other
- 2 1 4
2 6 1 8
1 10 2 5
1 4 4 2
- 1 - 1
1 4 1 2
High 14 6 Low 4 4
High 26 - LOW 17 1
3 symptoms Low41-7
Averages High 16. 0
2. 9
11 5 3 2
0. 7 ? Low 5. 7 1. 9 1. 3 ?
5 4
? Rated Rated
No. of Agreements
T otal Percentage Agreement
81. 4
72. 0
65. 3
73. 7
L by L by Both N by
6 4 20 1
2 2 15 3
1 2 10 7
1 - 11 1
2 5 16 7
12 181
2 22
Percentage Agreement
18 + 4 = 22
25 + 1 = 26 83. 9
-
20 + 1 = 21 42. 5
20+1. 5=21. 5 69. 4
1 15 1
75. 0
1. 9 2. 3 15. 0 3. 0
23 27
21. 1 24. 5
50
45. 5
82. 1 84. 7 87. 1
75. 5 77. 2 78. 8
PSYCHOLOGICAL ILL HEALTH AND POTENTIAL FASCISM 93 I
PSYCHIATRIC CLINIC PATIENTS: MEN AND WOMEN COMBINED (N =59)
One Other
H and L Respectively
H and L Combined
12 + 3 = 15 38. 5
21+2. 5=23. 5 75. 8
16+4=20 43. 5 18+5. 5=23. 5
71. 4 75. 8
27 29
56 96. 4 94. 9 93. 5
48 78. 6
14+5=19 43. 5 67. 9 73. 7 20+4. 5=24. 5 79. 0
244785. 779. 7 23 74. 2
53. 6
? 932
THE AUTHORITARIAN PERSONALITY
there being only 54 per cent agreement. This figure is the result of a rather large number of disagreements between the two raters. The control rater said that she was quite unsure of her ratings because she felt that the inter- view material reflected more the physician's than the patient's attitude toward the complaints. Also, she conceived of "ego-alienness" as something unfavor- able from the mental hygiene point of view and was therefore more often willing to score "absence" of the trait than "presence. " Possibly the most important factor, which lowered both sets of reliabilities, is the relative broadness with which this variable was defined, including few behavioral criteria but leaving much to the rater's intuition.
With the exception of variable III, the ratings for all variables, as well as the over-all estimates, proved to be reliable measures.
5. RELA TIONSHIP BETWEEN RA TINGS AND ETHNOCENTRISM SCORE
The idea of the present rating method is the following: If the "blind" ratings of a given trait correlate significantly with another trait about which the raters had no information (in this case ethnocentrism), it is likely that the two traits are correlated in reality. The relationships to be described below are probably valid because we have eliminated many of the factors inherent in the rating process which could have influenced the results.
There are, however, many factors inherent in the rating method which, can still influence the "blind" ratings in such a way that statistical relation- ships of two traits are artificially raised or lowered. The possibility of arti- ficially high correlations is present particularly when several traits are rated at the same time. Here, the rating of one trait can easily influence the ratings of other traits. This factor (called halo effect) could have influenced the ratings of judges A and B. It was primarily for this reason that control raters, each of whom was trained to pay attention to and to recognize only one of
. the seven single traits, were used.
The ratings of (I) over-all highness and lowness, and (2) of the single
traits, were related to ethnocentrism in the following manner: For a given trait, we counted the number of subjects in the high E quartile who were judged to be "high" by the raters. To these were added half of the high-scor- ing subjects receiving a "neutral" rating. This sum, divided by the total num- ber of high scorers in the group, yielded the percentage of high scorers rated in the high direction. By the same procedure the percentage of low scorers rated in the low direction was obtained. The weighted average of these two percentages was then computed; this indicated the percentage of subjects related "correctly," that is, the percentage whose ratings were re- lated to E score in the expected way.
a. THE OvER-ALL RATINGs. Table I I (XXII) gives the agreements between the subject's standing on E and A's ratings. Rater A's guesses as to highness and lowness agreed with the subject's actual E score 86 per cent of the
? Variable
I. Main emphasis on somatic complaints
II. Intraception III. Ego-alienness
IV. Externalized theory of onset and causes
v. Spontaneous mention of unhappy childhood and family relations
ACtual Standing on E Scale
High Low
High
Low
High
Low
High Low
High Low
Rated Rated H L
Rated N
No. of Agreements
18-1/2 24-1/2
25 24
24-1/2 21
20-1/2 19
27 13-1/2
22-1/2 . 24:. . 112
No. o f Agreements Combined
43
49 45-1/2 39-1/2
40-1/2
47
Percentage Agreement
66. 1 'T9:lY
89. 3 77. 4
87. 5 67. 7
73. 2 61. 3
96. 4 43. 5
80. 4 7 9 . 0
T o t a l Percentage Agreement
72. 9 83. 1 77. 1 66. 9
68. 6 79. 7 84. 7 86. 4
VI. Cues regarding charac- High
13
2 23
1 23
3 26
1
7 3
2 2
0 0
ter structure
VII. Predominant type of symptoms
OV er-all A verages
Low
Hhh Low
High Low
High Low
26 50 92. 9 24 77. 4
25 51 89. 3
TABLE 11 (XXII)
THE AMOUNI' OF AGREEMENT BETWEEN RATER A' S ESTIMATE OF HIGH OR LOW ETHNOCENI'RISM, BASED ON ANALYSIS OF INTAKE INTERVIEWS, AND ETHNOCENTRISM AS MEASURED BY
THEE SCALE
PSYCHIATRIC CLINIC PATIENTS: MEN AND WOMEN COMBINED (N =59)
1567 5233
2422 5224
2323 9202
17 7
27 17
19 5
25 6
25 5
47 14 10
10
26
lJ3. 9
83. 7
09. ~
? I.
II. III. IV. V.
VI.
VII.
Main emphasis on somatic complaints
I n t r a c e p t i o n
E g o ? a l i e n n e s s
E x t e r n a l i z e d
of onset and causes
Spontaneous mention
of unhappy childhood and family relations
C u e s r e g a r d i n g character structure
Predominant type of symptoms
Means
TABLE 12 (XXII)
THE AMOUNT OF AGREEMENT BETWEEN ESTIMATES OF ETHNOCENTRISM, BASED ON RATINGS OF SINGLE VARIABLES FROM INTAKE INTERVIEWS, AND ETHNOCENTRISM AS MEASURED BY THE E SCALE
Variable Composite Ratinga Single V ariables:
Psychiatric Clinic Patients: Men And Women Combined (N =59)
Actual No. No. No. No. of Total Standing Rated Rated Rated No. of Agreements Percentage Percentage
on E Scale H L N Agreements Combined Agreement Agreement fH~i~g~h________~2~2~----~6~----*o________~2~2~. *o________~4~4~-~o________~7~1~-~o________~7~4~-~6---
Low 9 22 0 22. 0 78. 6
~H~i~g~h________~1~5~----~1~2r-----T1________~1~5~-~5________~3~8~? ~o________~5~5~. 4~------~6~4~-~0--- Low 8 22 1 22. 5 72. 6
~H~i~g~h~------~1~9~----~5~----~4~------~2~1r. ~o~------~3~8~-~5~------~7~5~-~o~------~6~5~-~3_ _ _ Low 13 17 1 17. 5 56. 5
;H~i~g~h~------~9~-----T14~----~5~------~1~1~. ? 5_ _ _ _ _ _ _ _ ~3~3~-~o~------~4~lr. ~1_ _ _ _ _ _ _ _ ~5~5~-~9_ _ _ _ Low 6 18 7 21. 5 69. 4
+H~i~g~h--------~2~1~-----i4~----*3_ _ _ _ _ _ _ _ ~2~2~. *5_ _ _ _ _ _ _ _ 24~2~-~o_ _ _ _ _ _ _ _ ~8~0~? ~4_ _ _ _ _ _ _ _ ~7~1~? ~2--- Low 7 15 9 19. 5 62. 9
~H~i~gh~-------72*6______~2~----0~-------72T6~. 0~------~37~. 5~------~9;2~. 9~------~6~3~. 6~-- Low 19 11 1 11. 5 37. 1
~H~i~gh~_ _ _ _ _ _ _ ; 1 ; 2 _ _ _ _ _ _ ~1~1~--~5~------~14~. 5~------~3~6~. 5~------~5T1~. 8~------~6~1~. 9~--
t h e o r y
Low
High Low
High Low
5 18 8 22. 0
22 6 0 22. 0 10 21 0 21. 0
17. 71 7. 7 2. 6 19. 0 9. 71 17. 4 3. 9 19. 4
43. 0
38. 4
71. 0
78. 6 72. 9 67. 7
67. 9 65. 0 62. 5
acomposite rating based on ratings by 7 independent raters on the -variables listed. A "High rating" received point, a "Neutral rating? ~ point; 4 points or more constituted a composite "High" score; everything below 4 a ? ? Low" score.
? PSYCHOLOGICAL ILL HEALTH AND POTENTIAL FASCISM 935
time. For the high scorers alone the agreement was 89 per cent, for the low scorers 84 per cent. These figures show again how closely ethnocentrism is related to personality factors, although the relationships are by no means perfect.
The remaining problem now is: How did raters A and B arrive at their rather accurate guesses about ethnocentrism from a short paragraph of inter- view material dealing almost exclusively with the subjects' complaints? Did they base their guesses mainly on the variables described in the manual, or did they inadvertently use other cues, such as type of language used and other cues that have not been made explicit but which they learned to associate with highness or lowness while interviewing subjects with known ethno- centrism scores?
The results obtained from the control raters, who had had no such previous experience with high- and low-scoring subjects, should help to decide this question.
