Inter-rater agreement in trait judgements from faces

Kramer, Robin S. S., Mileva, Mila and Ritchie, Kay L. (2018) Inter-rater agreement in trait judgements from faces. PLoS ONE, 13 (8). e0202655. ISSN 1932-6203

Full content URL:

Kramer, Mileva, & Ritchie 2018.pdf
Kramer, Mileva, & Ritchie 2018.pdf - Whole Document
Available under License Creative Commons Attribution 4.0 International.

Item Type:Article
Item Status:Live Archive


Researchers have long been interested in how social evaluations are made based upon first impressions of faces. It is also important to consider the level of agreement we see in such evaluations across raters and what this may tell us. Typically, high levels of inter-rater agreement for facial judgements are reported, but the measures used may be misleading. At present, studies commonly report Cronbach’s α as a way to quantify agreement, although problematically, there are various issues with the use of this measure. Most importantly, because researchers treat raters as items, Cronbach’s α is inflated by larger sample sizes even when agreement between raters is fixed. Here, we considered several alternative measures and investigated whether these better discriminate between traits that were predicted to show low (parental resemblance), intermediate (attractiveness, dominance, trustworthiness), and high (age, gender) levels of agreement. Importantly, the level of inter-rater agreement has not previously been studied for many of these traits. In addition, we investigated whether familiar faces resulted in differing levels of agreement in comparison with unfamiliar faces. Our results suggest that alternative measures may prove more informative than Cronbach’s α when determining how well raters agree in their judgements. Further, we found no apparent influence of familiarity on levels of agreement. Finally, we show that, like attractiveness, both trustworthiness and dominance show significant levels of private taste (personal or idiosyncratic rater perceptions), although shared taste (perceptions shared with other raters) explains similar levels of variance in people’s perceptions. In conclusion, we recommend that researchers investigating social judgements of faces consider alternatives to Cronbach’s α but should also be prepared to examine both the potential value and origin of private taste as these might prove informative.

Additional Information:© 2018 Kramer et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Keywords:inter-rater agreement, face perception, Cronbach’s ?, attractiveness, trustworthiness, dominance
Subjects:C Biological Sciences > C800 Psychology
C Biological Sciences > C830 Experimental Psychology
Divisions:College of Social Science > School of Psychology
ID Code:32979
Deposited On:23 Aug 2018 08:15

Repository Staff Only: item control page