Happy faces are categorized faster as “happy” than angry faces as “angry,” the happy face advantage. Here, we show across three experiments that the size of the happy face advantage for male Caucasian faces varies as a function of the other faces they are presented with. A happy face advantage was present if the male Caucasian faces were presented among male African American faces, but absent if the same faces were presented among female faces, Caucasian or African American. The modulation of the happy face advantage for male Caucasian faces was observed even if the female Caucasian/male African American faces had neutral expressions. This difference in the happy face advantage for a constant set of faces as a function of the other faces presented indicates that it does not reflect on a stimulus-dependent bottom-up process but on the evaluation of the expressive faces within a specific context.