Human faces and voices are rich sources of information that can vary in many different ways. Most of the literature on face/voice perception has focused on understanding how people look and sound different to each other (between-person variability). However, recent studies highlight the ways in which the same person can look and sound different on different occasions (within-person variability). Across three experiments, we examined how within and between-person variability relate to one another for social trait impressions by collecting trait ratings attributed to multiple face images and voice recordings of the same people. We find that within-person variability in social trait evaluations is at least as great as between-person variability. Using different stimulus sets across experiments, trait impressions of voices are consistently more variable within people than between people—a pattern that is only evident occasionally when judging faces. Our findings highlight the importance of understanding within-person variability, showing how judgments of the same person can vary widely on different encounters and quantify how this pattern differs for voice and face perception. The work consequently has implications for theoretical models proposing that voices can be considered “auditory faces” and imposes limitations to the “kernel of truth” hypothesis of trait evaluations.