We learn new faces throughout life, for example in everyday settings like watching TV. Recent research has shown that image variability is key to this ability: if we learn a new face over highly variable images, we are better able to recognize that person in novel pictures. Here we asked people to watch TV shows they had not seen before, and then tested their ability to recognize the actors. Some participants watched TV shows in the conventional manner, whereas others watched them upside down or contrast-reversed. Image variability is equivalent across these conditions, and yet we observed that viewers were unable to learn the faces upside down or contrast-reversed—even when tested in the same format as learning. We conclude that variability is a necessary, but not sufficient, condition for face learning. Instead, mechanisms underlying this process are tuned to extract useful information from variability falling within a critical range that corresponds to natural, everyday variation.