Passionate music engagement is a defining feature of music fans worldwide. Although benefits to psychosocial well-being are often experienced by fans of music, some fans experience maladaptive outcomes from their music engagement. The Dualistic Model of Passion proposes that two types of passion—harmonious and obsessive—are associated with positive and negative outcomes of passionate engagement, respectively. This model has been employed in research on passion for a wide range of pursuits including music performers, but not for passionate listeners. The present study employed this model to investigate whether (1) harmonious passion for music is associated with positive music listening experiences and/or psychological well-being and (2) obsessive passion for music is associated with negative music listening experiences and/or psychological ill-being. Passionate fans (n = 197) of 40 different musical genres were surveyed about their experiences when listening to their favorite music. Measures included the passion scale, affective experiences with music, and psychological well-being and ill-being. Results supported the Dualistic Model of Passion. Structural equation modeling revealed that harmonious passion for music predicted positive affective experiences which, in turn, predicted psychological well-being. Conversely, obsessive passion for music predicted negative affective experiences which, in turn, predicted psychological ill-being. The findings suggest that the nature of passionate engagement with music has an integral role in the psychological impact of music engagement and implications for the well-being of music fans.