While physical activity is beneficial for men with prostate cancer, too few perform sufficient activity for such benefit. This study examined perceptions of men with prostate cancer of their barriers and facilitators to physical activity, and how androgen deprivation therapy (ADT) may influence these perceptions. Two focus groups were conducted, involving six ADT and eight non-ADT patients respectively. Data were transcribed verbatim and themes developed using a general inductive thematic approach. Facilitators to physical activity common to both groups of cancer survivors included clinician and spousal involvement, with pre-existing co-morbidities and increased age cited as barriers by both groups. The ADT subgroup cited personal involvement as a facilitator to physical activity, with fatigue, reduced motivation and a relative lack of specific advice from their clinician as additional barriers. The non-ADT subgroup had no additional facilitators to physical activity but cited time constraints as a barrier. These results highlight the important role that cancer clinicians and spouses play in promoting physical activity for men with prostate cancer and how ADT may influence their other facilitators and barriers. As physical activity is beneficial for prostate cancer survivors, especially those on ADT, cancer clinicians should regularly discuss physical activity with their patients.