Construct validity and measurement invariance of MMIs: Do they allow meaningful group comparison?

Introduction
Multiple mini interviews (MMI) measure multiple non-cognitive attributes and are widely used for selection to healthcare programmes. When instruments aim to assess subjective attributes, challenges for the validity and comparability of scores emerge. Thus, it is important to provide evidence that constructs have the same meaning across groups and group differences are not contaminated by group-specific characteristics (aka measurement invariance (MI)). Otherwise, interpretation of scores for members of different groups is not meaningful.

The present study investigates whether:
a.there is evidence to support factorial validity of MMI.
b.MMI measure the same underlying construct for students with different characteristics?
c.there are differences in MMI performance according to gender, age, disability status, interview time and gender of selector.

Methods
635 candidates were interviewed on a 7-station MMI in 2019/2020. Confirmatory factor analysis (CFA) assessed the factorial structure of 28-item MMI. We examined and compared the goodness-of-fit of different models. MI analyses were performed by Multi Group Confirmatory Factor Analysis (MG-CFA). Changes in CFI ≤.01 were used to determine invariance. Mean differences between groups were tested with independent samples t-tests.

Results/Discussion
Among measurement models tested, both seven-factor first order model and higher order model provided good fit to data. Reliability estimates (Cronbach’s alpha) for all factors ranged from 0.85 to 0.90. MI analyses showed MMI measures the same construct for candidates from different groups. Women and mature candidates scored significantly higher than men and younger students. Morning MMI candidates performed better than afternoon candidates.

Conclusion
This study provides psychometric evidence supporting the factorial validity and measurement equivalence (invariance) of MMIs. It is encouraging that there was no difference between MMI performances of students with and without disabilities. Equally heartening, we found no difference on MMI scores between students marked by male and female selectors.