Funnel plots and risk-adjusted count data adverse events. A limitation of indirect standardisation

Reporting of hospital adverse event data is becoming increasingly mandated and this has motivated work on methods for the analysis and display of these data for groups of institutions. Currently, the method preferred by many workers is the funnel plot. Often, indirect standardisation is employed to produce these plots. It appears that, when used to display binary data such as surgical site infection or mortality data, the method is satisfactory. Increasingly, these data are risk-adjusted. However, risk adjustment of these data usually involves individual patients undergoing the same or similar procedures and the method does not appear to mislead. However, when dealing with count data such as bacteraemias it appears that this method can mislead, particularly where methods for risk adjustment of these data are used. Information about the hospitals or units of interest rather than individual patients is employed. For example, one hospital may have plastic and cardiac surgery units in which bacteraemias occur infrequently whereas another may provide treatment for renal failure (including transplantation) and have a large haematology-oncology unit (also including transplantation), each of which would expect higher bacteraemia rates. Moreover, the hospitals and units within them may differ substantially in size. It is well known that indirect standardisation can give biased results when denominators differ substantially. We illustrate this difficulty with risk-adjusted bacteraemia data from the Queensland Health Centre for Healthcare Infection, Surveillance and Prevention (CHRISP) database.