Automated coding of suicidal ideation in emergency department presentations: balancing interpretability and performance

Over 700,000 people die due to suicide annually according to the WHO, and prior attempts are regarded as one of the strongest risk factors. Emergency departments are a key point of contact for patients in mental health crises, but effective measurement of suicidal presentations to emergency departments (EDs) is hindered by documented low sensitivity of standardised tools and International Classification of Disease codes for these presentations. Recent research has applied combinations of natural language processing and modelling techniques to automate the manual review of unstructured fields, such as clinical notes. Research has not yet investigated how the performance of these automated tools varies with respect to the complexity or interpretability of text encoding and modelling techniques, which is particularly important in this sensitive health setting. The objective of this work is to determine the trade-off between performance and interpretability of techniques in this setting. We empirically assess the predictive performance of automated tools as a product of text encoding and modelling techniques on a manually coded dataset of 91,778 ED presentations to two public teaching hospitals on the Gold Coast (Australia). For text encoding, we consider phrase matching derived from expert input, term-frequency inverse document frequency, static embeddings via algorithms like word2vec, and contextual embeddings generated by a pretrained transformer (clinicalBERT). For modelling techniques, we consider logistic regression, decision trees, support vector machines, ensembles, custom neural networks, and finetuned transformers. Predictive performance is assessed via stratified five-fold cross-validation with accuracy, sensitivity, precision-recall curves and Hosmer-Lemeshow tests. The research will provide a framework for hospital decision-makers to determine the appropriate trade-off between competing needs for predictive accuracy and interpretability, supporting initiatives to improve measurement of suicidality. Findings highlight the improved predictive performance associated with both more complex text representations and modelling techniques.