Artificial intelligence versus humans: A comparative analysis of time, cost, and performance on a clinical code conversion task

Healthcare services generate and store large quantities of data, requiring significant resources to manually analyze and gain meaningful insights. Recent advancements in automation tools—such as generative artificial intelligence (GenAI)—provide new opportunities to reduce human labor. This study explores the potential utilization of GenAI for a healthcare data analysis task—specifically, the conversion of clinical data from one diagnostic classification system to another (i.e., the Australian extension of the Systematized Nomenclature of Medicine Clinical Terms to the International Classification of Diseases, 10th Revision, Clinical Modification)—and examines the time and cost benefits of performing this using GenAI compared to a human rater. Conversions were completed using three methods: manual conversion using the National Library of Medicine’s I-MAGIC tool, ChatGPT-4o, and Claude 3.5 Sonnet. The accuracy of the GenAI tools was mapped against the manually extracted codes and examined in terms of a perfect, partial, or incorrect match. Task completion time was recorded and extrapolated to calculate and compare the cost associated with each method. When compared to the manually extracted codes, Claude 3.5 Sonnet yielded the highest level of agreement over ChatGPT-4o, whilst being the most time- and cost-effective. GenAI tools have greater utility than they have currently been given credit for. The automation of big data healthcare analytics, whilst still the domain of humans, is increasingly capable of being undertaken using automation tools with low barriers to entry. The further development of GenAI’s capabilities, alongside the capability of the healthcare system to use it appropriately, has the potential to result in significant resource savings.