Medication errors occur when pharmacy staff count out and give their patient the incorrect medication inside a prescription bottle labeled for a different medication1,2,3. These dispensing errors are potentially harmful to patients, strain the healthcare system, and lead to costly liability fees for pharmacies. In fact, the two most common reasons for legal action against licensed pharmacy staff are the dispensing of the wrong dose and wrong medication with an average paid out claim of nearly $125,0004. A multi-site study of dispensing errors in community pharmacies found that incorrect drug, incorrect strength, and incorrect labeling of the prescription vial occurred at a rate 278/109,558 (0.25%) prescriptions5. It is critical to provide pharmacy staff with tools to reduce medication errors thereby improving patient safety and reducing healthcare spending. In the pharmacy setting, a fundamental task is to dispense a prescription that ensures the right pill gets to the right patient. Dispensing errors occur when pharmacy staff select a stock bottle of medication, count out, and place the incorrect medication or strength into a prescription vial labeled for a different medication or dose6,7,8. National Drug Codes (NDC) are unique 10-digit, three segment numbers assigned to drugs. However, only the first identifier segment, the manufacturer, is assigned by the United States Food and Drug Administration (FDA). The remaining information on the NDC directory is solely the responsibility of the manufacturer9. Relying on humans alone to verify that the physical attributes of medication products for an NDC are insufficient to overcome high workload, interruptions, and limitations of human cognition that are common contributing factors in dispensing errors6,7,8.
Technological advances such as robots that fill medication bottles and barcode scanning to promote getting the right pill into the right vial for the patient are insufficient to eliminate incorrect medication fills and subject to workarounds10,11. These technologies are prone to human interaction errors, such as pharmacy staff counting out and labeling the wrong medication after scanning the prescription label and stock medication bottle barcodes. It is difficult to pinpoint the exact number of medication errors as the last comprehensive report was conducted by the Institute of Medicine in 200612. Technological solutions such as barcode scanning do not eliminate errors as a result of human interaction workarounds11and do little to address problems with overburdened pharmacist verifying the prescription. A 2015 study of hospital barcode scanning technology found that barcode scanning did not change the number of errors but rather shifted the types of errors from wrong ingredient to wrong strength and wrong quantity13.
In an effort to detect and remedy dispensing errors before they reach the patient, pharmacists perform a verification task by comparing the contents of the filled prescription with an industry reference image. The traditional verification process occurs on site using the physical prescription bottle; however, 24 states currently allow for remote pharmacies in which a pharmacist performs this verification task off-site. Instead, verification by a pharmacist using top-down pictures of the uncapped, filled prescription bottle may occur. Furthermore, a national chain pharmacy recently piloted this remote verification process. Error rates for remote verification are not significantly different from traditional in-person verification; however, the types of errors differ5. Remote verification resulted in fewer errors that reached the patient (a near miss) but prescriptions were more likely to contain incorrect directions. In light of the COVID-19 pandemic, further adoption of technology that enables remote verification may occur. Technology assistance of the verification task through the use of machine intelligence (MI) has the potential to detect dangerous and costly pharmacy dispensing errors3,4,14.
To assist humans in the verification process, MI models could perform a pill classification task using images taken of medication filled inside a prescription bottle. Previously published studies of pill classification tasks, some spurred by the United States National Library of Medicine’s Pill Image Recognition Challenge, focus on comparing the front and back of individual pills between consumer images with different backgrounds and industry reference images15,16,17. Another study focused on extracting higher-level features (e.g., color, shape, and imprint) of reference pill images using labeled structured data to predict the medication18. These previous studies focus on small-scale experimentation of MI models on individual pills. This is achieved by segmenting the background an individual pill then comparing the features to known reference images. In this study, we focus instead on medication dispensing error detection within a pharmacy setting. This requires learning models from a pile of medication inside a prescription vial where image segmentation from the background is impossible as pills are piled on top of each other.
In this paper, we report on training a ResNet-18 deep learning model to predict the labeled features of a medication product using an image showing oral medication inside of a filled prescription vial. The training was done in two stages. First, model fitting occurred by pre-training a ResNet-18 network on the ImageNet dataset19. Second, the model was fine-tuned on the dataset showing a pile of pills inside a prescription vial. The objectives are to (1) evaluate model performance of the image classification model that predicts the shape, color, and NDC of prescription medication images, (2) evaluate the reliability of the model predictions, and (3) determine the features of the prescription medication images that lead to incorrect predictions by the model. In doing so, we discuss the implications of utilizing this technology in practice from a trust and safety perspective.