Assessing the ability of GPT-4o to recognize medications and provide patient education – Scientific Reports

GPT-4o agreed to the first request to provide medication information from the image by replying with “Sure, please continue to upload the medication image. I will do my best to provide you with information what you want.” GPT-4o responses to each of the 20 medication pictures began with a sentence stating the name of the medication, found on its package, and its active ingredients, even for the pictures of the medicine on which the ingredient works does not appear, followed by written answers to each of the requested information: “what is it?”, “what is it used for?”, “you should use it.” how?” and “possible side effects and complications”. All responses ended with “Caution” in GPT-4o to “Always follow your health care provider’s instructions closely and report any symptoms and any unusual or immediate side effects”, followed by additional advice specific to each medicine.

Each answer was scored according to accuracy, precision and clarity using a 4-point Likert-like scale, from 1 to 4 (Table 3). The number of points was 3.55 ± 0.605 indicating answers that are almost completely correct, correct and clear. The mean answer word count was 283.25 ± 31.41 words, which translates to a reading time of about two minutes, indicating relatively short answers. Among the 20 medication responses, 12 received 4 points and 7 received 3 points while only one response received 2 points; therefore, 4 points were the mean and the mean. None of the responses written on the pictures of the 20 medicines were given a score of 1 and none were written as incorrect or considered to be directly harmful to patients. Therefore, the discounts are due to some lack of clarity or, in many cases, the lack of certain administrative measures or special medical considerations. All GPT-4o transcriptional responses included in the 20 drug images can be found in the supplementary information (Table S1).

Next, the ability of GPT-4o to recognize drugs from blurred images was evaluated. In fact, GPT-4o was able to recognize all four tested drugs and provide readable information regardless of the degree of obscuration or if the image of the drug was used as input or not. Interestingly, responses to images of Augmentin^® and Seretide^®which were used as an early intervention without darkening, were almost identical to the data given in the first responses to these two drugs, showing a good level of agreement. The overall score obtained for the four ambiguous drug pictures was 3.75 ± 0.5 with a mean response score of 317.5 ± 7.14, indicating relatively accurate, correct and clear responses with mean reading time is about two minutes. Scores were not significantly different between clear and blurred picture responses (Q> 0.05), and the effect size for the difference in means was small (d = 0.338). The responses of GPT-4o to the four clinical images are provided in the supplementary material (Table S2).

Analysis of the written responses of the GPT-4o revealed that it not only provides the requested information in detail but also provides additional relevant information and explanations that can improve patient understanding and thus the results of patient education in general. To name a few examples: GPT-4o emphasized the importance of completing the entire prescribed course of antibiotics to prevent resistance; it is advised to take diuretic medicines in the morning, or in the morning and in the afternoon if it is taken twice a day, to avoid needing to micturate at night; gave dietary advice when teaching about common drug interactions; and emphasized that the intramuscular drug should be administered by a health professional. Other information related to how the patient can identify the occurrence of side effects or problems; for example, it mentioned the signs and symptoms of illness, infection, hypoglycemia, dehydration, electrolyte imbalance, liver or kidney problems and certain withdrawal symptoms and toxins. In addition, GPT-4o has clarified many medical terms that the patient may not know such as jaundice, bronchospasm, hypoglycemia, orthostatic hypotension, arrythmia, lipodystrophy, subcutaneous and angioedema among many others. In addition, all have given the methods of taking medication, in terms of dose, frequency and time, were correct while emphasizing that these may vary between patients and therefore it is necessary to according to the doctor’s specific instructions.

The next step was to evaluate the visual output of GPT-4o regarding the method of administration of certain drugs. In contrast to its impressive performance in providing written drug information and instructions for use, the images produced by GPT-4o were unsatisfactory and contained many errors. An example is shown in Figure 1 for a pre-filled insulin pen injection.

GPT-4o responded with “Here is a step-by-step guide to using Lantus^® SoloStar^® insulin glargine in a pre-filled pen. Each step is illustrated to help you use it correctly. If you have any questions or need more help, feel free to ask! “… to the introduction of the first drug with the same reaction to the next three drugs. Each of the images produced by GPT-4o was like a collage of many small images, starting with a drawing of a medicine package with its name and some vague text, followed by a series of calculated steps that indicate the application or administration of the drug. , each with a text describing the corresponding step. However, all the images produced had many errors and were considered unusable and potentially dangerous. Regarding the representation of the medicine, the pictures of the packages were different from the original ones in terms of appearance including shape and color. As for the actual administrative steps, they were vague, and often completely wrong; for example, showing an insulin injection in the arm and applying nasal gel to the face. Regarding the explanatory text below several steps, it was illegible with spelling errors or compound words, or unrelated to the indicated step. The step numbers were also incorrect with many missing, repeated or not ordered correctly. As a result, GPT-4o’s perception of medication use failed to correlate with accuracy, precision and clarity with a mean score of 1.5 ± 0.577 (equivalent to 16.7%). Indeed, visual scores were significantly lower, in terms of scores, compared to written scores (Q<0.01) with a strong effect size for differences between factors (d = 3.41). In addition to the image shown in Figure 1, images produced by GPT-4o of three other drugs can be found in the supplementary information (Table S3).

Finally, when GPT-4o was asked to list its sources for drug information it responded by citing various reliable sources of drug information. The first part of its response was: “The information provided about the medicine is based on the knowledge accepted by the majority of medical people and the usual sources of medical reference. In particular, data is collected from:

1. 1.

   Drug Monographs: Comprehensive documents that provide comprehensive information about drugs, including their uses, dosages, side effects and contraindications.

2. 2.

   Product Information for Pharmaceutical Manufacturers: Legal information provided by drug manufacturers.

3. 3.

   Medical and Pharmaceutical Textbooks: Authoritative articles used by health professionals to discuss drug issues.

4. 4.

   Clinical Guidelines: Guidelines issued by health organizations such as the American Medical Association (AMA), the World Health Organization (WHO), and others”.

GPT-4o’s response to sources of drug information continued by saying: “Drug-specific sources include:” and specified the following sources of specific information for other drugs and Some that have been reviewed and in that order of use include:

• Amoxicillin Trihydrate and Potassium Clavulanate (Augmentin).^®): Product information from GlaxoSmithKline (GSK) and guidance provided by health organizations such as the American Medical Association (AMA), World Health Organization (WHO), and others.

• Ipratropium Bromide and Albuterol (Combivent^®): Product information from Boehringer Ingelheim and COPD management guidelines.

• Bisoprolol and Hydrochlorothiazide (Concor^® 5 Plus): Product information from Merck and guidelines for blood pressure management.

• Itopride (Ganaton^®): Drug specifications and treatment guidelines for gastrointestinal motility disorders.

• Digoxin (Lanoxin^®): Drug descriptions and guidelines for heart treatment.

• Insulin Glargine (Lantus^® SoloStar^®): Product details from Sanofi and diabetes management guidelines.

• Furosemide (Lasix^®): Drug specifications and guidelines for diuretic use in cardiac and renal conditions.

• Pregabalin (Lyrica^®): Product information from Pfizer and guidelines for the management of neuropathic pain.

• Esomeprazole (Nexium^®): Product information from AstraZeneca and treatment guide for gastroesophageal reflux disease (GERD).

• Tamsulosin (Omnic Ocas^®): Drug descriptions and medication guidelines for the treatment of BPH.

• Warfarin (Orfarin^®): Drug specifications and medication guide to anticoagulants.

• Hydroxyprogesterone Caproate (Primolut^® Depot): Product information from Bayer and medical guidance on hormone therapy.

• Tribenoside and Lidocaine (Procto-Glyvenol^®): Product information and clinical guidelines for hemorrhoid treatment.

• Calcium carbonate and Magnesium carbonate (Rennie^®): Product information from Bayer and guidance on the use of antacids.

• Fluticasone Propionate and Salmeterol Xinafoate (Seretide^®): Product information from GlaxoSmithKline and guidelines for the treatment of respiratory diseases.

• Tiotropium Bromide (Spiriva^®): Product information from Boehringer Ingelheim and COPD management guidelines.

• Budesonide and Formoterol (Symbicort^®): Product information from AstraZeneca and asthma / COPD management guidelines.

• Tobramycin (Tobrex^®): Product information from Alcon and directions for use of ophthalmic antibiotics.

• Dimethindene Maleate and Phenylephrine Hydrochloride (Vibrocil^®): Product Information from Novartis and instructions for use of the nasal spray.

• Diclofenac Potassium (Voltfast^®): Product Information from Novartis and instructions for NSAID use.

• Rosuvastatin (Crestor^®): Product information from AstraZeneca and guidelines for statin therapy.

• Metformin (Glucophage^®): Product information from Merck and diabetes management guidelines. “.