An In-Depth Study of the Reliability of ChatGPT and Gemini in Addressing Complex Chemistry Questions in Instrumental Analysis

Abstract

Artificial intelligence (AI) has gained significant value in education and problem-solving across various fields. Advanced generative AI systems like ChatGPT (versions GPT-4 and GPT-4o) and Gemini AI provide notable advantages in tackling challenges, especially in chemistry. This research investigates how ChatGPT and Gemini AI help understand and solve chemistry problems, focusing on instrumental analysis for undergraduate pharmacy students. The study evaluates the problem-solving skills of both AI systems using a set of 120 multiple-choice questions (MCQs) randomly selected from exam-style queries. It also observes instances where ChatGPT, when asked the same questions twice, gave inconsistent answers - some correct and some incorrect. Gemini AI scored an overall 87.5% (105 out of 120 questions), while GPT-4 scored 84.2% (101 out of 120), and GPT-4o scored 85.3% (103 out of 120). The percentage of correct answers notably declined for calculation-based questions across all AI systems. Overall, GPT-4 and GPT-4o performed similarly in total correct answers, with the added benefit of providing logical, step-by-step explanations for their solutions. This feature is particularly useful for pharmacy students, as it can assist with calculations, improve understanding of concepts, and help overcome challenges in chemistry and instrumental analysis in pharmaceutical education. Therefore, AI can support chemistry and pharmacy students in performing calculations, understanding concepts, and resolving problems. However, issues such as response inconsistencies and domain-specific fine-tuning are recommended to be further addressed to maximize these systems’ potential.