PERFORMANCE OF CHATGPT-5.1 AND GEMINI 2.5 FLASH ON THE UKRAINIAN LICENSING INTEGRATED EXAMINATION “KROK 3”: A COMPARATIVE STUDY

Abstract

The development of large language models (LLMs) has opened new perspectives for their integration into healthcare. Previous studies have demonstrated the high effectiveness of leading LLMs, such as GPT-4, in passing standardized English-language medical examinations (e.g., the USMLE), showing accuracy levels comparable to those of qualified practicing physicians. However, data on the performance of these models in non-English, particularly Ukrainian, contexts remain limited. Aim. To evaluate the performance of contemporary LLMs in completing test items of the Ukrainian-language licensing integrated examination “Krok 3” and to analyze the impact of role-based instructions on the accuracy and stability of responses. Materials and Methods. The performance of ChatGPT-5.1 (OpenAI) and Gemini 2.5 Flash (Google) was assessed using a set of 150 official test items in the specialty “Internal Medicine” from the Ukrainian-language licensing integrated examination “Krok 3.” The models were tested in a standard mode and in a mode with the role-based instruction “act as a professional physician.” The percentage of correct answers, 95% confidence intervals (Wilson method), statistical differences (McNemar test), and agreement between modes (Cohen's weighted kappa, kw) were evaluated to assess accuracy and response stability. Results. Both models demonstrated high accuracy (>90%) in both operating modes. No statistically significant differences were found between the models or between modes with and without the role-based instruction (p > 0.05). The role-based instruction did not affect overall performance. However, response stability differed: ChatGPT showed moderate agreement between modes (kw = 0.41), whereas Gemini demonstrated high agreement (kw = 0.80). Conclusions. Contemporary LLMs are capable of performing Ukrainian-language licensing examination tasks from “Krok 3” with high accuracy, indicating effective multilingual adaptation in the field of medical education. Role-based instructions do not improve response accuracy in the single-best-answer MCQ format, although response stability may vary between models. These findings highlight the substantial potential of LLMs for use in Ukrainian medical education, preparation for licensing examinations, and as clinical decision-support tools, provided that further local validation is conducted. Key words: large language models, artificial intelligence, medical licensing examination, Krok 3, multiple-choice questions, role-based prompting, response stability.