Effective use of Item Analysis to improve the Reliability and Validity of Undergraduate Medical Examinations: Evaluating the same exam over many years: A different approach

Nadeem Alam Zubairi; Turki Saad AlAhmadi; Mohamed Hesham  Ibrahim; Moustafa Abdelaal Hegazi; Fahad Ussif Gadi

doi:10.12669/pjms.41.3.10693

Nadeem Alam Zubairi King Abdulaziz University
Turki Saad AlAhmadi 1. Department of Pediatrics, Faculty of Medicine in Rabigh, King Abdulaziz University, Saudi Arabia
Mohamed Hesham Ibrahim 1. Department of Pediatrics, Faculty of Medicine in Rabigh, King Abdulaziz University, Saudi Arabia
Moustafa Abdelaal Hegazi 1. Department of Pediatrics, Faculty of Medicine in Rabigh, King Abdulaziz University, Saudi Arabia
Fahad Ussif Gadi 1. Department of Pediatrics, Faculty of Medicine in Rabigh, King Abdulaziz University, Saudi Arabia

DOI: https://doi.org/10.12669/pjms.41.3.10693

Keywords: Difficulty Index, Discrimination Index, Distractor Efficiency, Exam Reliability, Item Analysis

Abstract

Objective: MCQ exams are part of end-module assessments in undergraduate medical institutions. Item Analysis (IA) is the best tool to check their reliability and validity. It provides the Reliability Coefficient KR20, Difficulty Index (DI), Discrimination Index (DISC), and Distractor Efficiency (DE). Almost all research papers on IA are based on single exam analysis. We examined the IA of multiple exams of the same module, taken in four years. We aimed to explore the required consistency over the years and the effectiveness of IA-based post-exam measures.

Methodology: Item Analysis of eight final MCQ exams of the Pediatric module from 2020-21 to 2023-24, at the Faculty of Medicine in Rabigh, King Abdulaziz University, Saudi Arabia, were included in the study.

Results: All exams had KR20 of 90 and above indicating excellent reliability. Difficulty levels were consistent except for a single year. Discriminative ability was maintained over the years. Only 28 out of 800 MCQs had a negative DISC. All exams maintained good DE. Only 15 MCQs over four years had zero DE. The practice of reviewing all Non-Functional Distractors yielded a gradual improvement in exam quality

Conclusion: Besides the IA of individual exams, it is also recommended that IA of the same exam be evaluated over 4-5 years to see consistency and trends towards improvement. It helps in improving the reliability and validity by addressing deficiencies and deviations from the recommended standards

doi: https://doi.org/10.12669/pjms.41.3.10693

How to cite this: Zubairi NA, AlAhmadi TS, Ibrahim MH, Hegazi MA, Gadi FU. Effective use of Item Analysis to improve the Reliability and Validity of Undergraduate Medical Examinations: Evaluating the same exam over many years: A different approach. Pak J Med Sci. 2025;41(3):810-815. doi: https://doi.org/10.12669/pjms.41.3.10693

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.