Reliability of non-pretested versus pretested questions in the applied knowledge test (AKT) of the MRCGP: evidence of quality assurance

Dixon, Hilton and Blow, Carol and Milne, Paul and Siriwardena, A. Niroshan (2014) Reliability of non-pretested versus pretested questions in the applied knowledge test (AKT) of the MRCGP: evidence of quality assurance. Education for Primary Care, 25 (3). pp. 149-154. ISSN 1473-9879

Documents
Reliability of non-pretested versus pretested questions in the applied knowledge test (AKT) of the MRCGP: evidence of quality assurance

Request a copy
[img] PDF
Dixon Reliability of AKT EPC 2014.pdf - Whole Document
Restricted to Registered users only until 31 December 2099.
Available under License Creative Commons Attribution Non-commercial No Derivatives.

143kB
Item Type:Article
Item Status:Live Archive

Abstract

The applied knowledge test (AKT) is a computer delivered multiple choice test offered three times a year. It forms part of the Membership of the Royal College of General Practitioners (MRCGP) examination, which licenses doctors for UK general practice. This paper investigates the effect of current quality assurance processes in the AKT on the reliability of questions which are not pretested.
Test questions in the AKT are based on the RCGP curriculum blueprint, referenced to current evidence, peer-reviewed and critically appraised before they are added to the question bank. There is a standardised process for question selection and test construction. Standard setting for the AKT follows a modified Angoff process where a panel of experts make individual judgements on each question of the test estimating the likelihood of the ‘just-passing candidate’ answering correctly.
In the three examinations delivered in 2011 (AKT 11 (January), 12 (May) and 13 (October)) Cronbach’s alpha was 0.88, 0.92, 0.91, respectively. For pretested questions, the predicted alpha was 0.90, 0.93, 0.92 and for non-pretested questions 0.84,
0.91, 0.92, respectively adjusting for test length using the Spearman-Brown formula.
Pretesting of questions is unnecessary provided there is a systematic process of question and test construction which quality assures the AKT. This ensures that pretesting does not delay the use of questions to examine current knowledge and guidelines.

Additional Information:What is already known in this area The AKT has consistently shown high reliability and includes pretested and non-pretested questions. What this work adds This study shows that the inclusion of non-pretested questions does not reduce the overall reliability of the test and supports their continued inclusion to test new clinical knowledge and guidelines. Innovative question formats using computer-based testing such as free text, ranking and algorithm completion, should be subject to similar scrutiny when non-pretested, to continue to ensure the validity and discrimination of the test.
Keywords:primary care, general practice, assessment, knowledge, examination, medical licensing
Subjects:A Medicine and Dentistry > A300 Clinical Medicine
Divisions:College of Social Science > School of Health & Social Care
ID Code:15923
Deposited On:01 Nov 2014 21:27

Repository Staff Only: item control page