Assuring the quality of an applied knowledge assessment for licensing purposes (Membership of the Royal College of General Practitioners, MRCGP) in UK general practice

Milne, Paul and Dixon, Hilton and Blow, Carol and Siriwardena, A. Niroshan (2012) Assuring the quality of an applied knowledge assessment for licensing purposes (Membership of the Royal College of General Practitioners, MRCGP) in UK general practice. In: 15th Ottawa Conference on the Assessment of Competence in Medicine and the Healthcare Professions, 9-13 March 2012, Kuala Lumpur Convention Centre, Malaysia.

Documents
Assuring the quality of an applied knowledge assessment for licensing purposes (Membership of the Royal College of General Practitioners, MRCGP) in UK general practice
[img]
[Download]
[img]
Preview
PDF
AKTKLfinal.pdf - Presentation
Available under License Creative Commons Attribution Non-commercial No Derivatives.

569kB

Full text URL: http://www.ottawaconference.org/

Abstract

Background
The Applied Knowledge Test (AKT) forms part of the tripos of the MRCGP exam. It is a computer-delivered licensing assessment of applied knowledge. This presentation demonstrates how the test quality is reviewed.
Summary of work
From the RCGP curriculum blueprint, the construction of test items is referenced to high quality evidence, peer reviewed and critically appraised before addition to the question bank. A standard process for question selection, proof reading and feedback to candidates supports its validity. New question formats include free text answers which enhance discrimination. Standard setting follows the modified Angoff process.
Summary of results
The test shows a high Cronbach’s alpha (0.88-0.92). Pre-trialling of questions, common to many postgraduate medical examinations, is deemed unnecessary with an alpha-coefficient for new questions in the most recent test > 0.8 (0.91 when adjusted for the test length). Computer delivered testing enables detailed analysis of patterns of timing and omission of items by candidates.
Conclusions
Systematic processes in test construction and computer delivery has enhanced continuous quality assurance of the AKT.

Take-home messages
Test performance relies on meticulous attention to detail in item writing and test construction. Use of un-trialled new questions enables reliable testing of emerging and changing knowledge.

Item Type:Conference or Workshop Item (Presentation)
Additional Information:Background The Applied Knowledge Test (AKT) forms part of the tripos of the MRCGP exam. It is a computer-delivered licensing assessment of applied knowledge. This presentation demonstrates how the test quality is reviewed. Summary of work From the RCGP curriculum blueprint, the construction of test items is referenced to high quality evidence, peer reviewed and critically appraised before addition to the question bank. A standard process for question selection, proof reading and feedback to candidates supports its validity. New question formats include free text answers which enhance discrimination. Standard setting follows the modified Angoff process. Summary of results The test shows a high Cronbach’s alpha (0.88-0.92). Pre-trialling of questions, common to many postgraduate medical examinations, is deemed unnecessary with an alpha-coefficient for new questions in the most recent test > 0.8 (0.91 when adjusted for the test length). Computer delivered testing enables detailed analysis of patterns of timing and omission of items by candidates. Conclusions Systematic processes in test construction and computer delivery has enhanced continuous quality assurance of the AKT. Take-home messages Test performance relies on meticulous attention to detail in item writing and test construction. Use of un-trialled new questions enables reliable testing of emerging and changing knowledge.
Keywords:primary care, general practice, assessment, knowledge, examination, medical licensing
Subjects:A Medicine and Dentistry > A300 Clinical Medicine
Divisions:College of Social Science > School of Health & Social Care
ID Code:5053
Deposited By: Niro Siriwardena
Deposited On:10 Apr 2012 17:25
Last Modified:13 Mar 2013 09:06

Repository Staff Only: item control page