Abstracts
Abstract
The purpose of medical licensing examinations is to protect the public from practitioners who do not have adequate knowledge, skills, and abilities to provide acceptable patient care, and therefore evaluating the validity of these examinations is a matter of accountability. Our objective was to discuss the Medical Council of Canada's Qualifying Examinations (MCCQEs) Part I (QE1) and Part II (QE2) in terms of how well they reflect future performance in practice.
We examined the supposition that satisfactory performance on the MCCQEs are important determinants of practice performance and, ultimately, patient outcomes. We examined the literature before the implementation of the QE2 (pre-1992), post QE2 but prior to the implementation of the new Blueprint (1992-2018), and post Blueprint (2018-present).
The literature suggests that MCCQE performance is predictive of future physician behaviours, that the relationship between examination performance and outcomes did not attenuate with practice experience, and that associations between examination performance and outcomes made sense clinically.
While the evidence suggests the MCC qualifying examinations measure the intended constructs and are predictive of future performance, the validity argument is never complete. As new competency requirements emerge, we will need to develop valid and reliable mechanisms for determining practice readiness in these areas.
Résumé
L’objectif des examens donnant lieu au titre de Licencié du Conseil médical du Canada est de protéger le public en garantissant que les praticiens possèdent les connaissances, les habiletés et les aptitudes nécessaires pour offrir des soins satisfaisants aux patients; par conséquent, l’évaluation de la validité de ces examens est une question de responsabilité. Notre objectif était de déterminer dans quelle mesure l’Examen d’aptitude du Conseil médical du Canada (EACMC), partie I, et l’EACMC, partie II reflètent le rendement futur des médecins dans leur pratique.
Nous avons examiné l’hypothèse selon laquelle des résultats satisfaisants aux EACMC sont des déterminants importants du rendement dans la pratique future et, ultimement, des résultats rapportés pour les patients. Nous avons examiné les écrits publiés avant l’introduction de l’EACMC,-partie II (avant 1992), post EACMC-partie II ci mais avant l’adoption du Plan directeur (1992-2018), ainsi que ceux publiés post adoption du Plan directeur (2018-présent).
La littérature suggère que la performance à l’EACMC permet de prédire les comportements futurs des médecins, que le rapport entre la performance à l’examen et les résultats dans la pratique perdure, et que les associations entre la performance à l’examen et les résultats sont liés sur le plan clinique.
Bien que les données probantes indiquent que les examens d’aptitude du CMC (EACMC) mesurent les concepts visés et permettent de prédire le rendement des médecins dans leur pratique future, la démarche de validité n’est pas complète. Au fur et à mesure que de nouvelles exigences en matière de compétences émergent, nous devrons élaborer des mécanismes valides et fiables pour déterminer la capacité à exercer dans ces domaines
Appendices
Bibliography
- Wenghofer EF. Research in medical regulation: an active demonstration of accountability. J Med Reg. 2015;101(3):13-7. https://doi.org/10.30770/2572-1852-101.3.13
- Swanson DB, Roberts TE. Trends in national licensing examinations in medicine. Med ed. 2016;50(1):101-14. https://doi.org/10.1111/medu.12810
- Bobos P, Pouliopoulou DV, Harriss A, Sadi J, Rushton A, MacDermid JC. A systematic review and meta-analysis of measurement properties of objective structured clinical examinations used in physical therapy licensure and a structured review of licensure practices in countries with well-developed regulation systems. PloS one. 2021;16(8):e0255696. https://doi.org/10.1371/journal.pone.0255696
- Archer J, Lynn N, Coombes L, Roberts M, Gale T, Regan de Bere S. The medical licensing examination debate. Regul Gov. 2017;11(3):315-22. https://doi.org/10.1111/rego.12118
- Archer J, Lynn N, Roberts M, Coombes L, Gale T, de Regand Bere S. A systematic review on the impact of licensing examinations for doctors in countries comparable to the UK. Plymouth: Plymouth University. 2015.
- Schuwirth L. National licensing examinations, not without dilemmas. Med ed. 2016;50(1):15-7. https://doi.org/10.1111/medu.12891
- Tsichlis JT, Del Re AM, Carmody JB. The past, present, and future of the united states medical licensing examination step 2 clinical skills examination. Cureus. 2021;13(8). https://doi.org/10.7759/cureus.17157
- Wormald BW, Schoeman S, Somasunderam A, Penn M. Assessment drives learning: an unavoidable truth? Anat sci educ. 2009;2(5):199-204. https://doi.org/10.1002/ase.102
- Murray DJ, Boulet JR. Anesthesiology Board Certification Changes: A Real-time Example of "Assessment Drives Learning." Anesthesiology. 2018;128(4):704-6. https://doi.org/10.1097/ALN.0000000000002086
- Norman G, Neville A, Blake JM, Mueller B. Assessment steers learning down the right road: impact of progress testing on licensing examination performance. Med teach. 2010;32(6):496-9. https://doi.org/10.3109/0142159X.2010.486063
- Boulet J, van Zanten M. Ensuring high-quality patient care: the role of accreditation, licensure, specialty certification and revalidation in medicine. Med ed 2014: 48: 75-86 https://doi.org/10.1111/medu.12286
- Schuwirth L. The need for national licensing examinations. Med ed. 2007;41(11):1022-3. https://doi.org/10.1111/j.1365-2923.2007.02856.x
- Boulet JR, Durning SJ. What we measure… and what we should measure in Med ed. 2019;53(1):86-94. https://doi.org/10.1111/medu.13652
- Archer J, Lynn N, Coombes L, et al. The impact of large scale licensing examinations in highly developed countries: a systematic review. BMC med ed. 2016;16(1):212-212. https://doi.org/10.1186/s12909-016-0729-7
- Roberts WL, Gross GA, Gimpel JR, et al. An investigation of the relationship between COMLEX-USA licensure examination performance and state licensing board disciplinary actions. Acad Med. 2020;95(6):925-30. https://doi.org/10.1097/ACM.0000000000003046
- Cuddy MM, Young A, Gelman A, et al. Exploring the relationships between USMLE performance and disciplinary action in practice: a validity study of score inferences from a licensure examination. Acad Med. 2017;92(12):1780-5. https://doi.org/10.1097/ACM.0000000000001747
- Norcini JJ, Boulet JR, Opalek A, Dauphinee WD. The relationship between licensing examination performance and the outcomes of care by international medical school graduates. Acad Med. 2014;89(8):1157-62. https://doi.org/10.1097/ACM.0000000000000310
- Hudson KM, Feinberg G, Hempstead L, Zipp C, Gimpel JR, Wang Y. Association between performance on COMLEX-USA and the American College of Osteopathic Family Physicians in-service examination. J Grad Med Ed. 2018;10(5):543-7. https://doi.org/10.4300/JGME-D-17-00997.1
- Arnhart KL, Cuddy MM, Johnson D, Barone MA, Young A. Multiple United States medical licensing examination attempts and the estimated risk of disciplinary actions among graduates of US and Canadian medical schools. Acad Med. 2021;96(9):1319-23. https://doi.org/10.1097/ACM.0000000000004210
- Patel MD, Heitkamp DE, Jordan SG. Re: The relationship between US medical licensing examination step scores and ABR core examination outcome and performance: a multi-institutional study. J Am Coll Radiol. 2021;18(1):8-9. https://doi.org/10.1016/j.jacr.2020.10.018
- Medical Council of Canada. The MCC suspends the delivery of the MCCQE Part II [Internet]. Available from: https://mcc.ca/news/mcc-suspends-delivery-of-mccqe-part-ii/ [Accessed on Nov 7, 2021].
- Medical Council of Canada. Medical Council of Canada Qualifying Examination (MCCQE) Part 1 Annual Technical Report 2019 [Internet]. 2019 Apr. Available from: https://mcc.ca/media/MCCQE-Part-I-Annual-Technical-Report-2019.pdf [Accessed on Apr 1, 2022].
- Medical Council of Canada. Medical Council of Canada Qualifying Examination (MCCQE) Part II Annual Technical Report 2018 [Internet]. 2018 Oct. Available from: https://www.mcc.ca/media/MCCQE-Part-II-Technical-Report-Oct-2018.pdf [Accessed on Apr 1, 2022].
- Medical Council of Canada. Blueprint Project: Examinations Blueprint and Content Specifications - Final Version [Internet]. 2014 Sep. Available from: https://mcc.ca/media/Blueprint-Report.pdf [Accessed on Nov 7, 2021].
- Pugh DM, Wood TJ, Boulet JR. Assessing procedural competence: validity considerations. Simul Healthc 2015;10(5):288-94. https://doi.org/10.1097/SIH.0000000000000101
- Cook DA, Brydges R, Ginsburg S, Hatala R. A contemporary approach to validity arguments: a practical guide to Kane's framework. Med ed 2015: 49: 560-575 https://doi.org/10.1111/medu.12678
- Standards for educational and psychological testing. Washington, DC: American Educational Research Association; 2014.
- Kane MT. Validating interpretive arguments for licensure and certification examinations. Eval Health Prof. 1994;17(2):133-59. https://doi.org/10.1177/016327879401700202
- Kane M. Validating high-stakes testing programs. Educ Meas: Issues Pract. 2002;21(1):31-41. https://doi.org/10.1111/j.1745-3992.2002.tb00083.x
- Clauser B, Margolis M, Swanson D. Issues of validity and reliability for assessments in medical education. In: Holmboe ES, Durning SJ, Hawkins RE, editors. Practical guide to the evaluation of clinical competence. Second edition. Philadelphia, PA: Elsevier; 2018. p. 22-36.
- Kane M. The argument-based approach to validation. School Psych Rev. 2013;42(4):448-57. https://doi.org/10.1080/02796015.2013.12087465
- Tamblyn R. Is the public being protected? prevention of suboptimalmedical practice through training programs and credentialing examinations. Eval Health Prof. 1994;17(2):198-221. https://doi.org/10.1177/016327879401700205
- Tamblyn R, Abrahamowicz M, Bartlett G, et al. Executive summary. Medical Council of Canada: The Quebec-Ontario Follow-up Study of the Association between Scores Achieved on the MCCQE Part II Examination and Performance in Clinical Practice [Internet]. 2011. Available from: https://mcc.ca/media/Tamblyn_Score-Association_MCCQE-Part-II_Clinical-Practice-Performance_2009.pdf [Accessed on Jun 5, 2021].
- Tamblyn R, Abrahamowicz M, Dauphinee D, et al. Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities. JAMA. 2007 Sep 5;298(9):993-1001. https://doi.org/10.1001/JAMA.298.9.993
- Wenghofer E, Klass D, Abrahamowicz M, et al. Doctor scores on national qualifying examinations predict quality of care in future practice. Med ed. 2009;43(12):1166-73. https://doi.org/10.1111/j.1365-2923.2009.03534.x
- Tamblyn R, Abrahamowicz M, Dauphinee D, et al. Influence of physicians' management and communication ability on patients' persistence with antihypertensive medication. Arch Intern Med. 2010;170(12):1064-72. https://doi.org/10.1001/archinternmed.2010.167
- Kawasumi Y, Ernst P, Abrahamowicz M, Tamblyn R. Association between physician competence at licensure and the quality of asthma management among patients with out-of-control asthma. Arch Intern Med. 2011;171(14):1292-4. https://doi.org/10.1001/archinternmed.2011.319
- Cadieux G, Abrahamowicz M, Dauphinee D, Tamblyn R. Are physicians with better clinical skills on licensing examinations less likely to prescribe antibiotics for viral respiratory infections in ambulatory care settings? Med care. 2011;156-65. https://doi.org/10.1097/MLR.0b013e3182028c1a
- Meguerditchian A-N, Dauphinee D, Girard N, et al. Do physician communication skills influence screening mammography utilization? BMC. 2012;12(1):1-8. https://doi.org/10.1186/1472-6963-12-219
- Sherbino J, Dore KL, Wood TJ, et al. The relationship between response time and diagnostic accuracy. Acad Med. 2012;87(6):785-91. https://doi.org/10.1097/ACM.0b013e318253acbd
- De Champlain AF, Streefkerk C, Roy M, Tian F, Qin S, Brailovsky C. Predicting family medicine specialty certification status using standardized measures: for a sample of international medical graduates engaged in a practice-ready assessment pathway to provisional licensure. J Med Reg. 2014;100(4):8-16. https://doi.org/10.30770/2572-1852-100.4.8
- Tamblyn R, Abrahamowicz M, Dauphinee WD, et al. Association between licensure examination scores and practice in primary care. JAMA. 2002;288(23):3019-26. https://doi.org/10.1001/JAMA.288.23.3019
- Cadieux G, Tamblyn R, Dauphinee D, Libman M. Predictors of inappropriate antibiotic prescribing among primary care physicians. CMAJ. 2007;177(8):877-83. https://doi.org/10.1503/CMAJ.070151
- De Champlain AF, Ashworth N, Kain N, Qin S, Wiebe D, Tian F. Does pass/fail on medical licensing exams predict future physician performance in practice? a longitudinal cohort study of Alberta physicians J Med Reg. 2020;106(4):17-26. https://doi.org/10.30770/2572-1852-106.4.17
- Kane MT, Crooks TJ, Cohen AS. Designing and evaluating standard-setting procedures for licensure and certification tests. Adv Health Sci Educ. 1999;4(3):195-207. https://doi.org/10.1023/A:1009849528247
- Price T, Lynn N, Coombes L, Roberts M, Gale T, de Bere SR, et al. The international landscape of medical licensing examinations: a typology derived from a systematic review. International J health policy and management. 2018;7(9):782. https://doi.org/10.15171/ijhpm.2018.32
- Breithaupt K. Medical Licensure Testing White Paper for the Assessment Review Task Force of the Medical Council of Canada. 2011.
- McConnell M, Gu A, Arshad A, Mokhtari A, Azzam K. An innovative approach to identifying learning needs for intrinsic CanMEDS roles in continuing professional development. Med ed online. 2018;23(1):1497374. https://doi.org/10.1080/10872981.2018.1497374