VLADGRIN / shutterstock.com
SAN DIEGO—The Next Accreditation System, the newest process for evaluating medical residency and fellowship programs and their participants, including future rheumatologists, was implemented in 2014 by the Accreditation Council for Graduate Medical Education (ACGME). The system introduced specialty-specific milestones designed to reflect significant points in professional development.
Milestones are defined by ACGME as a rubric for performance improvement in six core areas of medical practice: patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice. The milestones are accompanied by competency-based professional development outcomes that can be progressively demonstrated, documented and evaluated. They represent a change in how training programs approach curriculum development and the assessment of both trainees and training programs.
Through the evaluations of milestones by residents and fellows at the end of rotations, a lot of data are collected, evaluated and submitted to ACGME. The process is not just evaluative, but also formative, “providing granular detail of where trainees are at and how they can move forward,” Calvin R. Brown Jr., MD, professor of medicine and director of the rheumatology training program at Northwestern University, Chicago, said during the 2017 ACR/ARHP Annual Meeting Nov. 3–8. In this session, the Great Educational Debate: Milestones: Meaningful vs. Millstone, two veteran fellowship program directors, Dr. Brown and Simon Helfgott, MD, rheumatology fellowship director at Brigham & Women’s Hospital in Boston, explored the advantages and drawbacks of the training milestones approach for rheumatology fellowships.
For Dr. Brown, the problem with traditional graduate medical education (GME) is its dependence on what he described as a teabag model in which trainees are dipped into the hot water of the medical institution, with all of its faculty’s intellectual and scholarly rigor. After 24 months, when the tea is fully steeped, the trainee is deemed ready to practice medicine. “The teabag model is an antiquated approach,” Dr. Brown says. “I think we can do much better.”
ACGME’s new process calls for the specialty to put forth general competencies defining what a rheumatologist does.1 These competencies are spelled out through entrustable professional activities (EPAs) that describe the comprehensive abilities, knowledge and skills of a physician specialist who has attained mastery of the field.
“In rheumatology, we have defined 14 EPAs for our trainees,” Dr. Brown added. “The next question: How do we know our trainee is making progress on those 14 EPAs? The way we answer that is with the milestones.”
Each specialty’s Milestones Working Group developed milestones for its particular field. Pediatric rheumatology has 21 subspecialty milestones, and adult rheumatology has 23.
“Let’s talk about what a milestone is, centered around competencies and a very specific granular description of activities, to see if this individual is progressing toward competency,” Dr. Brown said. “I think these competencies are really important.” The purpose of milestones is to allow continuous monitoring of programs and trainees, with a rich descriptive developmental framework for clinical competency. This kind of purposeful evaluation of trainees’ abilities during GME is essential to their long-term success, with the ultimate goal of producing highly competent physicians, he said.
But what evidence exists that milestones are a meaningful alternative to ACGME’s previous processes for evaluating medical education? A growing body of literature supports milestones, Dr. Brown said. One recent article suggested that milestones are well suited to the needs and learning style of current learners from the millennial generation with its need for frequent formative feedback.2 Emergency medicine recently demonstrated validity for its milestones, although less data exist specific to rheumatology.3
ACGME’s recently issued the Milestones 2017 Annual Report, which summarizes the results of 48 peer-reviewed studies with data for the academic year July 2016 to June 2017. Across all specialties, the central tendency of the data shows general attainment of milestones across years in program. It also found variation in attainment across residents/fellows and programs that needs to be investigated further. The published evidence acknowledges a problem with inter-rater reliability for the milestones, Dr. Brown said. But ultimately, they have been shown to be meaningful and represent a step toward ensuring the collection and use of educational outcomes data that will provide a pathway toward improving evidence-based rheumatology education in the future.
Are Milestones Really Millstones for GME Faculty?
The biggest drawback to the milestones system, countered Dr. Helfgott, is its impracticality. The questions are time consuming and redundant. And often, faculty is unable to parse out meaningful differences in what the assessment questions are asking and what is expected from faculty evaluators. He called the ACGME milestones “millstones” around the necks of faculty evaluators and recommended changing trainee evaluation to make it simpler and easier for evaluators to use.
“These 24 questions that supervisors must assess and answer are where the problem lies,” Dr. Helfgott said. “In my department, faculty say they’re crazy. They ask me: ‘How do I recognize systems-based practice? Do we really need to subanalyze four different questions about professionalism?’ Because they are highly redundant and we don’t have time, there may be a tendency toward survey glaze and to give every trainee the same rating.”
Ultimately, Dr. Helfgott said, the vast majority of rheumatology fellows will move forward on schedule in their training, with maybe 1% turning out to be unsatisfactory or not appropriate for professional practice. “I dispute that milestones can effectively separate trainees on either end of the spectrum. I would argue these are not the tools we should be using.” A less cumbersome system could still identify that 1% without the onerous requirements of milestones for everyone else. Keep it simple, he urged, proposing a system that gives trainees an evaluation score of 1–5—perhaps with a positive/negative modifier—in six general areas of intellect, curiosity, clinical skills, procedural skills, scholarship and professionalism.
“The greatest problem we face in medicine is that we don’t really know the evidence,” Dr. Brown responded. He acknowledged the drawbacks of milestones, but added, “They will provide the basis of evidence that we need to move forward,” using data to drive future improvements in the program curriculum.
Larry Beresford is a freelance medical journalist in Oakland, Calif.
References
- Brown CR Jr, Criscione-Schreiber L, O’Rourke KS, et al. What is a rheumatologist and how do we make one? Arthritis Care Res (Hoboken). 2016 Aug; 68(8):1166–1172.
- Desy JR, Reed DA, Wolansky AP. Milestones and millennials: A perfect pairing—competency-based medical education and the learning preferences of Generation Y. Mayo Clinic Proceedings. 2017 Feb;92(2): 243–250.
- Beeson MS, Holmboe ES, Korte RC, et al. Initial validity analysis of the emergency medicine milestones. Acad Emerg Med. 2015 Jul;22(7):838–844.
