Dr. Caplan critiques the authors’ approach, explaining that they appear to have used the rating of evidence according to GRADE and back converted into the older ACC/AHA scheme. “But the two are not exactly the same, so they had to make a lot of assumptions, and it’s not exactly clear how they made those assumptions. That’s one issue.” However, he agrees with the general assessment that the guidelines are not predominantly supported by level A evidence (meaning RCTs). Dr. Caplan is coauthoring a response letter to the paper in JAMA Internal Medicine in collaboration with the current chair of the ACR Quality of Care Committee, John Fitzgerald, MD, PhD.
“These findings aren’t terribly surprising, and I don’t think they are unique to rheumatology, given a small discipline that is underfunded,” says Dr. Caplan. He points out that to conduct RCTs to address many of the questions included in guidelines would require immense amounts of funding that is often not available. He also notes it would be ethically challenging to pursue certain questions because existing effective medications might preclude a comparison with placebo, and thus only head-to-head data between agents are available.
Dr. Caplan also makes it clear that it is important to distinguish limits to the evidence base due to the state of the science from actual flaws in the guidelines themselves. “It’s not that the guidelines are flawed per se; it’s simply that they are the best available product for addressing the issue based on the limited evidence that is available. I would call a guideline flawed if it doesn’t adhere to a process or purports to make decision making in one way but really allows undue influence from pharma or allows votes in an inequitable manner. There are ways that guidelines can be flawed, but to call them flawed because the evidence base on which they’re established is limited is misleading.”
Unlike the ACC/AHA categorization of evidence used in the JAMA analysis, GRADE takes into account study quality when analyzing recommendations. Dr. Neogi points out that although RCTs provide the gold standard of evidence, they may have bias or other issues that can downgrade evidence quality. “On the other hand, a very well done observational study, a cohort study done very rigorously could be considered a slightly higher level of evidence because it dealt with all the different biases and nuances to make it more robust evidence. And the strength of the recommendation takes all that into consideration.” In other words, a trial that might technically qualify as level A evidence might not actually be as reliable as data from a study ranked as level B evidence under the ACA/AHA scheme.
