“We expected to find generally high-quality evidence to support these changes, because these devices are important to health, and many are implanted and are difficult and/or dangerous to remove,” Dr. Redberg tells Reuters Health by email. “We were surprised to discover that relatively few studies were randomized or blinded, which means that it is not known if the device was better than an alternative treatment [or better than no treatment], and whether any purported beneficial effect was actually due to the well-documented placebo effect of procedures and devices.”
She notes, “In addition, we were disturbed to find numerous discrepancies between the number of patients enrolled and the number of patients with data reported, indicating that there is loss [or incompleteness] of data evaluated by the FDA. This raises concerns about missing data and selection bias.”
Of the 78 supplements studied, 71 were supported by a single clinical study, and half of these studies enrolled 185 or fewer patients. Information about age and sex was lacking for many patients.
One in 12 studies did not specify the outcome they were measuring, and studies that did specify the outcome typically left some patients out of the analysis, including one study that didn’t analyze the results for 91% of the patients.
“I think the public assumes that medical devices currently on the market, particularly high-risk devices, have been approved based on a high standard to show safety and effectiveness before doctors can recommend and implant them,” Redberg says. “Our findings show that this assumption is often incorrect.”
“The public should understand that the standard of evidence for device approval and modifications is not high and is lower than is generally required for drugs, which is two randomized controlled trials,” she says. “Patients may wish to discuss the evidence for any particular device with their doctor as part of the informed consent discussion of risks and benefits of any recommended device.”
Dr. Robert M. Califf from Duke University School of Medicine, Durham, N.C., who wrote an editorial related to these reports, told Reuters Health by email, “I do not believe this needs to be high on the list of public concerns, but it is a motivation to examine how we generate evidence about the risks and benefits of medical products, and how we come to understand comparative risks and benefits. The system has come a long way, but it has a ways to go.”3
