A recent study published in The BMJ that questions the value of health care accreditation misses the mark. The study—“Association between patient outcomes and accreditation in U.S. hospitals: Observational study”—has methodological problems that bias it against accredited hospitals and does not support its conclusions.*
Problems with the methodology
The study’s researchers examined risk-adjusted, 30-day mortality and readmission rates for selected medical and surgical conditions among Medicare inpatients using billing data, but they compared two radically different groups of hospitals:
- State-surveyed hospitals—93 percent with fewer than 100 beds
- Joint Commission-accredited hospitals—often larger, with 66 percent having more than 100 beds
The difference in hospital size and teaching status is an important distinction because it eliminates any chance of valid results. Large hospitals, especially major teaching hospitals, care for the most severely ill patients, and the study used claims data that had no information about patients’ severity of illness. In fact, studies have shown that comparing hospital mortality rates without markers of severity of illness yields erroneous results.†
Major teaching hospitals comprised 216 of the Joint Commission-accredited hospitals examined in The BMJ, but none in the state-surveyed group. Similarly, only four state-surveyed hospitals had 400 or more beds versus 403 Joint Commission-accredited hospitals. It is impossible to make valid comparisons when the two groups are so radically different.
Also, in assessing outcomes, The BMJ study selected six “common and costly surgical procedures.”* Four of the six—coronary artery bypass graft surgery, open repair of abdominal aortic aneurysm, endovascular repair of abdominal aortic aneurysm, and pulmonary lobectomy—are rarely performed in hospitals with fewer than 100 beds. Only one procedure—hip replacement—is performed with any volume in small hospitals. The authors ignored this discrepancy, combining all operations together in their analysis.
Therefore, the authors’ conclusion that surgical mortality rates were identical at accredited organizations and state-surveyed hospitals was based almost entirely on the lack of differences in mortality for hip surgery (mortality rates 0.5 percent for accredited hospitals and 0.6 percent for state-surveyed hospitals).* For three of the five other surgical procedures, the results actually favored accredited hospitals, but this information was shared only in an appendix of the study only.
Furthermore, in studies of mortality risks following hospital admission, measuring the severity of the illness responsible for admission is consistently the strongest predictor of risk of death. The BMJ study omitted these measures of severity in its attempt to adjust for differences in risks of death or readmission.
Benefits of accreditation
Despite the study’s methodology issues, the data offered two key findings:
- Joint Commission-accredited hospitals demonstrated lower mortality than state-surveyed hospitals at conventional levels of statistical significance.
- Patients with medical conditions admitted to Joint Commission-accredited hospitals had lower readmission rates.
The study authors considered these differences “modest.” However, when applied to the more than 3 million patients with the medical conditions addressed in this study, the results show that patients treated in Joint Commission-accredited hospitals experience 12,000 fewer deaths and 24,000 fewer readmissions.
The Joint Commission believes those findings make a difference to patients and, therefore, support the argument that the overall conclusion presented by the authors that there is no measurable difference on outcomes between hospitals accredited by The Joint Commission and those that are state-surveyed is not supported by the data presented in this study.
The accreditation process offers additional benefits to hospitals. For example, many accredited institutions have praised Joint Commission surveyors for identifying quality and safety issues at their facilities, providing education to help them go beyond compliance with Joint Commission standards, and helping them to move toward high reliability.
The Joint Commission recently conducted a systematic review of the literature on accreditation and certification to determine the value of accreditation. Of the 170 published articles with a statistical analysis, the following results were noted:‡
- 135 (79 percent) had positive findings; that is, favored accreditation or certification
- 22 (13 percent) were neutral
- 9 (5 percent) had mixed results
- 4 (2 percent) were negative
Joint Commission accreditation will not improve all aspects of quality, so studies need to use the right measures. Scientific studies aiming to determine the value of accreditation should focus on the outcomes that accreditation is designed to improve.
Disclaimer
The thoughts and opinions expressed in this column are solely those of Dr. Pellegrini and do not necessarily reflect those of The Joint Commission or the American College of Surgeons.
*Lam MB, Figueroa JF, Feyman Y, Reimold KE, Orav EJ, Jha AK. Association between patient outcomes and accreditation in U.S. hospitals: Observational study. The BMJ. October 18, 2018. Available at: www.bmj.com/content/bmj/363/bmj.k4011.full.pdf.
†Baker DW, Chassin MR. Holding providers accountable for health care outcomes. Ann Intern Med. 2017;167(6):418-423.
‡The Joint Commission. Value of accreditation and certification literature database. Available at: https://manual.jointcommission.org/Accreditation/. Accessed January 3, 2019
