[News release] Credibility of Evidence: A Reconsideration of the Logic and Strength of Our Healthcare Decisions

From the 22 May 2015 HealthCare Blog post

By ROSS KOPPEL and STEVEN SOUMERAI

A few days ago, we wrote an editorial for US News and World Reports on the scant or dubious evidence used to support some healthcare policies (the editorial is reproduced in full below).  In that case, we focused on studies and CMS statements about a select group of Accountable Care Organizations and their cost savings. Our larger point however is about the need to reconsider the evidence we use for all healthcare-related decisions and policies. We argue that an understanding of research design and the realities of measurement in complex settings should make us both skeptical and humbled.  Let’s focus on two consistent distortions.

Evidence-based Medicine (EBM).  Few are opposed to evidence-based medicine.  What’s the alternative? Ignorance-based medicine? Hunches?  However, the real world applicability of evidence-based medicine (EBM) is frequently overstated. Our ideal research model is the randomized controlled trial, where studies are conducted with carefully selected samples of patients to observe the effects of the medicine or treatment without additional interference from other conditions. Unfortunately, this model differs from actual medical practice because hospitals and doctors’ waiting rooms are full of elderly patients suffering from several co-morbidities and taking about  12 to 14 medications, (some unknown to us). It is often a great leap to apply findings from a study under “ideal conditions” to the fragile patient. So wise physicians balance the “scientific findings” with the several vulnerabilities and other factors of real patients.  Clinicians are obliged to constantly deal with these messy tradeoffs, and the utility of evidence-based findings is mitigated by the complex challenges of the sick patients, multiple medications taken, and massive unknowns. This mix of research with the messy reality of medical and hospital practice means that evidence, even if available, is often not fully applicable. 

Relative vs. Absolute Drug Efficacy:

Let’s talk a tiny bit about arithmetic. Say we have a medication (called X) that works satisfactorily for 16 out of a hundred cases, i.e., 16% of the time.  Not great, but not atypical of many medications.  Say then that another drug company has another medication (called “Newbe”) that works satisfactorily 19% of the time. Not a dramatic improvement, but a tad more helpful (ignoring how well it works, how much it costs, and if there are worse side effects).  But what does the advertisement for drug “Newbe” say?   That “Newbe” is almost 20% better than drug “X.” Honest. And it’s not a total lie.  Three percent (the difference between 16% and 19%) is 18.75%, close enough to 20% to make the claim legit. Now, if “Newbe” were advertised as 3% better (but a lot more expensive) sales would probably not skyrocket. But at close to 20% better, who could resist?   

Policy:  So what does this have to do with healthcare policy?  We also want evidence of efficacy with healthcare policies but it turns out that evaluation of these interventions and policies is often harder to do well than are studies of drugs. Interventions and policies are introduced into messy pluralistic systems, with imprecise measures of quality and costs, with sick and not-so-sick patients, with differing resources and populations, with a range of payment systems, and so on and so on. Sometimes, randomized controlled trials are impossible.  But sometimes they are possible but difficult to effect. Nevertheless, we argue they are usually worth the effort. Considering the billions or trillions of dollars involved in some policies (e.g., Medicare changes, insurance rules) the cost is comparatively trivial.

But there’s another question: What if a decent research design is used to measure the effects of a large policy in a select population but all you get is a tiny “effect?”  What do we know? What should policymakers do? Here’s what we wrote in our recent editorial in the US News and World Report….

 

Accountable care organization (ACO) and medical home differences

By Kevin Fickenscher, MD at KevinMd.com

In the great healthcare alphabet soup, it’s easy to lose sight of the differences between proposed solutions for making healthcare more efficient and effective.

Rather than tackling payment reform in isolation of care delivery, Accountable Care Organizations (ACOs) and Medical Homes offer a consolidated approach to both issues. While the models are still developing, various pilot programs are being implemented around the country….

 

Read the article

Related articles

• What would an ACO would look like if it was truly patient centered? (kevinmd.com)
• The value of Accountable Care Organizations (ACOs) (kevinmd.com)
• AMA: Revisions, clarity needed before physicians form Medicare ACOs (kevinmd.com)
• Pay attention when your hear “ACO” or “Accountable Care Organization” (quinnscommentary.com)
• Post-HMO Health Care: Are ACOs the Answer? (time.com)
• Hitting the ACO Ground Running – Pt. 2 (primarycarefirst.wordpress.com)
• Post-HMO Health Care: Are Accountable Care Organizations the Answer? (time.com)
• Post-HMO Health Care: Are Accountable Care Organizations the Answer? (time.com)