Monday, September 6, 2010

Quality assessment in primary care: an imperfect science

Several years ago, when I was a very junior faculty member at a well-regarded medical school, I was asked to give a lecture to first-year students about, among other things, medical licensure and specialty board certification. I considered this task to be unenviable because these processes 1) are boring; and 2) don't happen until after medical school graduation, and thus are probably the last things on the minds of first-years who are spending most of their waking hours memorizing anatomical and biochemical trivia.

So, to get them interested, I began the lecture with a question: how does society assure that medical schools like their own produce "good" doctors who are worthy of the trust that patients have traditionally placed in them? More to the point, how do processes such as licensure and board certification assure the quality of practicing physicians? As it turns out, not well at all. Only the worst offenders (doctors who abuse or divert prescription painkillers, for example) are typically disciplined by state medical boards, and there was no publicly available information then or now that would allow an average citizen to determine how his or her doctor measured up against others.

More recently, in an attempt to provide feedback about their performance, some institutions have started giving primary care (and other) physicians "report cards" that summarize their performance on objective measures of care quality; for example, what percentage of patients with a hypertension diagnosis met blood pressure goals at their most recent visits. In theory, this type of reporting should motivate physicians who receive lower scores than their peers to improve their practices, and, therefore, the quality of care they provide their patients.

A few weeks ago, general internist Danielle Ofri, MD, PhD (a frequent guest contributor to Common Sense Family Doctor) wrote a commentary in the New England Journal of Medicine that portrayed current methods for assessing quality in individual primary care physicians as inadequate and potentially misleading. Ofri writes:

Quantitative analysts ... will see it as a sign of medical arrogance that physicians insist that everyone simply trust us to do the right thing because we are such smart and noble people. I've always wanted to ask these analysts how they choose a physician for their sick child or ailing parent. Do they go online and look up doctors' glycated hemoglobin stats? Do they consult a magazine's Best Doctor listing? Or do they ask friends and family to recommend a doctor they trust? That trust relies on a host of variables - experience, judgment, thoughtfulness, ethics, intelligence, diligence, compassion, perspective - that are entirely lost in current quality measures. These difficult-to-measure traits generally turn out to be the critical components in patient care.

Notwithstanding the firestorm of blog responses that Dr. Ofri's editorial provoked (most of whom misunderstood her as arguing for discarding quality measurement in general, rather than the crude tools being used to measure it in primary care), she has a point. Quality measurement in primary care needs to move beyond what is easy to measure. For example, current metrics target underuse of health services (e.g. not ordering enough recommended screening tests) rather than overuse (e.g. ordering tests too often or for no good reason). They have a myopic focus on individual chronic conditions rather than assessing overall quality of care for a patient with multiple diseases, where some disease-oriented goals might be inappropriate. They provide retrospective information rather than real-time feedback. And they fail to measure at all many of the personal qualities that most patients would agree are essential to a good primary care physician.

This doesn't mean that we should throw up our hands and stop trying. Not at all - in fact, I think it's critical for family physicians to be open to novel methods of quality measurement to in order to support our role in the changes that will be taking place in the U.S. health system over the coming years and decades. As I've said in this blog again and again, there will be no real health reform without meaningful investment in good primary care. To that statement I now add this corollary: there will be no meaningful investment in good primary care until we develop much better tools to measure what, exactly, good primary care is.


  1. This faith in measurement tools is misguided. Human relationships cannot be successfully subjected to them. What can be measured, basic clinical knowledge and maybe basic responsibility: are you respectful, on-time, all that can be measured. Primary care is part science, part art, part faith. The provider is part scientist, part adjunct family member, part artist, part priest. Relationships between doctor and patient are like the different images in a kaleidescope and one should not try to force them into some kind of universal mold beyond that of basic competence.

  2. I believe we may actually have the tools necessary to measure good primary care. Many countries track their primary care delivery very well with patient-oriented evidence. It is the United States that doesn't look at the quality of primary care in an effective way. We know a lot about subspecialty care but we don't know much about primary care in the US simply because we fail to ask about it.
    Our ability to assimilate data about overuse is simply "underutilized." For example, we do know that overreliance on subspecialists for follow-up care results in increased costs with lower quality, higher morbidity, and higher mortality. We also know that complication rates are directly related to the number of physicians taking care of a person after correcting for severity of illness. This data is very easy to accumulate and is an excellent example of the ability to look at lack of stewardship of scarce resources. Interestingly, one's overuse is another's underuse because the resources are actually finite. So it really depends on your paradigm.

  3. Quality Questions

    Do medical schools choose the physicians most likely to have the people skills necessary for physicians or does their focus on academic preparation get in the way of people skills selections? Medical schools focus on academic preparation. Parents of children with top scores become quite irate when their children fail to gain admission. Top scores have yet to be linked to better quality in physicians. The case can be made that those admitted are the least like normal Americans and are least likely to be aware of the needs of normal Americans.

    Why do medical schools fail to focus on a strategy of admissions that focuses on sufficient academics with superior people skills instead of the current focus on superior academics with some consideration of people skills? Current medical education fails to focus on the physicians needed for most Americans to have basic health access. With top scores, top grades, and most prestigious colleges and parents shaping admission, the lowest levels of basic health access workforce are guaranteed.

    Does medical education help screen out the physicians most likely to have quality problems before medical school? Physicians with communication skills deficits are known to have more adverse events. Despite this understanding, efforts to screen out these medical students during medical school or prior to entry remain limited.

    Why do studies continue to promote physicians as quality determinants and why do studies blame physicians for quality problems when physicians are such a small portion of the quality equation? An appropriate quality equation for use in regressions would be

    Quality = patient factors + health team factors + system factors + physician

    With highest status patients, the patient and health care team and system factors cancel out leaving the physician factor.

    With lower and middle income patients, the patient factors (social determinants of health) are dominant and also impact health care team and system designs. The physician is a small component in such situations

    This illustrates the folly of pay for performance or other attempts to link pay to quality. Quality is linked primarily to social determinants. The physicians associated with lower and middle income patients, already disadvantaged by the design and lack of health care team and lack of health care spending, will be disadvantaged further by serving where most needed.

    I have a more important question. How do I convince medical students to keep learning about people skills and clinical skills when the physicians seen at hospitals do 2 minute examinations and 5 minute total interactions with patients with numerous omissions and errors in clinical skills and communication skills?

    Three month medical students in their first OSCE perform at 85 to 95% levels in basic history and physical examination skills. The physicians observed by me in recent encounters performed at 20 to 40% levels in their real life OSCE clinical encounters. Missing pieces involve introductions, asking permissions, making the patient comfortable, past history, social history, and review of systems. The examinations were totally deficient. These are physicians that would not trip any quality indicators. Their documentation is likely to be very good. But they had the bad fortune of interacting with a physician evaluator.

  4. A nice work, much needed at the current time

    "In summary, our study demonstrates that patient panel characteristics are associated with the relative measured quality of physicians within a large academic primary care network. Adjustment for differences in patient panel characteristics resulted in significant reclassification of top tier vs bottom tier physicians. To the extent that health systems reward physicians for higher measured quality of care, lack of adjustment for patient panel characteristics may penalize physicians for taking care of more vulnerable patients, incentivize physicians to select patients to improve their quality scores, and result in the misallocation of resources away from physicians taking care of more vulnerable populations. Conversely, adjustment for patient panel characteristics may remove the incentive to improve care or may inappropriately reward lower-quality physicians caring for more vulnerable patients. Efforts to improve quality of care must address both fairness of physician clinical performance assessment and the design of incentive schemes to both provide equitable distribution of resources and reduce disparities in care for vulnerable patients."

    Notice that attempts to separate translation services, types of clinicians, or various other interventions are extremely difficult to impossible due to the complexities of the studies required. Pooled data involving millions of patients would be needed for many of these comparisons.

    This should not prevent attempts and understanding, but studies clearly must be conservative regarding various claims that are commonly made.

  5. Thanks, Bob. I read the JAMA study, and am planning to discuss it further in a future post. Clearly it suggests both the need to exercise caution when measuring quality of care in different socioeconomic environments and begs the question of what clinical outcomes are or aren't under a primary care clinician's control.