Friday, November 18, 2011

"Pure Custer": our obsession with a flawed screening test

In the face of accumulating evidence and a U.S. Preventive Services Task Force finding that PSA screening for prostate cancer does more harm than good, the most frequent response I hear from physicians who continue to defend the test is that PSA is all we have, and that until a better test is developed, it would be "unethical" to not offer men some way to detect prostate cancer at an asymptomatic stage. (However, these physicians for the most part don't question the ethics of not offering women screening for ovarian cancer, which a recent randomized trial concluded provides no mortality benefit but causes considerable harms from diagnosis and treatment.)

I'm currently reading historian Stephen Ambrose's dual biography of Oglala Sioux leader Crazy House and Civil War cavalry general George Armstrong Custer, whose troops were routed by the Sioux at the famous Battle of Little Bighorn in 1876. One premise of the book is that the same aggressive instincts that served Custer so well during the Civil War - to always attack, even when the strength and disposition of his enemy was unknown - became fatal flaws when he became an "Indian fighter." For most of his post-Civil War career, Custer and his men blundered around the Great Plains looking for someone to fight, and not particularly caring if the Indians he engaged in battle were actually at war with the U.S. Army. In one telling description of Custer's first major Western engagement, Ambrose writes:

Here was audacity indeed. ... Custer had no idea in the world how many Indians were below him, who they were, or where he was. His men and horses were exhausted. ... He was going to attack at dawn from four directions at once. He had made no reconnaissance, held nothing back in reserve, was miles away from his wagon train, and had ordered the most complex maneuver in military affairs, a four-pronged simultaneous attack. It was foolish at best, crazy at worst, but it was also magnificent and it was pure Custer.

If readers of American Indian descent will kindly forgive my making this analogy with their 19th century ancestors, this passage is strikingly similar to the way we diagnose and manage prostate cancer. The vast majority of American Indians by this time had either signed peace treaties or were content to leave settlers alone. Under pressure to "do something" about a few troublesome tribes, however, the U.S. Army sent the overaggressive Custer out to do battle with whatever "warriors" he could find, assuming that in the process he would either kill, capture, or scare off those who aimed to do them harm.

That's pretty much what we do by deploying the PSA test to screen for prostate cancer. We cast as wide a net as possible, doing harm at every step of the way: false positives, adverse effects of prostate biopsies, and overdiagnosis and overtreatment of abnormal-appearing cells that we identify - usually inaccurately - as potentially lethal. For every man whose life may be extended by treatment, 30 to 50 will be treated for no benefit, and 10 to 20 will sustain permanent physical harm. And our continuing obsession with this flawed screening test not only flies in the face of evidence, it's pure Custer.


  1. Dr. Lin:

    While I agree that PSA screening presents some difficult risk tradeoffs, I think the statistics you present understate the relative benefits vs. costs.

    Your statistics appear to be based on the European study's estimate that the NNT (number needed to treat) is 48 treated to 1 life saved. But this is as of a 9 year average followup. If one looks at the European study, the survival probabilities in the screening vs. control arms only begin to diverge as of 7 years after screening begins, and are clearly widening their divergence after 9 years.

    One study has found that as of 12 years (the most years of followup possible with any reasonable sample size in the European study), the NNT is 18.

    Another study concludes that the 20% reduction in prostate cancer mortality as of a 9 year average followup in the European study is probably over 50% as of 12 years.

    A just-published study uses the European study results to project mortality over 25 years, and finds a NNT of 9 or less.

    Therefore, the ratio of lives saved to side effects is considerably higher than has been portrayed in much of this discussion.

    By the way, I am NOT a urologist. Rather, I am a prostate cancer survivor who is also an economist. I have tried to use my econometrics training to be as educated as possible about the statistics on this disease.

  2. Thanks for these thoughtful comments. I agree that it's possible that longer follow-up of the ERSPC study may show a decrease in the number needed to treat to prevent one PrCa-related death, which is why I gave a range of 30-50 rather than the NNT of 48 quoted in their 2009 NEJM publication. However, I seriously doubt that it will be much lower than 30, as this JNCI analysis found that even if the entire decline in U.S. PrCa mortality was attributed to screening and treatment of screen-detected cancers, the NNT would still be greater than 20 (and that's assuming NO improvements in treatment effectiveness over 25 years, which is doubtful):

    Also, I don't think that you can ignore the results of the U.S. PLCO study. Yes, there was lots of PSA testing in the "control" group and the finding of no benefit had wide confidence intervals, but they still detected thousands more cases of PrCa in the screened group, which should have at least showed a trend toward benefit if screening "works." And actually, if you break down the ERSPC study into its component parts, the finding of benefit is driven almost entirely by the huge benefit (44% PrCa mortality reduction) seen in the Swedish center. Why PrCa screening should be so incredibly effective in Sweden but not in neighboring Finland is a mystery that calls into question the overall result of the trial.

    My full analysis of the screening trials is now posted on the USPSTF website:

  3. Dr. Lin:

    Thank you for your response.

    We could get into discussing the merits of the studies I cite versus the studies you cite, but I think the real issue is how we should deal with relative medical risks, along with the additional uncertainty due to dueling studies.

    The USPSTF’s draft recommendation gives PSA screening a D rating. This means that the USPSTF is saying that there is moderate or high certainty that harms of screening outweigh benefits. I simply don’t see how such a rating is justified by the scientific evidence. Given the variation in the results of various studies, and the variation in the values that individual men place on risk of death versus risk of side-effects, I think it is unreasonable to talk about “certainty”, even moderate certainty, that harms outweigh benefits.

    Therefore, for example, there are some good studies that find an NNT of 48 after 9 years of follow-up, 18 after 12 years of follow-up, and possible an NNT of 9 in the long-term. I think it is reasonable to attach some non-negligible probability to the proposition that these studies have their numbers right. Of course, there is some non-zero probability that these studies are wrong. However, it is by no means “certain” that these numbers are wrong.

    Some men may view these NNTs as justifying both screening and treatment. Others will not, based on their individual values.

    For example, one way that these NNTs could be achieved is if treatment prompted by screening reduces the probability of death by 2% after 9 years, 5% after 12 years, and by 10% in the long-term. If so, this reduced risk of death must be compared with a 30 to 50% risk of serious side-effects.

    Some men would rather accept these 2/5/10% risks of death rather than the 30/50% risks of serious side-effects. Other men have the opposite preferences, and would view the reduced risk of death as outweighing the risk of side-effects.

    In any event, I don’t see how there is “certainty” that harms outweigh benefits for each individual man. I suspect a non-trivial fraction of men would choose reduced risk of death, and a non-trivial fraction would choose reduced risk of side-effects.

    Therefore, I think it is far more reasonable for the USPSTF to give PSA screening a C rating, which says that PSA screening should be based more on individual situations, which includes the preferences of individual men over these relative risks.


    Tim Bartik

    1. "the variation in the values that individual men place on risk of death versus risk of side-effects"

      But that is not the issue because the "side-effects" also include death and unless there is a demonstrated improvement in overall survival (which there isn't) then we don't know whether the number of lives that are possibly saved from prostate cancer are not lost from another cause.

      So the issue is not one of death vs. non-death death side-effects. This is a common misrepresentation.