Observational studies of treatment outcomes must be viewed with caution, investigators urge. In a report published online April 21 in Cancer, researchers showed that selection bias continues to be a major problem in observational data.
“The strong yet implausible treatment effects observed in our analyses should reinforce the caution and modesty of all investigators assessing outcomes from observational data,” write the authors, led by Sharon Giordano, MD, from the University of Texas MD Anderson Cancer Center, in Houston.
However, health claims from observational data can persist long after being debunked in randomized controlled trials. According to a study published in December in the Journal of the American Medical Association, observational results are frequently cited in later articles even after trials contradicting the results have been published (JAMA. 2007;298:2517-2526).
This can lead eventually to a decrease in the frequency of citations, write those authors, led by Athina Tatsioni, MD, from the University of Ioannina, in Greece. However, this might occur with considerable delay, they note, and a segment of the literature continues to cite the articles.
In this new study, Dr. Giordano and her team encourage skepticism. And, in clinical situations where trials are nonexistent and observational studies are necessary, they suggest that disease-specific, other-cause, and overall survival be provided in studies of treatment outcomes.
Selection Bias Bypassing Safeguards
There is growing interest in the use of observational data to study cancer outcomes. This is driven in part by the availability of population-based data, such as that from the Surveillance, Epidemiology, and End Results (SEER) tumor registry in the United States, which has been linked to Medicare charge data.
These databases provide excellent external validity and they encourage the study of populations that often are not included in clinical trials, such as the elderly, minorities, and patients with comorbidities.
Large administrative datasets can also provide information on patterns of care and treatment adherence. They can be used to detect rare toxicities and can help researchers compare adverse events across different patient populations.
But recently, databases are being used to compare the effects of different treatments on overall survival, the researchers point out. This approach has been used across many tumor types, including breast, lung, colon, rectal, prostate, and ovarian.
The researchers presented several examples, including reanalyses of previously published data. They hypothesized that the usual means of dealing with selection biases, such as controlling for patient and tumor characteristics using multivariate and propensity analyses, would not eliminate improbable results.
Call to Include Disease-Specific, Other-Cause, and Overall Survival
“For the first example, we selected a situation in which we believed that selection biases might produce implausible results compared with results from a randomized controlled trial,” write the investigators. “For the second and third examples, we reanalyzed previously published data.” In all cases, they examined cancer mortality, noncancer mortality, and overall mortality.
“We reasoned that any real benefit of cancer therapy could be manifested only through differences in cancer-specific mortality,” they note. “Selection biases might result in differences in noncancer mortality that would be as great or greater than the differences in cancer mortality.” This would call into question the reliability of using mortality end points to assess treatment efficacy in nonrandomized data, they explain.
In their first analysis, the researchers looked at data on androgen deprivation in men with stage 3 prostate cancer. Randomized clinical trials have shown that treatment can improve survival.
When the investigators analyzed data from the SEER registry of more than 5000 men, they found that men treated with androgen deprivation actually had a higher risk for death from prostate cancer than men who did not receive therapy.
The researchers then reanalyzed data from a previously published study of more than 43,000 men with localized prostate cancer (JAMA. 2006;296:2683-2693). Like the original study, the researchers’ analysis revealed that men who were treated for prostate cancer with surgery or radiation experienced lower mortality rates.
However, they also found that in many cases, the cause of death was something other than prostate cancer, such as diabetes or pneumonia.
In the final example, the investigators reanalyzed data from a published study on the effects of fluorouracil-based chemotherapy in colon cancer (Ann Intern Med. 2002;136:349-357).
They came to the same conclusion as the original study — that chemotherapy for node-positive colon cancer is associated with improved survival. But they found that the link between the treatment and survival was strongest for noncancer deaths.
“Investigators clearly are aware of these potential biases and use statistical techniques to address them,” the researchers point out. Multivariate analyses, stratification, matching, restricting, and propensity analyses often are used to adjust for information available in the datasets, such as age, ethnicity, neighborhood socioeconomic level, and previous diagnoses, procedures, and hospitalizations. “Nevertheless,” they warn, “unmeasured confounders are likely to persist.”
The researchers have disclosed no relevant financial relationships.
Cancer. Published online before print April 21, 2008. Abstract
Reviewed by Dr. Ramaz Mitaishvili
