Reporting of Multivariable Methods in the Medical Literature

Introduction

Clinicians are expected to interpret the results of studiesfound in the medical literature, and multivariable statistical techniques are often used to assess complex associations.^1-3Although generally accepted methodological criteria exist for the application of multivariable analysis,^1,2these criteria may not always be applied, or at least not reported. For clinicians who encounter such analyses, medical training offers little instruction in multivariable methods.^4,5Accordingly, if authors do not conduct and report the application of methodological criteria appropriately, the results of a study may be misinterpreted or perhaps be incorrect.¹

The objectives of this review were to (1) assess the frequency of multivariable methods reported in the medical literature, (2) quantify reporting of methodological criteria applicable to most multivariable models, and (3) determine if assumptions specific to logistic regression or proportional hazards analysis (Cox regression) were reported.

RESULTS

A total of 452 abstracts of original research articles (listed at www.cerc.med.va.gov) were reviewed, with 26% (n = 119) from British Medical Journal, 23% (n = 105) from Journal of the American Medical Association, 22% (n = 100) from New England Journal of Medicine, 20% (n = 90) from Lancet, and 8% (n = 38) from Annals of Internal Medicine (published semimonthly). Multivariable methods were reported in 60% (n = 272) of the articles, including 28% (n = 77) using more than one multivariable method; 2% (n = 9) reported a multivariable analysis for bivariate ("unadjusted") purposes only. For the elements included on the data extraction form, the average percent agreement was 96.9%, and the average κ statistic was 0.91, indicating "almost perfect" agreement.⁹

As shown in Table 1, logistic regression (33%, n = 89/272) and proportional hazards analysis (28%, n = 76/272) were the most frequently reported methods. Other less common methods were found in 8% (n = 23) of the studies, including Weibull regression¹⁰and accelerated failure time models.^11,12Three percent of the studies (n = 7) used an unspecified multivariable modeling technique that precluded further review; 1 study reported the use of more than 5 different methods.

Table 2 shows the adequacy of reporting of the 265 articles with clearly stated multivariable methods. The coding scheme for variables was described in 84% (n = 222) of the studies evaluated. The numbers of events per variable were described in 79% (152/192) of studies with categorical outcome events, and 45% (n = 118) of the studies reported data on testing for interaction terms. The model selection process was described in 15% of studies (n = 41); assessing model "fit" or other mechanisms of model validation were described in 10% of studies (n = 27), including techniques such as bootstrapping¹³(n = 4) and the Hosmer-Lemeshow test¹⁴(n = 13). The text described issues relating to colinearity in 9% (n = 24) of the studies, and a method for dealing with outliers was found in 5% (n = 12). Of note, 5% (n = 13) of the studies failed to report any of the criteria; only 1 study met all of the criteria.

We also looked for discussion of assumptions specific to logistic regression and proportional hazards analysis. Among studies reporting data from logistic regression models, 13% (12/89) discussed the interpretation of odds ratios; 21% (16/76) of studies using proportional hazards analysis discussed the proportional hazards assumption. As examples of analytic strategies, 1 study reported using Poisson regression rather than logistic regression because the outcome event was common, and another study reported use of logistic regression because the proportional hazards assumption was not met. Examples of how the proportional hazards assumption was tested were use of log-log plots and use of the Schoenfeld residual test.¹⁵

DISCUSSION

In a review of prominent medical journals, we found that multivariable methods of data analysis were used frequently, with logistic regression and proportional hazards analysis the most commonly reported methods. "Any mention" of methodological criteria applicable to most multivariable models varied widely, suggesting an opportunity for improved reporting (and possibly conduct) of these methods. In addition, model assumptions specific to logistic regression and proportional hazards were infrequently discussed.

The results of our review can be considered in the context of a prior review¹finding an 18% prevalence of 4 common multivariable methods in Lancet and New England Journal of Medicine as of 1989. In the current review (as of 2006), we found that 60% of studies used multivariable methods. Across the 2 time periods, logistic regression and proportional hazards analysis remained the most frequently used methods. The range of "adequate" reporting of general criteria in the current analysis ranged from 5% (method for evaluating outliers) to 84% (coding of variables). Among the criteria evaluated in both reviews, most were met more frequently in the current review. For example, testing for interactions was evident in 45% of articles in the current review, versus 27% of articles in the earlier review; only reporting of model selection process occurred less frequently in the current versus former review (15% vs 86%, respectively). The frequency of reporting of the proportional hazards assumption remained approximately the same (and was "low") in both reviews.

Reviews of multivariable analysis have been published in the medical specialty¹⁶and obstetrics-gynecology literature,¹⁷with similar findings of incomplete reporting. Other research has focused on the statistical review process at the level of the journal or authors. In a masked before-and-after study,¹⁸the peer review and editing process at the Annals of Internal Medicine were found to improve the quality of reporting of multivariable methods. In another study¹⁹of 114 journals responding to a survey, only one third of journals required statistical review for all accepted manuscripts. Finally, a study²⁰of 704 authors submitting to either of 2 general medical journals found that 73% received input from a methodologist (most often a biostatistician or epidemiologist); papers without methodological input were more likely to be rejected without review.

As a potential limitation of our work, the manual search, although extensive, may not have identified all possible studies using multivariable methods. In addition, the 10% random review indicated minimal interobserver variability, but some studies may still have been misclassified. As a strength of our study, we "gave credit" if any information was presented for the various methodological criteria. For example, mentioning the number of outcome events per independent variable was considered awareness of the issue; we did not apply a threshold value (eg, 10 events per variable^7,8) for "appropriateness." Similarly, a figure showing the relationship of 2 factors with an outcome variable was considered evidence of assessing interactions, even if not mentioned in the text.

This review suggests the continued need for more complete reporting of multivariable methods in the medical literature. Inadequate reporting of these methods increases the potential for unclear or misinterpreted results. Efforts to standardize reporting of multivariable methods would improve the quality of publications and assist clinicians in reaching appropriate conclusions.