Abstract
This study used the inverse variance heterogeneity (IVhet) model to conduct a reanalysis of a recent meta-analysis that reported a positive association, based on the random-effects (RE) model, between obesity and the incidence of type 2 diabetes and coronary heart disease, but not all-cause stroke, in adults. Data emanated from a recent meta-analysis of five Mendelian randomization studies representing 881,692 adults. Results were pooled using the IVhet model and reported as OR’s and 95% CI. Small-study effects were examined using the Doi plot and Luis Furuya-Kanamori (LFK) index. Influence analysis was also conducted. The association between obesity and type 2 diabetes, coronary heart disease, and all-cause stroke was, respectively, 1.38 (95% CI 1.00 to 1.90, p=0.05, I2 =93%), 1.10 (95% CI 0.90 to 1.35, p=0.35, I2 =87%), and 1.02 (95% CI 0.95 to 1.09, p=0.64, I2 =0%). Compared with the original RE model, results were similar for all-cause stroke, but point estimates for type 2 diabetes and coronary heart disease were smaller (29.3% and 9.8%) with wider (7.0% and 14.7%), overlapping CI. Major asymmetry suggestive of small-study effects was observed (LFK=3.59). With the exception of one study for type 2 diabetes, results remained uncertain (overlapping 95% CI) when each study was deleted from the model once. A lack of certainty exists regarding the association between obesity and the incidence of type 2 diabetes, coronary heart disease, and all-cause stroke in adults.
Significance of this study
What is already known about this subject?
The incidence of obesity, type 2 diabetes, coronary heart disease, and all-cause stroke in adults is high.
Observational studies examining the association between obesity and the incidence of type 2 diabetes, coronary heart disease and all-cause stroke traditionally suffer from confounding and/or reverse causation.
Mendelian randomization studies limit confounding and/or reverse causation.
What are the new findings?
Meta-analytic results of Mendelian randomization studies using the inverse variance heterogeneity (IVhet) model suggest greater uncertainty regarding the association between obesity and the incidence of type 2 diabetes and coronary heart disease when compared with previous meta-analytic results using the traditional random-effects model.
Using the Doi plot and Luis Furuya-Kanamori (LFK) index, small-study effects suggestive of publication bias were found.
Significance of this study
How might these results change the focus of research or clinical practice?
Meta-analytic research should consider using the IVhet model to pool results, as well as the Doi plot and LFK index for examining small-study effects.
Future research should integrate different sources of evidence, that is, triangulation, to try and reach some formal consensus regarding the association between obesity and the incidence of type 2 diabetes, coronary heart disease and all-cause stroke in adults.
While clinicians, overall, should probably continue to counsel obese patients to reduce their adiposity, there may be certain subpopulations who may not be negatively affected, or may even benefit, from higher levels of adiposity.
Introduction
Cardiovascular disease is the number one cause of death among adults in the USA, with cerebrovascular disease and diabetes mellitus ranked fifth and seventh, respectively.1 In addition, the prevalence of obesity among adults in the USA is high, with an estimated 39.6% of adults classified as obese in 2015–2106.2 Given conflicting findings regarding the association between obesity and cardiovascular outcomes, a recent aggregate data meta-analysis of five Mendelian randomization trials3–7 representing 881,692 subjects reported a ‘statistically significant’ and positive association between obesity, as assessed by body mass index (BMI) in kg/m2, and the incidence of type 2 diabetes (OR=1.67, 95% CI 1.30 to 2.14, p<0.001) and coronary heart disease (OR=1.20, 95% CI 1.02 to 1.41, p=0.03), but not all-cause stroke (OR=1.02, 95% CI 0.95 to 1.09, p=0.65).8 It was also concluded from visual inspection of a funnel plot, a plot that included more than the results from the meta-analysis, that the risk of small-study effects (publication bias, etc.) was low.8 The major strength of this meta-analysis8 was the focus on Mendelian randomization studies, a study design that uses genetic variation as a natural experiment to examine the associations between potentially modifiable risk factors and health outcomes in observational data.9
A major advantage of the Mendelian randomization approach is the ability to limit confounding and/or reverse causation, a major potential problem in conventional observational studies.9 As an example, Holmes et al 5 conducted Mendelian analyses using a genetic score comprising 14 BMI-related single nucleotide polymorphisms (SNPs) in order to examine the role of BMI on type 2 diabetes, coronary heart disease, and stroke among 34,538 European-descent individuals. They found a positive association between higher BMI and an increased odds for type 2 diabetes but not coronary heart disease or stroke.5 Like any statistical approach, however, Mendelian analysis depends on certain assumptions. These include (1) relevance (the genetic variants are associated with the risk factor of interest), (2) independence (there are no unmeasured confounders of the associations between genetic variants and the outcome(s) of interest), and (3) exclusion restriction (the genetic variants affect the outcome(s) of interest only via their effect on the risk factor of interest).9
Despite the strength of the Riaz et al 8 meta-analysis in limiting studies to those that conducted Mendelian analysis, results were pooled using the traditional random-effects (RE) model of DerSimonian and Laird.10 The RE model was developed to overcome a major limitation of the traditional fixed-effect (FE) model, that is, the assumption that all included studies share the same common treatment effect.10 Thus, the traditional FE model accounts for only within-study variance, while the RE model accounts for both within-study and between-study variance. However, the RE model assumes that the true between-study differences in treatment effects follow a normal distribution with a common variance.10 Unfortunately, the assumption of normally distributed RE is untenable.11 Consequently, as heterogeneity increases, CI coverage decreases well below the nominal level, the end result being potential overconfidence in any conclusions drawn.11 To address this shortcoming, a more robust approach, the inverse variance heterogeneity (IVhet) model, has recently been developed.12 The IVhet model, details of which have been described elsewhere, addresses the problem of underestimation of the statistical error and spuriously overoptimistic estimates associated with the RE model by using an estimator under the FE model assumption with a quasi-likelihood-based variance structure.12 Simulation studies have demonstrated that this estimator retains correct coverage probabilities and a lower observed variance than the RE model estimator, irrespective of the amount of heterogeneity.12 For example, when applied to the often-used meta-analytic data set on intravenous magnesium for the prevention of mortality after a myocardial infarction, the pooled ORs and 95% CIs were 0.71 (95% CI 0.57 to 0.89) based on the RE model and 1.01 (95% CI 0.71 to 1.46) based on the IVhet model.12 Based on the IVhet model, both the point estimate and CI demonstrate more uncertainty regarding the benefits of intravenous magnesium for the prevention of mortality after myocardial infarction.12
An important component of meta-analysis is an examination for potential small-study effects (publication bias, etc.).13 The presence of small-study effects such as publication bias usually suggests an overestimate of treatment effect benefits.13 In the original meta-analysis by Riaz et al,8 the often-employed funnel plot was used.13 Based on visual inspection, it was suggested that low bias existed.8 No quantitative approach such as Egger’s p test14 was reported.8 While the traditional funnel plot is widely used, it is sometimes difficult to determine the presence of asymmetry and subsequent small-study effects such as publication bias.15–17 A funnel plot consists of a measure of precision, for example SEs from each study, on the y-axis and the treatment effects from each study on the x-axis. The reasoning behind the funnel plot is that smaller studies with less precision will scatter more widely at the bottom of the plot, while larger studies with greater precision will be grouped more closely together at the top of the plot, thus representing an inverted funnel and symmetry given that one would not expect an association between the magnitude of effect and precision.17 However, if asymmetry does exist, the presence of such an association and potential small-study effects such as publication bias may exist, although it may also be nothing more than the result of chance.13 In addition, concerns have been raised about Egger’s p test and its power to detect asymmetry, especially when the number of studies is small,18 something that is common in meta-analysis.19 Recently, a new graphical approach, the Doi plot, as well as a quantitative measure, the Luis Furuya-Kanamori (LFK) index, have been developed to improve this assessment.17 The Doi plot has been shown to represent a significant improvement over the traditional funnel plot when determining funnel plot asymmetry visually. It replaces the typical scatterplot approach of precision on the y-axis and treatment effect on the x-axis with a normal quantile (Z-score converted from percentiles) on the y-axis and treatment effects on the x-axis.17 As a result, it is easier to detect asymmetry visually.17 In addition, the LFK index has been shown to outperform Egger’s p test for the quantitative identification of asymmetry and possible small-study effects, including when the number of studies is small.17 The LFK index quantiles two regions of the Doi plot with respect to their areas under the plot and the number of studies in each arm.17 An LFK index of 0 is considered to represent perfect symmetry.17 Cutpoints ±1, greater than ±1 but within ±2, and greater than ±2 are considered to represent no, minor, and major asymmetry, respectively.17 When compared with Egger’s p test, the LFK index had superior areas under the receiver operating characteristic curve (0.74–0.88 vs 0.58–0.75), as well as greater sensitivity (71.3%–72.1% vs 18.5%–43.0%).17 However, specificity was greater for Egger’s p test (87.6%–90.0% vs 64.7%–87.1%).17
Thus, given (1) the importance of determining the association between obesity and type 2 diabetes, coronary heart disease, and stroke, (2) the strength of Mendelian randomization analyses for observational studies,9 (3) the importance of meta-analysis in evidence-based decision-making,20 21 and (4) the need to use the most robust meta-analytic methods available to provide the most valid information possible to both researchers and practitioners, the primary purpose of the current study was to use the IVhet model as well as the Doi plot and LFK index to reanalyze the studies included in the meta-analysis by Riaz et al 8 regarding the association between obesity and type 2 diabetes, coronary heart disease and all-cause stroke in adults,8 including a qualitative comparison with the RE model, funnel plot, and Egger’s p test.
Methods
Data for this brief report were derived from a recent meta-analysis that included five Mendelian randomization studies representing 881,692 subjects in which the association between obesity (BMI) and the incidence of type 2 diabetes, coronary heart disease and all-cause stroke was examined.8 This study was based on a meta-analysis of existing, publicly available aggregate data.8 The mean study age of the subjects was 60 years (range 50–64) and adjustment was made for an average of 47 SNPs (range 9–97).8 Further details are described in the original article.8
The effect size of interest was the ORs, with higher ORs indicative of greater risk. Data were extracted from data provided in the original article.8 Abstracted ORs and their 95% CIs22 were then pooled using the recently developed IVhet model.12 Two-tailed alpha values were generated for all results. The results from IVhet analyses were also compared with previously reported RE results qualitatively.8 Heterogeneity across studies was analyzed using Cochran’s Q statistic,23 which assesses the presence of heterogeneity, while the I2 test was used to measure the amount of heterogeneity, that is, inconsistency.24 Similar to the Riaz et al 8 meta-analysis, I2 values between 25% and 50%, 50% and 75%, and greater than 75% were considered to represent mild, moderate, and severe inconsistency, respectively. Absolute between-study heterogeneity was assessed using tau-squared (τ2). τ2 is the extent of variation among the effects observed in different studies (between-study variance).25 It denotes the absolute value of the true variance, that is, heterogeneity, across the different studies.25
Small-study effects (publication bias, etc.) were examined across all meta-analytic results using a recently developed graphical and quantitative method, details of which can be found elsewhere.17 Briefly, a Doi plot was created to visualize asymmetry, while the LFK index was used to quantify asymmetry of small-study effects from the Doi plot.17 LFK indexes within ±1, greater than ±1 but within ±2, and greater than ±2 were considered to represent no, minor, and major asymmetry, respectively.17 For comparative purposes, a funnel plot and Egger’s p (one-tailed) were also generated in order to qualitatively compare them with the Doi plot and LFK index, respectively.
Given the small number of studies, influence analysis was also conducted with each study deleted from the model once. All data were analyzed using MetaXL V.5.3.26 Congruent with recent recommendations from the American Statistical Association and with the aim of delivering more accurate information for both practice and research, the focus was on the uncertainty in the evidence and avoidance of the term ‘statistical significance’ as well as the dichotomization of p values.22 27
Results
A forest plot of the results is shown in figure 1 and table 1. As can be seen, the 95% CIs of the ORs included 1 for each subgroup. There was greater certainty with respect to between-study heterogeneity for type 2 diabetes and coronary heart disease but less for all-cause stroke. Inconsistency based on I2 was considered severe for type 2 diabetes and coronary heart disease but non-existent for all-cause stroke. τ2 was 0.06 for type 2 diabetes, 0.03 for coronary heart disease, and 0 for all-cause stroke. When pooled IVhet results were compared with the original RE findings,8 results were the same for all-cause stroke, but point estimates for type 2 diabetes and coronary heart disease were smaller (type 2 diabetes, 1.38 vs 1.67; coronary heart disease, 1.10 vs 1.20), with wider CI that included 1 (type 2 diabetes, 1.00 to 1.90 vs 1.30 to 2.14; coronary heart disease, 0.90 to 1.35 vs 1.02 to 1.41), suggesting greater uncertainty based on the IVhet model (table 1, figures 1 and 2). Also, as can be seen in figures 1 and 2, a comparison of weights between the IVhet and RE models for each study within each comparison demonstrated notable differences, the result of the different weighting approaches used for each model. More specifically, differences in weights of greater than 10% were noted for type 2 diabetes3 5 6 and coronary heart disease,5–7 but not all-cause stroke.
As can be seen in figure 3, major asymmetry across all groups, as indicated by the lack of an inverted funnel shape, was suggestive of small-study effects. This was supported by an LFK Index of 3.59 (major asymmetry). An examination of the traditional funnel plot in figure 4 reveals greater difficulty in identifying small-study effects compared with the Doi plot. In addition, Egger’s regression p value was 0.07, suggestive of small-study effects but not as convincing as the LFK index of 3.59. With each study for each group deleted from the model once, greater certainty (non-overlapping 95% CIs) was observed for type 2 diabetes when the study by Lyall et al 6 was excluded but not when any of the other studies were excluded for type 2 diabetes, coronary heart disease, and all-cause stroke (results not shown).
Discussion
Based on Mendelian randomization studies, the current findings suggest greater uncertainty than the meta-analysis by Riaz et al,8 with respect to the association between obesity, as assessed by BMI, and the incidence of type 2 diabetes and coronary heart disease, but not all-cause stroke, in adults. This is evidenced by the smaller point estimates and wider 95% CI for type 2 diabetes as well as coronary heart disease . For all-cause stroke, results were similar, with uncertainty observed for both (95% CI overlapping 1). The qualitative (Doi plot) and quantitative (LFK index) findings of major asymmetry suggestive of small-study effects are in opposition to the previous meta-analysis of no small-study effects based on only the funnel plot,8 possibly because of the greater ability to visually detect small-study effects using the current approach. However, another possibility is that the funnel plot by Riaz et al 8 appeared to include additional outcomes beyond those that were quantitatively synthesized and reported in their forest plot. A third possibility is that the original funnel plot appeared to be generated using the SE of the log OR on the y-axis and the OR on the x-axis. Thus, the original funnel plot results generated by Riaz et al 8 may not be well suited for comparison with the Doi plot used in the current investigation. However, when the current authors generated a funnel plot (see figure 4) using the same data as the Doi plot, it was clear that the ability to detect small-study effects was enhanced using the Doi versus the funnel plot approach. For quantitative analyses, the LFK index was suggestive of major asymmetry, while Egger’s p was also suggestive of small-study effects, although the magnitude of such appears to favor the LFK index. Finally, with the exception of only one excluded study for type 2 diabetes,6 greater uncertainty remained for type 2 diabetes as well as coronary heart disease and all-cause stroke when each study was deleted once.
From the investigative team’s perspective, the three major strengths of the current meta-analysis were use of the more robust IVhet model12 to pool results, as well as the Doi plot and LFK index to examine for potential small-study effects.17 Use of the IVhet model in the current study provides more robust information than the RE model used in the original meta-analysis for at least three reasons. First, pooled IVhet estimates, when compared with RE estimates, have been shown to favor larger trials.12 Second, when compared with the RE model, IVhet results have been shown to produce a more conservative CI and correct coverage probability.12 Third, IVhet estimates, when compared with RE estimates, have been shown to generate a lower observed variance, regardless of the degree of heterogeneity, an important factor in meta-analysis.12
Useof the Doi plot and LFK index17 to examine for potential small-study effects provided more robust information than the traditional funnel plot and Egger’s p test.14 More specifically, the ability to visually determine asymmetry and possible small-study effects was enhanced using the Doi plot versus the traditional funnel plot. In addition, the LFK index appeared to outperform Egger’s p value for detecting asymmetry, especially with respect to magnitude.
From a clinical perspective, the results of the current study suggest greater uncertainty than the original meta-analysis8 regarding the association between obesity and type 2 diabetes, coronary heart disease, and all-cause stroke. However, these findings should not be interpreted as a contraindication for recommending reductions in adiposity among obese adults. This is especially true given the association between obesity and all-cause mortality in adults,28 although a recent meta-analysis did conclude that weight loss increases all-cause and cardiovascular disease mortality in overweight or obese patients with diabetes.29 Clearly, additional research is needed regarding this apparent obesity paradox. Along those lines, it is suggested that future research integrate different sources of evidence, that is, triangulation,30 to try and reach some formal consensus regarding the association between obesity and the incidence of type 2 diabetes, coronary heart disease and all-cause stroke in adults. Until that time, it would seem prudent, overall, to advise obese patients to reduce their adiposity given the many deleterious consequences associated with such, although there may be subpopulations of people who may not be negatively affected, or may even benefit, from higher levels of adiposity.31
While the current results are noteworthy, they should nevertheless be interpreted with caution given the small number of studies included for each group and the resultant inability to examine for sources of heterogeneity and inconsistency, a finding that was considered severe for both type 2 diabetes and coronary heart disease. In addition, the strengths, for example, including only Mendelian randomization studies, and limitations, for example, ecological fallacy, in the original meta-analysis8 also exist in the current study.
Potential limitations also exist with respect to the methods employed in the current meta-analysis. For example, when compared with the RE model, the IVhet model will not always produce a more conservative point estimate,32 although this was not the case for the current analyses. Second, while the Doi plot enhances the ability to visually detect potential small-study effects, it is still qualitative in nature and thus lends itself to some subjectivity in the interpretation of findings. Third, while the LFK index, when compared with Egger’s p value, has been shown to better discriminate asymmetry and have higher sensitivity, its specificity is lower than the Egger’s p value.17 Finally, while the authors believe that the IVhet model, Doi plot, and LFK Index provide significant enhancements over existing methods, it is important to understand that no perfect test exists for either pooling results in a meta-analysis or for detecting small-study effects.
In conclusion, these new findings suggest a lack of certainty, based on Mendelian randomization studies, with respect to the association between obesity and selected cardiovascular outcomes in adults. A need exists for future research on this topic, including, but not necessarily limited to, the triangulation of different types of evidence before more definitive conclusions can be reached. It is also suggested that future meta-analyses consider using the IVhet model to pool results, as well as the Doi plot and LFK index when examining for small-study effects.
Footnotes
Contributors GAK was responsible for the conception and design, acquisition of data, analysis and interpretation of data, drafting the initial manuscript and revising it critically for important intellectual content. KSK was responsible for the conception and design, acquisition of data, drafting the initial manuscript and revising all drafts critically for important intellectual content. BLS was responsible for the conception and design, drafting the initial manuscript and revising all drafts critically for important intellectual content. All authors read and approved the final manuscript.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Ethics approval Given that this study was based on a meta-analysis of existing, publicly available aggregate data, this study was exempt from institutional review board approval.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Data are available upon reasonable request.
Patient consent for publication Not required.