Currently, teachers experience a general problem, a surge of published systematic reviews where slightly modified search criteria have led to slightly different bunch of papers with slightly different results of meta-analyses. Systematic reviews have once been welcomed as valuable tool to either end a story once and forever (if evidence for or against a certain treatment or association was overwhelming), or call for more conclusive randomized controlled trials (RCT) after still open questions had been identified. If, after any new RCT, editors of our professional journals would accept considering a new systematic review for publication, which basically ruminates already published RCT summaries but adds just another study without changing main conclusions, it will in fact become difficult to keep pace with what some call “emerging evidence”.
One main reason why evidence based medicine has to be taught to undergraduate students is to provide future health care workers with proper tools and train specific skills to conduct brief systematic reviews of identified randomized controlled trials themselves. Here on this blog, I had posted a couple of quick examples, see here, here and here.
In particular the latter of the above examples has dealt with the question whether the large multi-center trial by Engebretson et al. (2013), which had reported no effects of periodontal therapy on HbA1c levels in diabetics, would nullify the conclusion (that nonsurgical periodontal treatment may reduce HbA1c levels by about 0.4%) of previous meta-analyses of smaller and mainly single-center RCTs with similar settings.
Engebretson et al. (2013) had listed possible shortcomings of their study. However, that oral hygiene of study participants had not improved was considered by most of our professional leaders scandalous. Further issues for unprecedented criticism included “nearly normal” HbA1c levels at the outset and extreme obesity of participants. Engebretson’s unwelcome results had been reviled by an armada of 21 editors of our key journals, presidents of our main scientific societies, and further periodontal experts. Criticism had culminated in a very strange recommendation.
“Given the inconlusive nature of these data, we recommend that the existing body of evidence in which meta-analyses consistently conclude that successful periodontal therapy appears to improve glycemic control, should instruct us until results from future studies are reported. We urge all interested parties to refrain from using these study results as a basis for future scientific texts, new research projects, guidelines, policies, and advice regarding the incorporation of necessary periodontal treatment in diabetes management.” (My emphasis.)
In other words, forget about Engebretson et al. and continue quoting more favorable results from existing meta-analyses of RCTs on the effect of periodontal therapy on diabetes control. A quick analysis revealed that it won’t nullify a mean HbA1c reduction in diabetics by nonsurgical periodontal therapy, but that considerable heterogeneity was introduced by including Engebretson’s study which may in fact lower the grade of evidence. I had entered meta-data of Engebretson and Kocher 2013 in an amazing tool for meta-analysis and had added findings by Engebretson et al. (2013). That might have been premature, see below.
While most of us rather want to wait for a respective update of the previous report by the Cochrane Collaboration, a surge of new systematic reviews of trials considering the effects of nonsurgical periodontal therapy on HbA1c levels in diabetics have been published in the meantime. Entering terms “periodontal” AND “diabetes” in PubMed and filtering for article type “systemic review” yield four papers after Engebretson’s paper which was published in December 2013. Not always were meta-analyses done and negative results, on HbA1c levels after periodontal treatment, by Engebretson et al. were not always included. Very heterogeneous results demand indeed a closer look at and careful appraisal of the original papers.
When comparing systematic reviews by Engebretson and Kocher 2013, Liew et al. (2014) and the latest report by Wang et al. (2014), and checking original articles, it turned out that Engebretson and Kocher (2013) contains at least two numerical errors in their meta-analysis of studies with 3 months duration. They claim that their largest study, Sun et al. (2011) had reported high standard deviations for HbA1c reductions in diabetics of 0.99 and 0.94 in patients having received periodontal treatment (including extraction of hopeless teeth and periodontal surgery “if indicated”) or not, respectively. However, as reported in the original paper, respective standard deviations were much smaller, just 0.18 and 0.12. Engebretson and Kocher’s erroneous standard deviations led to considerably wide 95% confidence interval (CI) of their reported difference between test and control patients in the study by Sun et al. (2011) of -0.36% HbA1c, -0.66; -0.06. Another error concerns the study by Kiran et al. (2005), where HbA1c reduction amounts to -0.86% in the periodontal treatment group whereas Engebretson and Kocher (2013) report -0.80%. That led to the erroneous difference between treatment and control groups of -1.11 and erroneous 95% CI of -1.94; -0.28. The correct point estimate and 95% CI is given in Wang et al. (2014): -1.17 (-2.00; -0.34). Liew et al. (2014) also report a faulty result when estimating the 95% CI as -2.00; 0.34 [sic].
The systematic review by Wang et al. (2014) has been published in PLOS ONE. The paper includes Engebretson’s trial as well as a few new RCTs while it excludes studies which had reported follow up exams at 4, or less than 3 months (authors considered only studies reporting 3 and 6 month results after periodontal therapy). They end up with very familiar conclusions: HbA1c may be decreased by 0.36% on average (95% confidence interval -0.52%; -0.19%) 3 months after periodontal treatment while results after 6 months were not conclusive. Moreover, heterogeneity of studies was in fact now significant.
What is very much concerning, however, is the quality of the published text. This is not a well-written paper. Supposed results were reported correctly, poor English language may discredit any conclusions. The article’s editor is biostatistician Dr. Yu-Kang Tu of Taiwan, who has enlightened Periodontology with explaining valuable, if partly incomprehensible, statistical aspects, but who has also most critically written about pitfalls of certain statistical analyses in Dentistry. Had he had no chance to return the manuscript in another round of reviews to a native English speaking expert? In the paper, “root planning” [sic] appears once more, something which had made it even into titles of certain papers, see for instance here, due to the auto-correct function of Microsoft WORD.
The surge of systematic reviews on one and the same topic with slightly different inclusion and exclusion criteria (Liew et al. did not even mention the study by Sun et al. where tooth extractions and periodontal surgery, “if indicated”, had been done) has not kept an armada of our “thought leaders” from using all of them as “emerging evidence” in their unprecedented and politically motivated attack on unwelcome and, in their opinion, easily to argue away results by Engebretson et al. (2013). A closer look at the quality of these papers is, however, most justified.
During the joint EFP/AAP workshop of 2012 in Segovia, Spain, there would have been plenty of time to review more carefully Engebretson and Kocher’s hastily written systematic review before getting it published in two of our hardcore journals. Well, that particular workshop had a strong agenda, and any errors in the calculation of overall effects would not have altered already drafted conclusions, I’m afraid.
23 March 2015 @ 9:51 am.
Last modified March 23, 2015.