INTRODUCTION

---

Systematic reviews and meta-analyses aggregate the findings of multiple published works while taking into account the quality of evidence of each individual source[1]. In addition, meta-analysis offers the benefit of combining data from individual studies into a larger sample, providing greater power to detect important differences in outcomes of interest [2,3]. Because of this, systematic reviews and meta-analyses are generally regarded as important components of the collective body of orthopedic literature and are frequently utilized for clinical practice guidelines and recommendations [4-6].

However, performance of a quality systematic review or meta-analysis is contingent on many factors. Vital components of the process include appropriate study selection, quantitative and/or qualitative analysis, and data reporting. The methodological and reporting quality of these studies is key to their ability to generate unbiased results[7]. Performed poorly, these of studies may lead to inaccurate conclusions [8,9]. Because of this, it is important for authors to understand and adhere to available quality guidelines, such as the preferred reporting items for systematic reviews and meta-analyses (PRISMA) statement and the Cochrane guidelines[7,10,11].

The issue of methodological quality in systematic reviews and meta-analyses has been explored previously in the orthopedic literature. In 2001, Bhandari et al. reviewed 40 meta-analyses published between 1969 and 1999 and found that the majority had major methodological flaws [5]. Similarly, in 2010 Dijkman et al. reviewed orthopedic meta-analyses published during the years 2005 and 2008, and they concluded that a large percentage had major to extensive methodological flaws [12]. More recently, in 2013, Gagnier et al. performed a review of systematic reviews and meta-analyses in the five highest impact factor orthopedic journals, utilizing the PRISMA checklist and the Assessment of Multiple Systematic Reviews (AMSTAR) index [13]. They found that on average 68% of PRISMA checklist items and 54% of AMSTAR criteria were present, leading the authors to conclude, that the overall methodological and reporting quality remained "less than optimal"[13]. These studies all highlights the necessity of strict adherence to appropriate methodological construct to ensure the quality of orthopedic systematic reviews and meta-analyses. Their findings are especially pertinent in light of the increasing number of systematic reviews and meta-analyses being published (an increase of 67% and 132%, respectively, between 2010 and 2014, compared to only a 27% increase for all PubMed-indexed items)[9].

However, it is uncertain whether these prior studies, and similar works across other medical disciplines, have impacted the quality of systematic reviews and meta-analyses in the orthopedic literature[9]. Furthermore, while providing recommendations to authors, they failed to offer guidance for readers and reviewers evaluating systematic reviews and meta-analyses. If the quality of studies being produced remains variable, it is incumbent on readers and reviewers to be able to critically evaluate quality when considering a study's conclusions. The aim of our study is to examine methodological and reporting quality of orthopedic systematic reviews and meta-analyses published in 2016, across a range of orthopedic subspecialties, and recommend a quality rating strategy for readers and reviewers.

METHODS

---

We drafted a protocol outlining a search, data collection, and analysis strategy before initiation of our literature search. This protocol was registered on PROSPERO, an international registry of systematic review protocols. We conducted our review utilizing standard methodology as outlined in the Cochrane Handbook and reported results in accordance with the Preferred PRISMA guidelines[14,15].

Search strategy

One subspecialty society was chosen by faculty at our institution to represent each of the 10 orthopedic surgery subspecialties with accredited fellowships in the US, as outlined by the Accreditation Council for Graduate Medical Education (ACGME). Shoulder and elbow surgery, which is ACGME accredited under the classification of orthopedic sports medicine and adult reconstructive surgery, was included as a distinct subspecialty[16]. The affiliated journals of each chosen subspecialty society were identified: Orthopedic Trauma (Journal of Orthopedic Trauma); Orthopedic Sports Medicine (The American Journal of Sports Medicine); Hand Surgery (Journal of Hand Surgery); Adult Reconstructive Orthopedic Surgery (The Journal of Arthroplasty); Orthopedic Surgery of the Spine (The Spine Journal); Shoulder and Elbow Surgery (Journal of Shoulder and Elbow Surgery); Pediatric Orthopedic Surgery (Journal of Pediatric Orthopedics); and Foot and Ankle Orthopedic Surgery (Foot and Ankle International); Musculoskeletal Oncology (Clinical Orthopedics and Related Research). If a subspecialty society had more than one affiliated journal, the journal with the highest impact factor, per the 2015 Thomson Reuters ISI index, was chosen[17]. In addition, the highest impact orthopedic surgery journal (The Journal of Bone and Joint Surgery) was reviewed and included in our analysis.

Online archives of selected journals were searched for articles published during the calendar year 2016. Initial search was performed by reviewing the title and abstract for each study. To ensure completeness of study identification, a search of Medline was also performed using the keywords "systematic review OR meta-analysis" and "journal name OR journal abbreviation" and "2016." All articles that stated in the title or methodology section that they were a systematic review and/or meta-analysis were identified. Full text of identified articles was subsequently reviewed to ensure that the article met inclusion criteria. Because our study aimed specifically to evaluate published literature, no attempts were made to obtain unpublished manuscripts.

To be included in the study, an article must have been explicitly described as a systematic review and/or a meta-analysis (including the terms "systematic review" and/or "meta-analysis") in the title, introduction, or methods section. Excluded from analysis were published abstracts for which full-text manuscripts, published in 2016, were not available. Studies were also excluded if they did not include a formal search and/or data analysis strategy (i.e. expert opinion literature reviews). Eligibility criteria for each study were reviewed by two authors (STK and MJS). Disagreements between reviewers were resolved by consensus.

Data extraction and assessment of methodological quality

All data extraction was performed by two authors (STK and MJS) utilizing a uniform data extraction worksheet. Extracted data included primary author, author's hospital and/or institutional affiliation, journal of publication, number of primary studies included, type of study designs included, reported level of evidence, protocol registration, financial support, and conflicts of interest reporting. Methodological and reporting quality was assessed by two authors (STK and MJS) using the guidelines in the PRISMA statement, as well as the AMSTAR scale[7,11,18,19]. Disagreements between reviewers were resolved by consensus.

The PRISMA statement is an evidenced-based guideline for the performance and reporting of systematic reviews and meta-analyses[11]. It consists of a 27-item checklist (composed of seven main categories) and a flow chart. A PRISMA score (1-27) was assigned by reviewers based on presence or absence of each element within the paper [11]. Item number 5 on the PRISMA checklist, relating to protocol, and registration was scored "yes" if the study explicitly described a protocol, regardless of whether or not the protocol was published. Information on protocol registration was captured as a separate data item on our uniform data collection worksheet[11].

The AMSTAR tool is a validated 11-item assessment of methodological quality that generates a score of (0-11) [18,19]. AMSTAR scores were assigned by reviewers based on presence or absence of each criteria. Disagreements between reviewers were resolved by consensus. To avoid ambiguity with regard to the interpretation of AMSTAR item number 5, we chose to grade this item as "yes" if studies excluded at the full-text review stage were referenced, consistent with the recommendation of prior authors who have critically evaluated the AMSTAR [20-22] The AMSTAR checklist items are provided in Figure 1.

Figure 1

Data analysis

Because some items in both the PRISMA and AMSTAR pertain to quantitative analysis of data and may be relevant only to meta-analyses, we analyzed our data both in aggregate and separately for systematic reviews (studies containing only qualitative analysis) and meta-analyses (studies containing quantitative analysis). Distribution of PRISMA checklist items and AMSTAR scores was described using mean, standard deviation, and range. We also categorized studies based on the relative proportion of PRISMA and AMSTAR items present, to define a quality rating scale corresponding to the interquartile range for each metric. Interrater reliability was assessed using kappa. Correlation of PRISMA and AMSTAR scores for our aggregate sample was assessed using Pearson's correlation. One-way analysis of variance (ANOVA) was used to evaluate the association of journal of publication with PRISMA and AMSTAR scores. Statistical significance was set at P < 0.05. Statistical analysis was performed using Stata Version 14 (StataCorp. College Station, TX: StatCorp LP, 2016).

RESULTS

---

Initial search results yielded 104 papers [Figure 2]. Full-text review excluded seven articles,[23-29] leaving 97 articles for inclusion in the study (46 systematic reviews and 51 meta-analyses) [Appendix]. Description of studies by journal of publication is provided in Table 1. Only 43% (42/97) of studies reported level of evidence, with reported level of evidence ranging from 2 to 4 for systematic reviews and 1-4 for meta-analyses. Twelve percent (12/97) of studies were registered on PROSPERO (nine systematic reviews and three meta-analyses).

Figure 2    Table 1

Interrater agreement was strong for both PRISMA (k = 0.85 [95% confidence interval [CI] 0.82-0.88), P < 0.001) and AMSTAR (k = 0.94 [95% CI 0.90-0.98], P < 0.001). Correlation between PRISMA score and AMSTAR scores was also strong (r = 0.83, P < 0.001) [Figure 3]. Mean PRISMA score for our aggregate sample of systematic reviews and meta-analyses was 21 (±3.7, range 9-27), and mean AMSTAR score was 5.3 (±2.3, range 0-10). Itemized distribution of AMSTAR scoring for systematic reviews and meta-analyses is provided in Figure 4.

Figure 3    Figure 4

Among the 46 systematic reviews, 63% (29/46) provided detailed descriptions of included studies. Of these 29, 55% (16/29) included randomized control trials, 45% (13/29) included prospective cohort studies, 62% (18/29) included retrospective cohort studies, and 41% (12/29) included case series. Only 17% (5/29) included randomized control trials exclusively. For systematic reviews, the mean PRISMA score was 18.6 (±3.2, range 9-27). The lowest quartile of PRISMA scores was 9-16, the interquartile range was 17-20, and the highest quartile was 21-27. The mean AMSTAR score was 4.2 (±2.2, range 0-9). The lowest quartile of AMSTAR scores was 0-1, the interquartile range was 2-6, and the highest quartile was 6-9. ANOVA demonstrated that mean scores did not vary significantly by journal of publication for either PRISMA scores (F (9,36) = 0.95, P = 0.497) or AMSTAR scores (F (9,36) = 1.12, P = 0.376).

Among the 51 meta-analyses, 82% (42/51) provided detailed descriptions of included studies. Of these 42, 80% (33/42) included randomized control trials, 56% (23/42) included prospective cohort studies, 54% (22/42) included retrospective cohort studies, and 20% (8/42) included case series. Only 25% (13/42) included randomized control trials exclusively. For meta-analyses, the mean PRISMA score was 23.3 (±2.7, range 16-27). The lowest quartile of PRISMA scores was 16-20, the inter-quartile range was 21-25, and the highest quartile was 26-27. The mean AMSTAR score was 6.4 (±1.9, range 2-10). The lowest quartile of AMSTAR scores was 2-4, the interquartile range was 5-7, and the highest quartile was 8-10. Mean PRISMA and AMSTAR scores by journal are provided in Table 2. ANOVA demonstrated that mean scores did not vary significantly by journal of publication for either PRISMA scores (F (7,43) = 1.12, P = 0.366) or AMSTAR scores (F (7,43) = 2.12, P = 0.062).

Table 2

DISCUSSION

---

Our review demonstrates a wide range of methodological and reporting quality among included studies. Furthermore, these findings suggest that the overall quality of orthopedic systematic reviews and meta-analyses has not improved since this issue was first raised, nearly two decades ago.[5,12] Our findings are consistent with those reported by similar studies evaluating systematic reviews and meta-analyses in orthopedics and other areas of medicine [13,30-36]. Moreover, the scores reported in this study are similar to those published by Shea et al., for studies reviewed in both their initial publication and external validation studies of the AMSTAR, in which they reported ranges of scores from of 0 to 10 and mean scores of 5.4 and 4.6, respectively [18,19].

One important consideration when evaluating the findings of this study is interpretation of individual quality metric questions. Two specific items, of note, are AMSTAR questions 5 and 11. Gagnier et al., in their 2013, study reported 86% of studies fulfilled item 5, and 86% of studies fulfilled item 11; however, in our analysis, these items were fulfilled in far fewer studies (12% [12/97] for item 5 and 3% (3/97) for item 11) [13]. With regard to AMSTAR item 5, we suggest interpretation of this criteria in accordance with the recommendations of Burda et al. and Wegewitz et al [20,22]. They suggest the item be interpreted such that it is graded as "yes" if studies excluded at the full-text review stage are referenced. We believe this item is important, as it relates to the transparency and reproducibility of the study selection process. AMSTAR item 11 asks specifically about inclusion of conflict of interest for the systematic review or meta-analysis authors and for the included studies. While most studies included a conflict of interest statement, very few referenced any potential conflicts of interest among the studies they included in their analysis. We believe this is also an important item, particularly if the systematic review or meta-analysis is including studies related to a product or service for which some of the existing published evidence may be biased by a pertinent conflict of interests [18,22]

One of the more surprising findings of our analysis was that only 63% (29/46) of systematic reviews and 82% (42/51) of meta-analyses provided detailed descriptions of the types of studies included. This suggests a need for greater transparency regarding included studies, as it facilitates critical evaluation by readers and promotes a greater degree of confidence in the results of a systematic review or meta-analysis[14]. Furthermore, surprising was the large percentage of both systematic reviews (41%) and meta-analyses (20%) which included case series in their analysis. This contrast with the relatively low percentage of systematic reviews (17%) and meta-analyses (25%) which included randomized control trials exclusively and suggests that the primary source data utilized in many of these studies may be suboptimal [37]. In addition, our study found that 57% (55/97) of studies did not report level of evidence. Because both systematic reviews and meta-analyses can range in level of evidence (between 2-4 and 1-4 in our study, respectively), we recommend that all of these studies clearly state their level of evidence, as determined by the authors and/or editors of the publishing journal.

There are several limitations of the current study. This study does not provide an exhaustive assessment of all systematic reviews and meta- analyses published in all orthopedic journals. However, our process of study selection was designed to provide a broad overview, applicable to a wide range of orthopedic providers. As such, this work represents an important cross-sectional assessment of the current state of quality in this domain of literature. Another important consideration is that the quality of published systematic reviews and meta-analyses is partially dictated by the editorial process of the publishing journal and may vary among journals not included in our study. However, among our sample, journal of publication was not significantly associated with variation in either PRISMA or AMSTAR scores, despite the fact that the journals, we included covered a breadth of subspecialty interests and had a wide range of impact factors (1.330-5.163) [17]. This supports the generalizability of our findings, and we believe that the 97 articles included in our study are likely representative of published orthopedic systematic reviews and meta-analyses.

CONCLUSIONS

---

Despite the stated limitations, our findings demonstrate that there remains substantial variability in methodological and reporting quality of systematic reviews and meta-analyses in the orthopedic literature. This suggests that prior calls for greater adherence to methodologic guidelines have been largely ineffective. The continued variability in quality, combined with the large volume of systematic reviews and meta-analyses being published, places a strong impetus on readers and reviewers to evaluate and formulate their own conclusions about the quality of these studies. To facilitate this process, we devised a strategy that allows assignment of a three-tier (low, moderate, or high) quality rating, using PRISMA and AMSTAR scores. Our findings indicate that a score =21 PRISMA items and =6 AMSTAR items for systematic reviews and =26 PRISMA items and =8 AMSTAR items for meta-analyses are associated with the top quartile of quality reporting and methodology. A score =16 PRISMA items and =1 AMSTAR items for systematic reviews and =20 PRISMA items and =4 AMSTAR items for meta-analyses is associated with the lowest quartile. Incorporating these score ranges into a single metric, we produced a simple grading scale [Table 3] by which readers and reviewers can critically assess and communicate study quality for orthopedic systematic reviews and meta-analyses.

Table 3

Studies are considered a "systematic review" if they include only qualitative analysis of primary sources, studies are considered a "meta-analysis" if they included quantitative analysis primary sources. Quality grade is determined by the lowest scored metric.

REFERENCES

---

1. Mulrow CD. Rationale for systematic reviews. BMJ 1994;309:597-9.
2. Der Simonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials 1986;7:177-88.
3. Cohn LD, Becker BJ. How meta-analysis increases statistical power. Psychol Methods 2003;8:243-53.
4. Mullen PD, Ramirez G. The promise and pitfalls of systematic reviews. Annu Rev Public Health 2006;27:81-102.
5. Bhandari M, Morrow F, Kulkarni AV, Tornetta P 3rd. Meta-analyses in orthopaedic surgery. A systematic review of their methodologies. J Bone Joint Surg Am 2001;83a:15-24.
6. Kuehn BM. Iom sets out "gold standard" practices for creating guidelines, systematic reviews. JAMA 2011;305:1846-8.
7. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Ann Intern Med 2009;151:264-9, W64.
8. Bhandari M, Montori VM, Devereaux PJ, Wilczynski NL, Morgan D, Haynes RB, et al. Doubling the impact: Publication of systematic review articles in orthopaedic journals. J Bone Joint Surg Am 2004;86-A:1012-6.
9. Ioannidis JP. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q 2016;94:485-514.
10. Higgins J, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses [journal article as teaching resource, deposited by John Flynn]. Br Med J 2003;327:557-60.
11. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Ann Intern Med 2009;151:W65-94.
12. Dijkman BG, Abouali JA, Kooistra BW, Conter HJ, Poolman RW, Kulkarni AV, et al. Twenty years of meta-analyses in orthopaedic surgery: Has quality kept up with quantity? J Bone Joint Surg Am 2010;92:48-57.
13. Gagnier JJ, Kellam PJ. Reporting and methodological quality of systematic reviews in the orthopaedic literature. J Bone Joint Surg Am 2013;95:e771-7.
14. Higgins JP, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Hoboken, NJ: Wiley Online Library; 2008.
15. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Ann Intern Med 2009;151:264-9.
16. Daniels AH, DiGiovanni CW. Is subspecialty fellowship training emerging as a necessary component of contemporary orthopaedic surgery education? J Grad Med Educ 2014;6:218-21.
17. Reuters T. 2015 ISI Thomson Reuters Journal Citation Reports; 2016.
18. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 2007;7:10.
19. Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, et al. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol 2009;62:1013-20.
20. Burda BU, Holmer HK, Norris SL. Limitations of A measurement tool to assess systematic reviews (AMSTAR) and suggestions for improvement. Syst Rev 2016;5:58.
21. Faggion CM Jr. Critical appraisal of AMSTAR: Challenges, limitations, and potential solutions from the perspective of an assessor. BMC Med Res Methodol 2015;15:63.
22. Wegewitz U, Weikert B, Fishta A, Jacobs A, Pieper D. Resuming the discussion of AMSTAR: What can (should) be made better? BMC Med Res Methodol 2016;16:111.
23. Elmallah RK, Cherian JJ, Meneghini RM, Hozack WJ, Westrich GH, Mont MA, et al. How to approach a recalled dual modular hip implant: An update. J Arthroplasty 2016;31:2646-52.
24. Jacofsky DJ, Allen M. Robotics in arthroplasty: A Comprehensive review. J Arthroplasty 2016;31:2353-63.
25. Feeley BT, Agel J, LaPrade RF. When is it too early for single sport specialization? Am J Sports Med 2016;44:234-41.
26. Matz PG, Meagher RJ, Lamer T, Tontz WL Jr., Annaswamy TM, Cassidy RC, et al. Guideline summary review: An evidence-based clinical guideline for the diagnosis and treatment of degenerative lumbar spondylolisthesis. Spine J 2016;16:439-48.
27. Moghimi MH, Reitman CA. Perioperative complications associated with spine surgery in patients with established spinal cord injury. Spine J 2016;16:552-7.
28. Labbe JL, Peres O, Leclair O, Goulon R, Scemama P, Jourdel F, et al. Posterior C1-C2 fixation using absorbable suture for Type II odontoid fracture in 2-year-old child: Description of a new technique and literature review. J Pediatr Orthop 2016;36:e96-100.
29. Leopold SS. Editor's spotlight/Take 5: When can I drive after orthopaedic surgery? A systematic review. Clin Orthop Relat Res 2016;474:2553-6.
30. Xia L, Xu J, Guzzo TJ. Reporting and methodological quality of meta-analyses in urological literature. Peer J 2017;5:e3129.
31. Pollock M, Fernandes RM, Hartling L. Evaluation of AMSTAR to assess the methodological quality of systematic reviews in overviews of reviews of healthcare interventions. BMC Med Res Methodol 2017;17:48.
32. MacDonald SL, Canfield SE, Fesperman SF, Dahm P. Assessment of the methodological quality of systematic reviews published in the urological literature from 1998 to 2008. J Urol 2010;184:648-53.
33. Park JY, Lee KH, Ku YJ, Cho SG, Kim YJ, Lee HY, et al. Characteristics, trends, and quality of systematic review and meta-analysis in general radiology between 2007 and 2015. Acad Radiol 2017;24:1013-22.
34. Campbell JM, Kavanagh S, Kurmis R, Munn Z. Systematic reviews in burns care: Poor quality and getting worse. J Burn Care Res 2017;38:e552-67.
35. Schiegnitz E, Kammerer P, Al-Nawas B. Quality assessment of systematic reviews and meta-analyses on biomarkers in oral squamous cell carcinoma. Oral Health Prev Dent 2017;15:13-21.
36. Sharma R, Vannabouathong C, Bains S, Marshall A, MacDonald SJ, Parvizi J, et al. Meta-analyses in joint arthroplasty: A review of quantity, quality, and impact. J Bone Joint Surg Am 2011;93:2304-9.
37. Egger M, Schneider M, Smith GD. Spurious precision? Meta-analysis of observational studies. BMJ (Clinical research ed) 1998;316:140-4.