Prevalence of spin reporting bias in hip labrum reconstruction: a qualitative assessment of systematic reviews and meta-analyses

INTRODUCTION

The hip labrum plays an important role in the stability, kinematic function, durability, proprioception, and the suction seal of the hip joint.^1-3 Nepple et al and Phillipon et al found that the hip labrum was the primary hip stabilizer to distraction forces during the initial 6 mm of distraction.^2,6 Therefore, management of hip labral injuries has been shifting from debridement/excision to repair/reconstruction so that the hip labral seal may be preserved.^4,5 Labral preservation with repair or fixation had been the gold standard treatment for labral tears; however, the introduction of the labral reconstruction procedure by Philippon et al has brought changes and questions to the topic of management of hip labral tears.²⁶ Reconstruction has typically been saved for irreparable labral tears, but some recent studies show potential efficacy for reconstruction in reparable labral tears at the time of index hip arthroscopy.^2-6

Orthopaedics is an ever-changing field, and as more surgical techniques emerge, reviewing all primary research regarding a specific topic can be time consuming. This has led to increased production of and reliance on systematic reviews and meta-analyses for reviewing literature.^7-10 Recognition of bias and critical appraisal of systematic reviews is important for physicians to determine clinical management decisions, which has led to the creation of scoring tools to critically appraise studies.¹¹

Spin has been defined as “a specific way of reporting, intentional or not, to highlight that the beneficial effect of the experimental treatment, in terms of efficacy or safety, is greater than that shown by the results.”¹²

It has been found to play an important role in physician interpretation of articles.¹³ Consequently, evaluation of spin should be part of a physician’s toolkit when critically appraising literature. Yavchitz et al reported a rubric of the 9 most severe forms of spin found in the abstracts of biomedical research.¹²

The purpose of this study was to identify the presence of spin in abstracts of systematic reviews and meta-analyses regarding hip labrum reconstruction. Our hypothesis was that spin would be present in 33% of the abstracts of systematic reviews and meta-analyses regarding this topic.

METHODS

Search strategy

A systematic search conducted on January 9^th, 2022 using MEDLINE, Embase, Web of science, and Google Scholar. Following keywords used: “labrum”, “reconstruction” or “repair”, and “systematic review” or “meta-analysis”. After removing duplicates, each article was screened by title and abstract by 2 authors independently in a masked fashion to determine eligibility. Once screening was complete, consensus was reached by both reviewers regarding which articles met inclusion criteria.

Inclusion criteria

All systematic reviews and meta-analyses regarding hip labrum reconstruction conducted on human subjects that were available in English were included in this study. There was no exclusion based on date of publication or country of origin. All articles that were not systemic reviews or meta-analyses on this topic were excluded.

Data extraction and review

Two authors were trained on the topic of spin and AMSTAR-2 by reviewing current articles published regarding these 2 topics.^{3,8,9,16,25,26,30} After training, each article regarding hip labrum reconstruction was evaluated separately by two independent reviewers for the 9 most severe types of spin commonly found in abstracts (Figure 2), as previously described by Yavchitz et al.¹² Each systematic review was also evaluated with AMSTAR 2, a 16-question survey used for critical appraisal of systematic reviews.²⁶ The AMSTAR 2 survey is used to provide a confidence rating for all systematic reviews, ranging from critically low to high. In order to determine the types of spin and the AMSTAR-2 score, a thorough review of each article in its entirety was required. After thorough review of each article in its entirety, the abstract was again reviewed to assess the types of spin found in the abstract. Other variables recorded included, year of publication, journal, journal impact factor, reported COI, adherence to PRISMA guidelines, journal endorsement of PRISMA, number of citations, and average number of citations per year. The citations were recorded on March 27, 2022, and Google Scholar was used to determine number of citations for all articles. Data was extracted independently by two authors. Meeting was held after data extraction to reconcile any discrepancies in data extraction and review. If a discrepancy was unable to be reconciled between two authors, 3^rd author consulted until consensus was reached.

Statistical analysis

Descriptive statistics were used to calculate frequency of spin and its subtypes. Data was imported into RStudio for all further analysis. Only 10 meta-analyses or systematic reviews on the topic of hip labrum were found during our database search. As such, analyses to determine the association between spin and AMSTAR-2 grade, number of citations, average citations per year, and journal impact factor were underpowered.

RESULTS

The electronic database search was performed on January 9^th, 2022 and resulted in 1,148 articles, of which 10 met the inclusion criteria (Figure 1).^4,5,14-21 Year of publication ranged from 2013-2021. In regards to PRISMA, 80% (8/10) of articles mentioned adherence to PRISMA guidelines, while none of journals were official endorsers of PRISMA. Total no. of citations of each article ranged from 0-126, with average number of citations 26.7. The number of citations per year ranged from 0-12.6, with an average of 5.4 citations per year. Of the reviewed articles, 70% (7/10) reported if there were any conflicts of interest.

Spin in articles

The 70% (7/10) of the articles were found to have at least one form of spin in the abstract (Figure 2). Of the 9 types of spin assessed, type 5 (“conclusion claims the beneficial effect of the experimental treatment despite high risk of bias in the primary studies”) was the most commonly found (60%, 6/10). Spin type 3 (“selective reporting of or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention”) was found in 40% (4/10), type 8 (“conclusions extrapolate the review’s findings from a surrogate marker or a specific outcome to the global improvement of the disease”) was found in 30% (3/10), and type 7 spin (“Conclusion extrapolates the review’s findings to a different intervention”) was found in 10% (1/10). No articles were found to contain spin type 1, 2, 4, 6, or 9. Of note, 20% (2/10) of articles contained 3 different types of spin, 30% (3/10) of articles contained 2 different types of spin, and 20% (2/10) articles contained 1 type of spin. Due to the limited number of studies, we were unable to determine any relationship between presence of spin and number of citations per year, journal impact factor, source of funding, AMSTAR-2 grade, or adherence to PRISMA guidelines.

AMSTAR 2 rating

The AMSTAR 2 survey is used to provide a confidence rating for all systematic reviews, ranging from critically low to high. AMSTAR 2 ratings showed that 50% (5/10) were “critically low,” 40% (4/10) were “low,” and 10% (1/10) were “moderate.” No articles were rated “high” according to the AMSTAR 2 scoring system, which means that each systematic review contained 1 critical flaw or multiple non-critical flaws. No significant association was found between presence of spin and AMSTAR 2 rating. Of the articles reviewed, 90% (9/10) framed their research question in a population-intervention-comparator-outcome (PICO) format. All articles provided information on potential COI and funding.

DISCUSSION

Critical appraisal of the published literature is an important skill for practicing evidence-based medicine; therefore, recognizing spin in research is crucial. The purpose of this study was not to determine the indications or outcomes of hip labrum reconstruction; it was to determine the prevalence of spin bias in the current literature regarding the topic. Overall, review of this topic shows level III and level IV evidence that it is an acceptable treatment alternative for labral repair in patients with irreparable labrums.^4,5,14-21

However, it is important to recognize that the current literature regarding outcomes also includes a multitude of concomitant pathologies and procedures, as such, it is our recommendation to determine surgical candidacy on a patient-to-patient basis with review of outcomes specific to patients with similar pathology. This will likely require review of primary articles as opposed to utilizing the systematic reviews or meta-analyses on the topic.

Our study found at least 1 type of spin in 70% (7/10) of abstracts of systematic reviews and meta-analyses regarding hip labrum reconstruction. We found that 50% had multiple types of spin, with 20% having 3 types of spin. Our results are similar to analyses of spin currently found in orthopaedic and other medical literature. Checketts et al found spin in 58.7% of lower extremity joint randomized controlled trial (RCT) abstracts, Arthur et al found spin in 44.8% of orthopaedic RCT abstracts, and Jones et al found spin in 34.2% of systematic reviews and meta-analyses pertaining to proximal humerus fractures.^22-24 Similar rates of spin have been found in other specialties, with 70% in ENT, 56% in psychiatry, and 31% in dermatology.^25-27

Two potential causes of the high incidence of spin within our study is the relative novelty of hip labrum reconstruction and the strict indications for the procedure. These 2 factors lead to a lower number of randomized control trials, smaller patient cohorts, relatively short duration of follow up, and insufficient data/studies to create comprehensive systematic reviews and meta-analyses. Of note, 40% (4/10) of the systematic reviews assessed in our study include a disclaimer within the abstract addressing the current state of evidence regarding this topic. For example, Orner et al stated in the conclusion of their abstract, because of study heterogeneity, low level of evidence, and high risk of bias, [treatment options] were unable to be directly compared.¹⁸ However, 60% (6/10) of the articles make recommendations or draw conclusions within the abstract without including a disclaimer regarding the the paucity of literature including primarily articles with level 3 or level 4 evidence in the form of retrospective case series or retrospective cohort studies.

The lack of adequate data regarding this topic leads to type 5 spin, “conclusion claims the beneficial effect of the experimental treatment despite high risk of bias in the primary studies.” An example of type 5 bias is found in the study by Bessa et al which concludes in the abstract that, autograft reconstruction of the acetabular labrum results in significant improvement in patient reported outcomes.⁵ However, their study contained primary studies with a high risk of bias due to having only 7 observational studies. None of the studies included in the systematic review by Bessa et al met the methodological index for non-randomized studies (MINORS) criteria for methodological quality.^2,5 One way to mitigate or prevent type 5 spin is by including a statement within the abstract as done by Al Mana et al long-term follow-up with higher quality studies was not available in the literature based on this review.⁴

Type 3 spin (“selective reporting of or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention) was found in 40% (4/10) of the studies in our analysis. One example of type 3 spin is found in the conclusion of an article’s abstract that suggests a “higher rate of conversion to total hip arthroplasty in the labral resection group” when compared to the labrum reconstruction group.²⁰ However, within the text of the manuscript they admit that only 1 of the 7 studies analyzed showed this increased conversion to THA in the resection group. They also describe no difference in patient reported outcomes between resection and reconstruction group; yet do not mention this finding within the abstract, but instead focus on the beneficial effect of reconstruction.

Another example of type 3 spin is in the abstract by Trivedi et al which states that there is decreased osteoarthritis after labrum reconstruction.²¹ However, they do not address this variable within the manuscript, or describe how this conclusion was made. Seven of the eleven studies (64%) analyzed in their study had an average follow up of less than 3 years, a short amount of time to the grade osteoarthritis.

The reasons behind spin in published research is not well studied but could be due to an unconscious bias in researchers, a desire to produce a more “impactful result” that would more likely get published or read, or perhaps a conscious attempt to influence readers in a particular direction.^12,29,30 Despite mechanism and intention of spin, it has the ability to influence the physicians reading the scientific literature.¹⁵ It is important for authors of systematic reviews and meta-analyses to temper drawing conclusions that are not supported by the articles included in the analysis. It is equally important for these authors to assess and recognize bias and spin within the included articles.

Bias can be more problematic when found within the abstract. The studies reviewed in our analysis acknowledged the paucity of unbiased primary studies on the topic of labrum reconstruction in the manuscript; however, excluding qualifying statements within the abstract may affect a reader’s opinion on the subject. As noted, with increasing publications in orthopaedic literature, it is possible that there is an increased reliance on abstracts; in fact, one study showed that 75% of surveyed family medicine physicians used abstracts as their primary source for guiding their treatment.³¹ Because abstracts play a vital role in conveying the results of research in a succinct way, it is important for them to report the evidence without bias or spin.

Our analysis also found that most systematic reviews regarding hip labrum reconstruction received an AMSTAR 2 rating of “critically low” to “moderate,” with no studies receiving a “high” rating. Although we did not find any significant association between AMSTAR 2 rating and prevalence of spin, the overall low ratings seem to correlate with the overall high prevalence of spin. As an example, 50% (5/10) of articles did not report adherence to item 2 on the AMSTAR 2 survey, requiring protocol to be registered before commencement of the review. This resulted in at least 1 critical flaw present in at least half of the articles.²⁶ The low ratings could be due to insufficient methodological rigor in research pertaining to a relatively novel topic such as hip labrum reconstruction, or it could be due to the relatively strict criteria set out by AMSTAR 2.

This study, along with previous studies in orthopaedics and other fields, reiterates the growing need for awareness of spin in medical literature. Increased education of spin can elevate a reader’s ability to critically appraise an article, also diminishing the distorting effects of spin. As physicians become more aware of the prevalence of spin and its potential effects, both the prevalence and its effects will likely decrease through the peer review process.

Limitations

Although we tried to make analysis as methodological as possible, evaluation and classification of spin remains a subjective evaluation. Furthermore, evaluation of AMSTAR 2 rating also involves a degree of subjectivity. Due to the relative novelty of hip labrum reconstruction, there was a paucity of systematic reviews and meta-analyses available for analysis, resulting in a relatively low sample size of 10 articles. Yavchitz et al also reported some limitations to their classification of spin in their original article, stating that their ranking of 9 most severe types of spin was based on individuals with expertise in the field of systematic reviews, but were unable to directly evaluate severity based on impact on readers. Finally, we only evaluated for the 9 most severe types of spin as reported by Yavchitz et al and therefore other types of spin were not evaluated.

CONCLUSION

We found that the majority (70%) of systematic reviews and meta-analyses regarding hip labrum reconstruction contained spin within their abstracts. Readers, especially physicians reading literature looking to make clinical decisions, should learn to critically appraise articles for the presence of spin and the potential effects of spin. By increasing awareness of spin in medical literature, we can potentially decrease its prevalence in medical literature and its resultant effect on clinical management.

Funding: No funding sources

Conflict of interest: None declared

Ethical approval: Not required

REFERENCES

1. 1. Alzaharani A, Bali K, Gudena R, Pamela R, Dragana P, John RM, et al. The innervation of the human acetabular labrum and hip joint: an anatomic study. BMC Musculoskelet Disord. 2014;15:41.
2. 2. Nepple JJ, Philippon MJ, Campbell KJ, Grant JD, Kyle SJ, Robert FL, et al. The hip fluid seal--Part II: The effect of an acetabular labral tear, repair, resection, and reconstruction on hip stability to distraction. Knee Surg Sports Traumatol Arthrosc Off J ESSKA. 2014;22(4):730-6.
3. 3. Philippon MJ, Nepple JJ, Campbell KJ, Grant JD, Kyle SJ, Robert FL, et al. The hip fluid seal--Part I: the effect of an acetabular labral tear, repair, resection, and reconstruction on hip fluid pressurization. Knee Surg Sports Traumatol Arthrosc Off J ESSKA. 2014;22(4):722-9.
4. 4. Al Mana L, Coughlin RP, Desai V, Simunovic N, Duong A, Ayeni OR. The Hip Labrum Reconstruction: Indications and Outcomes-an Updated Systematic Review. Curr Rev Musculoskelet Med. 2019;12(2):156-65.
5. 5. Bessa FS, Williams BT, Polce EM, Mansueto N, Flávio LG, Gustavo L, et al. Indications and Outcomes for Arthroscopic Hip Labral Reconstruction with Autografts: A Systematic Review. Front Surg. 2020;7:61.
6. 6. Philippon MJ, Briggs KK, Hay CJ, Kuppersmith DA, Dewing CB, Huang MJ. Arthroscopic labral reconstruction in the hip using iliotibial band autograft: technique and early outcomes. Arthrosc J Arthrosc Relat Surg Off Publ Arthrosc Assoc N Am Int Arthrosc Assoc. 2010;26(6):750-6
7. 7. Bastian H, Glasziou P, Chalmers I. Seventy-Five Trials and Eleven Systematic Reviews a Day: How Will We Ever Keep Up? PLOS Med. 2010;7(9):e1000326.
8. 8. Cook DJ, Mulrow CD, Haynes RB. Systematic Reviews: Synthesis of Best Evidence for Clinical Decisions. Ann Intern Med. 1997;126(5):376-80.
9. 9. Shea BJ, Grimshaw JM, Wells GA, Maarten B, Neil A, Candyce H, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7(1):10.
10. 10. Zhang H, Han J, Zhu YB, Wan-Yee L, Myron ES, Guo-Qiang X, et al. Reporting and methodological qualities of published surgical meta-analyses. J Clin Epidemiol. 2016;70:4-16.
11. 11. Shea BJ, Reeves BC, Wells G, Micere T, Candyce H, Julian M, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomized or non-randomized studies of healthcare interventions, or both. BMJ. 2017;358:j4008.
12. 12. Yavchitz A, Ravaud P, Altman DG, David M, Asbjørn H, Toby L, et al. A new classification of spin in systematic reviews and meta-analyses was developed and ranked according to the severity. J Clin Epidemiol. 2016;75:56-65.
13. 13. Boutron I, Altman DG, Hopewell S, Vera-Badillo F, Tannock I, Ravaud P. Impact of spin in the abstracts of articles reporting results of randomized controlled trials in the field of cancer: the SPIIN randomized controlled trial. J Clin Oncol Off J Am Soc Clin Oncol. 2014;32(36):4120-6.
14. 14. Ayeni OR, Alradwan H, de Sa D, Philippon MJ. The hip labrum reconstruction: indications and outcomes- -a systematic review. Knee Surg Sports Traumatol Arthrosc Off J ESSKA. 2014;22(4):737-43.
15. 15. Forster-Horvath C, von Rotz N, Giordano BD, Domb BG. Acetabular Labral Debridement/Segmental Resection Versus Reconstruction in the Comprehensive Treatment of Symptomatic Femoroacetabular Impingement: A Systematic Review. Arthrosc J Arthrosc Relat Surg Off Publ Arthrosc Assoc N Am Int Arthrosc Assoc. 2016;32(11):2401-15.
16. 16. Maldonado DR, Kyin C, Chen SL, Philip JR, Jacob S, Mitchell BM, et al. In search of labral restoration function with hip arthroscopy: outcomes of hip labral reconstruction versus labral repair: a systematic review. Hip Int J Clin Exp Res Hip Pathol Ther. 2021;31(6):704-13.
17. 17. Maldonado DR, Kyin C, Shapira J, Rosinsky PJ, Walker-Santiago R, Domb BG. Circumferential and Segmental Labral Reconstruction: A Systematic Review. Orthopedics. 2021;44(6):361-6.
18. 18. Orner CA, Patel UJ, Jones CMC, Giordano BD. Segmental and Circumferential Acetabular Labral Reconstruction Have Comparable Outcomes in the Treatment of Irreparable or Unsalvageable Labral Pathology: A Systematic Review. Arthrosc J Arthrosc Relat Surg Off Publ Arthrosc Assoc N Am Int Arthrosc Assoc. 2022;38(4):1341-50.
19. 19. Rahl MD, LaPorte C, Steinl GK, O’Connor M, Lynch TS, Menge TJ. Outcomes After Arthroscopic Hip Labral Reconstruction: A Systematic Review and Meta-analysis. Am J Sports Med. 2020;48(7):1748-55.
20. 20. Safran N, Rath E, Haviv B, Atzmon R, Amar E. The Efficacy of Labral Reconstruction: A Systematic Review. Orthop J Sports Med. 2021;9(2):2325967120977088.
21. 21. Trivedi NN, Sivasundaram L, Su CA, Derrick K, Shane JN, Richard CM, et al. Indications and Outcomes of Arthroscopic Labral Reconstruction of the Hip: A Systematic Review. Arthrosc J Arthrosc Relat Surg Off Publ Arthrosc Assoc N Am Int Arthrosc Assoc. 2019;35(7):2175-86.
22. 22. Checketts JX, Riddle J, Zaaza Z, Boose MA, Whitener JH, Vassar MB. An Evaluation of Spin in Lower Extremity Joint Trials. J Arthroplasty. 2019;34(5):1008-12.
23. 23. Arthur W, Zaaza Z, Checketts JX, Austin LJ, Kevin M, Clint B, et al. Analyzing Spin in Abstracts of Orthopaedic Randomized Controlled Trials with Statistically Insignificant Primary Endpoints. Arthrosc J Arthrosc Relat Surg Off Publ Arthrosc Assoc N Am Int Arthrosc Assoc. 2020;1443-50:e1.
24. 24. Jones C, Rulon Z, Arthur W, Ryan O, Jake C, Byron D, et al. Evaluation of spin in the abstracts of systematic reviews and meta-analyses related to the treatment of proximal humerus fractures. J Shoulder Elbow Surg. 2021;30:2197-205.
25. 25. Cooper CM, Gray HM, Ross AE, Tom AH, Jaye BD, Cole W, et al. Evaluation of spin in the abstracts of otolaryngology randomized controlled trials. The Laryngoscope. 2018;2036-40.
26. 26. Jellison S, Roberts W, Bowers A, Tyler C, Jason B, Cole W, et al. Evaluation of spin in abstracts of papers in psychiatry and psychology journals. BMJ Evid-Based Med. 2019:178-81.
27. 27. Morgan G, Tremayne K, Ryan O, Wade A, Taylor CR, Micah H, et al. Evaluation of Spin in the Abstracts of Systematic Reviews and Meta-Analyses Focused on the Treatment of Acne Vulgaris: Cross-Sectional Analysis. Tob Prev Cessat. 2021;7:35.
28. 28. Slim K, Nini E, Forestier D, Kwiatkowski F, Panis Y, Chipponi J. Methodological index for non-randomized studies (minors): development and validation of a new instrument. ANZ J Surg. 2003;73(9):712-6.
29. 29. Fletcher RH, Black B. “Spin” in scientific writing: scientific mischief and legal jeopardy. Med Law. 2007;26(3):511-25.
30. 30. Weed DL. Preventing scientific misconduct. Am J Public Health. 1998;88(1):125-9.
31. 31. Barry HC, Ebell MH, Shaughnessy AF, Slawson DC, Nietzke F. Family physicians’ use of medical abstracts to guide decision making: style or substance? J Am Board Fam Pract. 2001;14(6):437-42.