Patient-generated health communication: a novel readability analysis of online reviews in orthopaedic surgery

INTRODUCTION

Health literacy is essential for effective health communication. It influences patient outcomes, adherence to treatment, and overall satisfaction with healthcare services.¹ Numerous health organizations, including the National Institutes of Health (NIH) and the American Medical Association (AMA), have consistently recommended that health education materials be composed at or below a sixth-grade reading level to ensure they are accessible to the average American.^2-4 However, despite these recommendations, several studies have found that many patient resources continue to be composed at readability levels that exceed these guidelines, potentially limiting their effectiveness.^5,6

The literature has consistently highlighted the disparity between the average American literacy level and the complexity of health-related written communication. For example, Ghanem et al in 2024 determined that web-based patient education materials, including those provided by major orthopaedic organizations, such as the American Academy of Orthopaedic Surgeons (AAOS) and the American Orthopaedic Society for Sports Medicine (AOSSM), are written at a reading level significantly higher than recommended.⁷ The average readability of AAOS and Orthopaedic Trauma Association (OTA) patient education materials, for instance, were found to be at the ninth- and eighth-grade levels, respectively, far exceeding the sixth-grade recommendation. The basic premise of these studies is that the sample populations used in national surveys, whose findings underpin the current readability standards, are representative of online consumers of health information and readers of patient education materials (PEM).

While much attention has been paid to the complexity of physician-generated content, little is known about the readability of online PEM content by consumers. Although making PEM readable for patients with limited literacy is advantageous, there are some trade-offs.^4,8 Lowering text readability might lead to bland text that lacks sufficient information to keep readers engaged.^3,9,10 Patients who engage in online-seeking PEM could be more educated and able to tolerate higher readability PEM than the general population.

Our literature review did not identify any studies that examined the readability levels of patients who actively sought health information online. This may be due to the inherent difficulty in identifying whether patients or caregivers are actively seeking online health information. Additionally, most online health information-seeking activities are anonymous, making it challenging to identify who consumes the content. This anonymity complicates the efforts to accurately survey and assess the readability of individuals seeking health information. We hope to bridge this gap by conducting this study to explore the readability of patient-written online reviews of orthopaedic surgeons as a form of natural health communication, aiming to understand how patients convey health experiences in their own words to inform communication strategies in digital health contexts.

Study objective

Unlike physician-created content, whose readability has been studied extensively, there is a notable gap in the literature regarding the readability of content written by the patients themselves. This study aimed to address this gap by evaluating the readability of online reviews by patients of American Association of Hip and Knee Surgeons (AAHKS).

This novel approach seeks to enhance our understanding of how patients interact with online health information and could provide valuable insights into the alignment between patient-generated content and educational materials provided by healthcare professionals.

METHODS

Study design and data collection

This cross-sectional observational study was designed to evaluate the readability of online patient reviews for orthopaedic surgeons and to compare these findings with the readability of standardized PEM. This study was conducted in accordance with institutional ethical guidelines, and data collection focused on publicly available online content.

Surgeon selection

Orthopaedic surgeons were identified using the AAHKS “Find a Doctor” tool. The surgeons were randomly selected using a stratified sampling approach to ensure a geographically diverse and representative sample size. Specifically, for every 20 surgeons listed in each state, one surgeon was randomly chosen using a computer-generated random-number algorithm. This method was employed to reduce selection bias and ensure coverage across different regions and practice settings.

Review selection and data extraction

For each selected surgeon, the most recent patient review containing a minimum of 50 words was retrieved from Healthgrades.com, ensuring that only substantive reviews were included in the analysis. The corresponding star rating (on a scale of 1 (low) to 5 (high)) and the reviewer’s ZIP code were also recorded. If no qualified review was available, an alternative surgeon from the same state was randomly selected.

Readability analysis

The collected reviews were subjected to a comprehensive readability analysis using a suite of validated tools, including the Flesch-Kincaid grade level (FKGL). These tools were chosen for their widespread use and ability to assess different aspects of text complexity.

Each review was independently processed to calculate readability scores, which were then averaged to determine the overall readability level of the patient population.

Socioeconomic data collection

To explore the potential socioeconomic factors influencing readability, the median household income for each review’s corresponding ZIP code was extracted from the most recent U.S. Census Bureau data.⁴

This data was used to categorize reviews into income brackets for further analysis based on the recommended income levels from the U.S. Census Bureau’s 2022 Income in the United States Report.¹¹ The income brackets included lower, lower-middle, middle, upper-middle, and upper classes.

Statistical analysis

Statistical analyses were performed to explore the correlations between readability scores and various factors, including review length, star rating, geographic region, and median household income. Pearson's correlation coefficient was used to analyze continuous variables, and ANOVA was used to compare categorical variables. A one-way sample t-test was used to determine whether the average FKGL literacy level of the patients in this study was significantly different from the sixth-grade reading level recommended by readability guidelines.^2,12-14 Statistical significance was set at p<0.05. Additionally, this study examined the alignment between the readability of patient reviews and that of OrthoInfo materials. This comparative analysis aimed to assess whether the current educational resources are appropriately tailored to the literacy levels of the patient population.

Sensitivity analysis

A sensitivity analysis was conducted to determine the robustness of our findings, including subgroup analyses based on region, income bracket, and surgeon. This analysis was conducted to identify potential confounding factors and validate the generalizability of the study results.

RESULTS

A total of 114 patient reviews and ratings (n=114) of orthopaedic surgeons were compiled according to our random protocol. The readability analysis of patient-generated online reviews for orthopaedic surgeons revealed a mean FKGL of 7.46 (SD=2.49), with a median grade level of 7.24. A one-sample t-test indicated that the mean FKGL readability score (M=7.46, SD=2.49) in this study population was significantly higher than the sixth-grade reading level recommended by the NIH and the AMA (t (114) =6.251, p<0.001), with a medium effect size (d=0.585) (Figure 1).

Figure 2 illustrates that 88% (100/114) of the reviews were composed at a level above the fifth grade, 73% (83/114) exceeded the sixth-grade level, 56% (64/114) surpassed the seventh-grade level, and 34% (39/114) were at or above the eighth-grade level.

Further statistical analysis using ANOVA revealed no significant associations between the readability level of the reviews (using the FKGL readability score) and the patient-assigned star ratings for surgeons (p=0.976), the geographic region of the reviewer (p=0.697), or the median household income of the reviewer’s ZIP code (p=0.720). No significant associations were found between the region of the reviewer and the rating given to the surgeon (F (3, 110) =1.091, p=0.356) or FKGL of the review (F (3,110) =1.283, p=0.284), as shown in Figures 3 and 4, respectively.

The average median household income based on the ZIP code was $78,276 (SD $32,471). The average star rating out of five stars given to the orthopaedic surgeon was 4.44 (SD=1.32). A one-way ANOVA was performed to assess the impact of median household income on the average surgeon rating, which revealed a statistically significant difference between income groups (F (108, 5) =9.124, p=0.010). However, when the median household income was grouped into five separate Census Bureau income brackets, no significant difference was found between the income brackets and the average surgeon ratings (F (4, 109) =0.918, p=0.456) or the FKGL readability scores in their review (F (4, 109) =1.027, p=0.397).

The descriptive statistics of surgeon ratings, along with additional readability scores of patient reviews for other FKGL, are listed in Table 1.

DISCUSSION

The main findings of this study were that orthopaedic patients who engage online are able to write at a level much higher than the sixth-grade reading level recommended by leading public health organizations and, therefore, might be able to use health communication materials written at a more complex level.³ The average FKGL literacy level found in this study was 7.46, or between seventh- and eighth-grade reading levels. This study also found that approximately 73% (88/114) of the patient reviews had a literacy level at or above the recommended sixth-grade level. These data challenge the assumption that health-related communications must strictly adhere to lower readability standards, suggesting that patients are engaging with and producing content at a higher literacy level than previously anticipated.

Clear communication between healthcare providers and patients is essential for achieving optimal health outcomes.^1,15 Numerous studies have consistently demonstrated that inadequate communication can lead to poor treatment adherence and lower patient satisfaction, emphasizing the importance of providing easily understandable information.^16,17 As medical knowledge and technology continue to evolve, the complexity of information shared with patients has increased, making it more challenging for the average individual to fully comprehend their health and treatment options.^4,18

In 2024, Ghanem et al. determined that the average FKGL for trauma-related web-based patient communication content from the American Academy of Orthopaedic Surgeons (AAOS) and the Orthopaedic Trauma Association (OTA) was 8.9 (SD=0.74) and 8.1 (SD=1.14), respectively.² This finding is not unique; hundreds of studies published since the initial 2008 AAOS forum publication on this topic have consistently reported this disparity in online PEM.⁸ Our findings indicate that the communication style and language level used in online PEM may still need to be slightly simplified, but not necessarily to as low as the sixth-grade reading level. This reinforces the idea that communication strategies should consider actual patient language patterns and engagement, rather than relying solely on legacy literacy thresholds.

There was also no significant association between a patient's FKGL literacy level and the geographical area of the reviewer, median household income, or patient-assigned star rating for surgeons. These findings suggest that patient literacy, as reflected in online reviews, operates independently of these variables. The significant difference that we observed between the median household income and average surgeon rating (p=0.010) suggests that household income has a significant positive influence on how participants rated the surgeon. However, the lack of a significant correlation between readability and socioeconomic factors further implies that patients can consistently write at higher levels of readability, regardless of their background or satisfaction with care.

Traditional readability tools, such as the FKGL, have been criticized for their limitations in capturing the true comprehension and structural nuances of texts. Despite these criticisms, these tools offer an objective measure of text complexity and have been widely validated in healthcare settings. Our study leveraged these tools to compare the readability levels of patient-generated content with those of educational materials provided by physicians. Despite their imperfections, the argument for using these conventional tools lies in their objectivity and consistency of evaluation. However, these tools may not fully capture the communicative value of patient-generated texts.

Our study has significant limitations. Not all reviews may have been left by patients, as some were likely written by caregivers. Although this might be viewed as a weakness, we believe that it represents the target population more accurately. Caregivers who go online are often deeply involved in patients' healthcare choices and decision-making, serving as an integral part of our target population in terms of PEM. Additionally, the recommendations of this study should be applied specifically to online health communication materials. Printed PEM must still adhere to current guidelines, as a significant proportion of our patients face socioeconomic disadvantages, belong to racial/ethnic minorities, are aged 65 years and older, live in rural areas, or are immigrants—groups that are more likely to experience limited literacy. The readability tools we used, such as FKGL, were originally developed more than 75 years ago, long before the advent of computers. They were purposefully simplistic to enable manual calculations. With the advent of software, readability can be calculated much faster. However, the basic limitations of these readability tools remain true to this day: they are formulaic and rely on sentence structure and word complexity based on syllables.

The advent of generative artificial intelligence (AI) offers an opportunity to move beyond traditional readability scales and approach readability and comprehension of health communication from a new angle. First, natural language processing (NLP) models can analyze text in a more nuanced manner by considering context, intent, and tone rather than simply sentence length and syllable count. This allows for a more sophisticated understanding of whether a text is truly accessible to the intended audience. Second, AI can offer personalized readability adjustments by tailoring PEM to the literacy level of individual patients or caregivers, which is a major leap forward in health communication.¹⁹ Finally, AI-driven PEM offers real-time adaptability, allowing content to be dynamically modified based on user interactions. For instance, it can provide simpler explanations when a user struggles to grasp a concept or offer more detailed information when necessary or upon request. These advancements have the potential to revolutionize how we think about and create patient-centered communication, making it more accessible, relevant, and effective for a broader range of individuals.

This is the first study of its kind to analyze the readability of patient-generated online content across any medical specialty, providing insights that could reshape standards for online PEM. By aligning the complexity of educational materials with the demonstrated literacy capabilities of patients, the quality of health communication can be enhanced. Oversimplification to a fifth- or sixth-grade reading level can strip away the essential context and depth, resulting in overly generic and less informative content.^3,8 This study underscores the need to reassess and potentially revise current readability guidelines to better serve and empower patients through more sophisticated and comprehensive educational resources.

CONCLUSION

This study indicated that the online population of patients and caregivers may be able to comprehend PEM at a significantly higher level than the recommended sixth-grade reading level. Our results suggest that online health communication materials can be crafted at a higher readability level, enriching content without sacrificing patient comprehension or engagement. Simplifying materials to a lower reading level can eliminate essential context and depth. By aligning the complexity of health materials with the demonstrated writing capabilities of patients, and possibly using generative artificial intelligence tools, we can create health information that better supports patient education and literacy.

Funding: No funding sources

Conflict of interest: None declared

Ethical approval: The study was approved by the Institutional Ethics Committee

REFERENCES

1. 1. Mitchell JP. Association of provider communication and discharge instructions on lower readmissions. J Healthc Qual. 2015;37:33-40.
2. 2. Institute of Medicine (US) Committee on Health Literacy. Health Literacy: A Prescription to End Confusion. Nielsen-Bohlman L, Panzer AM, Kindig DA, editors. Washington (DC): National Academies Press (US). 2004.
3. 3. Weiss BD. Health literacy: help your patients understand: a continuing medical education (CME) program that provides tools to enhance patient care, improve office productivity, and reduce healthcare costs: American Medical Association Foundation and American Medical Association. 2003.
4. 4. Badarudeen S, Sabharwal S. Assessing readability of patient education materials: current role in orthopaedics. Clin Orthop Relat Res. 2010;468:2572-80.
5. 5. Roberts H, Zhang DF, Dyer GSM. The Readability of AAOS Patient Education Materials: Evaluating the Progress Since 2008. J Bone Joint Surg-Am. 2016;98:6.
6. 6. Rooney MK, Santiago G, Perni S, Horowitz DP, McCall AR, Einstein AJ, et al. Readability of patient education materials from high-impact medical journals: a 20-year analysis. J Patient Exp. 2021;8:2374373521998847.
7. 7. Ghanem D, Covarrubias O, Maxson R, Sabharwal S, Shafiq B. Readability of Trauma-related Patient Education Materials From the American Academy of Orthopaedic Surgeons and Orthopaedic Trauma Association Websites. J Am Acad Orthop Surg. 2024;32:e642-50.
8. 8. Badarudeen S, Sabharwal S. Readability of patient education materials from the American Academy of Orthopaedic Surgeons and Pediatric Orthopaedic Society of North America web sites. J Bone Joint Surg Am. 2008;90:199-204.
9. 9. Jiggins K. A content analysis of the Meaningful Use clinical summary: do clinical summaries promote patient engagement? Prim Health Care Res Dev. 2016;17:238-51.
10. 10. Wang LW, Miller MJ, Schmitt MR, Wen FK. Assessing readability formula differences with written health information materials: application, results, and recommendations. Res Social Adm Pharm. 2013;9:503-16.
11. 11. Semega J, Kollar M. Income in the United States: 2021. US Census Bureau, Current Population Reports. 2022;60-76.
12. 12. Clear Communication National Institutes of Health (NIH). National Institutes of Health. 2023. Available at: https://www.nih.gov/institutes-nih/nih-office-director/office-communications-public-liaison/clear-communication. Accessed on 12 May 2025.
13. 13. Weiss BD, Coyne C, Michielutte R, Davis TC, Meade CD, Doak LG, et al. Communicating with patients who have limited literacy skills - Report of the National Work Group on Literacy and Health. J Fam Pract. 1998;46:168-76.
14. 14. Kutner M, Greenberg E, Baer J. National Assessment of Adult Literacy (NAAL). A first look at the literacy of America’s adults in the 21st century: Washington DC: National Center for Education Statistics, US Department of Education. 2005.
15. 15. Sabharwal S, Badarudeen S, Unes Kunju S. Readability of online patient education materials from the AAOS web site. Clin Orthop Relat Res. 2008;466:1245-50.
16. 16. Weiss BD. Health literacy. American Medical Association. 2003;253:358.
17. 17. Nielsen-Bohlman L. Health Literacy: A Prescription to End Confusion: National Academies Press. 2004;345.
18. 18. Miskiewicz M, Capotosto S, Ling K, Hance F, Wang E. Readability Analysis of Patient Education Material on Rotator Cuff Injuries From the Top 25 Ranking Orthopaedic Institutions. J Am Acad Orthop Surg Glob Res Rev. 2024;8:20240509.
19. 19. Ribeiro LFR, Bansal, Mohit, Dreyer, Markus. Generating Summaries with Controllable Readability Levels. 2023. Available at: https://arxiv.org/abs/2310.10623. Accessed on 12 May 2025.