Development, validation, and reliability testing of the standard treatment workflow checklist

INTRODUCTION

Standard treatment workflows (STWs) are systematically developed documents designed to assist clinicians in providing better and more appropriate healthcare. There is standard treatment guidelines (STGs) and procedures from many organizations, however, they are dense, complex, and difficult to review. As a result, clinicians need some simple, basic, and easy-to-reference guidance materials that can be viewed in several ways. To meet this requirement, the Indian Council of Medical Research (ICMR) collaborated with the National Health Authority, the Government of India, and the World Health Organization (WHO) India Country Office to develop STWs. These workflows have been designed for common and serious diseases clinicians encounter across all healthcare system levels, from primary to tertiary care.¹

The STWs are expected to bring a structured and accurate case management protocol in a concise format. They are essential for achieving uniform standards of care in the Indian public healthcare system, assisting with effective administration, and promoting proper utilization of drugs, healthcare services, and other diagnostics. These STWs, prepared by national specialists with significant expertise, evaluate the feasibility of implementation within India's healthcare system. Each STW is a one-page document with key indicative actions. As of July 30, 2024, STWs for 157 disease conditions across 28 specialties, compiled into four volumes, were available in both print and digital formats.

The primary purpose of STWs is to guide clinicians in the rational use of medicines, ultimately benefiting patients and working towards universal health coverage. In the health care system, STWs assist in formulating and budgeting services, becoming crucial for overseeing and approving processes in government-funded health insurance plans.²

Validation of STWs is paramount since no such concise, nationwide documents existed before their introduction by ICMR. Studies should ensure that STWs meet the desired quality and performance criteria, enabling intended users to utilize them effectively. While validated tools are available to ensure guideline transparency, quality, and completeness, directly using these tools to evaluate STWs is not feasible. These tools are designed to evaluate the detailed process from development to the final implementation of comprehensive practice guidelines, whereas STWs deliver critical information concisely, serving as quick reference guides.^3,4

To address this evidence gap, we have developed a checklist for evaluating and appraising STWs in practice. Ensuring the validity and reliability of this new checklist is crucial to obtaining relevant information on STWs. Validity ensures that the collected data accurately covers the intended area of investigation, measuring what is intended to be measured. Reliability describes the “consistency and stability of measurements”. A measurement tool is considered reliable if it provides consistent results under constant conditions.⁵ Testing for both validity and reliability is essential to ensure that the developed checklist effectively evaluates and appraises STWs. Therefore, this study aims to develop, validate, and assess the reliability of the checklist developed to evaluate STWs.

METHODS

This methodology will be a part of a mixed-method explanatory study aimed at validating STWs and assessing their feasibility of implementation in diverse healthcare settings, both public and private, across the selected regions. This study was conducted in the first quarter of 2024. The study employs a cross-sectional design conducted in three phases: Phase One: Checklist Development. Phase Two: Validation of the checklist, focusing on content validity. Phase Three: Testing the reliability of the checklist, specifically its internal consistency. A detailed flow of the methodology is illustrated in Figure 1.

Phase 1: checklist development

Content and item generation: The initial checklist items were developed by reviewing STW guidelines, conducting a comprehensive literature review on treatment workflow validation, consulting with experts, and examining healthcare quality assurance practices. The checklist is in a closed-ended format with three response options per item, where respondents select the option that best describes their answer. We developed the checklist following AGREE II guidelines for rigorous methodology and transparency. This included gathering and synthesizing evidence and continually refining items. By adhering to AGREE II, we ensured high quality, and reliability in evaluating treatment workflows.⁴

We generated around 34 items, focusing on adherence to evidence-based guidelines and regulatory compliance. We then created a checklist using Google Forms, ensuring accessibility via computers, smartphones, or tablets.

Pre-testing

Comprehensiveness, clarity, and response of the checklist: We pre-tested the checklist's comprehensiveness, clarity, and response options with the help of 50 experts. These aspects are defined as follows: 1) Comprehensiveness: Do the items cover all important aspects?, 2) Clarity: Are the items clear and easy to understand?, 3) Response Options: Are the response options appropriate and relevant to the constructs measured in the checklist?

Phase 2: checklist validation

A panel of healthcare professionals specialized in clinical practice, workflow management, and quality assurance conducted the content validation for the checklist. Domain experts from various departments were chosen based on their professional experience in relevant fields. They individually reviewed each item for essentiality and relevance, providing open-ended comments as needed. All experts gave informed consent via the Google Form before participating. A larger number of experts were involved to minimize agreement by chance. Each item was independently rated using a 4/3-point scale, guided by a scoring table in the Google Form to ensure consistency.

Content Validity Index (CVI): CVI method assesses “how well each item reflects the intended construct”. Each item is rated by experts on a scale of 1 (not relevant) to 4 (very relevant). CVI is a common indicator of content validity, and it is calculated using Item-level-CVI (I-CVI) and Scale-level-CVI (S-CVI).

To compute I-CVI, we divide the number of experts who rate item 3 or 4 by the total number of experts. It involves dichotomizing ratings by combining 3 and 4 as relevant, and 1 and 2 as irrelevant, then calculating the agreement among experts. The I-CVI measures expert agreement on item relevance, with a range from zero to one, reflecting the degree of consensus among experts. Items with an I-CVI more than 0.79 are considered relevant, while those with values between 0.70 and 0.79 may require improvements, and items less than 0.70 are deleted.⁶

S-CVI assesses overall content validity. It can be calculated using “S-CVI/UA (Universal Agreement) or S-CVI/Ave (Average)”. S-CVI/UA is the proportion of items evaluated as "very relevant" (I-CVI = 1) among all items, whereas S-CVI/Ave is the average of all I-CVIs. An S-CVI/UA ≥0.8 or an S-CVI/Ave ≥0.9 depicts strong content validity, ensuring the items accurately represent the intended content.

Kappa statistics: While CVI is widely used for content validity, Wynd et al suggested using the kappa statistic to account for chance agreement.⁷ The kappa statistic measures agreement beyond chance using the following method:

This formula uses PC (probability of chance agreement), N (number of experts), and A (number of experts who consider an item relevant). Kappa values more than 0.74 are superior, 0.60 to 0.74 are good, and 0.40 to 0.59 are acceptable.

Content validity Ratio (CVR): “CVR measures the essentiality of each item, with scores of 1 (not essential), 2 (useful but not essential), and 3 (essential)”. Ratings are dichotomized (combining 1 and 2 as non-essential), and agreement among experts is calculated accordingly. CVR is then computed using the following formula.

Where; Ne is the number of experts marked "essential" and N is the total number of experts. CVR scores range from -1 (complete disagreement) to +1 (complete agreement), with higher scores indicating greater agreement on an item's necessity. The CVR score is compared with the Lawshe Table to determine statistical significance. All the definitions are provided in the Table 1.

Note: The definition and formula were based on the recommendations of Lynn et al⁹, Davis et al¹⁰, Polit et al¹¹

Phase 3: reliability testing (pilot testing)

After validating the checklist, a pilot test was conducted with the final version of the checklist among 30 doctors providing services at tertiary care centers. A sample size of 30 was chosen based on literature review.¹² Internal consistency and reliability are evaluated using correlation analysis, such as Cronbach's alpha and item-total correlations. Item-total correlations below 0.3 indicate an insufficient correlation.¹³ Cronbach's alpha evaluates response consistency on a scale of 0 to 1, with values closer to 1 indicating more reliability. The usually accepted minimum is 0.7.¹⁴

Data analysis

Data were exported from Google Forms, cleaned in Excel, and analyzed using SPSS version 17.0.

RESULTS

Phase 1

Comprehensiveness, clarity, and response results: Comprehensiveness: Of the 50 experts, 46 (90.2%) agreed that the items cover all important aspects. Clarity: Approximately 49 experts (96.1%) confirmed that the items are clear and easy to understand. Response Options: About 49 experts (96.1%) affirmed that the response options are appropriate and relevant to the constructs measured.

Phase 2

Socio-demographic details: A total of 43 experts participated in the study. Specifically, 3 experts evaluated only the CVI, 18 evaluated only the CVR, and 20 evaluated both the CVI and CVR, resulting in a total of 23 experts evaluating the CVI and 40 evaluating the CVR. The sample size for performing content validation was calculated by taking 10% of the original study's sample size (N=400). Descriptive statistics summarized the expert’s demographics. The mean age was 41.7 (SD = 11.8) years, with an average of 12.8 (SD = 10.4) years of experience. Most experts were male (67.4%) and held MD and MS qualifications (65.1%). Common specialties included paediatrics (16.3%), general medicine (11.6%), and nephrology (9.3%) and the remaining are from various other departments. Many experts worked in tertiary care settings (81.4%) and were employed in the private sector (65.1%). Professors and Assistant Professors each comprised 20.9% of the group. Demographic details are presented in Table 2.

I-CVI= item-level content validity index; CVR= content validity ratio; S-CVI/Ave= scale-level content validity index/average, S-CVI/UA = scale-level content validity index/ universal agreement, CVR/Ave = Average content validity ratio

Content validity results

Content validity (CVI and CVR) was calculated for all 34 items. All the results are presented in Table 3.

I-CVI results: The I-CVI analysis, given in Table 1, shows that all items were deemed important, with I-CVI values ranging from 0.96 to 1.00. Specifically, 29 items had an I-CVI of 1.00, whereas five had an I-CVI of 0.96. All I-CVI values exceeded the 0.79 criterion, hence all items were considered relevant, and none were eliminated.

S-CVI results: This checklist has high content validity, as indicated by the S-CVI/UA and S-CVI/Ave metrics. The S-CVI/UA is 0.85, computed as the ratio of items having an I-CVI of 1.00 (29 items) to the total of 34 items, which is greater than the recommended threshold of 0.8. The S-CVI/Ave is extremely high at 0.99, calculated by dividing the sum of all I-CVIs (33.7) by the total number of items, which is above the 0.9 criterion. The Kappa values for the items ranged from 0.95 to 1.00, falling into the "excellent" category based on established benchmarks.

CVR results: The CVR scores ranged between 0.75 and 1.00, with every single item classified as "essential." Specifically, 24 items (70.6%) had a CVR of 1.00, nine items (26.5%) had a CVR of 0.85 to 0.95, and only one item (2.9%) had a CVR of 0.75. The average CVR for all items was 0.98. All CVR values exceeded the 0.29 threshold required for a group of 40 experts, hence every item is considered essential.¹⁵

Checklist refinement

Given the high CVI, CVR, and Kappa statistics values, the checklist was not refined, and a reliability assessment was performed in the next phase.

Phase 3

Socio-demographics: The pilot test involved 30 study participants. The mean age (SD) of the participants was 30 (35.1) years. Of these participants, 16 (53%) were male and 14 (47%) were female. Among them, 11 (37%) reviewed STWs for dermatology, 5 (17%) reviewed STWs for psychiatry, and the remaining participants reviewed STWs for various other departments.

Reliability: The checklist was divided into two parts: part one consisted of 23 items with 3 response options, and part two comprised a 10-item Likert scale. Reliability testing was conducted separately for each part.

Part 1: Part 1 of the scale indicated strong internal consistency, with a Cronbach's alpha value of 0.861, indicating a reliable and consistent measurement. Most items' corrected item-total correlations were above 0.3, showing they contributed appropriately to the scale. Although a few items (e.g., items 1, 7, and 21) had lower item-total correlations, their impact on overall reliability was minimal, and they were not removed because experts identified them as important. When an item is deleted, the Cronbach's alpha values range from 0.844 to 0.869. None of the items, if removed, significantly enhance the overall alpha. This suggests that all items are reasonably contributing to the reliability of the scale. Results are presented in Table 4.

Part 2: The 10-item Likert scale exhibited strong internal consistency, with Cronbach's alpha at 0.963, indicating high reliability in measuring the intended construct. All items showed strong item-total correlations, and removing any item would not significantly affect the overall reliability. Results are presented in Table 5.

DISCUSSION

The development and validation of a checklist for evaluating STWs were methodically executed to ensure its reliability and relevance across diverse clinical settings. This study demonstrated the efficacy of the developed checklist through rigorous phases of literature review, pre-testing, content validation, and reliability testing. The initial phase focused on generating relevant items through an extensive literature review and expert consultations. This approach ensured that the checklist covered all critical aspects of STWs, providing a robust foundation for subsequent validation steps.

The content validation process involved 43 experts who assessed the items for relevance and essentiality. CVR and CVI were pivotal in determining the relevancy and essentiality of each item. The high I-CVI values (0.96 to 1.00) and CVR values (above 0.85) reflect a consensus among experts on the essentiality of the items. The S-CVI scores for “universal agreement (S-CVI/UA) and average (S-CVI/Ave)” are 0.83 and 0.99, respectively. These indicators show a high level of agreement among experts, which validates the checklist's content. Cronbach's alpha for the two parts of the checklist were 0.863 and 0.961, respectively, based on reliability testing using correlation analysis involving 30 participants. These results signify high internal consistency, confirming that the checklist items are reliably measuring the intended aspects of STW validation.

Some studies complement one another by focusing on distinct areas of clinical practice and validation, such as implementing guidelines and evaluating new technologies. While some highlight the actual application of treatment procedures, others stress the significance of strong validation methodologies to ensure reliability and effectiveness. Similar studies have undertaken validation processes for clinical tools. For instance, Zadeh et al conducted a content validity study for a nursing assessment tool, achieving high I-CVI values and demonstrating the tool's reliability.¹⁶ Similarly, a study on scale development by Grover et al emphasized the systematic approach in formulating standard treatment guidelines, ensuring their applicability across various healthcare levels.² Sharma et al undertook a study to analyze the adoption of STG in India and identified various facilitators and barriers. The findings highlighted the importance of accessible medical information and the practical challenges faced by clinicians in implementing these guidelines.¹⁷

Bano et al conducted research comparing different diagnostic and treatment approaches, aiming to identify new gold standards for translational research. The study emphasized the need for proper validations to ensure the effectiveness of new methodologies.¹⁸ Another study evaluated the accuracy and trueness of digital impressions compared to traditional gypsum checks. This study provided insights into the clinical application of digital technologies and their reliability in dental practice.¹⁹

These studies complement each other by addressing different aspects of clinical practice and validation, from guideline implementation to the evaluation of new technologies. While some focus on the practical adoption of treatment workflows, others emphasize the importance of robust validation methods to ensure reliability and effectiveness. Hence, the validation of STWs in India is crucial for ensuring quality healthcare delivery and achieving universal health coverage. The development of comprehensive STGs in India has been a significant endeavor, facing challenges such as a lack of common STG, a weak understanding of the concept, and managing consensus between specialists and generalists.²

Conformance-checking methodologies using neural networks have been proposed to ensure adherence to standard treatment plans extracted from reputable healthcare websites. Reviewing available guidelines globally has highlighted the need for coordinated efforts, standardized development processes, and a designated authority for STG development to ensure evidence-based, rigorously developed guidelines with wide accessibility and acceptance.²⁰ Validating STWs in India and aligning with similar articles, the healthcare system can enhance implementing evidence-based methods will enhance patient outcomes and aid in achieving universal health coverage. This study is crucial because it addresses a gap by developing a valid checklist for the validation of STWs. However, it has some limitations. We intend to do the Principal Component Analysis and exploratory factor analysis as part of improving the checklist with the continuous dynamic data being obtained from clinicians. Since the study involves ever-evolving STWs it becomes necessary to improve the quality of the checklist alongside.

CONCLUSION

The study's findings indicate that the developed checklist has strong content validity and reliability, as evidenced by the I-CVI, S-CVI, CVR, and Cronbach's alpha scores. As of our knowledge, this is the first checklist developed in India to assess the validity of STWs. We anticipate that the developed checklist will serve as a standard tool for evaluating the validity of STWs, as it has been illustrated with high validity and reliability. We utilized this checklist to conduct our further study on the validity of the STWs currently in practice.

ACKNOWLEDGEMENTS

We would like to thank all the participants who participated in the study.

Funding: This study was funded by the ICMR

Conflict of interest: None declared

Ethical approval: The study was approved by the AMTZ IC SCR with the clearance number REGN. #ECR/1702/INST/AP/2022

REFERENCES

1. 1. STWs. Indian Council of Medical Research | Government of India. Available at: https://www.icmr.gov.in/stws. Accessed on 15 May 2024.
2. 2. Grover A, Bhargava B, Srivastava S, Sharma LK, Cherian JJ, Tandon N, et al. Developing Standard Treatment Workflows-way to universal healthcare in India. Front Public Health. 2023;11:1178160.
3. 3. EQUATOR Network. A Reporting Tool for Practice Guidelines in Health Care: The RIGHT Statement. Available at: https://www.equator-network.org/reporting-guidelines/right-statement/. Accessed 12 July 2024.
4. 4. EQUATOR Network. The AGREE Reporting Checklist: a tool to improve reporting of clinical practice guidelines. Available at: https://www.equator-network.org/reporting-guidelines/the-agree-reporting-checklist-a-tool-to-improve-reporting-of-clinical-practice-guidelines/. Accessed 24 June 2024.
5. 5. Taherdoost H. Validity and reliability of the research instrument; how to test the validation of a questionnaire/survey in a research. Int J Acad Res Manage. 2016;5:28-36.
6. 6. Shi J, Mo X, Sun Z. Content validity index in scale development. Zhong Nan Da Xue Xue Bao Yi Xue Ban. 2012;37(2):152-5.
7. 7. Wynd C, Schmidt B, Schaefer M. Two quantitative approaches for estimating content validity. Western J Nurs Res. 2003;25:508-18.
8. 8. Yusoff MS. ABC of content validation and content validity index calculation. Educat Medi J. 2019;11(2):49-54.
9. 9. Lynn MR. Determination and quantification of content validity. Nurs Res. 1986;35(6):382-5.
10. 10. Davis LL. Instrument review: Getting the most from a panel of experts. Applied Nursing Research. 1992;5(4):194-7.
11. 11. Polit DF, Beck CT, Owen SV. Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Res Nurs Health. 200;30(4):459-67.
12. 12. Bujang MA, Omar ED, Foo DHP, Hon YK. Sample size determination for conducting a pilot study to assess reliability of a questionnaire. Restor Dent Endod. 2024;49(1):e3.
13. 13. Jansen M, Doornebosch AJ, de Waal MW, Wattel EM, Visser D, Spek B, et al. Psychometrics of the observational scales of the Utrecht Scale for Evaluation of Rehabilitation (USER): Content and structural validity, internal consistency and reliability. Arch Gerontol Geriatr. 2021;97:104509.
14. 14. Gottems LBD, Carvalho EMPD, Guilhem D, Pires MRGM. Good practices in normal childbirth: reliability analysis of an instrument by Cronbach’s Alpha 1. Rev Lat Am Enfermagem. 2018;26:e3000.
15. 15. Ayre C, Scally A. Critical Values for Lawshe’s content validity ratio. Measurement and Evaluation in Counseling and Development. 2013;47:79-86.
16. 16. Zamanzadeh V, Ghahramanian A, Rassouli M, Abbaszadeh A, Alavi-Majd H, Nikanfar AR. Design and implementation content validity study: development of an instrument for measuring patient-centered communication. J Caring Sci. 2015;4(2):165-78.
17. 17. Sharma S, Pandit A, Tabassum F. Potential facilitators and barriers to adopting standard treatment guidelines in clinical practice: An Indian context. Int J Heal Care Quality Assur. 2017 ;30(3):285-98.
18. 18. Bano R, Gupta S, Shekhar C. Translational research in biomedical sciences in India: Challenges, observations & national perspectives. Ind Med Res. 2020;152(4):335.
19. 19. Barile G, de Rosa G, Papadia P, Muci G, Capodiferro S, Corsalini M. Clinical comparative study for validation of digital impression reliability with the gypsum check: a simple and fast way to evaluate the trueness and accuracy of implant-supported rehabilitation. Appli Sci. 2023;13(13):7358.
20. 20. Lahariya C, Sharma S, Agnani M, de Graeve H, Srivastava JN, Bekedam H. Attributes of standard treatment guidelines in clinical settings and public health facilities in India. Indian J Community Med. 2022;47(3):336-42.