Evaluation of method of cohort research articles using Q-coh assessment tool

INTRODUCTION

Observational studies are an important category of study designs. To address some investigative questions in plastic surgery, randomized controlled trials are not always indicated or ethical to conduct. Instead, observational studies may be the next best method to address these types of questions. Well-designed observational studies have been shown to provide results similar to randomized controlled trials, challenging the belief that observational studies are second-rate. Cohort studies and case-control studies are two primary types of observational studies that aid in evaluating associations between diseases and exposures. The differentiating characteristic between observational and experimental study designs is that in the latter, the presence or absence of undergoing an intervention defines the groups. By contrast, in an observational study, the investigator does not intervene and rather simply “observes” and assesses the strength of the relationship between an exposure and disease variable. Three types of observational studies include cohort studies, case-control studies, and cross-sectional studies.

Case-control and cohort studies offer specific advantages by measuring disease occurrence and its association with an exposure by offering a temporal dimension (i.e., prospective or retrospective study design). Cross-sectional studies, also known as prevalence studies, examine the data on disease and exposure at one particular time point. Because the temporal relationship between disease occurrence and exposure cannot be established, cross-sectional studies cannot assess the cause-and-effect relationship. Results from observational studies are often criticized for being vulnerable to influences by unpredictable confounding factors. But recent studies have challenged this notion, showing comparable results between observational studies and RCTs. Observational studies can also complement RCTs in hypothesis generation, establishing questions for future RCTs, and defining clinical conditions. Well-designed randomized controlled trials (RCTs) have held the pre-eminent position in the hierarchy of EBM as level I evidence. However, RCT methodology, which was first developed for drug trials, can be difficult to conduct for observational studies like behavior related areas like speech and hearing. Instead, well-designed observational studies, recognized as level II or III evidence, can play an important role in deriving evidence for speech and hearing areas.

The word “cohort” has been adopted into epidemiology to define a set of people followed over a period of time. The modern epidemiological definition of the word now means a “group of people with defined characteristics who are followed up to determine incidence of, or mortality from, some specific disease, all causes of death, or some other outcome.” Cohort studies are particularly advantageous for examining rare exposures because subjects are selected by their exposure status. Additionally, the investigator can examine multiple outcomes simultaneously. Disadvantages include the need for a large sample size and the potentially long follow-up duration of the study design resulting in a costly endeavor. If the study is prospective type then it will be very expensive to conduct the study. If the study of retrospective type then the main disadvantage is that the study will have less control over variables.

The process of setting up a cohort study in speech and hearing leads to the creation of a unique, large-scale data set which will be available for researchers to access now and in future. As well as exploring predictive factors, the data can be used to explore the impact of interventions in relation to individual differences. Findings from these investigations can be used to provide information on sample criteria and definitions of intervention and dosage which can be used in future trials. The observational cohort study is a useful alternative design to explore questions around prevalence, risk factors and intervention for clinical groups where robust research data are not yet available. Findings from such a study can be used to guide service-delivery decisions and to determine power for future clinical trials. All the data for the study is collected by investigators according to the specific protocol for the study. Also because all subjects are disease free and followed until the disease develops. Exposure is assessed before the disease develops. Incomplete and inadequate reporting of research hampers the assessment of the strengths and weaknesses of the studies reported in the literature. Readers need to know what was planned (and what was not), what was done, what was found, and what the results mean. Recommendations on the reporting of studies that are endorsed by leading medical journals can improve the quality of reporting. The common tools used to evaluate the effective method used in observational cohort studies in speech and hearing sciences is STROBE (strengthening the reporting of observational studies in epidemiology).

Aim

Aim of current study was to investigate the quality of the method adopted by cohort studies using the assessment tool Q-coh and evaluating the reliability of the assessment tool.

METHODS

Study location and duration

The study was conducted at Dr. S R Chandrasekhar Institute of Speech and Hearing, Bengaluru from October 2022 – December 2022.

Selection criteria of reviewers

Raters should be possessing experience in the field of research and should have more than two years of experience in research and in the field of speech and hearing.

Materials

Tool name: Quality of cohort studies (Q-coh: a tool to screen the methodological quality of cohort studies), author: Jarde et al Q-coh consist of 26 items with 7 inferences along with those 5 more items was also provided for checking the design characteristics of the study. The items in the Q-coh were grouped into seven domains; representativeness, comparability of the groups, quality of the exposure measure, maintenance of the comparability, quality of the outcome measure, attrition and statistical analysis and also each items in the tool had selective options. For checking the quality, the selected articles were given to few reviewers. The responses from each reviewer was collected in excel sheets. The reviewers were asked to select appropriate option for each item in the tool after reviewing the articles. The overall quality of the articles was determined based on the responses received from the reviewers. If one or none of the domain is negatively reviewed then the articles is considered to be of “good quality”. If two of the domains are negatively reviewed then the article is considered to have “acceptable quality”. If more than 2 domains are negatively reviewed then the article is said to be of “poor quality”. After the quality evaluation of the studies, the responses were subjected to statistical analysis such as inter rater reliability for checking the agreement between the raters and reliability of the tool.

Statistical analysis

Cronbach’s alpha was determined to check the reliability of the tool Q-coh. Inter rater reliability was performed to measure the degree of agreement between the raters, Kappa statistic value was also determined which defines the strength of agreement between the raters. The results are interpreted based on “Kappa interpretation by Landis & Koch, 1977”.

RESULTS

Inter rater reliability

Cronbach's alpha coefficient describes reliability of multiple items in a scale or it describes the internal consistency of a scale, questionnaire etc. Alpha value greater than 0.6 indicates acceptable quality of the tool, alpha value less than 0.6 indicate poor quality of the tool or very poor internal consistency. The (Table 1) represents Cronbach’s alpha of the tool across different selected articles.

Cronbach’s alpha values were found to be 0.804, 0.886 and 0.667 when different articles with cohort design used respectively; this indicates acceptable and good level of internal consistency within the scale. Overall Cronbach’s alpha value was 0.785 which also indicates the same.

The (Table 2) gives the agreement scores across raters. The interpretation of Kappa statistic score according to Landis & Koch, 1977 is given in (Table 3), which had been used as the result interpretation for the present study. From the results (Table 2) it was found that there exists ‘Fair to Moderate’ agreement between the raters for article 1, ‘Moderate to Substantial’ agreement between the raters for article 2. But for article 3, the range of agreement was between ‘No to Moderate’. The algorithm which was given by Q-coh checklist authors was followed to evaluate the quality of the selected articles. If none or one of the domains is negatively evaluated then the overall quality is considered to be good. If two of the domains are negatively evaluated then the quality is said to be acceptable. If more than two domains are negatively evaluated then the article is said to have poor quality. The quality assessment result for the articles is given in (Table 4). By following the algorithm, article 1 & 2 are found to be of good quality and article 3 is found to have acceptable quality.

DISCUSSION

In speech and hearing, randomized control trial cannot be done due to ethical issues and the study design. Hence cohort studies are mostly selected in order to conduct research in speech and hearing. Several assessment tools are available to check the quality of research studies. Tools are designed purposefully for a particular study design. Q-coh Jarde et al claimed to assess the methodological quality of primary studies with cohort designs. One of the main advantages of Q-coh is that the checklist is easily accessible for an investigator and it also contains all the required domains which can help us to investigate the quality of selected articles and also the results showed that there was a moderate agreement between the raters, instead of strong agreement. This can be due the variation in experience level in research among the raters. Experience in research may not be sufficient to review the articles. From the results it was noted that the investigators could correctly identify the studies which were poorly designed and accurately designed. Thus, we can say that the Q-coh is relevant and can be recommended as a good tool to evaluate the methodological quality of cohort studies. Also noted that this checklist can be only used for prospective studies, there were some` characteristics which doesn’t support retrospective studies.

Limitations

The moderate agreement among the raters when Q-Coh was used indicated that the interpretation of results may vary among different investigators. This points out the potential source of bias that could affect the reliability of the tool.

CONCLUSION

The present study was conducted to check if the checklist Q-coh is applicable to assess the methodological quality of cohort research studies. The average agreement between the raters for article 1, article 2 and article 3 were 51%, 66% and 33% respectively. To define the strength of agreement, kappa statistics was calculated and from the output of Kappa statistics a fair to moderate agreement was observed for article 1, for article 2 it was between moderate to substantial, while for article 3 it was between no to moderate. By judging the proportion of agreement between the raters for each article it was evident that level knowledge in the respective field is an important factor which could influence the results. Furthermore, based on the algorithm which was provided by the authors of Q-Coh, the quality of the research studies was interpreted. Article 1 & 2 belongs to the category of ‘Good’ and article 3 belongs to the category of ‘Acceptable’.

Funding: No funding sources

Conflict of interest: None declared

Ethical approval: The study was approved by the Institutional Ethics Committee

REFERENCES

1. 1. Jardea A, Losillaa JM, Vivesa J, Rodrigob MF. Q-Coh: A tool to screen the methodological quality of cohort studies in systematic reviews and meta-analyses. Available at: https://www.redalyc.org/pdf/337/33726322002.pdf. Accessed on 20 February 2023.
2. 2. Ma LL, Wang YY, Yang ZH, Huang D, Weng H, Zeng XT. Methodological quality (risk of bias) assessment tools for primary and secondary medical studies: what are they and which is better? Mil Med Res. 2020;7(1):7.
3. 3. Rao A, Brück K, Methven S, Evans R, Stel VS, Jager KJ, et al. Quality of Reporting and Study Design of CKD cohort studies assessing mortality in the elderly before and after strobe: A Systematic Review. PLoS One. 2016;11(5):e0155078.
4. 4. Margulis AV, Pladevall M, Riera-Guardia N, Varas-Lorenzo C, Hazell L, Berkman ND, et al. Quality assessment of observational studies in a drug-safety systematic review, comparison of two tools: the Newcastle-Ottawa Scale and the RTI item bank. Clin Epidemiol. 2014;6:359-68.
5. 5. Dreyer NA, Bryant A, Velentgas P. The GRACE Checklist: A Validated Assessment Tool for High Quality Observational Studies of Comparative Effectiveness. J Manag Care Spec Pharm. 2016; 22(10):1107-13.
6. 6. Kim MB, Zhang Y, Chang Y, Ryu S, Choi Y, Kwon MJ, et al. Diabetes mellitus and the incidence of hearing loss: a cohort study. Int J Epidemiol. 2017; 46(2):717-26.
7. 7. Liu W, Dong H, Yang L, Zhao H, Dong W, Yang Y. Severity and Its Contributing Factors in Patients With Vestibular Migraine: A Cohort Study. Front Neurol. 2020;11:595328.
8. 8. Marsden J, Pavlou M, Dennett R, Gibbon A, Knight-Lozano R, Jeu L, et al. Vestibular rehabilitation in multiple sclerosis: study protocol for a randomised controlled trial and cost-effectiveness analysis comparing customised with booklet based vestibular rehabilitation for vestibulopathy and a 12 month observational cohort study of the symptom reduction and recurrence rate following treatment for benign paroxysmal positional vertigo. BMC Neurol. 2020; 20(1):430.