Interrater reliability of client categorization system as a psychiatric status rating scale in measuring psychiatric patients’ health status

INTRODUCTION

An important nursing competency is the capacity to identify the severity of a client's health status as this will affect the planning of goals and appropriate interventions for a client. One of the crucial abilities of nurses in detection is the capacity to classify the condition of a client. There are a variety of instruments for classifying a client’s condition, based on the goals desired by the instrument maker.

Literature related to instrument of client status in mental health setting however, mostly outdated. Since 1979, more than 1,000 hospitals have used methods to classify the time required for healthcare processes.¹ The reason for these classification methods is firstly, because the use of funds is no longer based on historic costs but rather the primary reason is more related to the type of client being managed and secondly is related to resources that need to be used in treating patients.² Fries also stated that the diagnosis related groups (DRGs) cannot be applied to chronic or long-term care settings where the length of stay (LOS) varies.²

Various classifications have different purposes, such as those for diagnosing. In addition, there are also classifications that have been developed for identifying the most common symptoms and behavioral patterns of clients, for example the BPRS (Brief Psychiatric Rating Scale).³ Other authors state that the BPRS is used to measure significant clinical progress.⁴

In nursing, a psychiatric patient classification system is generally intended to be used as a basis for improving efficiency and allocation of nursing staff.⁵ The design of this classification system was to develop a classification system that is based on nursing care activities for patients and non-nursing activities. Both kinds of activities are identified, defined and organized into two categories.⁵

Classifications can also be the basis for calculating costs.² It can be added here that a patient classification system can also be an accurate prediction regarding staff needs and facilitating budget calculations.⁶ In addition, classification systems can also be used as the basis for determining the length of time needed to care for patients and as a tool to measure the workload of nurses.^7,8 Patient classification systems that were previously developed include the Nursing Patient Classification Form, the GRASP system, the Patient Classification System (PCS), the Patient Classification system, and the Resource Utilization Groups T-18 (RUG T-18).^1,6-10

Although various classification systems have been developed, the application of them have been deemed unsatisfactory as they have failed to address the critical needs of patients.¹¹ Patient classification systems such as those mentioned above are intended to depict the needs of patients in regard to healthcare and not for measuring the provision or quantity of nursing duties based on the average time required for treatments.¹¹

Based on some of these reasons, researchers subsequently developed a psychiatric patient classification system called the Client Categorization System (CCS) (see Annexure 1. CCS as a research instrument).¹² The choice of the term 'category' is because essentially the validity of any classification of psychiatric illness is based on two approaches, namely a clinical knowledge approach (criteria approach or categorical/typological approach) and a mathematical and statistical method approach (dimensional approach).¹³ Therefore, according to the author the right approach is to use the clinical knowledge, thus this classification system uses the term 'category'.

An instrument that is developed needs to go through a trial process in order to determine whether the instrument is valid and reliable to be used as a measuring tool. “Reliability refers to the precision of measurement or the reproducibility of the scores obtained with the examination”.¹⁴ Using interrater reliability is one of method in measuring reliability. Stronger agreement between raters toward instrument can be referred to higher interrater reliability.¹⁵ Two commons type of statistical test to measure interrater reliability are kappa and percent agreement.^16-18 Morris et al, also mentioned that the benefit of using percent agreement is that it will show whether there is any problem or not with kappa value.^16,19 This research measured this instrument to determine the interrater reliability of the CCS by using the Cohen-Kappa reliability formula and percent agreement.

METHODS

This was descriptive quantitative non-experimental research using a cross-sectional design. Data was collected by twelve undergraduate students on a psychiatric internship program during a clinical placement. Those students were familiar with CCS instruments since in academic stage. The study was approved by the Research Ethics Board at the Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia.

The sample size was calculated according to Gorsuch and Hatcher in which the minimum ratio between number of items in scale and number of people in sample should be 1:5.^20,21 The total number of items in this study was 11 so that the total minimum number of samples needed was

55. In this study, however, the sample size was taken based on the number of CCS measurement, instead of using the number of patients. Patients in this study consisted of 14 males and 16 females. Measurements were made 98 times because several clients were measured more than once in different shifts. There were six pair of raters in this study. The number of clients, measurements and raters are presented in the Table 1.

Instrument

The instrument used in this research was the Client Categorization System (CCS), which was designed by the researcher (Intansari Nurjannah).

This instrument classifies client categories into four parts, namely crisis, acute, maintenance and health promotion categories. Clients are categorized in the crisis if they have a score ≥120, or if at the time of the first screening they were identified with having suicidal tendencies. Clients are categorized as acute if they have a score of 60-119 and clients are categorized as maintenance if they have a score of 31-59. Clients are categorized in the promotion of health if they have a score of 0-30.

This categorization is based on the opinion of the instrument makers. The variables used in the scoring in this instrument included the behavioral tendency to injury oneself or others, communication, social interaction, Activities of Daily Living (ADL) (eating, bathing, dressing), sleep/rest, medication, and scheduled activities (eating, bathing, dressing).

This instrument was modified from Patient Classification System designed by Stuart and Sundeen.²² Patient Classification System is an instrument which has four categories (first, second, third and fourth categories). Each category has different need of care for each shift (8 hours). Below is the table of each category (Table 2) in Patient Classification System.²²

Adapted from Stuart G, W, Sundeen S, J. Principles & Practice of Psychiatric Nursing. St. Louis: Mosby Year Book; 1995

Data analysis

Data analysis was conducted using the STATA program to determine the Kappa coefficient. The Cohen-Kappa formula consists of:²³

Note: K=Kappa Coefficient; Pa=Percentage of interrater measurement consistency; Pe=Percentage of interrater measurement change

Some scholars have interpreted the Kappa value. Following are the Kappa value interpretations of several scholars in Table 3.²⁴ Data also was analysed using percent agreement. Percent agreement formula as follows:²⁵

Acceptable percent agreement only if the value is >70%.²⁷

RESULTS

The value scores of the two raters were analyzed using the Cohen Kappa formula and percent agreement. Information on the Kappa scores and percent agreement can be seen in Table 3. The calculation results using the Cohen-Kappa formula showed that the interrater reliability value for the total score was: 0.59, which can be categorized as moderate and percent agreement 77.78% indicated acceptable.

DISCUSSION

The research results demonstrate that the CCS is an instrument that has interrater reliability in categories with a score of 0.59, which according to Cohen Kappa values can be interpreted as moderate. Based on Landis and Koch, this instrument can be considering acceptable (κ≥0,41).²⁶ Meanwhile the result of percent agreement is 77.78% and it also considers as acceptable according to Osborne.²⁷ The difference in sharpness in data will influence interrater reliability.²⁸

Based on literature searched, it was difficult to find a similar research. Most of literature was outdated and did not use interrater reliability by using two statistical methods (kappa and percent agreement) to measure interrater reliability. However, there was one research conducted to measure reliability instrument. This instrument called Subjective Patient Classification was designed by Morath et al.⁸ The acuity level of this instrument reached 0.001 confidence interval. Morath’s instrument, however has five level classifications which CCS has 4 classification. Thus, it is difficult to compare these instruments in terms of their reliability. Other research related to classification system also was conducted by Parloca and Henry.²⁹ However, this research was conducted to find out patient problem using specific classification (Georgetown home health care classification system) but it did not measure the instrument itself. Another research about validity and reliability was research conducted by Schroder also measure validity and interrater reliability of patient clasification systems (PCS).¹ This study rated 141 patients with the number of raters varied by unit from minimum of three to maximum of six. Each patient rated varied from 7-26 times more than this study. The raters were clinicians. Researchers used Intra-class correlation for analysis and showed adequate reliability (R1=0.36, Rk=0.85) of this instrument. It is however, difficult to compare with this study as items in PCS was slightly different with CCS. The method of analysis was also different. The item of PCS in the instrument which five consists categories also not all similar with CCS.

Following, the calculation results for each variable in the CCS according to the Kappa assessment only found one variable which has a different value of kappa and percent agreement. This variable is Scheduled Activity: Bathing with Kappa 0.5098 (accepted, if >0.41) while the percent agreement is 69, 39% (unacceptable). Although this phenomenon is rarely occurring, this result from kappa and percent agreement occur if there is an influence of prevalence index and index bias.³⁰ As low value of interrater reliability is not acceptable in clinical setting due to the consequence for patient outcomes, researcher then examine raters’ characteristic for more consideration.¹⁵ Based on Mc Hugh, kappa statistic need to be chosen to determine interrater reliability rather than percent agreement, if there is much guessing among the raters.⁵ It is because the kappa was designed to take account of the possibility of guessing, while percent agreement does not take account of the possibility that raters guessed on scores.¹⁵

In this study, researcher then use kappa statistic as based on decision on this conflict because raters were not well trained or expert in clinical setting.³¹ Although raters were familiar with the instruments however, they were not familiar in taking care of ‘real’ patient. The reliability of this item (Scheduled Activity: bathing) then considering as acceptable.

Literature shows that there are 4 keys that can affect interrater reliability according to Neuendorf.³² First, the scheme of an instrument must be designed and established properly to produce an easy and accurate assessment. The CCS was developed per stage of criteria and was supplemented with an explanation of the scoring and the assessment formulas. One other variable which is Social interaction has kappa value 0.4901. Although this value is accepted but it considered low and need to be improved.

Another factor that led to the Cohen Kappa total score in the CCS being categorized as moderate was because both the raters had training (in academic setting) and they were already familiar with the CCS tool. Even though they are familiar with the tools but they are not familiar in handling ‘real’ patients. Rater training is able to reduce the amount of variation in assessments of two raters.¹⁵ Untrained raters tend to produce a low value of interrater reliability. Third, a rater’s fatigue is a factor that affects the stability of an assessment. This research encountered several obstacles that required the raters to conduct assessments that exceeded their scheduled shifts, thus the possibility of fatigue affecting the stability of the assessment may have occurred.

One of the debates related to the measurement of interrater reliability regards those doing the measuring, the raters. Who the raters are and how qualified and experienced they are affects the reliability.²³ This research used raters that can be classified as good in terms of the same experience and knowledge regarding the CCS, and the same level of familiarity in taking care of psychiatric patient. This reinforces that the interrater reliability value of the CCS was not influenced by characteristic differences of the raters.

The number of sample may also lead to low reliability of this instrument. As Eye and Mun states that the minimal sample to get kappa value 0.4 is 102 measurements times, meanwhile in this study the number measurements was 98.³³

CONCLUSION

In conclusion, this research finds that CCS instrument has an acceptable score in interrater reliability both using Cohen’s-Kappa formula and percent agreement. Although this CCS can be used in clinical setting, several items in this CCS instrument are able to be improved by undertaking further research. For better result, future research on this instrument also need to consider raters and the number of sample/measurement.

Funding: No funding sources

Conflict of interest: None declared

Ethical approval: The study was approved by the Institutional Ethics Committee Faculty of Medicine Universitas Gadjah Mada, Yogyakarta, Indonesia

REFERENCES

1. 1. Schroder PJ, Washington PW, Deering DC, Ciyne L. Testing validitiy and reliability in a psychiatric patient classification system Psychiatric nurses thest their patient classification system. Nursing Management. 1986;49(1):49-54.
2. 2. Fries BE, Nerenz DR, Falcon SP, Ashcraft MLF, Lee CZ. A classification system for long-staying psychiatric patients. Medical Care. 1990;28(4):311-23.
3. 3. Overall JE. The brief psychiatric rating sclae in psychopharmacology research. Modern Problems of Pharmacopsychiat. 1974;7(67-68).
4. 4. Lachar D, Bailley SE, Rhoades HM, Varner RV. Use of BPRS-A percent change scores to identify significant clinical improvement: accuracy of treatment response classification in acute psychiatric inpatients. Psychiatric research. 1999;89:259-68.
5. 5. Eklof M, Qu W. Validating a Psychiatric patient Classification System. JONA. 1986;16(5):1986.
6. 6. Croft A. A psychiatric patient classification system that works! Nursing Management. 1993;24(11):66.
7. 7. Iglesia C, Villa AM. A system of patient classification in long-term psychiatric inpatient: Resources Utilization Groups T-18 (RUG T-18). 2005.
8. 8. Morath J, Fleischmann R, Boggs G. A missing consideration: The psychiatric patient classification for scheduling staffing systems. Perspective in Psychiatric Care. 1989;XXV(3, 4):40 - 7.
9. 9. Ehrman ML. Using a factored patient classification system in psychiatry. Nursing Management. 1987;18(5):48-53.
10. 10. Ringerman ES, Luz S. A psychiatric patient classification system. Nursing Management. 1990;21(10):66-71.
11. 11. O'Leary C. A psychiatric patient classification system. Nursing Management. 1991;22(9):66.
12. 12. Nurjannah I. Treatment guidelines for mental illness patient: management, nursing process and nurse-patient therapeutic relationship. Yogyakarta: Mocomedia. 2004.
13. 13. Robins E. Categories Versus Dimensiions in Psychiatric Classification. 1976;8:39-55.
14. 14. van der Vleuten C. Validity of final examinations in undergraduate medical training. BMJ. 2000;321(7270):1217.
15. 15. McHugh ML. Interrater Reliability: The Kappa Statistic. The Journal of Croatian Society of Medical Biochemistry and Laboratory Medicine. 2012:276-82.
16. 16. Morris R, MacNella P, Scott A, Treacy P, Hyde A, O'Brien J, et al. Ambiguities and conflicting results: The limitations of the Kappa Statistics in establishing the interrater reliability of the Irish nursing minimum data set for mental health: A discussion paper. Inter J Nursing Studies. 2008;45:645-7.
17. 17. Craddock J. Interrater reliability of psychomotor skill assessment in athletic training: ProQuest. 2009.
18. 18. Cargo M, Stankov I, Thomas J, Saini M, Rogers P, Mayo-Wilson E, et al. Development, interrater reliability and feasibility of a checklist to assess implementation (Ch-IMP) in systematic reviews: the case of provider based prevention and treatment progrmas targeting children and youth. Med Res Meth. 2015;15:73.
19. 19. Morris R, MacNeela P, Scott A, Treacy P, Hyde A, O’Brien J, et al. Ambiguities and conflicting results: The limitations of the kappa statistic in establishing the interrater reliability of the Irish nursing minimum data set for mental health: A discussion paper. Inter J Nursing studies. 2008;45(4):645-7.
20. 20. Gorsuch RL. Factor Analysis. Hillsdale. NJ: Erlbaum Associates. 1983.
21. 21. Hatcher L. A step by step approach to using the SAS System for Factor Analysis and Structural Equation Modeling. Cary, N, C: SAS Institute, Inc; 1994.
22. 22. Stuart GW, Sundeen SJ. Principles and Practice of Psychiatric Nursing. St. Louis: Mosby Year Book; 1995.
23. 23. Rushforth HE. Objective Structured Clinical Examination (OSCE): Review of Literature and Implications for Nursing Education. Nurse Education Today. 2007:481-90.
24. 24. McCray. Assessing Interrater Agreement For Nominal Judgement Variables. Language Testing Forum. 2013;15-7.
25. 25. House A, House B, Campbell M. Measures of inter observer agreement: Calculation formulas and distribution effects. J Behavioral Assessment. 1981;3(1):37-57.
26. 26. Landis J, Koch G. The measurement of observer agreement for categorical data. biometrics. 1977:159-74.
27. 27. Osborne JW. Best practices in quantitative methods: Sage. 2008.
28. 28. McHugh M. Interrater reliability: the kappa statistic. Biochemia Medica. 2012;22(3):276-82.
29. 29. Parloca PK, Henry SB. The usefulness of Georgetown Home Health Care Classification System for Coding Patient Problems and Nursing Interventions in Psychiatric Home Care. Computers in Nursing. 1998;16(1):45-52.
30. 30. Feinstein A, Cicchetti D. High agreement but low kappa: I. The problems of two paradoxes. J Clinical Epid. 1990;43(6):543-9.
31. 31. Nurjannah I, Siwi S, M. Guidelines for analysis on measuring interrater reliability of nursing outcome classification. International Journal of Research in Medical Sciences. 2017;5(4):1169-75.
32. 32. Neuendorf KA. The Content Analysis Guidebook. Thousand Oaks, CA: Sage; 2002.
33. 33. Eye AV, Mun EY. Analyzing Rater Agreement: Manifest Variable Methods. New Jersey: Lawrence Erlbaum Associates; 2005.

ANNEXURE

Annexure 1: CCS as Research instrument.

Client Category System (CCS) in Psychiatric (English Version) By Intansari Nurjannah

Client Identity::…………………………

Initial Screening:

• a. Does a client have a tendency/idea to commit suicide? Yes/No (If the answer yes, then client is automatically in the crises condition/category IV)
• b. Does a client have a tendency to abscond? Yes/No (If the answer is yes, continue the screening below, if the score is in health promotion or maintenance then client need to be discharged soon)

Note: Initial screening for suicide risk only can be used if client has an idea of committing suicide but not because the order of hallucination.

Score 0-30 : 1^st Client category (Health promotion)

Score 31-59 : 2^nd Client category (Maintenance)

Score 60-119 :3^rd Client category (Acute)

Score ≥120 : 4^th Client category (Crisis)

Related to the risk for absconding, if the client have a risk for absconding but in the 1 and 2 category, so nurse needs to assess more detail related to the risk for absconding to determine what the appropriate observation for client or for discharging client.

* This score is also applied to client who has a high risk because of his/her physiological condition

Score 0 : Physically unable to harm self/other/impossible to cause an injury

Score 16: There is a tendency to do violence but client understand the consequency and does not want to do it

First level of hallucination: The content of hallucination considered as making client feels happy (client smile or laugh), hallucination is experienced when client usually alone

Second lavel of hallucination: The content of hallucination considered as disgusting, insulting, critizing, cursing or blaming

Third level of hallucination: Hallucination starts to ask client to do something, the content of hallucination may attracts client and client may feel lonely if hallucination dissapear, or client may feel afraid.

The fourth lavel of hallucination: Hallucination may threats client if client does not want to follow hallucination's instruction

Score 34 : There is a strong tendency to act violently but client has not acted yet because no opportunity or client who has a risk for injury because of unclear vision, dementia, delirium etc.

Score 50 = Have acted violently (up to 3 days) or doing violently (present) to others or self (but not committing suicide)

** More explanation for

Score 0 = If client is willing to be involved in one group consists of more than five person, client can/willing to participate/willing to be involved in that group

Score 5 = The condition that client is willing to interact with everyone but always in 1-1 interaction

Score 10 = Client is only willing to interact with one person (one subject), if in 1-1 interaction then there is another person is involved then client does not want to continue interaction/mute/or refuse

*** If in the ADL ability, there is an abnormal sign (for example taking a bath, eating, clothing/get dressed which in abnormal frequency/too much) without any reasonable reason so client will be scored 7 (need a help to do activity in a normal frequencies standard)

**** More explanation for sleeping variable, sleeping variable not only refer to sleeping but also in having a rest

Score 0 = For assessment in morning and afternoon shift: client understand the need to have a rest and willing to have a rest if needed

Score 3 = For assessment in morning and afternoon shift: client understand the need to have a rest but need a motivation in case client need to have a rest

Score 7 = For assessment in morning and afternoon shift: client does not understand the need to have a rest and need nursing intervention (for example accompanying client) so client is willing /able to have a rest if needed

Score 10 = Client unconsciousness (GSC < 8)

At morning and afternoon shift: Client need to have a rest because of his/her physical condition (for example client could not sleep in the previous night) but client unable to have a rest except if pharmacology is given or client who needs restraint to make sure he/she can have a rest

*****If medicine/therapy is given does not covered three shift, then the score referred to previous shift

****** Unable to follow the instruction/direction/guidance in the consciousness or unconsciousness state or because of physiological cause