Assessment of sigma metrics results of serum glucose and lipid profile tested by automated chemistry analyzer in medical city hospitals in Iraq

INTRODUCTION

The purpose of a clinical laboratory test is to evaluate the pathophysiologic condition of an individual patient to assist with diagnosis, to guide or monitor therapy, or to assess risk for a disease or for progression of a disease. To have value for clinical decision making, an individual laboratory test result must have total error small enough to reflect the biological condition being evaluated.¹ Quality control (QC, also called internal quality control or statistical process control) is a process to periodically examine a measurement procedure to verify that it is performing according to pre-established specifications.¹ Its also defined as overall system of activities whose purpose is to control the quality of a product or service so that it meets the needs of users. The aim is to provide quality that is satisfactory, adequate, dependable and economic.²

Sigma metrics (six sigma) is a management strategy that seeks to improve the quality of process outputs by identifying and removing the causes of defects (errors) and minimizing variability in manufacturing and business processes. Sigma metrics places analytical characteristics within the framework of clinical requirements. The six-sigma idea asserts an association between the numbers of product defects, wasted operating costs and levels of customer satisfaction. It can be inferred that as sigma increases, the consistency and steadiness of the test improves, thereby reducing the operating costs. The sigma scale is easily interpreted and appreciated by laboratories. Sigma values can be calculated for both qualitative and quantitative assays. The Sigma scale provides guidelines for assay improvement and monitoring. Total testing process is a multistep process that begins and ends with the needs of the patient.³

The number of steps may vary according to test types and laboratory organization. We can describe nine activity steps in laboratory medicine, test selection and ordering a laboratory test request, collecting the sample (serum, plasma, urine and so on), identification, transport the sample to laboratory, preparation of the sample, analysis, reporting test results, interpretation of test results, action. Historically in clinical laboratories, the total testing process was assumed to consist of only three phases: preanalytical phase, analytical phase and post-analytical phase. The errors can occur in any of the above-mentioned steps. To overcome the serious errors originating in clinical laboratories, a new perspective and approach seem to be essential. All laboratory procedures are prone to errors because in many tests, the rate of human intervention is higher than expected. It appears that the best solution for analyzing problems in clinical laboratories is the application of Sigma metrics methodology.⁴

Sigma metrics is an evolution in quality management that is being widely implemented in business and industry in the new millennium. Six sigma metrics are being adopted as the universal measure of quality to be applied to their processes and the processes of their suppliers, Six Sigma provides a more quantitative framework for evaluating process performance and more objective evidence for process improvement, any process can be evaluated in terms of a sigma metric that describes how many sigma’s fit within the tolerance limits. The power of the sigma metric comes from its role as a universal measure of process performance that facilitates benchmarking across industries.⁵

Sigma metrics methodology first adopted by Motorola at 1986, Motorola was greatly impacted by the quality improvements in foreign products. Under the leadership and support of Bob Galvin, The immediate origin of Six Sigma can be traced to its early roots at Motorola and specifically to Bill Smith (1929 - 1993), Bill Smith was an employee of Motorola and a Vice President and Quality Manager of Land based Mobile Product Sector. The core principle of the latent defect theory is that variation in manufacturing processes is the main culprit for defects, and eliminating variation will help eliminate defects, which will in turn eliminate the wastes associated with defects, saving money and increasing customer satisfaction, variation is measured in terms of sigma values or thresholds, the threshold determined by Smith and agreed to by Motorola is 3.4 defects per million opportunities (3.4 DPMO), which is derived from sigma shifts from specifications. Motorola adopted the concepts and went on to win the first ever Malcolm Baldrige Excellence Award in 1988, just two years after Bill Smith’s introduction of Six sigma.⁶ In health care systems the first application describing sigma metrics in a healthcare laboratory was published by Nevalainen et al in the year 2000. This application focused on preanalytical and postanalytical processes.⁷ The popularity of Six sigma principles as an approach to quality management in health care has grown in the last 10 years. In 2000, there were one article published describing specifically the application Six sigma, according to a PubMed search utilizing the phrase “six-sigma, sigma metrics, laboratory.” This number has risen to 105 articles published in 2016/2017.⁸

Six sigma concepts can be better understood and explained using mathematical term Sigma and Normal Distribution. Sigma is a Greek symbol represented by "σ" refer to standard deviations. A Six sigma process means that 6 standard deviations fit on each side of the mean, between the mean and the specification limits. Six Sigma equates in percentage terms to 99.9997% accuracy or to 3.4 defects per million opportunities to make a defect.⁶

The aim of the study is to assess sigma metrics results of serum glucose and lipid profile tested by automated chemistry analyzer in Medical City hospitals in Iraq.

METHODS

Study design

Sigma metrics estimation was done by using the auto analyzer Dimension RxL Max Integrated Chemistry System from Siemens Healthcare Diagnostics at the Teaching Laboratories of the Medical City, Baghdad, Iraq.

Estimation of sigma metrics according to Westgard by the following steps

• • Define goals for intended use (TEa),
• • Validate method performance (CV%, bias %),
• • Sigma calculation,
• • Select Westgard rule.

Total allowable error (TEa)

Westgard was the first to introduce the concept of total error in 1974, Total Allowable Error(TEa), total analytic error (TAE) or Allowable Total Error (ATE) defined as an analytical quality requirement that sets a limit for both the imprecision (random error) and inaccuracy (systematic error or bias) that are tolerable in a single measurement or single test result to insure clinical usefulness.

Total allowable error was obtained according to Ricos specification (2014) update, shown in Table 1. Except for LDL because it’s not included in the EQA program.

Calculate the analyzer’s observed total error TE(obs)%, using measured CV% and measured bias%, according to the formula:

TE(obs)% = 2CV% + Bias%

Compare measured TEobs% to TEa%. If TEobs% < TEa % (or very close to it), then the quality requirement is met and instrument is considered suitable for measurement of that analyte.

Internal quality control material

For internal quality control, we used Human Assayed Multi-sera provided from RANDOX Laboratories.

External quality control material

For external quality assessment, we used The RIQAS General Clinical Chemistry EQA programme provided from RANDOX Laboratories. RIQAS is the largest international EQA scheme in the world.

Measurements of analytes

All the biochemical parameters were analyzed on automated analyzers (Siemens Dimension RxL Max) from May-July 2017 using standard IFCC methods. Prior to analysis, manufacturer instructions were followed regarding calibration, maintenance and controls. Serum glucose concentration was determined by using the hexokinase method, cholesterol by Dimension-Siemens reagents, triglycerides by L/G Kinase EP and HDL by Direct HDL PPD

Estimation of coefficient of variation (CV%)

• • The Coefficient of Variation (CV%) is the ratio of the standard deviation to the mean and is expressed as a percentage. The CV% allows us to make easier comparisons of the overall precision. CV%= (SD/mean) x 100%
• • At first, we use the manufacturer assigned values of the internal QC materials, this is because the manufacturer range is wide, after 20 readings from (22 May-20 June 2017) the calculated mean, SD and range were obtained, then the calculated range was used (21 June-26 July 2017)
• • QC material was analyzed daily for 20 days and plotted as the Y axis, because assuming a Gaussian or normal distribution, it would be expected that about 68% of the points fall within the mean ± 1 SD, 95% within the mean ± 2 SD.
• • QC results plotted on Levey Jennings chart for both normal (level 2) and pathological (level 3) for each analyte (glucose, cholesterol, triglycerides, and HDL) and using the calculated range (Table 2) as lower and upper limits.
• • Levey-Jennings chart used to evaluate run quality
• • looking for systematic error and random error
• • Westgard rules applied to evaluate the quality of analytical runs
• • At the end of the 20 days, mean, SD and CV% were calculated.
• • CV% for level 2 and level 3 were calculated for each analyte, then the mean of both readings was used and compared with desirable specifications for imprecision.

Estimation of bias

• • Bias was estimated utilizing EQAS (RIQAS) data as % deviation from the consensus mean of the participating labs. EQAS provides a means of assessing the analytical performance of a laboratory compared to other laboratories utilizing the same methods and instruments.
• • Proficiency Testing (PT) Provider used was RANDOX International External Quality Assessment Scheme (RIQAS) which is the largest international EQA scheme in the world. It is used by more than 40,000 laboratory participants in 124 countries.

Estimation of sigma metrics

Sigma-metric = (TEa% – Bia%)/CV %, Figure 1.

The Six Sigma scale typically runs from zero to six, but a process can actually exceed six sigma, if variability is sufficiently low as to decrease the defect rate.

In industries outside of healthcare, 3 sigma is considered the minimal acceptable performance for a process. When performance falls below 3 sigma, the process is considered to be essentially unstable and unacceptable.

Westgard rule selection

From the Westgard OPSpecs Charts QC planning tool and the Sigma Metrics formula's it was deduced that every Westgard rule has its own Sigma value.

• • 6-sigma quality requires only a single control rule, 13s, with 2 control measurements in each run one on each level of control). The notation N=2 R=1 indicates that 2 control measurements are needed in a single run
• • 5-sigma quality requires 3 rules, 13s/22s/R4s, with 2 control measurements in each run (N=2, R=1).
• • 4-sigma quality requires addition of a 4th rule and implementation of a 13s/22s/R4s/41s multirole, preferably with 4 control measurements in each run (N=4, R=1), or alternatively, 2 control measurements in each of 2 runs (N=2, R=2),
• • <4-sigma quality requires a multirule procedure that includes the 8x rule, which can be implemented with 4 control measurements in each of 2 runs (N=4, R=2) or alternatively with 2 control measurements in each of 4 runs (N=2, R=4).

Statistical analysis

Calculation of mean, SD, CV, TE and sigma metrics was done by using Microsoft excel 2010, Statistical analysis for plotting of Levey Jennings chart was done by using Microsoft excel 2010, Plotting of OPSecs chart was done using Westgard EZ Rules 3 software.

RESULTS

We have analyzed 4 analytes over a period of 2 Months (22 May-26 July 2017) and assessed for sigma metrics. In order to calculate sigma, we have calculated mean, standard deviation (SD), coefficient of variation (CV%) and bias % (Table 3). SD quantifies how close numerical values are in relation to each other. Since SD typically increases as the concentration of analyte increases, CV% can be regarded as statistical analyzer. Since CV% is the ratio of two, it cancels that effect. CV% is therefore standardization of the SD that allows comparison of variability estimates regardless of analyte concentration. CV% is dimensionless and does not vary with changes in measurement units. Precision is closeness of agreement between independent, repeated results obtained from the same sample under specific conditions. Lesser the CV%, better is the precision.

* according to Westgard.com

Bias was calculated as the average of % deviation of 2 samples of PT/month according to RIQAS.

We have obtained lowest the total CV% for glucose (1.28) and triglycerides (1.9) which is below the optimum (I)% followed by cholesterol (2.08) and HDL (2.2) which are below the desirable (I)%. For bias the lowest value for cholesterol (1.4) which is below the optimum (I)% followed by HDL (3.6) and glucose (3.9) and triglycerides (7.1) which are below the desirable (I)%.

For total error, we have obtained the total error for triglycerides (10.9) which is below the optimum (TEa)%, then HDL (8), cholesterol (5.56) and glucose (6.46). Which are below the desirable (TEa)%. For sigma metrics, in our study triglycerides has the highest sigma value (10.9) followed by HDL (3.4), cholesterol (3.4) and lastly glucose (2.5). Calculated and manufacturer range, mean, SD for L2, L3 QC materials were shown in Table 2, CV, bias, total error and sigma metrics for glucose and lipid profile with the allowable limits Table 3, OPSpesc charts, Figures 2-5.

DISCUSSION

A good laboratory practice requires that laboratories design their quality control (QC) procedures to assure that reported patient results meet the quality required for their intended use.⁹ The Sigma metrics is based on the statistical concept: laboratory errors can be reduced by maintaining 6 standard deviations between the parameter average and its upper and lower limits.

The sigma metrics represent the correlation among numbers of product defects, wasted operating costs and customer satisfaction. Therefore, as sigma increases, the consistency, reliability, steadiness and overall performance of the test improves, thereby decreasing the operating costs.¹⁰ When the method quality goals are set at six-sigma, stringent internal QC rules are mandatory. However, false rejections rate should also be kept in mind which can be minimized by relaxing control limits up to 3 SD. On other hand, if method is performing at sigma level below 3, it will require to implement a newer and better method because quality of the test cannot be assured even after multiple QC cycles.¹¹

A previous comparison study using the same manufacturer (Siemens) the sigma metric values for these analtytes are as follow: triglycerides (8.6), HDL (3.4), cholesterol (4.8) and glucose (3.1).¹²

Sigma values are useful for guiding QC strategy design. For a high sigma process, it is relatively easy for the laboratory to design a QC procedure, to detect any out-of-control condition that could pose a significant risk of producing unreliable results. A relatively large out-of-control condition would have to occur before there would be much chances of producing results that contained errors that exceeded the TEa specification and it is easy to design QC procedures that can detect large out-of-control conditions. The sigma metrics values are useful in setting the internal QC acceptability criteria.¹³

The six-sigma idea asserts an association between the numbers of product defects, wasted operating costs and levels of customer satisfaction. It can be inferred that as sigma increases, the consistency and steadiness of the test improves, thereby reducing the operating costs. As sigma increases, the consistency, reliability, steadiness and overall performance of the test improves, thereby decreasing the operating costs.¹⁰

The Sigma metrics reported here reflect assay performance at the time the data was collected and thus represent a “snapshot”. Naturally, performance can change over time for a variety of reasons (e.g., reagent lot to lot variation). Periodic calculation of sigma metrics is appropriate to determine if assay quality has been maintained, has decreased, or has improved. Sigma metrics thus represent another quality assurance tool to be monitored periodically to assess changes in assay quality

In industries outside healthcare, three sigma is considered the minimal acceptable performance for a process. When performance falls below three sigma, the process is considered to be essentially unstable and unacceptable.¹⁴

In healthcare, the sigma metrics used for Westgard rule selection. So, in our study the rule is,

• • For triglycerides (sigma>6) its excellent tests, so evaluate with 2 QC/day and 1:3.5s rule,
• • For HDL and cholesterol (sigma 3.4), use multi rules with 2 levels of qc/day,
• • For glucose (sigma<3), use max qc, 3 levels, 3 times a day.

Consider testing specimens in duplicate. Total quality management works on plan, do, check and act rules whereas sigma metrics works on define, measure, analyze, improve, control.¹⁵

when process performance is validated against Westgard rules or any other quality criteria for acceptability of control data, probability for rejection and probability of error detection are of paramount importance. The term probability of false rejection (Pfr) is used to describe a situation where there are no analytical errors present except for the inherent imprecision or random error of the method. Probability of error detection (Ped) is the term used to describe where an analytical error occurs in addition to the inherent random error. For achievement of world class quality, it is desirable to have a high probability of error detection and a low probability of false rejection.¹⁶

Despite there is violation of Westgard rules but these could be a false rejection results as much as these values are below the TEa, and this can be confirmed by using a software for calculation of Pfr and Ped.

The main limitations of our study for assessment of sigma metrics are:

• • The bias that is estimated as %deviation from the EQAS is based on peer group consensus mean rather than an accuracy based program,
• • The lack of knowledge about the corresponding Pfr and Ped for the different analytes due to lack of appropriate software as a result of financial constraints. This would have made our results and interpretation more explicit and ultra-precise.
• • The sigma metric results are widely variable due to lot to lot variation, environmental factors and depend on which specification is used (Ricos or CLIA).

CONCLUSION

Sigma metric was excellent for triglycerides (>6) and acceptable for cholesterol and HDL (>3). Sigma metric was desirable for cholesterol and HDL (>3), so need application of multi rules. Sigma metric was poor for glucose (<3) so need improvement of QC methods.

ACKNOWLEDGEMENTS

Authors would like to thank Dr. Zina Hasan, who has been most generous in giving her time, encouragement, support, and advice. Author deeply indebted to Dr. Ann khazaal who has great credit for the success of this work and for her continuous guidance, assistance, and encouragement. Author would like to thank the medical staff in the Teaching Laboratories for assistance.

Funding: No funding sources

Conflict of interest: None declared

Ethical approval: The study was approved by the Institutional Ethics Committee

REFERENCES

1. 1. McPherson RA. Henry’s Clinical Diagnosis and Management by laboratory methods. 2017. 23e. Chapter 10.
2. 2. Glossary of Terms for Quality Assurance and Good Laboratory Practices. United Nations Publication; 2009.
3. 3. Barr JT. The total testing process and its implications for laboratory administration and education. Clin Lab Manage Rev. 1994;8:526-42.
4. 4. Adiga US. Sigma metrics in clinical chemistry laboratory-A guide to quality control, Al Ameen J Med Sci. 2015;8:281-7.
5. 5. Burtis CA. Tietz textbook of clinical chemistry and molecular diagnostics. 2012. Chapter 8.
6. 6. Vivekananthamoorthy N. Lean Six Sigma, Six Sigma Projects and Personal Experiences. chapter 1, 2011.
7. 7. Nevalainen D. Evaluating laboratory performance on quality indicators with the six sigma scale, Arch Pathol Lab Med. 2000;124:516-9.
8. 8. Sigma metric laboratory. Available from: https://www.ncbi.nlm.nih.gov/pubmed/?term=sigma +metric+laboaratory. accessed June 2017.
9. 9. Westgard S. Six Sigma Metric Analysis for Analytical Testing Processes. Abbott MS; 2009.
10. 10. International Organization for Standardization Medical laboratories -particular requirements for quality and competence. ISO 15189. International Organization for Standardization (ISO), Geneva; 2012.
11. 11. Tetrault G. Evaluating laboratory performance with the six sigma scale. Arch Pathol Lab Med. 2000;124:1748-9.
12. 12. Westgard JO. The quality of laboratory testing today an assessment of σ metrics for analytic quality using performance data from proficiency testing surveys and the CLIA Criteria for Acceptable Performance J Clin Pathol. 2006;125:343-54.
13. 13. Hedwig CM. Measurements for 8 Common Analytes in Native Sera Identify Inadequate Standardization among 6 Routine Laboratory Assays. Clin Chem, 2014;60:855-63.
14. 14. Westgard JO. Six sigma quality design and control, 2^nd ed. Westgard QC Inc; 2006.
15. 15. Westgard JO. Internal quality control: planning and implementation strategies,Ann Clin Biochem. 2003;40:593-611.
16. 16. Westgard JO. Performance characteristics of rules for internal quality control: probabilities for false rejection and error detection, Clin Chem. 1977;23: 1857-67.
17. 17. Westgard S. Improve Lab Operations with Six Sigma Metrics. American society for clinical pathology, annual meeting; 2011.