NKK og NOKLUS, oktober 2011

Transkript

1 NKK og NOKLUS, oktober Kjære deltaker, Du har vært med i en pilot EKV-studie utført i juni. Pilotutsendelsen har vært et samarbeidsprosjekt mellom NKK, NOKLUS og Laboratory for Analytical Chemistry ved universitetet i Gent, Belgia. Hensikten har vært, ved bruk av kommutable (pasientlike) pasientsera, å undersøke nåværende analysekvalitet og status mht. standardisering på de store analyseinstrumentene i medisinske laboratorier. Resultater fra prosjektet vil kunne være et viktig bidrag til å bedre analysekvaliteten mht. standardisering og riktighet mellom ulike instrument og metoder innen medisinsk biokjemi. Dette kan være et incitament for den enkelte produsent til å bedre analysekvaliteten og for NKK til å bedre sine EKV-program. Laboratoriet mottok native sera på tørris og EKV-prøver fra utsendelse 3 i - nivå-programmet for medisinsk biokjemi (5) fra Labquality. Alle prøvene ble analysert for Ca, Mg, albumin og protein ulike dager i på forhånd angitt rekkefølge,. dag i omvendt rekkefølge fra. dag. Alle laboratoriene ble bedt om å oppgi resultatene uten bruk av lokale korreksjoner men oppgi hvilke korreksjoner laboratoriet benytter i rutine. Etter denne introduksjonen vil du finne en generell rapport på engelsk som innbefatter hele studien forfattet av Dietmar Stöckl. På neste års NKK-møte vil det bli arrangert en workshop hvor alle resultater og presentasjonen av dem blir gjennomgått i detalj. I vedlegg mottar du lenker til individuelle Excel-rapporter, laget av Dietmar Stöckl, for hver av de fire målte komponentene. Navn på filene er Nativ_ Komponent.xls (basert i hovedsak på. duplikat), Nativ Komponent.xls (basert i hovedsak på. duplikat) og Nativ_ Komponent B.xls (basert i hovedsak på beste replikat evt. slengere er fjernet) hvor Komponent kan være Ca, Mg, albumin eller Protein. I Carapporten (Nativ_Kasium.xls) har NKK satt inn norske kommentarer og forklaringer til innholdet i enkeltceller. Les derfor denne Ca-rapporten først slik at du lettere forstår de andre rapportene. Hvis du ikke har slengere i dataene dine, skal det være unødvendig å se på andre enn Nativ_ Komponent.xls-fila altså fila som viser data fra. replikat. Alle celler med et rødt øre oppe til høyre har en kommentar. For de kommentarene som ikke vises (pga. plassmangel), må du føre musepila over cellen for at kommentaren skal vises. Alle kommentarene kan skjules/vises ved å gå inn på Se gjennom og klikke på Vis alle merknader (Office 7). Øverst til venstre i Excel-arket Your Data (celle A) skal du angi et tall som representerer ditt laboratorium klikk pilene i cellene C-C3 slik at ditt Labqualitynummer vises i celle A. Alternativt kan du finne ditt Labquality-nummer (Code) i den sorterte tabellen til høyre tallet til høyre for dette (Lab) kan du taste inn i celle A. Da vises ditt laboratoriums data i regnearket.

2 NKK og NOKLUS, oktober Her er en foreløpig og kortfattet forklaring til de individuelle Excel-rapportene: I rapportene blir ManuMean brukt som referanseverdi for hver prøve. ManuMean er gjennomsnitt av resultatene fra alle produsentene hvor hvert metodesystem er vektet likt. NB! Gruppen Roche Cobas Integra er ikke inkludert, primært fordi den kun inneholdt 5 instrument, og sekundært for ikke å få overrepresentasjon av Roche-metoder (iflg. Dietmar Stöckl som har behandlet dataene). Man får ingen sammenligning mot egen metode, men metodene er sammenlignet i den generelle rapporten. Excel-rapportene inneholder til venstre i arket Your data grafiske presentasjoner av eget resultat (markert med rødt symbol) i forhold til resultatene fra alle deltakerne: - Øverst: Et plott av alle deltakernes gjennomsnitt (av alle native sera) sortert i stigende rekkefølge - I midten: Frekvensfordeling av alle deltakernes % bias fra ManuMean - Nederst: Frekvensfordeling av alle deltakernes korrelasjonskoeffisienter Average (plottet øverst til venstre) og Bias (%) (plottet i midten til venstre) er beregnet på grunnlag av gjennomsnitt av duplikatmålinger. Under første plott på venstre side er Bias % for ditt laboratorium oppgitt basert på gjennomsnitt av duplikat og fra. duplikat ( st Rep ). I plottet av korrelasjonskoeffisienter (plottet nederst til venstre) er primært første duplikatmåling brukt (dersom den første måling er en slenger er den. brukt). Korrelasjonsanalysen inkluderer i tillegg regresjonsdata og de tre plottene på høyre side: I korrelasjonsanalysen vist i plottene til høyre er primært første duplikatmåling brukt (dersom den første målingen er en slenger er den. brukt): - Øverst: scatterplott med regresjonslinje m/konfidensintervall (CI) for eget resultat (. duplikat) vs ManuMean. I tillegg er grenser for ± 5 % rundt identitetslinjen ( Line of equality ) markert. - I midten: differanseplott (%) for eget resultat vs ManuMean. Grenser for ± 5 % og ± 3 % er markert. - Nederst: residualplott hvor % avvik fra regresjonslinjen (linjen y=) er plottet. ±TE (total error) fra Ricos er også plottet. Rapporten gir også informasjon relatert til referanseintervall: Laboratoriets referanseintervall, det mest brukte referanseintervallet i denne undersøkelsen og anbefalt referanseintervall fra NORIP. I tillegg vises hvor mange resultater som ligger utenfor eget referanseintervall (eller det mest brukte hvis eget ikke er oppgitt) og gjennomsnitt antall prøver utenfor for alle laboratorier. Antall ManuMean ( vår referanseverdi) som ligger utenfor det mest brukte referanseintervallet, vises også. Ideelt skal det være ett resultat utenfor referanseintervallet, men man må huske på at dette er prøver fra USA mens referanseintervallet gjelder for ditt laboratorium. I vurdering av resultatene er det brukt ulike kvalitetsindikatorer som indikerer FAIL/PASS av egne resultat. Disse kvalitetsindikatorene er skjønnsmessig valgt (Stöckl). Følgende kvalitetsindikatorer og FAIL -grenser ble brukt til vurdering av laboratoriets analysekvalitet ( Lab Quality indicators ):

3 NKK og NOKLUS, oktober - CV >,5 % (variasjonskoeffisient innen serie/for repeterbarhet - beregnet på grunnlag av differansen mellom duplikatmålingene) FAIL tyder på stor upresishet på ditt laboratorium. - Gjennomsnittlig differanse mellom duplikater >,5 % - FAIL tyder på drift i instrumentet - Gjennomsnittlig bias (beregnet fra duplikater) fra metodegjennomsnitt > 3 % - FAIL tyder på bias større enn 3 % i forhold til metodegjennomsnitt. - Trend; r >, (beregnet fra % differanse mellom duplikater og prøvenummer) FAIL tyder på drift i serien Videre ble følgende kvalitetsindikatorer brukt for å sammenligne laboratoriets eget resultat med ManuMean ( Quality indicators for overall performance ): - Korrelasjon: r <,9 FAIL tyder på stor spredning rundt regresjonslinjen - Bias ± CI > 3 % (basert på differanse mellom gj.snitt av egne duplikater og ManuMean ved t- test) FAIL tyder på signifikant bias > 3 % (5 % signifikansnivå) - Bias ± CI > 3 % ved laveste og høyeste konsentrasjon (RANGE LIMITS) predikert ved regresjonslinje og konfidensintervall FAIL tyder på at regresjonslinjens konfidensintervall ved høyeste og laveste konsentrasjon ligger signifikant utenfor 3 % avvik - 3 resultat med totalfeil > ± 5 % - FAIL tyder på dårlig nøyaktighet - Slope (inklusiv konfidensintervall) <,9 eller >, FAIL tyder på konsentrasjonsavhengig bias Kvalitetsindikatorer for bias er basert på gjennomsnitt av duplikatmålinger mens alle andre er basert på første duplikatmåling primært (dersom en av målingene er en slenger blir den ikke brukt). I Excel-rapporten er det ingen data om EKV-prøvene og deres kommutabilitet (pasientlikhet) disse er bare beskrevet i den generelle rapporten under Commutability med oppsummering i tabell 7 og plottene vist i figurene 3 og i Annex. Disse plottene gir en umiddelbar forståelse av hvor godt EKV-prøvene representerer reelle prøver. En kort konklusjon er gitt i Summary and Conclusions. Noen korte forklaringer: Med peer group menes den gruppen man sammenlignes med. Understrekninger i tabeller, hvis ikke spesielt forklart, betyr at verdien er spesielt kommentert i teksten. Når sign. er angitt i teksten menes signifikant på 5 % nivå Vennlig hilsen Sverre Sandberg, NOKLUS Thomas Helge Røraas, NOKLUS Gunn BB Kristensen, NKK Pål Rustad, NKK 3

4 NKK og NOKLUS, oktober Pilot study: Use of frozen, single donation sera for External Quality Assessment of clinical laboratories in Norway Author: Dietmar Stöckl Norsk kvalitetsforbedring av laboratorievirksomhet utenfor sykehus ( NOKLUS, (Sverre Sandberg, sverre.sandberg@isf.uib.no, Thomas.) Laboratory for Analytical Chemistry (LAC) of Ghent University (UGent) (Linda Thienpont, linda.thienpont@ugent.be), in cooperation with STT Consulting ( (Dietmar Stöckl, dietmar@stt-consulting.com). Norsk Klinisk-kjemisk Kvalitetssikring ( NKK, (Pål Rustad, prustad@furst.no)

5 CONTENT NKK og NOKLUS, oktober Introduction... Study design... Laboratories and requested information... Analytes and measurement design... Samples... 7 DATA ANALYSIS (ONLY native samples, except for commutability)... 7 Laboratory information... 7 Target values... 7 Peer group targets... 7 Overall target... 7 Laboratory assessment... 7 Quality indicators for run-to-run quality and peer-group performance... 7 Quality indicators for overall performance (note, they include assay bias)... System assessment... Commutability... Laboratory report (EXCEL-file)... Results and Discussion... 9 Laboratory information... 9 Systems and tests... 9 Reference intervals... 9 Target values... Laboratory assessment... Quality indicators for run-to-run quality and peer-group performance... Quality indicators for overall performance (note, they include assay bias)... 3 Laboratory averages... Factors... System assessment... 5 Calcium... 7 Magnesium... Albumin... 9 Protein... System assessment from laboratory quality indicators... Within-run performance... Fail-percentages for run-to-run quality and peer-group performance... Fail percentages for overall performance... Commutability... SUMMARY AND CONCLUSIONS... 3 Laboratory performance... 3 System performance... 3 Commutability... 3 References... 3 ANNEX... Commutability

6 INTRODUCTION NKK og NOKLUS, oktober External quality assessment (EQA) investigates the quality and status of standardization of both in vitro diagnostic (IVD) assays and clinical laboratories. EQA typically makes use of processed samples (stripped, supplemented, delipidated, lyophilized etc.), rather than of native samples. Processed samples may be non-commutable with IVD assays and, therefore, not appropriate to provide unbiased data on quality and standardization of assays. In addition, EQA schemes typically only use a limited number of samples, for example, representing a low, mid and high concentration of an analyte. Again, this is a limitation for generating transparent and reliable data about the performance of both IVD assays and of the laboratory. Therefore, NOKLUS, UGent and STT Consulting agreed to perform a pilot EQA study with a panel of native sera and typical EQA materials with the following objectives: Create a reliable database documenting the current quality and standardization status on commutable sera, with sufficient attention to performance characteristics such as imprecision, accuracy, sample-related effects and specificity. Compare the performance of IVD assays not confounded by commutability issues. Assess the standardization status. Assess commutability of common EQA samples. Identify quality benchmarks. The overarching aim after accomplishment of the above objectives is that the study serves as incentive for the individual IVD manufacturer to improve quality and for NOKLUS and NKK to improve the Norwegian EQA scheme. Below we report on the results of the pilot study conducted in June. STUDY DESIGN LABORATORIES AND REQUESTED INFORMATION All Norwegian laboratories participating in Labquality s clinical chemistry EQA schemes were invited to take part in the study. The laboratories were asked to provide information about i) the test (principle, instrument, reagent and calibrator lot), ii) the measurement sequence for the analytes, iii) factors used for reporting patient results and iv) reference intervals for a year old male. ANALYTES AND MEASUREMENT DESIGN The analytes covered were S-calcium, S-magnesium, S-albumin and S protein. Two measurements were performed per sample, but each replicate in a different run. In the first run, the samples were measured upwards (# to #) with the EQA samples positioned after sample #7 and #. In the second run the samples were measured in reverse order. A template for reporting of the results was provided.

7 SAMPLES NKK og NOKLUS, oktober A panel of sera from Solomon Park Research Laboratories (Kirkland, WA, USA) was used. Blood collections and preparation of serum (approximately ml per unit) were done using the CLSI C37-A protocol$, however, without filtration and with addition of human thrombin (from Sigma, U/mL plasma), to ensure clotting within 3 hours at room temperature. The serum of each donation was assessed and found negative in the usual serological testing (HIV, HBsAG, HCV and Rapid Plasma Reagin). Samples were aliquoted ( ml), frozen and stored at -7 C, then shipped on dry ice to NKK that subsequently shipped the samples frozen to the study participants. Each of them received aliquots per sample. The two EQA materials were normal and spiked human serum pools, filtered (. m) and frozen (Labquality). DATA ANALYSIS (ONLY NATIVE SAMPLES, EXCEPT FOR COMMUTABILITY) LABORATORY INFORMATION The information of each laboratory comprised information about the test, factors and reference interval. From this data, peer groups (n 5, except Abbott magnesium: n = ) were identified for laboratories that used the same test combination for what concerns instrument, reagent and calibrator. Available lot information, measurement sequence and factors applied for reporting patient results were used as cross-check to verify discrepant results. Reference intervals were presented in histograms. The most frequently used intervals were identified and applied for laboratories that had failed to provide this information. TARGET VALUES PEER GROUP TARGETS Peer group targets were calculated for each sample as average of the mean of duplicates. Grossly outlying replicates (calcium 7, magnesium, albumin 5, protein ; by visual detection) and missing replicate values (calcium, magnesium, protein ) were substituted with the value of the other replicate. Peer groups were investigated for outliers by means of the Grubbs test. OVERALL TARGET Overall target values ( ManuMean ) were calculated as average of the peer group means in order to give the same weight to each manufacturer. Note, the Integra peer group was NOT included, firstly because it contained only 5 laboratories, secondly to prevent an overrepresentation of the Roche systems. LABORATORY ASSESSMENT QUALITY INDICATORS FOR RUN-TO-RUN QUALITY AND PEER-GROUP PERFORMANCE To judge the run-to-run and peer-group performance the following indicators ( FAIL -limits) were used: i) CV (.5 %), calculated from the differences between the replicates on different runs; ii) 7

8 NKK og NOKLUS, oktober average difference between replicates (.5 %); iii) average bias (calculated from duplicates) from the peer group target (3 %, paired t-test); iv) trend r (r >.), calculated from % difference between replicates and sample number; v) outliers in the differences between replicates identified with the Grubbs test (when present, the replicate causing the outlier was identified). QUALITY INDICATORS FOR OVERALL PERFORMANCE (NOTE, THEY INCLUDE ASSAY BIAS) Quality indicators for overall performance are based on the ManuMean. The indicators ( FAIL - limits) were i) correlation (r <.9); ii) overall bias (3 %, paired t-test); ii) bias at the range limits of the sample concentration (3 %, including the confidence interval of the regression line); iii) number of results with a total error > 5 % ( 3 results); slope (<.9 or >., including confidence interval of the slope). Note, all indicators for systematic error (bias, bias at range limits, slope) are based on the mean of the duplicates, while the others (correlation and total error) are based on the first replicate ( nd if the st is an outlier). Note, the number or results outside the laboratory s own reference interval and an overview about the distribution of the laboratory averages are given for information only. Fail percentages are tabulated and histograms are given to document the distribution of laboratory correlation, bias and number of results outside a generic total error limit of 5 %. SYSTEM ASSESSMENT System comparability and quality is assessed by correlation and regression analysis of the peergroup means versus the overall mean (ManuMean). Data are also graphically presented by means of %-difference plots. System performance is also derived from the laboratory quality indicators. COMMUTABILITY To assess commutability of the EQA samples, regression analysis of the peer group data versus the ManuMean (native samples only) was performed, followed by investigation whether the peer group results of the EQA samples were within the prediction interval of the regression (EP approach). Data are graphically presented by use of a difference plot including the prediction interval. LABORATORY REPORT (EXCEL-FILE) The individual laboratory receives reports that for each analyte measured give the PASS / FAIL decisions for the above Quality Indicators. The report also contains a graphical presentation of the value and position of the individual laboratory: in the continuum of all averages in the frequency distribution of all %-biases from the ManuMean in the frequency distribution of all correlations to the ManuMean For the calculation of the average and % bias, the mean of the replicates was used, while the correlation analysis was done with the st replicate ( nd if st is an outlier). Correlation analysis includes: regression data (equation, confidence interval of regression line) scatterplot with generic limits of ± 5 % %-difference plot with generic limits of ± 5 % and ± 3 %

9 NKK og NOKLUS, oktober %-residuals plot with the Ricos biological TE limit Finally, the report gives information related to the reference interval: the laboratory s reference interval the number of sample results observed outside this reference interval reference intervals most frequently used in this study and generally used (NORIP) The laboratories can access their own results and those of all other laboratories by a dynamic spinner in the EXCEL-templates. RESULTS AND DISCUSSION LABORATORY INFORMATION SYSTEMS AND TESTS The table below shows the number of systems (company) used in the study for each analyte. In total 5 laboratories participated: 55 measured the samples for calcium, 3 for magnesium, 5 for albumin and 9 for protein (total number of tests = 3). Table. Systems and tests used in the study Company/System n Calcium Magnesium Albumin Protein Abbott Architect Ortho Vitros Roche Cobas Roche Integra Roche Modular Siemens Advia 7 Others Horiba (n =), Hitachi (Roche) (n = ), Dimension RxL (n = ) & Vista (Siemens) (n = ) REFERENCE INTERVALS The two most frequently used reference intervals were for calcium.5.5 mmol/l and.5.55 mmol/l, for magnesium.7.9 mmol/l and.7.95 mmol/l, for albumin 3 5 g/l and 3.5 g/l and for protein 7 g/l and 79 g/l. Typically, the most frequently used reference intervals are similar to the ones established in the NORIP project. The broadest ranges reported were for calcium.5. mmol/l, magnesium.7.5 mmol/l, albumin 35 5 g/l and protein 7 g/l. 9

10 NKK og NOKLUS, oktober Table. Most frequently used reference intervals in the study and those established in the NORIP project Low High Range % Calcium Study ( of 53 laboratories) NORIP Magnesium Study (3 of laboratories) NORIP Albumin Study (3 of 55 laboratories) NORIP Protein Study (33 of 7 laboratories) 7 3 NORIP 7 3 NORIP (Nordic Reference Interval Project; More detailed information about the reference intervals used by the different laboratories can be found in the Annex: Fig. Distribution of reference intervals (lower limit, upper limit and range). TARGET VALUES Overall target values (ManuMean) were calculated from the average of the Abbott Architect, Ortho Vitros, Roche Cobas, Roche Modular and Siemens Advia peer groups. The following table gives the number of laboratories that constituted the peer groups for the different analytes. Table 3. Peer groups with indication of number of tests done Peer group Calcium Magnesium Albumin Protein Abbott Architect Ortho Vitros Roche Cobas 3 3 Roche Modular Siemens Advia Note, the numbers for calcium and magnesium differ slightly from those in the above table because some laboratories applied heterogeneous tests (different manufacturers for system and reagent/calibrator). A more detailed insight in the distribution of the peer group means will be given below under System assessment. The rationale for using the ManuMean as target was the assumption that globally operating manufacturers have standardized their systems by use of a trueness-based reference measurement system to comply with the requirements of the European Directive 9/79 (see also BIPM/JCTLM database; calcium: atomic absorption spectrophotometry; magnesium: atomic absorption spectrophotometry; albumin: nephelometry, turbidimetry; protein: spectrophotometry). We also supposed that laboratories using a homogeneous system should reflect that standardization status.

11 NKK og NOKLUS, oktober The table below shows the concentrations (ManuMean) for the native samples in ascending order and per analyte. Concentrations outside the most frequently used reference interval are underlined. Because the samples were from apparently healthy individuals 95 % ( of ) of the concentrations were expected to be in the reference interval (although these are samples from USA and the reference intervals are established for Nordic individuals). For calcium, however, 3 samples had a concentration below the lower limit of the most frequently used reference interval; for albumin sample had its value outside the lower and upper limit; none of the sample concentrations was outside for magnesium; were outside the low limit for protein. Table. Concentrations (ManuMean) for the native samples in ascending order; concentrations outside the most frequently used reference interval are underlined. Calcium Magnesium Albumin Protein Sample (mmol/l) Sample (mmol/l) Sample (g/l) Sample (g/l) #.9 #.7 #9 35. #9 53. #9. #.7 # 3. #. #.3 #.75 # 37. #.3 #.5 #7.7 # 3. #5.3 #.5 #. # 3.7 #7.7 #7.9 #. #7 39. # 3. #.9 #5. #.3 #.3 #. #. #3. #.5 #. #.3 #.7 #. #. #3.3 #. # 5. #5. #7. #. #3. #3. #. #.3 #7. #7. #5. #. #. #5. #.5 #7. #3.7 #. #3.5 #5 3. # 7. #.3 #9. #5 3. #5 7. #.3 #.7 # 3. # 7.7 #9.3 #9.9 #3 3.3 #. #3.33 #.9 # 5. # 9.7 #.3 #.9 #9 5.7 #9 7.3 LABORATORY ASSESSMENT QUALITY INDICATORS FOR RUN-TO-RUN QUALITY AND PEER-GROUP PERFORMANCE Table 5 shows the median CV, mean difference of replicates and mean difference from peer group. The average median CV is. %, with a maximum of. % for magnesium. The average mean difference of the replicates is. % and similar for all analytes. The average mean difference from the peer group is. % and also similar for all analytes.

12 NKK og NOKLUS, oktober Table 5. Medians for CV, mean difference of replicates and mean difference from peer group Calcium Magnesium Albumin Protein Average Median CV ( %)..... Diff Repl ( %) Diff Peer ( %).... More detailed information can be found in the Annex: Fig. 7. Distribution of median CV, mean difference of the replicates and mean difference from the peer group (limits and fail percentages indicated). Laboratory quality was further investigated by FAIL percentages (limits) for CV (.5 %), mean difference of replicates (.5 %), mean difference from peer group (3 %, paired t-test), trend (r >.) and outliers (Grubbs test) (see table below). All FAIL limits were chosen generic (the same for all analytes) and to obtain realistic failure rates. Table. Fail-percentages of indicators for run-to-run quality and peer-group performance (CV, mean difference of replicates, mean difference from peer group, trend and outliers) Calcium Magnesium Albumin Protein Average CV Diff Rep Diff Peer Trend Outlier Average More detailed information about the fail percentages can be found in the Annex: Table 9. Fail events for mean difference within peer group, mean difference between replicates, CV, outliers and trend. The Annex also contains examples of laboratories that showed trends and outliers: Fig.. Examples of laboratories where trends have been observed; Fig.. Examples of laboratories where outliers have been observed. In total, failures were observed for 3 tests, amounting to an average of.3 % per indicator. Failures occurred most frequently for magnesium (average for all indicators 7.9 %; note the high failure rate of % for CV), then for calcium (7 %), protein (5. %) and albumin (.9 %). The indicator with the most frequent failure for all analytes was the peer difference (average. %), followed by the CV (7.7 %). The other failure rates were in the order of 5 %. Note that the CV indicator is sensitive to imprecision, systematic differences between the runs, trends and outliers. One peer group failure (Albumin, Lab ) was related to the application of a factor for reporting patient results. The results of that laboratory were corrected, accordingly. Other peer group failures could not be related to the application of factors for patient reporting. Seven of the 5 laboratories accounted for of the 3 failures (3 or more failures), without those, the average failure/criterion was reduced to 3.7 % (see table 9). Note that the failure rates are strongly indicative for poor laboratory performance, however, some influence of the test system

13 NKK og NOKLUS, oktober cannot be ruled out (see also below under System assessment ). We found, for example, that the outliers occurred at random (in different laboratories in different runs), suggesting that they should be attributed to the system rather than to the laboratory (see Table in Annex). Table 7. Laboratories with 3 failures EQA code Company Instrument Fails () 3 Siemens Advia 5 3 () 7 Roche Cobas 3 (3) 5 Siemens Advia 3.5 () 5 Siemens Advia (5) 3 Roche Cobas () 75 Roche Hit 9 (7) Siemens Dim RxL.5 () ()Non-integer fail numbers because outliers were counted as.5 (twice the probability of occurrence of the other indicators) ()Problems with calcium and albumin. (3)Problems with albumin and protein. ()Problems with protein and magnesium. (5)Mainly due to calcium measurements. ()Mainly due to calibration on nd day. (7)Severe problems with magnesium. ()Mainly due to poor performance on st day. It should be mentioned here that laboratory 73 most probably mixed-up samples # and #7 and that laboratory probably mixed-up samples # and # for calcium analysis. QUALITY INDICATORS FOR OVERALL PERFORMANCE (NOTE, THEY INCLUDE ASSAY BIAS) Quality indicators for overall performance are based on the ManuMean. The indicators (fail limits) are correlation (r <.9), bias (sign. >3 % by paired t-test), bias at range limits (3 %, regression confidence interval), total error ( 3 results >5 % difference) and slope (sign. <.9 or >.). All indicators for systematic error (bias, bias at range limits, slope) are based on the mean of the duplicates, while the others are based on the st replicate ( nd if st was an outlier). Except the limit for the residuals, all fail limits were chosen generic (the same for all analytes) and to obtain realistic failure rates. The limits for the residuals were the allowable total error derived from the biological variation of the analytes (Westgard page). The bias and bias at range limits (3 %) were matched to the run-to-run and peer performance limits. The fail percentages for these indicators (see table below) are much higher than for those considered above, in particular for bias, bias at range limits and total error and for magnesium and albumin (>3 %). This indicates an overruling effect of the system (see also below System assessment ). The relative high fail percentages for protein, however, are mainly due to laboratories 5, 55, 5 (EQA codes: 75,, 3). Note, the influence of the test system could not be investigated because no peer group could be established for them. 3

14 NKK og NOKLUS, oktober Table. Fail percentages for indicators of overall performance Calcium Magnesium Albumin Protein Average Correlation (r) Bias Bias at range limits Total error 3 Slope. 5.. Average 35 The Annex also contains Fig : Distribution of laboratory correlation, bias and number of results outside a generic total error limit of 5 % (based on replicate without outlier). LABORATORY AVERAGES There was a considerable variation in the averages of the laboratories for all analytes (see Table and 9). Table 9. Distribution of laboratory averages Calcium Magnesium Albumin Protein (mmol/l) (mmol/l) (g/l) (g/l) Average Minimum Maximum A graphical overview of the distribution of averages can be found in Annex: Fig.. Overview of the distribution of laboratory averages. FACTORS Table lists the factors that laboratories apply for patient reporting and the respective system biases found in this study (note: no factors were reported for protein). Factors between.9 and. are probably lot related because they may be even smaller than the manufacturer s own specifications for lot-to-lot variations. The factors used for the Abbott Architect magnesium and albumin assays are indeed justified because of the high biases observed in this study. Other factors are reasonable, however, may be unnecessary in view of the small biases observed and lot-tovariation (Siemens Advia albumin, laboratories). Some factors, even, seem to be counterproductive (indicated by ). The factors for the Roche magnesium reagent on Abbott Architect and for the Audit calcium reagent on the Roche Modular cannot be interpreted by use of a bias criterion of the parent assay. Note, magnesium for laboratory corresponds to a typical Roche assay; calcium for laboratory is already high, a factor of.7 would be clearly inappropriate.

15 NKK og NOKLUS, oktober Table. Factors for reporting patient results and respective assay bias. The bias is calculated NOT using the factor. Calcium Magnesium Albumin Laboratory Factor Bias ( %) Factor Bias ( %) Factor Bias ( %) 55 Abbott [.]#? Abbott Abbott Abbott Ortho.9. [.] -. Ortho [.9]. [.9] -. Modular.7$? 9 Advia [.95] Advia.93. Advia Advia.9. Cobas.9. #Roche reagent; $Audit reagent. SYSTEM ASSESSMENT The quality and comparability of the systems was, in first instance, assessed by comparison of the peer group data with the ManuMean (see also above Target values ). Peer groups, generally, were quite homogeneous (reagents and instrument from the same producer). There was only one borderline outlying laboratory in the Ortho Vitros Albumin group (Grubbs test: 3.5 g/l significant, 3. g/l not significant) (see Fig below). The data of this laboratory were retained for the peer group despite the borderline outlying value. The Abbott Architect peer group for magnesium was not included in the ManuMean because it was a outlier in the Grubbs test. Also problematic was the peer group for the Siemens Advia system (n = ), which showed a relative high variation (3 laboratories had peer biases of -5.3 %, 3.7 % and 3.5 %). Fig. Distribution of peer groups for Ortho Vitros albumin and Abbott Architect and Siemens Advia magnesium. 5. Albumin OrthoLabs (g/l) Magnesium AdviaLabs (mmol/l) Albumin OrthoMean (g/l) 5 Magnesium SiemensAdviaMean (mmol/l)

16 NKK og NOKLUS, oktober Data were investigated by regression, correlation, calculation of bias and calculation of the individual %-differences. In a second instance, system quality and comparability were assessed by use of the Laboratory quality indicators.

17 CALCIUM NKK og NOKLUS, oktober All systems are within the generic bias limit of 3 % (maximum bias = -. %, Roche Modular) but only one is within the biological bias limit of. % (Roche Cobas). The bias is nearly constant over the range and no outlying samples were observed. The comparability of the assays could be improved in view of the small biological bias limit. The observed values, however, are small and may be within the lot-to-lot variation of many manufacturers. Table. Peer groups versus ManuMean (x) regression, correlation and bias data System (y) Regression Correlation Bias ( %) Abbott Architect y =.99 x Ortho Vitros y =. x Roche Cobas y =. x Roche Integra y =. x Roche Modular y =.97 x Siemens Advia y =.99 x Fig. Difference ( %) of peer groups from the ManuMean; indicated are generic (3 %, red long-broken lines) and biological limits (. %, blue short broken lines) for systematic error. Abbott difference (%) Ortho difference (%) Calcium ManuMean (mmol/l) Calcium ManuMean (mmol/l) Cobas difference (%) Integra difference (%) Calcium ManuMean (mmol/l) Calcium ManuMean (mmol/l) Modular difference (%) Siemens difference (%) Calcium ManuMean (mmol/l) Calcium ManuMean (mmol/l)

18 MAGNESIUM NKK og NOKLUS, oktober Except the Abbott Architect (bias =. %), all systems are borderline or within the generic bias limit of 3 % and three within the biological bias limit of. % (Ortho Vitros, Roche Cobas and Roche Modular). The bias is nearly constant over the range and significant sample-related effects are only observed for the Abbott Architect assay. Table. Peer groups versus ManuMean regression, correlation and bias data System Regression Correlation Bias ( %) Abbott Architect y =.97 x Ortho Vitros y =. x Roche Cobas y =.93 x Roche Integra y =.97 x Roche Modular y =. x Siemens Advia y =. x..99. Fig 3. Difference ( %) of peer groups from the ManuMean; indicated are generic (3 %, red long-broken lines) and biological limits (. %, blue short broken lines) for systematic error. Abbott difference (%) Magnesium ManuMean (mmol/l) Ortho difference (%) Magnesium ManuMean (mmol/l) Cobas difference (%) Magnesium ManuMean (mmol/l) Integra difference (%) Magnesium ManuMean (mmol/l) Modular difference (%) Magnesium ManuMean (mmol/l) Siemens difference (%) Magnesium ManuMean (mmol/l)

19 ALBUMIN NKK og NOKLUS, oktober The comparability of the assays is poor (Roche represented by only BCG methods). Three systems are outside the generic bias limit of 3 % (Abbott Architect, Roche Integra, Roche Modular) and none is within the biological bias limit of.3 %. For two systems, the bias is concentration dependent (significant slope/intercept combinations: Ortho Vitros and Siemens Advia) and significant samplerelated effects may be present for the Abbott Architect assay. Table 3. Peer groups versus ManuMean regression, correlation and bias data System Regression Correlation Bias ( %) Abbott Architect y =. x Ortho Vitros y =.7 x Roche Cobas y =.95 x Roche Integra y =.99 x Roche Modular y =.9 x Siemens Advia y =. x Fig. Difference ( %) of peer groups from the ManuMean; indicated are generic (3 %, red long-broken lines) and biological limits (.3 %, blue short broken lines) for systematic error. Abbott difference (%) Ortho difference (%) Albumin ManuMean (g/l) Albumin ManuMean (g/l) Cobas difference (%) Albumin ManuMean (g/l) Integra difference (%) Albumin ManuMean (g/l) Modular difference (%) Albumin ManuMean (g/l) Siemens difference (%) Albumin ManuMean (g/l) 9

20 PROTEIN NKK og NOKLUS, oktober The comparability of the assays is excellent. All systems are within the generic bias limit of 3 % (maximum bias = - %, Roche Integra) and 5 systems are within the biological bias limit of. %. The bias is nearly constant over the range and significant sample-related effects are observed for the Ortho Vitros assay and for sample for the Abbott Architect and Siemens Advia systems. Table. Peer groups versus ManuMean regression, correlation and bias data System Regression Correlation Bias ( %) Abbott Architect y =.9 x Ortho Vitros y =.9 x Roche Cobas y =.97 x Roche Integra y =.9 x Roche Modular y =.97 x Siemens Advia y =. x Fig 5. Difference ( %) of peer groups from the ManuMean; indicated are generic (3 %, red long-broken lines) and biological limits (. %, blue short broken lines) for systematic error. Abbott difference (%) Ortho difference (%) t-protein ManuMean (g/l) t-protein ManuMean (g/l) Cobas difference (%) Integra difference (%) t-protein ManuMean (g/l) t-protein ManuMean (g/l) Modular difference (%) Siemens difference (%) t-protein ManuMean (g/l) t-protein ManuMean (g/l)

21 NKK og NOKLUS, oktober SYSTEM ASSESSMENT FROM LABORATORY QUALITY INDICATORS Within-run performance While the systems typically showed good within-run performance, it was surprising to see relatively many outliers, in particular, for the calcium and protein assays (calcium, magnesium 3, albumin 5, protein 9). We attributed the outliers to the systems because they were randomly distributed over the laboratories during the runs (see Table in Annex). Therefore, we selected the run without outliers for investigating overall laboratory performance. When selecting these runs, the good within-run performance is reflected in the fact that only few laboratories had > residual outside the biological total error limits (calcium; of 55, magnesium: 3 of 3, albumin: of 5, protein: of 9). Fail-percentages for run-to-run quality and peer-group performance The quality indicators for run-to-run quality and peer-group performance (CV, mean difference of replicates, mean difference from peer group, trend and outliers) are strongly indicative for laboratory performance, however, the quality of the laboratory also reflects the intrinsic quality of the system (for example, imprecision, calibration stability, robustness in different laboratories). When these indicators are grouped by system, relative high fail percentages are observed for the Abbott Architect magnesium assay, the Roche cobas albumin assay and the Siemens Advia calcium and magnesium assays (see table below). Table 5. Fail-percentages of indicators for run-to-run quality and peer group performance (CV, mean difference of replicates, mean difference from peer group, trend and outliers) grouped by systems Calcium Magnesium Albumin Protein Average Abbott Architect Ortho Vitros Roche cobas Roche Modular Siemens Advia Average Integra data not representative due to low number of laboratories Fail percentages for overall performance It has been discussed above already that the quality indicators for overall performance (based on the ManuMean) showed very high fail percentages, in particular, for those that are sensitive to systematic errors (bias, bias at range limits, total error and in particular), indicating an overruling system effect. Grouping these indicators for the systems (see table below) shows that the failures are mainly due to the magnesium assays of Abbott Architect and Roche Cobas and for the albumin assays of Abbott Architect and Roche Modular. To a lesser extent, they are due to the Ortho Vitros Albumin assay and the Siemens Advia magnesium assay. The failures are typically related to the bias of the systems versus the ManuMean. However, for what concerns the Ortho Vitros Albumin assay and the Siemens Advia magnesium assay, they are due to the peer group bias itself.

22 NKK og NOKLUS, oktober Table. Fail percentages for indicators of overall performance (correlation, bias, bias at range limits, total error and slope) grouped by systems Calcium Magnesium Albumin Protein Average Abbott Architect Ortho Vitros Roche Cobas Roche Modular 7 9. Siemens Advia. 5 Average The Annex also contains Fig. Distribution of laboratory correlation, bias and number of results outside a generic total error limit of 5 % (based on replicate without outlier). COMMUTABILITY The native samples were prepared according the CLSI C 37 protocol and are considered commutable. The two EQA materials were normal and spiked human serum pools, filtered (. m) and frozen. The table below shows that the EQA material is generally commutable, or shows deviations from the native materials that are not considered significant (PI in table 7:.. %). EQA material shows significant non-commutability for the Abbott Architect calcium assay (. %), the Ortho Vitros and Siemens Advia magnesium assays (5. % and. %) and the Abbott Architect and Ortho Vitros assays for albumin (5. % and 7. %). The material may also be noncommutable for the Abbott Architect magnesium assay (3. %), however, the prediction interval is rather high (3. %) so that the difference is not significant. Table 7. Commutability data for the EQA sera # & # (%-difference at concentration of EQA sample for non commutable; ok: commutable) Calcium Magnesium Albumin Protein # # PI # # PI # # PI # # PI Abbott Architect... ok [3.] 3. ok ok.7 Ortho Vitros ok ok.5 ok 5.. ok 7.. ok ok. Roche Cobas ok ok.3 ok ok. ok ok. ok ok.7 Roche Modular.5 ok.5.7 ok. ok ok. ok ok.5 Siemens Advia ok ok. ok.. ok ok 3.. ok. Prediction interval (%) at the mean of the data The Annex shows a graphical presentation of the commutability data: Fig 3: Commutability - Calcium and magnesium (EQA samples in pink) and Fig : Commutability - Albumin and protein (EQA samples in pink; sample for protein corrected for Abbott, Modular and Advia).

23 NKK og NOKLUS, oktober SUMMARY AND CONCLUSIONS LABORATORY PERFORMANCE Laboratory performance was dominated by the performance of the system they used (note: peer groups could not be established for laboratories). Nevertheless, the somewhat higher peer group failure may indicate room for improvement of quality control and quality assurance procedures. Seven laboratories were identified with poorer analytical performance in comparison to the other ones, however, system effects cannot be excluded for some of them. It seemed that some sample mix-up occurred in laboratories. SYSTEM PERFORMANCE The systems showed a, generally, good within-run performance (good precision, low number of residuals outside biological total error limits). Unfortunately, the calcium and protein assays were prone to outliers. Some run-to-run and peer performance problems were observed for the Abbott Architect magnesium assay, the Roche Cobas albumin assay and the Siemens Advia calcium and magnesium assays. Comparability of the assays was excellent for protein, moderate for calcium (maximum bias -. % for Roche Modular), and problematic for albumin (-7.3 % Abbott Architect, 5. % Roche Modular, % Roche Integra). For what concerns magnesium, a high bias was observed for the Abbott Architect assay (7.3 %). COMMUTABILITY EQA sample is sufficiently commutable to assess system bias in external quality assessment. EQA sample shows moderate to strong matrix effects for some calcium, magnesium and albumin assays. The use of this material for the assessment of system bias in external quality assessment is questionable. REFERENCES. Clinical and Laboratory Standards Institute (CLSI). Preparation and Validation of Commutable Frozen Human Serum Pools as Secondary Reference Materials for Cholesterol Measurement Procedures; Approved Guideline. CLSI document C37-A (ISBN ). CLSI, Wayne, PA, USA,

24 NKK og NOKLUS, oktober ANNEX Fig.. Distribution of reference intervals (lower limit, upper limit and range).

25 NKK og NOKLUS, oktober Fig. 7. Distribution of median CV, mean absolute difference of the replicates and mean absolute difference from the peer group (limits and fail percentages indicated). 5

26 NKK og NOKLUS, oktober Table. Laboratory EQA code and run where an outlier occurred Calcium Magnesium Albumin Protein st day nd day st day nd day st day nd day st day nd day

27 NKK og NOKLUS, oktober Table 9. Fail events for mean difference with peer group, mean difference between replicates, CV, outliers and trend. Code Manuf System Peer Bias ( %) IDiffI Rep ( %) CV ( %) Outl %-diff repl Trend 559 Abbott C Ca Mg Alb Pro Ca Mg Alb Pro Ca Mg Alb Pro Ca Mg Alb Pro Ca Mg Alb Pro 55 Abbott CI 3. yes 553 Abbott CI yes Abbott CI yes yes Abbott I.7 Abbott CI yes 53 Abbott CI 55 Abbott CI Ortho Vitros 3.5 yes 7 II Ortho Vitros 7 Ortho Vitros Ortho Vitros 3.7 yes Ortho Vitros 5 Ortho Vitros 5 Ortho Vitros. 57 Ortho Vitros 5 Roche Modular Roche Modular yes 5 Roche Modular Roche Modular yes Roche Modular 3.5. Roche Modular 55 Roche Modular yes 55 Siemens Advia yes yes 37 Siemens Advia yes 9 Siemens Advia 3 Siemens Advia 5.9. yes yes 9 Siemens Advia Siemens Advia Siemens Advia Siemens Advia.5 3. yes yes Siemens Advia 5 Siemens Advia Siemens Advia yes 55 Roche Cobas Roche Cobas. 7 Roche Cobas 3 Roche Cobas yes Roche Cobas. Roche Cobas.5 7 Roche Cobas Roche Cobas 7 Roche Cobas Roche Cobas 5 Roche Cobas Roche Cobas.7 yes 3 Roche Cobas Roche Int.3 yes 79 Roche Int yes Roche Int yes yes 9 Roche Int Roche Int Sample mix-up yes yes yes 5 Horiba 75 Roche Hit 9..3 yes yes yes Siemens RxL...7 yes yes yes 3 Siemens Vista.5. yes 7

28 NKK og NOKLUS, oktober Fig. Distribution of laboratory correlation, bias and number of results outside a generic total error limit of 5 % (based on replicate without outlier) Calcium & Magnesium. Calcium 5 Magnesium Frequency 5 3 Frequency 3 5 Calcium r (versus ManuMean) 5 Magnesium Frequency Frequency r (versus ManuMean) Bias (ManuMean, %) Bias (ManuMean, %) 5 Calcium Magnesium Frequency (TE >5%) Number of samples Frequency (TE >5%) Number of samples

29 NKK og NOKLUS, oktober Fig 9. Distribution of laboratory correlation, bias and number of results outside a generic total error limit of 5 % (based on replicate without outlier) Albumin & Protein. Albumin 5 t-protein Frequency 5 3 Frequency 3 5 Albumin 5 5 Albumin 5 t-protein Frequency (TE >5%) r (versus ManuMean) r (versus ManuMean) 5 t-protein Frequency Frequency Bias (ManuMean, %) Bias (ManuMean, %) Frequency (TE >5%) Number of samples Number of samples 9

30 Lab DiffReps (%) Lab 5 DiffReps (%) Lab DiffReps (%) Lab 3 DiffReps (%) NKK og NOKLUS, oktober Fig.. Examples of laboratories where trends have been observed. Calcium Magnesium R² = R² = Sample # Sample # Total protein Total protein R² = R² = Sample # Sample # 3

31 Lab 7 Residuals (%) Lab 55 Residuals (%) Lab 7 Residuals (%) Lab 55 Residuals (%) NKK og NOKLUS, oktober Fig.. Examples of laboratories where outliers have been observed. Calcium ManuMean (mmol/l) Magnesium ManuMean (mmol/l) Albumin ManuMean (g/l) Total protein ManuMean (g/l) 3

32 NKK og NOKLUS, oktober Fig.. Overview of the distribution of laboratory averages. 3

33 NKK og NOKLUS, oktober COMMUTABILITY Fig 3. Commutability - Calcium and magnesium (EQA samples in pink) Abbott Difference (mmol/l) Abbott Difference (mmol/l) Calcium ManuMean (mmol/l) Magnesium ManuMean (mmol/l) Ortho (mmol/l) Ortho Difference (mmol/l) Calcium ManuMean (mmol/l) Magnesium ManuMean (mmol/l) Cobas Difference (mmol/l) Cobas Difference (mmol/l) Calcium ManuMean (mmol/l) Magnesium ManuMean (mmol/l) Modular Difference (mmol/l) Modular Difference (mmol/l) Calcium ManuMean (mmol/l) Magnesium ManuMean (mmol/l) Advia Difference (mmol/l) Advia Difference (mmol/l) Calcium ManuMean (mmol/l) 33 Magnesium ManuMean (mmol/l)

34 NKK og NOKLUS, oktober Fig. Commutability - Albumin and protein (EQA samples in pink; sample for protein corrected for Abbott, Modular and Advia). Abbott Difference (g/l) Abbott Difference (g/l) Albumin ManuMean (g/l) t-protein ManuMean (g/l). 3. Ortho Difference (g/l) Ortho Difference (g/l) Albumin ManuMean (g/l) t-protein ManuMean (g/l) Cobas Difference (g/l) Cobas Difference (g/l) Albumin ManuMean (g/l) t-protein ManuMean (g/l) Modular Difference (g/l) Modular Difference (g/l) Albumin ManuMean (g/l) t-protein ManuMean (g/l).. Advia Difference (g/l) Advia Difference (g/l) Albumin ManuMean (g/l) t-protein ManuMean (g/l) 3