What is Attribute Agreement Analysis? When to work with it?

O'PEEP'S ATTRIBUTE AGREEMENT ANALYSIS EXPLANATION

Does the bathroom need cleaning, yes or no? To you it looks sparkling clean. (So cleaning? No!) But your spouse says “The bathroom is disgusting, Honey!” (Cleaning? Yes!).
So, how to find out whether your bathroom needs cleaning? You want to measure something that you cannot measure with a gage. You need to get to results like “yes”/”no” OR “super clean”/ “more or less clean”/ ”disgusting”. In such a case, the attribute agreement analysis will tell you how good is your measurement system.

Attribute Agreement Analysis is a Gage R&R Study for discrete data.

WORKING WITH ATTRIBUTE AGREEMENT ANALYSIS

Work with Attribute Agreement Analysis if you aim at doing a Gage R&R (Measurement System Analysis) for a test/measurement that does NOT lead to results you could gage with a measurement instrument. You need it if the results are qualitative. Examples:

Visual quality control of parts.
Inspection whether the cleaning was done well.
Check whether a document is understandable.

PREREQUISITE:

The criteria for the assessment is definable, e.g. by working with an expert or working under very good conditions. For example if high concentration and a very good visual capability is required, the inspection is done by a very reliable appraiser after the break.
To work with Attribute Agreement Analysis you must be able to set a standard.

STEP-BY-STEP APPROACH TO GET TO SAMPLES

Take parts representative of the process (usually 30 parts). These parts must contain good and bad samples.
Get an expert assessment or best case assessment of the parts and define a “standard”.
Let one appraiser rate all parts.
Repeatability: Repeat step 3 for the same appraiser (keeping the parts in random order to keep the appraiser “blind”) at least once, better twice.
Reproducibility: Let a second (and third) appraiser repeat steps 3 and 4.
If there is only one person and set of equipment doing the measurements you have to leave the reproducibility out of the study
Analysis

RUN THE ANALYSIS

What do you need to find out? Accuracy or Precision? Reproducibility or Repeatability?

CALCULATE ACCURACY

Understand how well your measurement system comes to the same results as your “standard”.

CALCULATE %R&R (PRECISION)

Expresses how precisely your different appraisers agree regarding their results and how precisely one and the same appraiser comes to the same results when measuring multiple times.

Total %R&R describes the level of agreement within/ between appraisers and a standard (if available)
Total %R&R must be 100% (this sounds very high but since there are only 30 parts here and the study conditions are different from reality every result below 100% will lead to a measurement system not worth trusting)
If Total %R&R is less than 100% investigate the results to see if this is a problem of repeatability, reproducibility or both

EXAMPLE

O’Peep wants to evaluate how well his quality control department assesses the taste of his famous O’Gin. He wants to know whether the appraisers in his Quality Control have the same exquisite sense of taste as he does. That gives him the Accuracy.

He would also like to find out about the Precision in his Quality Control.

So he takes 3 of his appraisers and makes them test 30 samples three times. To make it easier for us he has elaborated this sophisticated system to assess the Gin: “Very good”, “Okay” or “Very bad”.

He records the assessment given by the three appraisers in this table to easily compare them with the standard:

From this, he observes the following results and calculates accuracy and precision:

ACCURACY

PRECISION

Using a statistical tool to run a Gage R&R test he finds that the Reproducibility and Repeatability parameters are not met at 100%. This shows a clear lack of precision with the appraisers.

O’Appraiser 2 for instance, only assessed 17 out of 30 samples correctly across the three trials. And they all agreed only on 12 samples 100% of the time.

Want to learn more about Precision and Gage R&R? Click here

NUMBER OF CORRECT APPRAISALS VS. STANDARD

CONCLUSION

Is O’Peep happy about an accuracy of 90% when it comes to judging on the taste of his Gin? Good enough always depends on what you need to achieve.

Even though an accuracy of more than 90% seems to be quite good, the total lack of precision shows that this 90% is not trustworthy and that further measures are needed to improve the performance of the appraisers. This can be done through training and clearer guidance on how to assess the Gin.

Attribute Agreement Analysis