O'PEEP'S ATTRIBUTE AGREEMENT ANALYSIS EXPLANATION
Does the bathroom need cleaning, yes or no? To you it looks sparkling clean. (So cleaning? No!) But your spouse says “The bathroom is disgusting, Honey!” (Cleaning? Yes!) ... Assessing the bathroom cleanliness on a peaceful Sunday morning leads to a different result than assessing it on a Friday evening. So, how to find out whether your bathroom needs cleaning? You want to measure something that you cannot measure with a gage. You need to get to results like “yes”/”no” OR “super clean”/ “more or less clean”/ ”disgusting”.
Attribute Agreement Analysis is a Gage R&R Study for discrete data.
WHEN TO WORK WITH ATTRIBUTE AGREEMENT ANALYSIS?
Work with Attribute Agreement Analysis if you aim at doing a Gage R&R (Measurement System Analysis) for a test/measurement that does NOT lead to results you could gage with a measurement instrument. You need it if the results are qualitative. Examples:
PREREQUISITE:
The criteria for the assessment is definable, e.g. by working with an expert or working under very good conditions. For example if high concentration and a very good visual capability is required, the inspection is done by a very reliable appraiser after the break.
To work with Attribute Agreement Analysis you must be able to set a standard.
STEP-BY-STEP APPROACH TO GET TO SAMPLES
RUN THE ANALYSIS
What do you need to find out? Accuracy or Precision? Reproducibility or Repeatability?
CALCULATE ACCURACY
Understand how well your measurement system comes to the same results as your “standard”.
CALCULATE %R&R (PRECISION)
Expresses how precisely your different appraisers agree regarding their results and how precisely one and the same appraiser comes to the same results when measuring multiple times.
Total %R&R describes the level of agreement within/ between appraisers and a standard (if available)
Total %R&R must be 100 % (this sounds very high but since there are only 30 parts here and the study conditions are different from reality every result below 100% will lead to a measurement system not worth trusting)
If Total %R&R is less than 100% investigate the results to see if this is a problem of repeatability, reproducibility or both
EXAMPLE
O’Peep wants to evaluate how well his quality control department assesses the taste of his famous O’Gin. He wants to know whether the appraisers in his Quality Control have the same exquisite sense of taste as he does. That gives him the Accuracy.
He would also like to find out about the Precision in his Quality Control.
So he takes 3 of his appraisers and makes them test 30 samples three times. To make it easier for us he has elaborated this sophisticated system to assess the Gin: “Very good”, “Okay” or “Very bad”.
He records the assessment given by the three appraisers in this table to easily compare them with the standard:
From this, he observes the following results and calculates accuracy and precision:
NUMBER OF CORRECT APPRAISALS VS. STANDARD
ACCURACY
PRECISION
Using a statistical tool to run a Gage R&R test he finds that the Reproducibility and Repeatability parameters are not met at 100%. This shows a clear lack of precision with the appraisers.
O’Appraiser 2 for instance, only assessed 17 out of 30 samples correctly across the three trials. And they all agreed only on 12 samples 100% of the time.
Want to learn more about Precision and Gage R&R? Click here
CONCLUSION
Is O’Peep happy about an accuracy of 90% when it comes to judging on the taste of his Gin? Good enough always depends on what you need to achieve.
Even though an accuracy of more than 90% seems to be quite good, the total lack of precision shows that this 90% is not trustworthy and that further measures are needed to improve the performance of the appraisers. This can be done through training and clearer guidance on how to assess the Gin.
Your measurement system analysis is a bit more complicated and you would love to discuss it? Contact us!
Learn more about Measurement System Analysis in our eLearning or join one of our Green Belt or Black Belt Courses to become a certified Improvement Leader