February 2015

gageYou have just completed a Gage R&R study on one of your critical to quality measurement systems.  You want to find out how “good” that test method is.  You have done everything right.   You carefully selected the parts to reflect the range of production.  You carefully selected the operators to do the testing and randomized the run order for the parts.  You ensured that the operators didn’t know what part they were testing.  Each operator tested each part the required number of times.  Now, you are ready to analyze the results.  What method do you use?

You can analyze the Gage R&R study using one of the following analysis techniques:

  • Average and Range Method
  • ANOVA
  • EMP (Evaluating the Measurement Process)

All three techniques have been covered in detail in past publications.  This publication compares the output from the three techniques and attempts to decide which is best.  We will assume that we want to use the test method for process control.

In this issue:

 You may download a pdf copy of this publication here.

The Data

The data we will use is from the 4th edition of the Measurement Systems Analysis manual published by AIAG.   In this Gage R&R study, there are three operators and ten parts.  Each operator runs each part three times.   The data are shown in Table 1.

For example, operator A ran part 1 three times with the following results: 0.29, 0.41, and 0.64.  The data from this table are analyzed using each of the three Gage R&R analysis techniques using the SPC for Excel software.   Before we start, we will quickly review the sources of variation in a Gage R&R study.  There are four sources we primarily follow: repeatability, reproducibility, part, and total.  

Table 1: The Data

Op.Trial/Part12345678910
A10.29-0.561.340.47-0.80.020.59-0.312.26-1.36
A20.41-0.681.170.5-0.92-0.110.75-0.21.99-1.25
A30.64-0.581.270.64-0.84-0.210.66-0.172.01-1.31
B10.08-0.471.190.01-0.56-0.20.47-0.631.8-1.68
B20.25-1.220.941.03-1.20.220.550.082.12-1.62
B30.07-0.681.340.2-1.280.060.83-0.342.19-1.5
C10.04-1.380.880.14-1.46-0.290.02-0.461.77-1.49
C2-0.11-1.131.090.2-1.07-0.670.01-0.561.45-1.77
C3-0.15-0.960.670.11-1.45-0.490.21-0.491.87-2.16

 

The Sources of Variation in a Gage R&R Study

beakerThe two major sources of variability that we are interested in a Gage R&R study are the repeatability and reproducibility.

  1. Repeatability is the variation in the measurements obtained by one operator measuring the same item repeatedly.  This is also called measurement or equipment variation.
  2. Reproducibility is the variation of the measurement system caused by differences in the way operators perform the test.  It is the variation in the average values obtained by several operators while measuring the same item and is sometimes called the appraiser variation.

The combined repeatability and reproducibility make up the Gage R&R variability.  The third major source of variation is the part variation.  This variation is a measure of how much the parts vary and should be representative of what occurs in production if you are using the measurement system to control the process.  The last major source of variation is the total variation – which is a measure of the variation in all the results.

The relationship between the total, part and measurement system variation is given by the equation below

total variance equality equation

where the subscripts represent the source (t = total, p = part, and ms = measurement system).  Note that this equality is based on variances.  Remember that the variance is the square of the standard deviation (sigma).

Average and Range Gage R&R Analysis

calculatorThis methodology has been around for many years and was, for much of that time, the preferred method for analyzing a Gage R&R study - mostly for the ease of calculation.  That is no longer the case today.  One of our previous publications laid out the calculations using the Average and Range method in detail.

The average and range method forms subgroups based on each operator-part combination (e.g., one subgroup is A-1 for operator A and part 1).  The three trials from Table 1 make up that subgroup (0.29, 0.41, 0.64).  Subgroup averages and ranges are calculated.  Each operator’s average and range is then calculated.  The average range for the three operators is then found.  In this example, the average range is R = 0.342.  To find the repeatability (called EV = equipment variation by AIAG), this average range is multiplied by a constant, K1, that depends on the number of trials.  For 3 trials, K1 = 0.5908.  Thus,

EV = R (K1) = (0.342)(0.5908) =0.202

A word of caution here.  The value of EV does not represent a variance.  It represents a standard deviation.  This is the start of the problems associated with the average and range method.

The range in operator averages is then calculated.  This is called XDIFF and is 0.445 in this example. This value is used in the following equation to find the reproducibility or the appraiser variability (AV).

AV Equation

where K2 is a constant that depends on the number of operators (0.5231 for three operators), n is the number of parts (10) and r is the number of trials (3).  The value of AV for our example dada is 0.230

The Gage R&R value is then found by combining the EV and AV results using the following equation:

Gage R&R Equation

The part variation (PV) is found by determining the range in part values (Rp) and multiplying this range by a constant (K3) that depends on the number of parts.   For this example:

PV = Rp(K3) = (3.511)(0.3146) = 1.105

Finally, the total variation (TV) is then found by the following equation:

total variance calculation

Again, not that the above equation for TV is not the variance – but the variation represented by the standard deviation.  We can use these results to determine the % of variation (NOT variance) due to each source of variation.  The results are shown in Table 2. 

Table 2: Average and Range Method Gage R&R Results

Measurement Unit Analysis% Total Variation (TV)
Repeatability - Equipment Variation (EV) 
EV =0.202%EV=EV/TV = 17.61%
Reproducibility - Operator Variation (AV) 
AV = 0.230%AV =AV/TV = 20.04%
Repeatability & Reproducibility (R & R) 
R&R = 0.306% R&R =R&R/TV=26.68%
Part Variation (PV) 
PV=1.104PV =PV/TV=96.37%
Total Variation (TV) 
TV =1.146

 

percentThe key result that most people look at is the % R&R.  From Table 2, the % R&R has a value of 26.68%.  Note that the values in the second column do not add to 100%.

So, what does this value of %R&R mean?  The acceptance criteria from AIAG are given on page 78 of their measurement system analysis manual.  The criteria given there are reproduced in Table 3 below.

Table 3: AIAG Gage R&R Acceptance Criteria*

% R&RDecisionComments
Under 10Generally considered to be an acceptable measurements system.Recommended, especially useful when trying to sort or classify parts or when tightened process control is required.
10 to 30May be acceptable for some applications.Decision should be based upon for example, importance of the application measurement, cost of measurement device, cost of rework or repair. Should be approved the customer.
Over 30Considered to be unacceptableEvery effort should be made to improve the measurement system.

 

* from Measurement Systems Analysis, 4th Edition, 2010, AIAG

The manual does say that these criteria alone are not an acceptable practice for determining the acceptability of a test method.  They are just guidelines.  But, in reality, many people do just that.  So, with our value of 26.88% for the % R&R, we would conclude that the test method may or may not be acceptable – it depends on the situation. 

ANOVA Gage R&R Analysis

Analysis of variance (ANOVA) is a technique that examines what sources of variation have a significant impact on the results.  This approach actually adds another source of variation to the mix: the operator*part interaction.  This interaction is usually not significant so we will leave it out of this discussion.  What ANOVA does is compare the variation in part and operator results to the repeatability of the test method. 

The Gage R&R output for our example is shown below.  Remember, there is no operator*part interaction so it was taken out of the ANOVA table below.

Table 4: ANOVA Gage R&R without Interaction Report

SourcedfSSMSFp Value
Part988.3629.818245.6140
Operator23.1671.58439.6170
Repeatability783.1180.04  
Total8994.647  

 

gas analyzerThe first column is the source of variability.   Operator here represents the reproducibility.  The second column is the degrees of freedom associated with the source of variation. This is a measure of the amount of data present.  The third column is the sum of squares.  This is a measure of the variation in the data for that source.

The fourth column is the mean square associated with the source of variation. The mean square is the estimate of the variance for that source of variability (not necessarily by itself) based on the amount of data we have (the degrees of freedom). So, the mean square is the sum of squares divided by the degrees of freedom.  We use the mean square information to estimate the variance of each source of variation – this is the key to analyzing the Gage R&R results.

The fifth column is the F value. This is the statistic that is calculated to determine if the source of variability is statistically significant. It is based on the ratio of two variances (or mean squares in this case).  The last column is the p value – a value ≤ 0.05 is considered significant.  So, both the parts and operator have a significant effect on the results.

With ANOVA, you determine the % of the total variance (not standard deviation) due to each source.  The following equations can be used to calculate the variances when there is no operator*part interaction.  The repeatability variance is simply the mean square of the repeatability source of variation.

repeatability ANOVA

The reproducibility comes from the mean square of the operators (with n = number of trials and p = number of parts):

reproducibility ANOVA

The part variance comes from the mean square of the parts (where o = number of operators).

parts ANOVA

The results are shown in Table 5.  The calculations are covered in our September 2012 publication.

Table 5: % Contribution to Variance by Source

SourceVariance% Contr.
Total Gage R&R0.09147.76%
Repeatability0.04003.39%
Reproducibility0.05154.37%
Part-to-Part1.08692.24%
Total Variation1.178100.00%

 

From this analysis, the % Gage R&R is 7.76%.  The AIAG reference manual does include ANOVA as a way of analyzing a Gage R&R study.  In fact, using these same data, the manual now says that the test method is acceptable since the % Gage R&R is below 10.  What?  How can it be one thing with the Average and Range method and another with the ANOVA?  You probably already know the answer, but we will review it later.  Next, we look at the EMP methodology.

EMP Gage R&R Analysis

evaluate stopwatch figureOur last two publications took an in-depth look at the EMP methodology.  The EMP methodology is similar to the ANOVA method in that it determines the variances due to the different sources of variation and determines the % contribution due to each source.  Like the Average and Range method, it uses subgroups of data to determine the variance due to the various sources of variation.   It does not take into account the operator*part interaction. 

The approach, not surprisingly since it is Dr. Donald Wheeler’s approach, includes the use of control charts.  A range chart is made based on the subgroups composed of each operator-part combination.  As long as the range chart is in statistical control, the repeatability can be estimated from the average range (using Dr. Wheeler’s nomenclature):

EMP repeatability

where d2 is a control chart constant that depends on subgroup size (the number of trials).  The numerical results of the calculations are shown in the table below.

The range of operator averages is used to find the reproducibility using the following:

EMP reproducibility

where R0 is the range of the operator averages,  d2* is a bias correction factor that depends on the number of operators, n = number of trials, o = number of operators, and p = number of parts.

The combined R&R variance is the sum of the repeatability variance and the reproducibility variance:

EMP gagerr

The range of the p part averages is used to determine the product variance using the following:

EMP part variance

The total variance is the sum of the product variance and the combined Gage R&R variance:

EMP total variance

The variances for each source of variation are shown in Table 6.  The last column is the % of total variance due to each source of variation.

Table 6: Contribution to % Variance using EMP Methodology

ComponentVariance% of Total
Repeatability0.04073.1%
Reproducibility0.05314.1%
R&R0.09387.2%
Product1.21692.8%
Total1.310

 

These results are very close to those obtained from the ANOVA Gage R&R methodology.  Now, lets compare the results.

Comparison of Results

The results are compared in Table 7.  The source is given in the first column.  The average and range method results are given first.  There has been an addition to the results for the Average and Range method.  The first two columns under the Average and Range results are based on the calculations shown above - which use the standard deviation for the results.  Those standard deviations were squared to generate variances and then the % of Total Variance was calculated for the Average and Range method.

Table 7: Comparison of Results

Average and Range   ANOVA EMP 
SourceVariation% of Total VariationVariance% of Total VarianceVariance% of Total VarianceVariance% of Total Variance
Repeatability0.20217.61%0.043.11%0.043.39%0.04073.10%
Reproducibility0.2320.04%0.054.03%0.05154.37%0.05314.10%
R&R0.30626.68%0.097.13%0.09147.76%0.09387.20%
Product1.10496.37%1.2292.80%1.08692.24%1.21692.80%
Total1.146 1.31100.00%1.178100.00%1.31100.00%

 

Note that the columns for variance and % of total variance are very close for all three methods. So, the three methods, when using the variance, generate very similar results.

compare figureObviously, the Average and Range approach of using the standard deviation gives significantly different results.  This is simply because the standard deviations are not additive.

standard deviation equation

So, the % of variation column does not sum to 100.  This makes it much more difficult to interpret the results.  Why AIAG continues to include the Average and Range approach in their manual is beyond me.  At a minimum, all they have to do is to square the results to convert the results to variances.  Bottom line:  If you must use the Average/Range method, do not pay attention to the % of variation information (based on standard deviation).  Use the % of variance information.  It is essentially the same with all three methods.

But the bigger question is how to interpret the results.  The criteria given by AIAG are just guidelines to consider (see Table 3).    But if you apply them directly to the variance, most of the % of variance range is unacceptable – anything over 30% is not acceptable. 

Dr. Wheeler’s EMP approach uses a completely different method.  He classifies the test method as a First, Second, Third or Fourth Class monitor based on the intraclass correlation coefficient (), which is the ratio of the part variance to the total variance:

intraclass coefficient equation

The subscripts are as follows: x = total variance, p = part variance, e = measurement system variance.  So the intraclass correlation coefficient is also equal to  1 minus the % of variance due to the measurement system (the % R&R). 

Table 8 shows how Dr. Wheeler suggests the results be interpreted.  The last column in the table was added to show the %R&R value and AIAG guidelines for acceptability.

Table 8: Interpreting the EMP Results

Type of MonitorReduction of Process SignalChance of Detecting ± 3 Std. Error ShiftAbility to Track Process Improvements% R&R/AIAG Guideline
0.8 to 1.0First ClassLess than 10%More than 99% with Rule 1Up to Cp800 to 20%/Acceptable to Marginal
0.5 to 0.8Second ClassFrom 10% to 30%More than 88% with Rule 1Up to Cp5020 to 50%/Marginal to Unacceptable
0.2 to 0.5Third ClassFrom 30% to 55%More than 91% with Rules 1, 2, 3 and 4Up to Cp2050% to 80%/Unacceptable
0.0 to 0.2Fourth ClassMore than 55%Rapidly VanishingUnable to Track80%  to 100%/Unacceptable

 

Note: table adapted from EMP III Evaluating the Measurement System, by Donald J. Wheeler, 2006 SPC Press.  

This table was described in detail in our previous publication.  Please refer to that publication for more information.  The first column lists the value of the Intraclass Correlation Coefficient.  The second column lists whether it is a First Class, Second Class, Third Class or Fourth Class monitor – with “First” being the best.  The rest of the table (except the last column I added) gives information about how much a reduction in process signal there is, the chance of detecting a major shift, and the ability to track process improvements.

You can see from the table, a First Class Monitor has a % R&R range of 0 to 20%.  Under the AIAG guideline, a test method is acceptable if the % R&R is 10% of less.  It is marginal in the range of 10 to 30%.  The % R&R for our example data is about 7%.  So, it is First Class Monitor and acceptable under AIAG guidelines.  This means that there is less than a 10% reduction in a process signal, there is a better than 99% chance of detecting a point beyond the control limits (Rule 1) and that the measurement system will be able to track process improvements up to Cp80.  Cp80 is calculated based on specifications and marks the point from the measurement system will move from a first class to a second class monitor.  Rules 2 to 4 are the zone tests.   Wow, a lot more information than the guidelines in Table 3.

But, if the result was 15% R&R, it would be marginal under AIAG guidelines but still be a First Class Monitor.  It appears to me that the AIAG guidelines are unduly restrictive.  A Third Class monitor would be unacceptable under the AIAG guidelines but from the table above still can be used to track a process.

So, what should you do to analyze a Gage R&R study?  Use the ANOVA or EMP method to analyze the Gage R&R study.  They will give similar results for % of variance.  The EMP method does have some control charting built into it which gives it the edge to me (see our last month’s publication).  Then interpret the results using Dr. Wheeler’s approach in Table 8.  Rate the test method as a First, Second, Third or Fourth Class monitor and then use the information in the table to understand what that means.

Summary

In our early publications, I said that a precise measurement system was one where the measurement system was in statistical control and the % of variance due to the measurement system was less than 10% of the total variance.   This would make it a Class One Monitor under Dr. Wheeler’s system.  So, was I wrong?  Well, I too was probably too restrictive.  I do believe that critical test methods should be monitored on an on-going basis by running a control and analyzing the results using an individuals control chart.  The objective should also be continuous reduction in the measurement system variability. 

But for Gage R&R studies, use the ANOVA or EMP methodology and interpret the results as a First, Second, Third or Fourth class monitor.  Forget about the Average and Range method results based on the standard deviation and their guidelines for what makes an acceptable measurement system.

Quick Links

SPC for Excel Software

Visit our home page

SPC Training

SPC Consulting

Ordering Information

Thanks so much for reading our publication. We hope you find it informative and useful. Happy charting and may the data always support your position.

Sincerely,

Dr. Bill McNeese
BPI Consulting, LLC

View Bill McNeese's profile on LinkedIn

Connect with Us

       

Comments (10)

  • anon

    From reading Wheeler, my understanding is that the fault here lies with the AIAG (using % of total standard deviation) rather than anything about the Average & Range method of interpretation.Running your Gauge R&R example data in Minitab 17, I found that its two available methods (ANOVA and Average & Range) give practically indentical results, where for both the reported % of variance is useful and the reported % of standard deviation isn't (and adds up to well over 100% for both methods).  

    Jul 02, 2015
  • anon

    My question is about how to handle a measurement that is a vector.   In my particular case, I am looking at an Unbalance reading of a rotor that has a magnitude and an angle.  I am only looking at Static Balance.  If I visualize the total tolerance zone of a certain magnitude, the tolerance zone would be a 360  degree tolerance zone of radius of what the maximum permissible deviation from the perfect part would be.   Now, if I wanted to find the  If I visualize the tolerance zone, I would see that it is basically a circle.  I would say that the diameter of the circle would be basically the diameter of the circle as described by the range of say 30 measurements of the same part.   Then, I would say that the P/Tol would be the area or the Repeatability/ Area of the total Tolerance Zone.    Do you agree with my assessment.   Is there a program developed for such a meaurement?

    Sep 25, 2015
  • anon

    Hi Terrance,

    I have not done anything with meaurements that are a vector and how to hanlde process capaiblity with that. I am sure that there are programs to handle it.  Try searching for multiple process capability vectors.

    Sep 26, 2015
  • anon

    Our manufacturing partner in Asia uses an Excel template for the Average and Range Method.  However, their template does not compute the part variation, nor total variation.  The % variation is done be dividing EV, AV, and R&R by the Tolerance (USL-LSL).  Does anyone endorse this particular statistic?

    Oct 21, 2015
  • anon

    It depends on what you are using the test for.  If you are using it for control charting and process capability, you want to use the total variation because you want the test to be able to tell the difference between samples from the process.  However, if you are just using the test to see if the product is in or out of spec, you use the tolerance range.  Both are widely used depending on the purpose of the test method.

    Oct 22, 2015
  • anon

    Dear Bill,Kindly share links or example where GRR has been done on two similar machines for destructive sampling. RegardsAshok

    Aug 24, 2017
  • anon

    Hello Ashok,

    I don't have any links on that.  Maybe some readers do.  

    Aug 25, 2017
  • anon

    Does anyone have the table of the d2* constants ?

    Feb 21, 2018
  • anon

    These are available in a number of books (like Dr. Wheeler's).   You call also search online for d2* constants to find some results.

    Feb 21, 2018
  • anon

    This is a very thorough article, easy to read and understand. I've been manipulating GRR template files without understanding the basics of it and found it confusing recently when I got different results using JMP software.This article helped me greatly understanding. Thank you so much!

    Sep 05, 2018

Leave a comment

Filtered HTML

  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <h1> <h2> <h3> <h4> <h5> <h6> <img> <hr> <div> <span> <strike> <b> <i> <u> <table> <tbody> <tr> <td> <th>
  • Lines and paragraphs break automatically.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
Image CAPTCHA
Enter the characters shown in the image.