HEATING, COOLING & WATER HEATING PRODUCTS DSQR Training Measurement System Analysis Fred Nunez Corporate Quality Goals At the end of this section youll be able to: describe the terms: accuracy, repeatability, reproducibility, stability and resolution of a measurement system conduct a Gage R&R study for continuous data 2 Cause & Effect Diagram for a Measurement System 3 Improving a Measurement System A measurement system consists of Measuring devices Procedures Definitions People To improve a measurement system, you need to Evaluate how well it works now (by asking how much of the variation we see in our data is due to the measurement system?). Evaluate the results and develop improvement strategies. 4 Properties of Measurement Systems Repeated measurements will disagree Means of repeated measurements will disagree Measurements made at different times, or by different operators, or on different instruments will disagree The measured value and the true value will disagree Waste due to poor quality test data Rejection of good material Acceptance of bad material Adjusting the process when not needed Failure to adjust when needed

Loss of goodwill between production and test people 5 Common Problems with Measurements Problems with the measurements themselves: 1. Bias or inaccuracy: The measurements have a different average value than a standard method. 2. Imprecision: Repeated readings on the same material vary too much in relation to current process variation. 3. Not reproducible: The measurement process is different for different operators, or measuring devices or labs. This may be either a difference in bias or precision. 4. Unstable measurement system over time: Either the bias or the precision changes over time. 5. Lack of resolution: The measurement process cannot measure to precise enough units to capture current product variation. 6 Desired Measurement Characteristic for Continuous Variables Good accuracy if difference is small Standard value 1. Accuracy The measured value has little deviation from the actual value. Accuracy is usually tested by comparing an average of repeated measurements to a known standard value for that unit. Observed value Good repeatability if variation is small * 2. Repeatability The same person taking a measurement on the same unit gets

the same result. Data from repeated measurement of same item 7 Desired Measurement Characteristic for Continuous Variables, cont. Data Collector 1 Data from Part X 3. Reproducibility Other people (or other instruments or labs) get the same average result you get when measuring the same item or characteristic. Data Collector 2 Data from Part X Good reproducibility if difference is small * * Small relative to a) product variation and b) product tolerance (the width of the product specifications) 8 Desired Measurement Characteristic for Continuous Variables, cont. Time 1 Observed value 4. Stability Measurements taken by a single person in the same way vary little over time. Time 2 Observed

value Good stability if difference is small * Small relative to a) product variation and b) product tolerance (the width of the product specifications) 9 Desired Measurement Characteristic for Continuous Variables, cont. X 5.1 X X X X 5.2 X X X X X X 5.3 X X X 5.4 X X 5.5 5. Adequate Resolution There is enough resolution in the measurement device so that the product can have many different values. Good if 5 or more distinct values are observed

10 Ways to See if the Measurement System is Adequate Accuracy Calibration and Gage Linearity Study (not covered here) Repeatability Gage R&R Study (covered next) Reproducibility Gage R&R Study (covered next) Stability Control Chart (covered in the module Patterns in data) Adequate Resolution With above tests 11 Gage R&R The Gage R&R study is a set of trials conducted to assess the repeatability and reproducibility of the measurement system. Multiple operators measure multiple units a multiple number of times. Example: 3 operators each measure 10 units 3 times each. Blindness is extremely desirable. It is better that the operator not know which of the test parts they are currently measuring. You analyze the variation in the study results to determine how much of it comes from differences in the operators, techniques, or the units themselves. 12 How a Gage R&R Study Works Select units or items for measuring that represent the full range of variation typically seen in the process. Measurement systems are often more accurate in some parts of the range than in others, so you need to test them over the full range. Have each operator measure those items repeatedly. In order to use Minitab to analyze the results, each operator must measure each unit the same number of times. It is extremely desirable to randomize the order of the units and not let the

operator know which unit is being measured. Minitab looks at the total variation in the items or units measured. 13 Adequate vs. Inadequate Measurement Systems Adequate Most of the variation is accounted for by physical or actual differences between the units. What Minitab calls part-to-part variation will be relatively large All other sources of variation will be small You can have higher confidence that actions you take in response to data are based on reality The measurement system has sufficient precision to distinguish at least four groups or categories of measurements. Inadequate Variation in how the measurements are taken is high. You cant tell if differences between units are due to the way they were measured, or are true differences You cant trust your data and therefore shouldnt react to perceived patterns, special causes, etc.they may be false signals The measurements fall into less than four categories. 14 Measurement System Indices %R&R Describes the variation of the measurement system in comparison to the part variation of the process %R & R

S measurement _ system S total %P/T or PTR Describes the variation of the measurement system in comparison to the part tolerances 56.15 * S measurement _ system %P / T Tolerances General guidelines for interpreting Gage R&R results. Unacceptable Desired 0% Acceptable 10% Borderline 20% 30% 100% 15 The Impact of GR&R on Specification Limits Three different parts: 1, 2, & 3 Each part is represented by a distribution representing known measurement variability, rather than an interval. For judging conformance to spec, the GR&R will not be a factor for parts 1 & 3. As in the previous example, the test result for sample #2 presents a distinct probability that the part may actually be out-of-specification, even though the test result is within spec.

Whenever a single test result falls directly on a spec limit, there is a 50:50 chance that the part may be in-spec or out-of-spec. This is true regardless of how good the GR&R is. For these reasons, it is always best for the process average to be at or near the target value. 16 Data for a Gage R&R Study Each operator measures each unit repeatedly. Data must be balanced for Minitabeach operator must measure each unit the same number of times. The units should represent the range of variation in the process. Operators should randomly and blindly test the units; they should not know which unit they are measuring when they record their results. 17 Plotting the Data From a Gage R&R Study Go to Minitab GRR Handout Here, two people each measured 5 units 4 times. Part tolerances are 12+/- 2. Unit Number Operator Measurement 1

1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 Joe Joe Joe Joe Sally Sally Sally Sally Joe Joe Joe Joe Sally Sally Sally Sally Joe Joe Joe Joe Sally Sally

Sally Sally 11.34 11.29 11.33 11.24 11.19 11.29 11.21 11.24 11.65 11.60 11.67 11.56 11.50 11.55 11.51 11.55 12.31 12.28 12.31 12.34 12.18 12.23 12.14 12.17 Unit Number Operator Measurement 4 4 4 4 4 4 4 4 5 5 5 5 5 5

5 5 Joe Joe Joe Joe Sally Sally Sally Sally Joe Joe Joe Joe Sally Sally Sally Sally 18 13.27 13.28 13.24 13.23 13.09 13.14 13.02 13.19 11.84 11.89 11.93 11.85 11.76 11.84 11.81 11.78 Plotting the Data From a Gage R&R Study Use the following Minitab command to plot the data. Open the worksheet: C:\SixSigma\Data\gagedemo.mtw Stat > Quality Tools > Gage Study > Gage Run Chart 19

Using Gage Run Chart Data columns from the worksheet You fill in this information 20 Output From Gage Run Chart Run chart of Measurement by Unit Number, Operator Gage name: Opacity Meter 3 Date of study: Reported by: Tolerance: Misc: 8/19/98 GRR Joe Sally Measurement 13 12 11 Unit Number 1 2 3

4 5 Very little difference is seen in repeated measurements by the same person or between people Each dot represents a single measurement by a single person. Repeated measurements by a single person are connected with a line. To analyze the chart, look at the spread of points within a series connected by lines (repeatability) and the difference between data from different operators (reproducibility). Here we see each unit (part number) has a tight spread of measurements with only a small operator effect. 21 Performing a Gage R&R Analysis Use the following Minitab command to conduct a detailed Gage R&R analysis. Stat > Quality Tools > Gage Study > Gage R&R Study (Crossed) 22 Using Gage R&R Study Use the ANOVA method since it allows for more precision and a look at operatorpart interactions Fill in 4 for Upper Spec - Lower Spec (the process specs are 12+/-2) 23 Interpreting the Gage R&R Output From Minitab Total Gage R&R plus Part-To-Part equals 100%

Gage R&R Source A2+A3 = Total Gage R&R Repeatability Reproducibility Operator Part-To-Part Total Variation A2+A3 +A5= These will match when there is no operator part interaction A1 A2 A3 A4 A5 A6 VarComp %Contribution (of VarComp) 0.00705 0.00183 0.00521 0.00521 0.55042 0.55747 1.26 0.33 0.94 0.94 98.74 100.00

Should give the width of 99.73% confidence interval = 6 x st. dev. Will not sum to 100 Source StdDev (SD) Study Var (6*SD) %Study Var % Tolerance (%SV) (SV/Toler) Total Gage R&R Repeatability Reproducibility Operator Part-To-Part Total Variation 0.083950 0.042835 0.072199 0.072199 0.741905 0.746640 0.50370 0.25701 0.43320 0.43320 4.45143 4.47984 11.24 5.74 9.67

9.67 99.37 100.00 12.59 6.43 10.83 10.83 111.29 112.00 Number of Distinct Categories = 12 %R&R Should be 4 or more for a healthy measurement system 24 %P/T Interpreting Gage R&R: Minitabs Graphical Output Conclusions Should be small relative to part-topart if measurement system is good Difference in values is part-topart variation This is an adequate measurement system for the range of units tested. B: No significant operatorpart interaction Difference in horizontal alignment is operator effect Should always be in control, meaning the

operators consistently measure the same way. If not, identify special cause. For good measurement systems, this will be out of control, indicating the parts are more variable than the measurement system (which is good) C: Measurement system can distinguish more than 4 categories D: No special causes within an operator for the same part, as evidenced by the Rchart If further measurement system refinement is desired: Parallel lines if no interaction A: Investigate the significant operator differences 25 Using the Gage R&R ANOVA Method Focus on the following: A) %P/T If > 30%, this indicates a poor measurement system to capture the actual process variation. B) Number of distinct categories If the number is < 4 it implies that the measurement variation is too large to adequately distinguish the part to part variation. C) The R chart by operator If it is stable, this tells that there are no special causes in the

measurement process that could be throwing off our calculations . 26 Determining the Tolerance Case 1 - Bilateral Specs: Upper Spec Limit (USL) and Lower Spec Limit (LSL) available Tolerance = USL minus LSL Case 2A - Unilateral Specs Target = 0; Have USL: Tolerance = USL minus Zero Case 2B - Unilateral Specs Have USL or LSL but not both: Tolerance = 2 * | Xbar Spec Limit | Case 3 No Specs, but have production data: Dont use %P/T Use only %GRR 27 Procedure for a GR&R Study 1. 2. 3. 4. 5. 6. 7. 8. Select the measuring equipment to be studied. Select three people for the study. Engineers and/or QA Technicians can be utilized to do preliminary studies to determine if gages, fixtures, and training are required to meet measurement requirements. Operators that will be doing the testing in production must be used for the final Gage R&R. IMPORTANT: Prepare a stepwise procedure to be used for the study. Illustrate the procedure with photos or sketches if necessary to make it clear how to properly perform the testing. If this testing, or the gage, is new to them, it would be advisable to conduct training, then allow them to practice using the gage and procedure so that they can develop their technique and familiarity with the method. Failure to do so may result in an

unacceptable %GR&R. Select 10 parts to be measured. Number the parts 1 to 10. (See NOTE) Allow each operator to measure each part in random order. Repeat step 5 a second time. Note: the operator should not have access to previous results. Repeat step 5 a third time. Note: the operator should not have access to previous results. Analyze the data using Minitab or similar software. 28 Procedure for a GR&R Study (Continued) Note 1: All 10 parts need not have the same part number if the target values are the same, or very similar in value. For example, we currently produce four different part numbers for skirt blank, but all four have the same 9.15 height dimension. If you cannot assemble the required 10 parts of the same part number, you could substitute different skirt blank parts to meet the requirement. Note 2: When selecting parts for a GR&R where multiple part numbers are available, e.g. 14, 16, 18 and 20 parts, where each has a different target dimension, select the part number with the dimension that will be the most difficult to measure accurately and precisely. For example, for the Skirt Blank, each part number has a different length. Select the part number with the longest length because this will generally be the most difficult one to measure. Note 3: When possible, select the parts that represent the actual variation of the process. Ideally, you would like the standard deviation of these 10 parts to be the same as the standard deviation of the process. Failure to select parts that represent the actual process variation will result in Gage R&R % of Study Variation values that may be of little use, with only the GR&R % of Tolerance value being useful. 29 Gage R&R Study - ANOVA Method Two-Way ANOVA Table With Interaction Source Sample Operator Sample * Operator Repeatability Total DF 9 2 18 60 89

SS 0.0006014 0.0000347 0.0000284 0.0001800 0.0008445 MS 0.0000668 0.0000173 0.0000016 0.0000030 F 42.3174 10.9844 0.5263 P 0.000 0.001 0.934 Two-Way ANOVA Table Without Interaction Source Sample Operator Repeatability Total DF 9 2 78 89 SS 0.0006014 0.0000347 0.0002084 0.0008445 Source Total Gage R&R Repeatability Reproducibility

Operator Part-To-Part Total Variation VarComp 0.0000032 0.0000027 0.0000005 0.0000005 0.0000071 0.0000103 MS 0.0000668 0.0000173 0.0000027 F 25.0066 6.4910 P 0.000 0.002 Values <0.05 indicate statistically significant differences. Here you can see a bias between operators, but no significant sample * operator interaction. This table will have slightly different values than the one above due to the way ANOVA handles the interaction. Gage R&R Source Total Gage R&R Repeatability Reproducibility Operator Part-To-Part

Total Variation 30 %Contribution (of VarComp) 30.72 25.97 4.75 4.75 69.28 100.00 Study Var StdDev (SD) (6 * SD) 0.0017780 0.0106678 0.0016346 0.0098079 0.0006993 0.0041961 0.0006993 0.0041961 0.0026697 0.0160184 0.0032076 0.0192456 This table shows the relative contribution of each component of variation. %Study Var (%SV) 55.43 50.96 21.80 21.80 83.23 100.00 %Tolerance (SV/Toler) 26.67 24.52 10.49 10.49 40.05 48.11 This value (26.67% of Tolerance) is the GR&R

for this study. This is the value to report. This value (55.43% of Study Variation) is high only because the 10 parts represented only a small portion of allowable tolerances. EXAMPLE 1 AS39583DSkirt BlankLength Gage name: Date of study: Reported by: John Smith Tolerance: AS39583D 56.612 +0.030/-0.015" Misc: Operators: A. Anderson, B. Black, C. Cooper #5432 Mitutoyo 60" Calipers 6/10/2004 1 4 Componentsof Variation 80 % Contribution Inchesby Sample 56.640 % Tolerance 40 0 56.630 Gage R&R 2 Sample Range

1 Reprod 1 Part-to-Part RChartby Operator 2 3 UCL=0.006950 0.0000 LCL=0 3 3 4 5 Sample 6 7 8 9 Inchesby Operator 56.640 56.630 1 XbarChartbyOperator 2 6

3 2 Operator 3 Operator* SampleInteraction Operator UCL=56.63505 _ _ X=56.63229 LCL=56.62953 56.628 2 56.635 0.0025 56.632 10 5 _ R=0.0027 1 Sample Mean Repeat 0.0050 56.636 56.635

Average Percent % StudyVar 56.636 1 2 3 56.632 56.628 1 10 2 3 4 5 6 Sample 7 8 9 31 EXAMPLE 1 Notes: 1 There are three components of variation - Repeatability, Reproducibility, & Part-To-Part. Each are shown as: % Contribution (orange), % of Study (green) & % of Tolerance (blue). Note that the contribution for repeatability is higher than that for reproducibility. Note that the % of Study is larger than % of Tolerance. This is because the parts selected for the study were not as variable as allowed by the part specifications. 2 Operator 3 had higher variability than operators 1 and 2. The Range chart has to be in-control for valid GR&R estimates. 3 Several out-of-control points on the Xbar chart indicates that the measurement process can distinguish between different parts. The goal is to have many points beyond limits. Had there been more variability in the ten parts, more would have plotted beyond the control limits. 4 This chart shows the variability in the sample averages and the individual values. 5 This chart shows the variability in the operator averages and individual values. It appears that Operator 3 is biased high. To verify, consult the ANOVA table. 6 This chart shows the Operator-Part interaction. You will have to look at the ANOVA table to determine if an operator is measuring a particular part differently than the others.

Minitab will also show the Number of Distinct Categories. This is the number of distinct categories of parts that the measurement process is currently able to distinguish. The lower the %GR&R, the higher this number will be. Ideally you should have 5 or more distinct categories. This example had only 2 distinct categories, but only because the variability of the 10 parts was small when compared to the allowable specifications. If the GR&R % of Tolerance is acceptable and GR&R % of Study Variation is too high, it means that your parts are too uniform to use the % of Study Variation as a reliable measure of gage capability. 32 Example #2 (%P/T = 12.3%) GR&R for Tank Height Gage Gage name: Date of study: Reported by: Tolerance: Misc: Tank Height Gage 8/5/2004 SM +/- 0.250" Components of Variation Measurement by Part Number 100 % Contribution Percent % Study Var % Tolerance 50 46.05 46.00 45.95

0 Gage R&R Repeat Reprod 1 Part-to-Part 2 R Chart by Operator Sample Range Don Martha Sean 0.02 3 UCL=0.02135 9 10 45.95 Don Martha Operator Xbar Chart by Operator Martha Sean Operator * Part Number I nteraction

Sean 46.05 Operator 46.05 _ _ UCL=46.0018 X=45.9895 LCL=45.9772 Average Sample Mean 8 46.05 LCL=0 Don 33 7 Measurement by Operator _ R=0.00653 0.00 45.95 5 6 Part Number 46.00 0.01

46.00 4 Don Martha Sean 46.00 45.95 1 2 3 4 5 6 7 Part Number 8 9 10 Example #3 (One Distinct Category, %P/T = 114%) GR & R fo r Ho le PN 5 7 6 8 9 G a ge n a m e : D a te o f stu dy : 6" C a lipe rs 9/ 3 / 200 4 Co m p o n e n ts o f Va ria tio n D ia m e te r b y Pa rt % Co n t r ib u t io n % S t u dy Va r Percent 100 0 .4 2 5

% T o le r a n ce 0 .4 2 0 50 0 0 .4 1 5 G a g e R& R Re p e a t Re p r o d P a r t- to - P a r t 1 2 3 Sample Range 2 U CL=0 .0 0 4 6 8 3 0 .0 0 2 _ R=0 .0 0 1 4 3 3 0 .0 0 0 LCL= 0 U CL=0 .4 2 6 1 1 _ _ X=0 .4 2 3 4 2 LCL= 0 .4 2 0 7 2 0 .4 2 0

0 .4 1 5 Two-Way ANOVA Table With Interaction SS 0.0000602 0.0000390 0.0000683 0.0000560 0.0002236 MS 0.0000067 0.0000195 0.0000038 0.0000019 9 10 1 2 Operator 3 O p e ra to r * Pa rt I n te ra ctio n O p e ra t o r 0 .4 2 4 1 2 3 0 .4 2 0 0 .4 1 6 0 .4 1 6 DF 9 2 18 30

59 8 0 .4 2 0 Average Sample Mean 3 0 .4 2 4 Source Part Operator Part * Operator Repeatability Total 7 0 .4 2 5 Xb a r Ch a rt b y O p e ra to r 2 6 D ia m e te r b y O p e ra to r 3 0 .0 0 4 1 5 P art R Ch a rt b y O p e ra to r 1 4

1 2 3 4 5 6 P art 7 8 9 10 Handling Interaction F 1.76363 5.13831 2.03348 P 0.146 0.017 0.041 If the ANOVA table shows a statistically significant interaction, it may or may not be real. If you cannot get it to repeat, conclude that it is probably not a true interaction. 34