DoE Basics - COJUG JEFF HESLOP MAY 3, 2017 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 1 Agenda Introduction to Experimentation One Variable at a Time Method Parts of an Experiment Planning for an Experiment Factorial Strategy Optimal Designs 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 2 Introduction to Experimentation

02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 3 Why does it take so long to learn? Two things must come together at the same time. An interested observed An exceptional event This joint occurrence is a rare event. Hence learning can be a very slow process. the cold winter temperatures [in Champagne, northeast France] prematurely halted fermentation in the cellars, leaving dormant yeast cells that would awaken in the warmth of spring and start fermenting again. One of the byproducts of fermentation is the release of carbon dioxide gas, which, if the wine is bottled, is trapped inside the wine, causing intense pressure. The pressure inside the weak, early French wine bottles often caused the bottles to explode, creating havoc in the cellars. If the bottle survived, the wine was found to contain bubbles, something that the early Champenois were horrified to see, considering it a fault. As late as the 17th century, Champenois wine makerswere still trying to rid their wines of the bubbles. the British were developing a taste for the unique bubbly wine. The sparkling version of Champagne continued to grow in popularity, especially among the wealthy and royal.

02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 4 Why does it take so long to learn? Experimenting brings the exceptional event and the interested observer together. Experimenting accelerates the rate of learning. 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 5 Speed of Learning Learning The greater the RPM of the experimental wheel, the faster you climb the knowledge ramp. Knowledge is competitive advantage.

Act Plan Study Do Time t0 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 6 Observational vs Experimental Studies OBSERVATIONAL STUDIES EXPERIMENTAL STUDIES Passive

Active Collect observations as the process is run normally Deliberately interfere with the process to watch the effects Process behavior charts will indicate exceptional events Forces the occurrence of an exceptional event in the presence of an interested observer May take a long time to bring and exceptional event together with an interested observer. 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 7 Iterative or sequential nature of learning

Theory, belief, or question plane Observation or data plane Begin with a theory. Collect data you expect to be consistent with your theory. Study the results. Keep or modify your theory. Repeat. Designed experiments accelerate this learning process. 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 8 History of Designed Experiments Agricultural era Sir Ronald Fisher. 1920s to early 1930s Randomization, replication, blocking Factorial design and analysis of variance Industrial era

George Box 1950s to 1980s Sequentiality, iteration Accelerated by the introduction of response surface methodology (RSM) Constrained by lack of statistical training, computer resources, and software 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 9 History of Designed Experiments Quality improvement era Genichi Taguchi, began late 1970s Robust parameter design. Making products and process insensitive to variation Peer reviews found Taguchis engineering methods sound but not his experimental strategy and statistical analysis methods Renewed interest era 1980s to today

Alternatives to Taguchi methods Great improvements in computer software Optimal designs Definitive Screening Designs Formal education in statistical methods, under grad and grad. 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 10 The aim of experimenters However, it is important that the experimenters do not lose sight of the fact that the ultimate aim is prediction of performance in the future. 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 11 One Variable At a Time (OVAT)

Method 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 12 One Variable At a Time (OVAT) 1 2 3 4 A + - B + - C Y - 10 - 20

- 5 + 15 Change one variable and hold all the others constant. Why do this? Calculate the effect for each variable. What percentage of the total number of Y observations did you use to calculate each effect? What is the effect of A if B is +? Effect? 02/07/2020 What percentage of the design space did you investigate? COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 13 Answers to OVAT efficiency questions What percentage of the total number of Y observations did you use to calculate each effect?

What is the effect of A if B is +? 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 14 Answers to OVAT efficiency questions What percentage of the design space did you investigate? -++ +++ -+B --02/07/2020 ++--+ +-+ A COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC

-+- C 15 Parts of an Experiment 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 16 One Factor Experiment Example Say we need to improve the yield of a process. We have always operated at 100 0F. We want to know if the yield will improve if we increase the temperature to 200 0F. Since we have never operated at 200 0F, how would you proceed? 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC

17 One Factor Experiment Example The response of an experiment is the measured outcome. What is our response in this case? ___________________ A factor is the name of the variable we intend to change. What is our factor in this case? _________________ We know our present setting is 100 0F. We have decided to change this to 200 0F. These settings of our factor are called levels. 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 18 One Factor Experiment Example The base experiment is one test at 100 0F and one at 200 0F. But because we planned to detect a change in yield of at least 5, we had to repeat each test case 4 times. This is called replication. Why do we replicate? Temp 100 200

Effect on yield Y1 65 92 Y2 59 97 Y3 68 94 Y4 Stdev 71 5.12 85 5.10 Avg std 5.11 The experimental stdev determines the height of the

fog bank. If the effect of temp does not rise above the fog, you cant see or detect it. Fog bank Noise Temp effect 02/07/2020 Before you estimate a difference you must first filter out the noise! COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 19 One Factor Experiment Example In what order should we conduct the tests? Time Series Plot of Yield 100 Yield 90 Average yield

at 200 0F Effect of operator confused with temp. 80 Average yield at 100 0F 70 60 100 100 100 100 200 200 200

200 Temp Temp Yield Oper 100 65 Sam 100 59 Sam 100 68 Sam 100 71 Sam 200 92 Sally 200 97 Sally 200 94 Sally 200 85 Sally Run all of the 100 100 0F tests then all of the 200 0F tests. After all, this is the easiest way to run all of the tests. Assume that the lurking variable, operator, makes the real difference. Without knowing that Sam

ran all of the 100 deg tests and Sally the 200 deg tests, we could wrongly conclude that the change was do to temperature when it was really an operator difference. 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 20 One Factor Experiment Example In what order should we conduct the tests? Randomize the run order to protect against lurking variables. Time Series Plot of Yield 100 Effect of operator confused with temp. Yield 90 80.25 80

77.5 70 60 100 200 200 100 100 200 100 200 Temp Temp 100

200 200 100 100 200 100 200 Yield Oper 65 Sam 92 Sam 94 Sam 59 Sam 71 Sally 85 Sally 68 Sally 97 Sally Randomize the order of the tests.

With randomization the effect of the lurking variable, operator, is now very small. 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 21 One Factor Experiment Example What is the effect of temperature on the yield? An effect in the average response of the high level of a factor minus the average response of the low level. Effect of temp on Yield 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC In this case 92.00 65.75 = 26.25. So, on average, we get a 26.25 point improvement in yield when we increase the temp from 100 to 200 0F.

22 One Factor Experiment Example A designed experiment has the following general sequence: Planning Data collection Analysis Conclusions 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 23 One Factor Experiment Example With more than one factor you will have to learn about: Factorial strategies Full and fractional factorials

Design resolution Confounding Modern design strategies Optimized designs Correlation matrix Model building 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 24 Planning for an Experiment 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 25 Experimentation steps 1.

Identify the objective of the experiment. 2. Define the response. 3. Define the factors and their levels. 4. Define the type of experimental design. 5. Identify possible lurking variables and how to protect against them. 6. Create the experimental design. 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC

26 Experimentation steps 7. Plan the data collection logistics. 8. Collect the data by conducting the experimental plan. 9. Analyze the data. 10. Perform verification runs. 11. Draw conclusions. 12. Leverage results. 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 27

Defining levels Remember that your are trying to create an exceptional event. So set your levels far enough apart so that you can detect the effect despite the noise in the experiment. Factors can be both quantitative and qualitative. So the levels of a factor will be consistent with the type of factor. 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 28 Experimental Validity Internal validity Did you protect against lurking variables with a method like randomization? External validity Do the results of the new process agree with the predictions of the experiment? 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 29

Sound advice All experiments are planned. The only question is were they planned well or not. What makes an experiment fail is poor planning, poor execution, or both. Planning is critical to success. Always validate your experimental results with verification runs. Verification is the confirmation that a product meets identified specifications. Validation is confirmation that a product appropriately meets its design function or the intended use. 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 30 Factorial strategy 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 31 Factorial designs When experimenting with more that one factor you must make sure that your factors are not

changing together. If they do, you wont know which one caused the effect on the response. Full factorial design for 2 factors at 2 levels each. A B -1 -1 1 -1 -1 1 1 02/07/2020 1

The levels are coded. -1 is called the low level while +1 is called the high level. For example for temperatures of 100 and 200, 100 would be coded -1 and 200 coded 1. What are the patterns in columns A and B? COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 32 Standard order A -1 1 -1 1 -1 1 -1 1 02/07/2020 B -1

-1 1 1 -1 -1 1 1 C -1 -1 -1 -1 1 1 1 1 Here is the standard order for three factors each at 2 levels. What patterns do you see in all three columns? Note that when adding a factor such as C, you repeat the 2 factor design below and add factor C. These are sometimes called geometric designs. The number of runs for factors with 2 levels is:

where k = no. of factors What would be the design if we added a forth factor? COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 33 Calculating main effects Run A B Y 1 -1 -1 70 Factor Name

Low High A Temp 100 200 B Flux A B 2 1

-1 65 In the first run we ran temp at 100 and flux at A. Our yield was 70%. This is repeated for all 4 runs. 3 -1 1 60 Recall that the main effect is the average response at the high minus the average at the low. 4 1 1 90

EffectA = - = 12.5 On average, when we change temp from 100 to 200 the yield increases 7.5. What is the effect of B? 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 34 What is an interaction? When the effect of one factor depends on the level of a second factor. Interaction Plot for Y Data Means A -1 1 90 What is the effect of B when A is +1? 65 - 70 = -5

85 Mean 80 What is the effect of B when A is -1? 75 70 90 60 = 30 65 60 -1 1 B 02/07/2020 The effect of B on the response depends on the level of A. So we have an AB interaction.

COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 35 Calculating interaction effects Run A B AB Y 1 -1 -1 1 70

2 1 -1 -1 65 3 -1 1 ? 60 4 1

1 ? 90 Factor Name Low High A Temp 100 200 B Flux

A B To calculate the interaction of A and B, named AB, first multiply across the rows of the coded factors for that interaction. Then calculate the effect as before. EffectAB = = 80 62.5 = 17.5 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 36 Advantages of factorial designs Run A B Y

1 -1 -1 70 2 1 -1 65 3 -1 1 60

4 1 1 90 Note that the number of s and +s are equal in each column. We call this a balanced design. It is one of the reasons that these designs are mutually orthogonal. When we calculate effects, no effect is confounded with any other effect. So we get clean estimates of each effect independent from all other effects. This allows a powerful yet simple analysis. Recall that in the OVAT design, we only used 50% of the data to estimate each effect. What percentage of the data was used here? And unlike the OVAT design, we can estimate the interaction. Note also that the calculated effect is robust. That is, it is the average effect regardless of the changes in the other factors. 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC

37 Design and analysis of a factorial experiment Cause-and-Effect Diagram Power A process engineer is working to improve etch rate for a single wafer plasma etching tool. Electrodes to cathode gap watts supplier Etch Rate

The team agreed to study electrode gap, gas flow, and power to the cathode. temp flow Gas 02/07/2020 Factor A B C COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC Factor Levels Name Gap, cm 0.8 Flow, SCCM 125 Power, W 275

+ 1.2 200 325 38 Create design in JMP 1. Select Full Factorial Design. 2. Complete entries as shown. 3. Click Continue 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 39 Create design in JMP 4. For this exercise choose Sort Right to Left for run order and

choose 1 replicate. 5. Then click Make Table. 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 40 Create design in JMP Here is the design in coded units. 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 41 Collect data Note one replication for each combination. Here is the design with the collected data. Open file: Etch Rate Full Factorial w

Responses.jmp 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 42 Analyze data 1. Go to Analyze and Fit Model 2. All the variables will load automatically. You have the 3 main effects and the 3 2-way interactions in the model. This is because you created the table from the JMP DOE platform. 3. Click Run. 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 43 Analyze data

Detectable zone Fog or noise Power, Gap, and the Gap*Power interaction were detected beyond the noise. 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 44 Analyze data 94% of the variation in the experiment is explained by the model. 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 45 Visual optimization Based on the main effects

and interactions plots, where would you set the Xs to maximize the etch rate? Use the dynamic capability on the Prediction Profiler. 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 46 Fractional Factorials 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 47 Why the need for fractional factorials? Remember that the number of runs is equal to 2k where k is the number of factors. Note how quickly the number of runs

makes the experimental unreasonable considering both costs and time. Experiments to study with more than 3-4 factors often require a reduction in the total number of runs. For example, how could we study 4 factors with only 8 runs? 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 48 How to create a one half fraction factorial Write the full factorial design for k-1 factors. Run A B C

1 -1 -1 -1 2 1 -1 -1 3 -1 1 -1

4 1 1 -1 5 -1 -1 1 6 1 -1 1 7

-1 1 1 8 1 1 1 02/07/2020 D Now multiply across the k-1 rows to create the codes for the 4th factor D. So I can study 4 factors with only 8 runs. For a full factorial, 4 factors would be 16 runs. So I have a 8/16 =1/2 fraction.

COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 49 Downside of fractional factorials It sounds very good but confounding is the price you pay when using a fractional factorial. Run 1 2 3 4 5 6 7 8 02/07/2020 A -1 1 -1 1 -1

1 -1 1 B -1 -1 1 1 -1 -1 1 1 C -1 -1 -1 -1 1 1 1 1 D =ABC

-1 1 1 -1 1 -1 -1 1 AB CD Since we multiplied ABC to get D, we say D is confounded with the ABC interaction. We are not concerned about this because 3-way interactions and higher are considered negligible. But what is confounded with the AB interaction? Fill in the codes for the AB and CD interactions. 2-way interactions can often be large, even bigger than main effects. Is this a concern?

COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 50 Color Map On Correlation JMP creates the color map of correlation using the design evaluation option. This is the color map for our one half fraction factorial design. A B C D AB Note that the correlations are all either 0 or 1. AC AD BC BD CD Optimal designs can reduce the number of runs by allowing partial

correlation. ABC ABD BCD 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 51 Optimal designs 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 52 Value of optimal designs In the analysis of experiments we are interested in creating models so that we can predict a responses with minimum bias and variation. Factorials and fractional factorials are only one way of doing this. Optimal designs use mathematical algorithms for accomplishing this.

02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 53 Models are built from the design and used for prediction In the analysis of experiments we are interested in prediction. We want to affect future product or service. To do this we need to build a mathematical model. So we want our model to predict a response with minimum bias and variation. Factorials and fractional factorial designs are only one way of doing this. Optimal designs offer many more options to accomplish the same thing. Analysis Design 02/07/2020 Model Y = -7.776 + 0.0 X1_-1 + 2.772 X1_1 + 0.0 X2_-1 + 2.151 X2_1 + 0.0 X3_-1 + 2.791 X3_1 + 0.0 X4_-1 + 8.899 X4_1

COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 54 When to use optimal designs Costs and resource restriction limit the number of runs required from standard designs. This is common. The experimental region has constraints within it. 1 Cant run the process in this region. B -1 -1 1 A The experiment requires a non-standard model. 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC

55 Optimality criterion D-Optimal Design This is the most common. It minimizes the variance of the model coefficients. With minimum variance in the model coefficients, you would expect low variation in the predicted responses. Works well with mixed level designs. I-Optimal Design I-optimal designs minimize the average prediction variance over the design space. Note that these approaches require that you know the true model before creating the design. 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 56 Creating an optimal design in JMP Say we want to estimate the main effects and 2-way interactions for the helicopter experiment as we did earlier. But now we also want to estimate the quadratic effect of wing length. This is not a standard design so we will use an optimal design created from the JMP Custom Design

platform. 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 57 Creating an optimal design in JMP 1. Select DOE then Custom Design 2. Click Add Factor and select Categorical, 2 Level. This adds the factor for material. 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 58

Creating an optimal design in JMP 3. Change Add N Factors to 2. Then click Add Factor and Continuous 4. Edit names of factors as shown below. Click Continue. 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 59 Creating an optimal design in JMP 5. Click on Interactions and select 2nd. This creates all of the 2-way interactions. 02/07/2020 6. Click on Powers and select 2nd. This created the quadratic term for both WL and BL.

COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 7. Select BL*BL and then click on Remove Term. We only want to estimate the quadratic effect of WL. 60 Creating an optimal design in JMP 8. Say we can only afford 10 runs. Enter 10 for User Specified. Click Make Design. 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 61 Creating an optimal design in JMP Note the 3 levels for WL since we

want to estimate the non-linear effects. 9. Click on Design Evaluation. Then select Color Map on Correlations. 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 62 Creating an optimal design in JMP 10. Click on Power Analysis. 11. Based on our anticipated effect sizes, we have good power for Material are lacking in power for WL. We might have to Go Back and add replication.

02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 63 Creating an optimal design in JMP 12. Note the correlations among the factors. Other than the diagonal, what is the strongest correlation? Is this acceptable? 13. Click make table. 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 64 Creating an optimal design in JMP Here is the design. Note that we now need a middle level for

Wing Length. So you will need to cut the length half way between the 3.00 and 4.75 inch length or 3.875 inches. This design is randomized so you wont have the same order as in this slide. Build your helicopters, test, and collect the data. Your instructor will show you how to analyze your data. 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 65 Summary Experiments bring rare events and interested observers together for accelerating learning. Using experimental designs is often a process of sequential learning. One Variable at a Time methods do not estimate interactions, study a small fraction of the design space, and fail to produce robust estimations of effects. Planning is essential to successful experimentation. Factorial designs are optimal designs. But they dont always fit the needs of the experimenter. Optimal designs can reduce the size of an experiment by permitting small correlation among the effects.

02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 66 Book Recommendations 02/07/2020 Douglas Montgomery, 8th ed. Peter Goos and Bradley Jones Box, Hunter, and Hunter, 2nd ed. C. F. Jeff Wu and Michael Hamada COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC

67 Book Recommendations Ron Moen, Thomas Nolan, Lloyd Provost Stephen Schmidt, Robert Launsby, Mark Kiemele 02/07/2020 COPYRIGHT 2017 VALUE CREATION INSTITUTE, LLC 68