Preparing Students to Incorporate Stakeholder Requirements in ...

Preparing Students to Incorporate Stakeholder Requirements in ...

Preparing Students to Incorporate Stakeholder Requirements in Aerospace Vehicle Design Alexandra E. Coso Dissertation Defense Presentation Friday, March 28th, 2014 Advisor: Dr. Amy Pritchett Committee: Dr. Brian German, Dr. Wendy Newstetter, Dr. Jerry Seitzman, Dr. Robin Adams, and Dr. Alan Jacobsen Image references at end of presentation 2 Stakeholders in Aircraft Design Pilots Ground Personnel Non-Users Manufacturers & Maintainers Design Firms Flight Attendants & Passengers Airlines Stakeholders: All those affected by the system. They may be users or non-users. They need not be clients or decision-makers. Stakeholders may be major or minor, and the

ways in which they interact with a large-scale system are myriad. - Gibson, Scherer, Gibson 2007, p.315 Coso - Ph.D. Thesis Defense - March 28th, 2014 3 Stakeholders in Complex System Design Stakeholders are critical to the successful design of products in many fields 1,2 International Civil Aviation Organization (ICAO) has called for more integrated consideration of stakeholders, through social sustainability efforts3 Human-related design criteria are part of certification requirements established by regulators AE industry members have explored a variety of tools and resources for better incorporating stakeholders Additional efforts are needed to examine how to better incorporate stakeholder requirements consistently within the design process4-6 Maguire, 2001; 2Green & Jordan, 1999; 3Hupe, 2011; 4Chua & Feigh, 2011; 5Pew, 2008; 6Landsburg et al., 2008 1 4 Engineering Design Education

Accreditation Board for Engineering and Technology (ABET) defines program outcomes that include need to graduate engineers who understand impact of design solutions on environment, economy, and society1 Recently, ASEE and NSF report based on a survey of industry representatives, highlights a need to improve how we prepare students to synthesize engineering, business, and societal perspectives in the design process2 Studies of design and engineering students pathways support the need to determine how to better prepare students to incorporate stakeholder considerations into design3-5 Movement within some fields of engineering to design learning environments which place a higher value on the needs and limitations of stakeholders6-7 1 ABET, 2011; 2ASEE, 2013; 3Hinkle, 2013; 4Dym et al., 2001; 5Ro et al., 2012; 6Jordan & Lande 2012; 7Zoltowski 2010; 8McMasters, 2003 5 Research is limited in terms of how to improve students understanding Engineering Design Education of the broader context of design and ability to balance the performance

and stakeholder-related considerations within a vehicle design8 Recently, ASEE and NSF report based on a Accreditation Board for Engineering and survey of industry representatives, Technology (ABET) defines program Research Goals and Contributions highlights a need to improve how we outcomes that include need to graduate (1) Examine how stakeholder considerations arestudents currently prepare to synthesize engineers who understand impact of integrated into aerospace vehicle systems design engineering, business and societal design solutions on environment, 1 perspectives in the design process2 economy, andhow society (2) Examine stakeholder considerations

are currently integrated into aerospace engineering design education (3) Design empirically-informed and theoretically-grounded Studies of design and explorations of engineering students pathways in support industry support educational interventions and assessment tools that the need students to determine how to better prepare students to incorporate stakeholder integration of stakeholder considerations considerations into design.3-5 (4) Isolate the characteristics of the interventions and learning environment that support students integration of There is a movement within some fields of engineering to design learning environments stakeholder considerations which place a higher value on the needs and limitations of stakeholders.6-7 1

ABET, 2011; 2ASEE, 2013; 3Hinkle, 2013; 4Dym et al., 2001; 5Ro et al., 2012; 6Jordan & Lande 2012; 7Zoltowski 2010; 8McMasters, 2003 6 Research Overview Phase I: Examination of Current Practice in Industry and Education RQ#1 What conditions enhance or inhibit the integration of stakeholder considerations into the practice of aerospace vehicle systems design? RQ#2 To what extent and how does the aerospace engineering design curricula take into account stakeholder considerations? RQ#3 To what extent do aerospace engineering students understand and take into consideration the effects of design decisions on stakeholders? Phase II: Design of Educational Interventions and Assessment Tool RQ#4 - What educational interventions can enhance students understanding of and ability to integrate stakeholder considerations into the design of an aerospace vehicle?

Coso - Ph.D. Thesis Defense - March 28th, 2014 7 Research & Presentation Overview Practice of Aerospace Engineering Design Phase I: Examination of Current Practice in Industry and Education Aerospace Engineering Design Curricula Educational Intervention Specifications Phase II: Design of Educational Interventions and Assessment Tool

Contributions and Future Work Aerospace Engineering Students Perceptions 8 Examination of Practice of AE Vehicle Design RQ#1 What conditions enhance or inhibit the integration of stakeholder considerations into the practice of aerospace vehicle systems design? Human factors & related research Aircraft design research Addressed need and provided recommendations for how to integrate stakeholder considerations into complex system design Examined stakeholder considerations within multidisciplinary design, aerodynamic analyses, and studies of the operational environment However, this literature has not empirically studied how stakeholder considerations are integrated within aerospace vehicle design industrys multidisciplinary design teams 9

Embedded Multiple Case Study Research Research Setting Setting Units of of Units Analysis Analysis Aerospace Vehicle Design Firm Time Frame: 3 Months Disciplinary Backgrounds Individuals First Round Interviews Semi-Structured Protocol Data Data Sources Sources Functional Groups Second Round Interviews Remaining Questions

Topics included: 1. Overview of the design process at the firm 2. The challenges affecting the integration of stakeholder considerations 3. Previously used interventions to improve integration Coso - Ph.D. Thesis Defense - March 28th, 2014 10 Results: Group/Team-Level Contributing Factors Group Structure That supports addressing stakeholder considerations Common Goal That addresses stakeholder considerations Pilot Compartment Group Design Engineers Human Factors Specialists InHouse Pilots Coso - Ph.D. Thesis Defense - March 28th, 2014 HF Specialists

Design Engineers Other Functional Groups 11 Results: Individual-Level & Interaction-Level Contributing Factors Differing Perspectives About collaborations with individuals from other disciplines Shared Language & Boundary Objects To support the exchange of ideas Image References included at end of presentation 12 Educational Intervention Specifications From Industry Case Study Group/Team-Level ID1. Stakeholder-centric learning environment Interaction-Level ID2. Language and vocabulary for discussing stakeholder considerations ID3. Tools and resources to support integration of stakeholder considerations Individual-Level ID4. Cross-disciplinary problem-solving experiences

ID5. Value of designs created from multiple perspectives Coso - Ph.D. Thesis Defense - March 28th, 2014 13 Research & Presentation Overview Practice of Aerospace Engineering Design Phase I: Examination of Current Practice in Industry and Education Aerospace Engineering Design Curricula Educational Intervention Specifications Phase II: Design of Educational Interventions and

Assessment Tool Contributions and Future Work Aerospace Engineering Students Perceptions 14 Examination of AE Design Education RQ#2 To what extent and how does the aerospace engineering design curricula take into account stakeholder considerations? A review of publically-available syllabi from top aerospace engineering programs and relevant published work Review of the design processes presented in classical aerospace design textbooks (including Roskam; Raymer; Nicolai & Carichner) commonly used within aircraft design capstone course Coso - Ph.D. Thesis Defense - March 28th, 2014 15 Components of AE Design Curricula

Elective course offerings & systems engineering courses Isolated Courses Isolated course may be insufficient in training students to integrate these topics into in their capstone courses and in the future 1 Quantitative approximations (i.e. surrogates) are used to model some stakeholder characteristics Design Textbooks Design Projects Mention of customer needs and certification at beginning of text, but not emphasized within the design processes themselves [with exception of Nicolai and Carichner, 2010] Distinctive from product design due to life span, purpose, and size of traditional aerospace vehicles Impacts extent to which students experience entire design cycle Contact with clients or stakeholders is limited 1 Peet & Mulder, 2004 16 Educational Intervention Specifications From Industry Case Study Group/Team-Level ID1. Stakeholder-centric learning environment

Interaction-Level ID2. Language and vocabulary for discussing stakeholder considerations ID3. Tools and resources to support integration of stakeholder considerations From Examination of Design Curricula and Students Prior Knowledge ED1. Awareness of contextual and stakeholder considerations ED2. Importance of considering broad context and wide variety of stakeholders ED3. Methods for incorporating novel or qualitative stakeholder considerations ED4. Interactions with clients or other stakeholders Individual-Level ID4. Cross-disciplinary problem-solving experiences ID5. Value of designs created from multiple perspectives Coso - Ph.D. Thesis Defense - March 28th, 2014 17 Research & Presentation Overview Practice of Aerospace Engineering Design Phase I:

Examination of Current Practice in Industry and Education Aerospace Engineering Design Curricula Educational Intervention Specifications Phase II: Design of Educational Interventions and Assessment Tool Contributions and Future Work Aerospace Engineering Students Perceptions 18

Engineering Students Prior Knowledge RQ#3 To what extent do aerospace engineering students understand and take into consideration the effects of design decisions on stakeholders? Design Understanding and Self-Efficacy Self-Efficacy in Design Scale11 and Design Ranking Test22 Understanding of Contextual Considerations Contextual Competence Scale33 and Open-Ended Follow-Up Question Perceptions of the Role of Stakeholders in Design Scenario-Based Design Task 1 Carberry et al., 2010; 2Adams & Fralick, 2010; 3Ro et al., 2012 19 Senior Aerospace Engineering Students Sample: 80 senior AE students enrolled in an aircraft design capstone course 8 women (~10%) 5 international students (~6%) 30 students (~38%) completed a previous design course 52 students (~65%) described having industry experience 50 students (~63%) took a problem-based learning course within the department Coso - Ph.D. Thesis Defense - March 28th, 2014 20 Design Understanding Important Design

Activities Most Important Design Activities Study Students (n = 80) Problem-Scoping Communication VT Seniors (n = 53) AE Professionals (n = 20) Using Creativity Generating Alternatives Planning Iteration Making Trade-offs Brainstorming Design Activities Not Considered Most Important Making Trade-offs Make Decisions Goal Set Test Evaluate Communicate Identify Constraints Understand the Problem

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Portion of Sample Coso - Ph.D. Thesis Defense - March 28th, 2014 21 Design Understanding Important Design Activities Least Important Design Activities Problem-Scoping Communication Study Students (n = 80) VT Seniors (n = 53) AE Professionals (n = 20) Visualizing Prototyping Iteration Making Trade-offs Consistent with aircraft design students from other aerospace

engineering departments Iterating Design Activities Not Considered Most Important Synthesizing Decomposing Imagining Building Abstracting Sketching 0% 10% 20% Coso - Ph.D. Thesis Defense - March 28th, 2014 30% 40% 50% 60% 70%

80% 90% 100% Percentage of Sample 22 Understanding of Contextual & Stakeholder Considerations Important Design Activities Problem-Scoping Communication Students consider Technical and logistical considerations Designs effects on customer Not Considered Most Important Iteration Making Trade-offs Economic and mission constraints Students do not see impact of Consistent with aircraft design students from other aerospace engineering departments

Coso - Ph.D. Thesis Defense - March 28th, 2014 Broader context (i.e. political, historical considerations) Stakeholders other than customer (i.e. pilots, maintainers, passengers) 23 Educational Intervention Specifications From Industry Case Study Group/Team-Level ID1. Stakeholder-centric learning environment Interaction-Level ID2. Language and vocabulary for discussing stakeholder considerations ID3. Tools and resources to support integration of stakeholder considerations Individual-Level ID4. Cross-disciplinary problem-solving experiences ID5. Value of designs created from multiple perspectives From Examination of Design Curricula and Students Prior Knowledge ED1. Awareness of contextual and stakeholder considerations ED2. Importance of considering broad context and wide variety of stakeholders

ED3. Methods for incorporating novel or qualitative stakeholder considerations ED4. Interactions with clients or other stakeholders ED5. Importance of iteration and tradeoffs Coso - Ph.D. Thesis Defense - March 28th, 2014 24 Phase I Contributions Practice of Aerospace Engineering Design Phase I: Examination of Current Practice in Industry and Education Aerospace Engineering Design Curricula Aerospace Engineering Students Perceptions Empirical evidence of conditions which

enhance or inhibit the integration of human factors and stakeholder considerations into the aircraft design process. Educational Intervention Specifications Evaluation method to explore engineering students perceptions of how stakeholder considerations are integrated into a complex system design 25 Research & Presentation Overview Requirements Lab Phase I: Examination of Current Practice in Industry and Education Educational Intervention Specifications Phase II: Design of Educational Interventions

and Assessment Tool Stakeholders in Design Labs Contributions and Future Work 26 Requirements Lab Situational Factors GOAL: Introduce students to requirements, their importance, and how and why stakeholder-requirements should be incorporated into the requirements definition phase of the design process Learning Objectives Assessment & Feedback Learning Theory & Activities Design Process Coso - Ph.D. Thesis Defense - March 28th, 2014

27 Requirements Lab Situational Factors Learning Objectives Student Characteristics (from In-Class Evaluation) Wide spectrum of design knowledge and industry experience Nature of the Subject Design should be flexible for use within other aerospace courses and programs Assessment & Feedback Learning Theory & Activities Context of Teaching/Learning Situation (Study-Specific) Two 50-minute lectures per week (80 students), One three-hour lab session (40 students per lab session) Design Process Coso - Ph.D. Thesis Defense - March 28th, 2014 28

Requirements Lab Situational Factors Learning Objectives Assessment & Feedback Learning Theory & Activities LO#3 - Students will be able to identify different stakeholders affected by a design problem Based on ID3 & ED3 LO#4 - Students will be able to discuss how requirements affect design and the design process Based on ED1 LO#6 - Students will be able to breakdown a problem and write requirements for performance and stakeholder considerations Based on ID1 Design Process Coso - Ph.D. Thesis Defense - March 28th, 2014 29 Requirements Lab Situational

Factors Learning Objectives Assessment & Feedback Learning Theory & Activities Opportunities for Direct & Immediate Feedback Class-wide discussions about requirements and stakeholders Forward-looking Assessment/Check for Understanding Design case provides an open-ended, real-life context for students to practice developing requirements and selecting verification and validation procedures Further Opportunities to Assess Students Understanding Anonymous Student Feedback Forms Design Process Coso - Ph.D. Thesis Defense - March 28th, 2014 30 Requirements Lab Adaptation of Kolbs Learning Cycle1 Situational Factors Learning

Objectives Active Experimentation (trying out what you have learned) Design case Group work Interactions with stakeholders Concrete Experience (doing/having an experience) Reflection on previous design experiences Related experience activity Group work Interactions with a client Assessment & Feedback Learning Theory & Activities Abstract Conceptualisation (learning from the experience) Class-wide discussion to define characteristics of well-written requirements Reflective Observation (reflecting on the experience) Class-wide discussion about

requirements and stakeholders Design Process 1 Kolb & Kolb, 2009 31 Requirements Lab Adaptation of Kolbs Learning Cycle1 Situational Factors Learning Objectives Assessment & Feedback Learning Theory & Activities Active Experimentation (trying out what you have learned) Design case Group work Interactions with stakeholders Concrete Experience

(doing/having an experience) Reflection on previous design experiences Related experience activity Group work Designed such that students will create knowledge about Interactions with a client stakeholders and requirements through activities that require them to experience and reflect on the relationships between stakeholders, requirements, and design Abstract Conceptualisation (learning from the experience) Class-wide discussion to define characteristics of well-written requirements Reflective Observation (reflecting on the experience) Class-wide discussion about requirements and stakeholders Design Process 1 Kolb & Kolb, 2009 32

Research & Presentation Overview Requirements Lab Phase I: Examination of Current Practice in Industry and Education Educational Intervention Specifications Phase II: Design of Educational Interventions and Assessment Tool Stakeholders in Design Labs Contributions and Future Work 33 Stakeholders in Design Labs GOALS:

Situational Factors Learning Objectives Have students consider how stakeholder requirements and concerns can be integrated throughout the aircraft design process Have students define how stakeholder requirements will be incorporated into the design process they will follow during their team design project Assessment & Feedback Learning Theory & Activities Design Process Coso - Ph.D. Thesis Defense - March 28th, 2014 34 Stakeholders in Design Labs Situational Factors

Learning Objectives Assessment & Feedback Learning Theory & Activities Student Characteristics (from In-Class Evaluation) Wide spectrum of design knowledge and industry experience Nature of the Subject Design should flexible for use within other aerospace courses and programs Context of Teaching/Learning Situation (Study-Specific) Two 50-minute lectures per week (80 students), One three-hour lab session (40 students per lab session) Implemented several weeks into the semester Design Process Coso - Ph.D. Thesis Defense - March 28th, 2014 35 Stakeholders in Design Labs Situational Factors Learning

Objectives Assessment & Feedback Learning Theory & Activities LO#1 - Students will be able to identify relevant stakeholders of a fixed wing design and explain how their concerns affect (or affected) the design solution Based on specifications ID1, ED1, & ED2 LO#2 - Students will be able to assess methods for integrating stakeholder considerations into the design process Based on ID1, ID3, & ED3 Design Process Coso - Ph.D. Thesis Defense - March 28th, 2014 36 Stakeholders in Design Labs Situational Factors Opportunities for Direct & Immediate Feedback Learning Objectives Assessment & Feedback

Learning Theory & Activities Student-generated design processes Case study evaluations Class-wide discussions Additional Checks for Understanding Case study evaluations Student reflections Anonymous student feedback forms Design Process Coso - Ph.D. Thesis Defense - March 28th, 2014 37 Stakeholders in Design Labs Situational Factors Social Constructivist Theory1 Knowledge is constructed through social interaction Based on ID4 and ID5 Learning Objectives Design as Reflective Practice2 Design requires negotiation among disciplines and perspectives

Assessment & Feedback Learning Theory & Activities Based on ED5 Design Process 1 Kim, 2001; 2Bucciarelli, 1988 38 Stakeholders in Design Labs Situational Factors Framing the Lessons Pre-Lab Reflection Individual Design Process Drawing Design Process Negotiation #1 Learning Objectives Assessment & Feedback

Learning Theory & Activities Design Process Lab #1 Coso - Ph.D. Thesis Defense - March 28th, 2014 39 Stakeholders in Design Labs Communicate Throughout! Situational Factors 1. Understand the Problem 2. Visualize & Brainstorm 3. Seek Information Learning Objectives Assessment & Feedback

Learning Theory & Activities Identify Constraints Iterating Evaluating Planning Make Initial Decisions Generate Alternatives Making Trade-offs Testing Building Design Process Coso - Ph.D. Thesis Defense - March 28th, 2014 40 Stakeholders in Design Labs Situational

Factors Framing the Lessons Pre-Lab Reflection Individual Design Process Drawing Design Process Negotiation #1 Learning Objectives Introduction to Stakeholder Mapping Tool for isolating stakeholder considerations that can affect a design Assessment & Feedback Learning Theory & Activities Design Process Lab #1 Coso - Ph.D. Thesis Defense - March 28th, 2014 41

Stakeholders in Design Labs Situational Factors Learning Objectives Stakeholder Mapping Technique (adapted from a technique used in Civil and Environmental Engineering) Non-Users (e.g. people who live near an airport) Wider seat widths Pilots Larger fuselage provides greater headroom when walking up and down the aisle Passengers Assessment & Feedback Government Regulators (e.g. FAA, EASA) Improved Passenger

Experience Commercial Aircraft Flight Attendants Airlines Learning Theory & Activities Maintainers/ Ground Personnel Suppliers Design Firm/ Manufacturers Design Process Coso - Ph.D. Thesis Defense - March 28th, 2014 42 Stakeholders in Design Labs Situational Factors Framing the Lessons Pre-Lab Reflection Individual Design Process Drawing

Design Process Negotiation #1 Learning Objectives Assessment & Feedback Learning Theory & Activities Design Process Introduction to Stakeholder Mapping Introduction to Case Study Assessment Case Study Assessment Tool for isolating stakeholder considerations that can affect a design Real-world examples of aircraft design process and impact of stakeholders on design Wrap-Up Lab #1

Coso - Ph.D. Thesis Defense - March 28th, 2014 43 Stakeholders in Design Labs Situational Factors Learning Objectives Framing the Lessons Case Study Presentations Introduction to 5 HCD methods Discussion of advantages and disadvantages of methods and applicability to aircraft design Modification of initial design processes based on lessons learned in labs Indications of where

stakeholders would be accounted for in the process Final reflections Anonymous Student Feedback Human-Centered Design Methods Assessment & Feedback Learning Theory & Activities Student groups deconstruct an aircraft case study Outcomes include a stakeholder mapping and a design process Design Process Negotiation #2 Conclusion Lab #2 Design Process Coso - Ph.D. Thesis Defense - March 28th, 2014 44

Contribution of Intervention Design Requirements Lab Phase I: Examination of Current Practice in Industry and Education Phase II: Design of Educational Interventions and Assessment Tool Stakeholders in Design Labs Empirically-informed and theoretically-grounded instructional designs that can be easily integrated into an aircraft design capstone course or adapted for complex systems design course 45 Research & Presentation Overview

Requirements Lab Phase I: Examination of Current Practice in Industry and Education Phase II: Design of Educational Interventions and Assessment Tool Stakeholders in Design Labs Implementation and Evaluation of Educational Interventions Contributions and Future Work 46 Evaluation of Educational Interventions: Research Questions

RQ#4 - What educational interventions can enhance students understanding of and ability to integrate stakeholder considerations into the design of an aerospace vehicle? RQ4.1 - What characteristics of the educational interventions support students abilities to integrate stakeholder considerations into the design of an aerospace vehicle? RQ4.2 - To what extent can the interventions help students integrate stakeholder considerations into the design of an aerospace vehicle? RQ4.3 - What characteristics of the engineering design learning experience and learning environment support or hinder students integration of stakeholder considerations into the design of an aerospace vehicle? Coso - Ph.D. Thesis Defense - March 28th, 2014 47 Research & Presentation Overview Requirements Lab Phase I: Examination of Current Practice in Industry and Education Phase II: Design of Educational Interventions and

Assessment Tool Evaluation of the Learning Environment Stakeholders in Design Labs Evaluation of Educational Interventions Contributions and Future Work Stakeholders in Design Rubric 48 Evaluation of Educational Interventions: Evaluation Framework1 Resources & Processes Students satisfaction with interventions Acquisition Achievement of learning objectives

Organization & Course Application Adoption of intervention content in class Factors within the learning environment Societal Impact after and outside the course RQ4.1 - What characteristics of the educational interventions support students abilities to integrate stakeholder considerations into the design of an aerospace vehicle? RQ4.2 - To what extent can the interventions help students integrate stakeholder considerations into the design of an aerospace vehicle? Adapted from Moon et al 2011 & Kaufman et al 1995 1 49 Results Summary: Data from Interventions Requirements Lab [data from design case & feedback forms] Students formulated well-constructed high-level requirements and many students were able to identify a diverse group of stakeholders Students considered both performance and stakeholder considerations and most groups included at least three different verification and validation procedures

Stakeholder-centric focus of the lab was not clear to students Stakeholders in Design Labs [data from all activities & feedback forms] Stakeholder-centric focus supported students integration of stakeholder considerations in a variety of different ways in final design processes Case studies illustrated challenges of cross-disciplinary design Misconceptions about HCD methods may have also developed over the course of these lab sessions Coso - Ph.D. Thesis Defense - March 28th, 2014 50 Results Summary: Post-Intervention Evaluation Design Understanding at Post-Evaluation Iteration: 23.8% of the students in the pre-evaluation selected it, as compared with 69.6% of students in the post-evaluation Making Trade-offs: 33.8% of the students in the pre-evaluation selected it, as compared with 48.2% of students in the post-evaluation Stakeholder Considerations Students reported significantly more stakeholder-related considerations in the post-evaluations as compared with the pre-evaluations Coso - Ph.D. Thesis Defense - March 28th, 2014 51 Research & Presentation Overview Requirements Lab Phase I: Examination of Current

Practice in Industry and Education Phase II: Design of Educational Interventions and Assessment Tool Evaluation of the Learning Environment Stakeholders in Design Labs Evaluation of Educational Interventions Contributions and Future Work Stakeholders in Design Rubric 52 Stakeholders in Design Rubric Stakeholder Integration

by Design Stage Examines students integration of stakeholder considerations into 3 stages of design Specifically, students intent, application of methods, and overall success Design Understanding Scores students based on their application of the design process 0pts: Lack of knowledge about the process 4pts: abstract the process and be a reflective designer REFERENCES: Zoltowkski, 2010; Crismond and Adams, 2012 Stakeholder Integration Scores students based on the extent to which they integrated stakeholder considerations 0pts: Lack of appreciation 4pts: Leveraging multiple perspectives for a more innovative solution

53 Examination of Students Team Design Projects Problem Definition Stage Technology Integration Stage Overall Design Most teams did not apply a design process that supported the integration of stakeholder considerations [as noted by 60.6% of judging panels scores] Most teams did not state an intention to incorporate stakeholder considerations [as noted by 67.6% of judging panels scores] Or apply a design process that supported integration of stakeholder considerations [as noted by 58.8% of judging panels scores] 68.6% of judges scores suggested that students were somewhat successful at integrating stakeholder considerations into their final design solutions Coso - Ph.D. Thesis Defense - March 28th, 2014 54 Examination of Students Team Design Projects Design Understanding [0 to 4pt Scale] Frequency of Scores for Students' Team Design Project Reports

18 16 Stakeholder Integration [0 to 4pt Scale] 12 10 8 6 4 2 0 Median: 2 Identifies stakeholder considerations at isolated points in the design process, but overall these considerations are not the basis for design decisions and are not addressed consistently through design Stakeholder Integration 14 Score Frequency Median: 2 Applies design process as presented in course and demonstrates how design solution improved through an iterative design process

Design Understanding 0 1 2 3 4 Score Example of Variations in Scores Across Teams Team E received 0s from all 3 judges Team C received two 3s and one 2 from their judges Coso - Ph.D. Thesis Defense - March 28th, 2014 55 Research & Presentation Overview Requirements Lab Phase I: Examination of Current Practice in Industry and Education Phase II:

Design of Educational Interventions and Assessment Tool Evaluation of the Learning Environment Stakeholders in Design Labs Evaluation of Educational Interventions Contributions and Future Work Stakeholders in Design Rubric 56 Evaluation of Educational Interventions: Evaluation Framework1 Resources & Processes Students satisfaction with interventions

Acquisition Achievement of learning objectives Organization & Course Application Adoption of intervention content in class Factors within the learning environment Societal Impact after and outside the course RQ4.3 - What characteristics of the engineering design learning experience and learning environment support or hinder students integration of stakeholder considerations into the design of an aerospace vehicle? Adapted from Moon et al 2011 & Kaufman et al 1995 1 57 Examination of the Learning Environment TASK

Characteristics of Educational Interventions Team Design Project Requirements Reliance on Design Process Structure Available Tools and Resources [Ames, 1992] Coso - Ph.D. Thesis Defense - March 28th, 2014 58 Examination of the Learning Environment TASK EVALUATION PRACTICES Characteristics of Educational Interventions Design of an Assessment Rubric Team Design Project Requirements Use of Project Requirements as Evaluation Strategy

Reliance on Design Process Structure Evaluation through Interactions with External Audience Available Tools and Resources [Ames, 1992] Coso - Ph.D. Thesis Defense - March 28th, 2014 59 Examination of the Learning Environment TASK EVALUATION PRACTICES DISTRIBUTION OF AUTHORITY Characteristics of Educational Interventions Design of an Assessment Rubric Team Design Project Requirements Use of Project

Requirements as Evaluation Strategy Reliance on Design Process Structure Evaluation through Interactions with External Audience Transition from Instructor-centered, Individual-based Learning Environment to Student-centered, Team-based Learning Environment Available Tools and Resources [Ames, 1992] Coso - Ph.D. Thesis Defense - March 28th, 2014 60 Additional Phase II Contributions Requirements Lab Phase I: Examination

of Current Practice in Industry and Education Phase II: Design of Educational Interventions and Assessment Tool Evaluation of the Learning Environment Stakeholders in Design Labs Empirical evidence of conditions within the learning environment that impact students integration of stakeholder considerations into a complex systems design process. Evaluation of Educational Interventions

Stakeholders in Design Rubric Assessment tool to support students understanding of how to design with stakeholders in mind, and to assess students abilities to consider the impact of complex system design decisions on stakeholders. 61 Research & Presentation Overview Phase I: Examination of Current Practice in Industry and Education Phase II: Design of Educational Interventions and Assessment Tool Contributions and Future Work

62 Research Contributions & Future Work Industry Case Study empirical evidence of conditions that enhance or inhibit the integration of human factors & stakeholder considerations into the aircraft design process Requirements Lab & Stakeholders in Design Labs - empirically-informed and theoreticallygrounded instructional designs that can be easily integrated into an aircraft design capstone course or adapted for complex systems design course. In-Class Evaluation & Stakeholders in Design Rubric evaluation methods to (1) explore engineering students perceptions of how stakeholder considerations are integrated into the vehicle design process, (2) support students understanding of how to design with stakeholders in mind, and (3) assess students abilities to consider the impact of complex system design decisions on stakeholders Examination of Learning Environment - empirical evidence of conditions within the learning environment that impact students integration of stakeholder considerations into a complex systems design process Coso - Ph.D. Thesis Defense - March 28th, 2014 63 Research Contributions & Future Work Discipline-based education research that informs teaching and learning within aerospace engineering and supports continued research into this interdisciplinary field Starting point for future research into pedagogical techniques which support students integration of stakeholder considerations into the design of complex systems Coso - Ph.D. Thesis Defense - March 28th, 2014 64 Acknowledgments NSF Graduate Research Fellowship under Grant No. DGE-0644493. NSF under Grant #EEC-1059472 to

the American Society for Engineering Education. Thank you to all of the interview participants, the instructors, and the students for their assistance with this study. Thank you to my amazing peer debriefers, Subject Matter Experts, and the former and current members of the Cognitive Engineering Center at Georgia Tech! Questions? Related Publications Journal Papers + (Submitted) Coso, A. and Pritchett, A. (2013) The Role of Design Teams in the Integration of Stakeholder Considerations. Journal of Aircraft Conference Proceedings + (Accepted) Coso, A. and Pritchett, A. (2014) The Development of a Rubric to Evaluate and Promote Students Integration of Stakeholder Considerations into the Engineering Design Process. American Society for Engineering Education: Proceedings of the 2014 ASEE Annual Conference and Exposition. Indianapolis, IN + Coso, A. and Pritchett, A. (2014) Incorporating Stakeholder Considerations in the Aircraft Design Process: A Focus on Aircraft Design Education. Proceedings of AIAA SciTech. National Harbor, MD. + Coso, A. and Pritchett, A. (2013) The Integration of Stakeholder Requirements within Aerospace Engineering Design Education. American Society for Engineering Education: Proceedings of the 2013 ASEE Annual Conference and Exposition. Atlanta, GA. Coso - Ph.D. Thesis Defense - March 28th, 2014 66

Image References http://claesjohnson.blogspot.com/2012_07_01_archive.html http://www.amazon.com/Airplane-Design-Part-Preliminary-Airplanes/dp/188488542X http://www.amazon.com/Mission-Analysis-Edition-TECHNOLOGY-LIBRARY/dp/079235 9011 https://www.aiaa.org/PubDetail.aspx?id=1796 http://www.aviationweek.com/Blogs.aspx?plckBlogId=Blog:27ec4a53-dcc8-42d0-bd3a -01329aef79a7&plckPostId=Blog:27ec4a53-dcc8-42d0-bd3a-01329aef79a7Post:6e334 c9f-0810-4874-a0be-d7e023b58135 http://orbiterchspacenews.blogspot.com/2010/11/nasa-selects-companies-for-advanc ed.html http://www.disinfo.com/tag/spacex/ http:// www.open.ac.uk/about/campus/jennie-lee-research-labs/files/jennieleeresearchlabs/i magecache/node_standard/HCI%202%20Lab-1.jpg http://www.manvitas.com/images/Corporate.jpg http:// www.inc.com/uploaded_files/image/speed-up-your-prototyping-process-pop_9632.jp g http://www.kelvinyoung.com/v6/work/storyboard_01.jpg Coso - Ph.D. Thesis Defense - March 28th, 2014 67

Recently Viewed Presentations

  • Studies in Philippians(18) - Rose Avenue

    Studies in Philippians(18) - Rose Avenue

    Studies in Philippians(18)Rejoice, but BewarePhilippians 3:1-2 Previously Paul is challenging these brethren to shine as lights in the midst of this crooked and perverse generation
  • Chapter 6: Developing Product and Brand Strategy

    Chapter 6: Developing Product and Brand Strategy

    Product Mix Width: Number of lines offered. Product line Depth: Number of products in a line. Product Mix and Product Lines Line extensions: occur when a company introduces additional items in the same product category under the same brand name...
  • Τίτλος Μαθήματος

    Τίτλος Μαθήματος

    O εσωτερικός έλεγχος ποιότηταςμε το διάγραμμα Levey-Jennings Διαγράμματα ελέγχου (control charts) To εργαλείο για την ανίχνευση των αναλυτικών σφαλμάτων και την διάκριση τους σε τυχαία και συστηματικά.
  • Estimated number of cancer cases attributable to infectious

    Estimated number of cancer cases attributable to infectious

    Ontario Cancer Registry, 2016 (Cancer Care Ontario) NOTES: Incidence rates are standardized to the age distribution of the 2011 Canadian population. Liver cancers exclude intrahepatic bile duct cancer. For ICD-O-3 site and histology codes for cancer type, see Table A4...
  • LHCb Book-Keeping - lhcb-comp.web.cern.ch

    LHCb Book-Keeping - lhcb-comp.web.cern.ch

    Create User Interface Move the actual information to the database Status Database model frozen the 4th of August Prototype based on the data find in the version of the actual database (15th of august) Creation in ORACLE of the different...
  • Unit 7: Adverbs and Prepositions - Sacred Heart Science

    Unit 7: Adverbs and Prepositions - Sacred Heart Science

    A prepositional phrase has a preposition and an object of the preposition. When the pronoun is the object of the preposition, then you should use an object pronoun. Practice
  • Identifying Tree Species From Bark

    Identifying Tree Species From Bark

    Found that GLCM only yielded 61% for the same dataset My Dataset Pictures of species in Central Park 2 Pictures for each individual tree Taken with iPhone, for each tree 1 picture using HDR setting, 1 with the standard Taken...
  • U.S. Rural vs. Non-Rural HIV Care Continuum Differences:

    U.S. Rural vs. Non-Rural HIV Care Continuum Differences:

    AETC Rural Health Committee Definition of "Rural" Rural is a geographic area that is populated with less than 50,000 people (or non-metropolitan for designated areas with less than 500,000 people) with one or more of the following geographical barriers:. Travel...