Addressing the Limitations of the Measurement of Team Cognition
Cognitive Task Analysis for Teams Nancy J. Cooke New Mexico State University CTA Resource Online Seminar October 11, 2002 Acknowledgements NMSU Faculty: Peter Foltz NMSU Post Doc: Brian Bell NMSU Graduate Students: Janie DeJoode, Jamie Gorman, Preston Kiekel, Rebecca Keith, Melanie Martin, Harry Pedersen US Positioning, LLC: Steven Shope UCF: Eduardo Salas, Clint Bowers Sponsors: Air Force Office of Scientific Research, Office of Naval Research, NASA Ames Research
Center, Army Research Laboratory October 11, 2002 Nancy Cooke 2 Overview What is team cognition? Q&A Shared mental models
Q&A Holistic CTA for teams Conclusions Q&A October 11, 2002 Nancy Cooke 3 What is Team Cognition? October 11, 2002 Nancy Cooke 4
Team Cognition in Practice October 11, 2002 Nancy Cooke 5 Experimental Context CERTT (Cognitive Engineering Research on Team Tasks) Lab A Synthetic Task Environment for the Study of Team Cognition Five Participant Consoles October 11, 2002 Experimenter Console Nancy Cooke
6 Defining Team a distinguishable set of two or more people who interact dynamically, interdependently, and adaptively toward a common and valued goal/object/mission, who have each been assigned specific roles or functions to perform, and who have a limited life span of membership Salas, Dickinson, Converse, and Tannenbaum (1992) October 11, 2002 Nancy Cooke 7
Defining Team Cognition It is more than the sum of the cognition of individual team members. It emerges from the interplay of the individual cognition of each team member and team process behaviors October 11, 2002 Nancy Cooke 8 Team Cognition Framework
Team Process Behaviors Individual knowledge Team Knowledge Team Performance October 11, 2002 Nancy Cooke 9 Team Cognition Framework
Collective level + + Team Process Behaviors Holistic Level Individual knowledge Team Knowledge Team Performance October 11, 2002
Limitations Measures tend to assume homogeneous teams Measures tend to target collective level Aggregation methods are limited Measures are needed that target the more dynamic and fleeting knowledge Measures are needed that target different types of long-term team knowledge A broader range of knowledge elicitation methods is needed A need for streamlined and embedded measures Newly developed measures require validation October 11, 2002 Nancy Cooke 12
Other Related Work Group Think (Janis, 1972) Distributed Cognition (Hutchins, 1991) Common Ground in Discourse (Clark & Schaefer, 1987; Wilkes-Gibbs & Clark 1992 ) Group Decision Support (Fulk, Schmitz, & Ryu, 1995) Social Decision Schemes (Davis, 1973; Hinsz, 1999) Transactive Memory (Wegner, 1986) Shared Mental Models (Cannon-Bowers, Salas, & Converse, 1993) October 11, 2002 Nancy Cooke 13
Why Do We Care? Outcome measures of team performance do not reveal why performance is effective or ineffective Team cognition is assumed to contribute to team performance Understanding the team cognition behind team performance should facilitate interventions (design, training, selection) to improve that performance October 11, 2002 Nancy Cooke 14 Team Cognition and Functions of Cognitive Task Analysis
Elicitation: Interviews, observations, think aloud used to make knowledge explicit Assessment: Judgments are made regarding specific elicited knowledge (e.g., accuracy, intrateam similarity) Diagnosis: Patterns in elicited knowledge (i.e. symptoms associated with dysfunctional or exceptional performance) are tied to a diagnosis October 11, 2002 Nancy Cooke 15 Questions or Comments? October 11, 2002
Nancy Cooke 18 Shared Sharing = to have compatible knowledge Sharing = to have the same knowledge Shared beliefs vs. To hold in common October 11, 2002
Share the pie To distribute Nancy Cooke 19 The Apples and Oranges Problem Measures to assess team knowledge often assume knowledge homogeneity among team members. Shared knowledge = similar knowledge Person A Person B
Accuracy is relative to single referent October 11, 2002 Nancy Cooke Referen t 20 Teams, by Definition, Consist of Apples and Oranges Airport Incident Command Center October 11, 2002 Nancy Cooke
Telemedicine 21 Shared Knowledge Knowledge Base Person A Person B Shared = Common October 11, 2002 Nancy Cooke 22
Shared Knowledge Shared = Complementary October 11, 2002 Nancy Cooke 23 Shared Knowledge Shared = Common and Complementary October 11, 2002 Nancy Cooke 24 Shared Knowledge
Common and Complementary Knowledge and Shared Perspectives/ Varied Granularity October 11, 2002 Nancy Cooke 25 Shared Knowledge Conflicting Knowledge Irrelevant Knowledge No Coverage Common and Complementary
Knowledge and Shared Perspectives October 11, 2002 Nancy Cooke 26 An Approach to the Apples and Oranges Problem Measures of team knowledge with heterogeneous accuracy metrics October 11, 2002 Nancy Cooke 27
Experimental Context Five studies: Two different 3-person tasks: UAV (Uninhabited Air Vehicle) and Navy helicopter rescue-and-relief Procedure: Training, several missions, knowledge measurement sessions Manipulate: co-located vs. distributed environments, training regime, knowledge sharing capabilities, workload October 11, 2002 Nancy Cooke 28 Experimental Context MEASURES Team performance: composite measure Team process: observer ratings and
critical incident checklist Other: Communication (flow and audio records), video, computer events, leadership, demographic questions, working memory Taskwork & Teamwork Knowledge, Situation Awareness October 11, 2002 Nancy Cooke 29 Long-term Taskwork Knowledge Factual Tests The camera settings are determined by a) altitude, b) airspeed, c) light conditions, d) all of the above.
Psychological scaling How related is airspeed to restricted operating zone? October 11, 2002 Nancy Cooke 30 Long-term Teamwork Knowledge Given a specific task scenario, who passes what information to whom? Teamwork Checklist ___AVO gives airspeed info to PLO ___DEMPC gives waypoint restrictions to AVO ___PLO gives current position to AVO AVO= Air Vehicle Operator
PLO = Payload Operator DEMPC = Navigator October 11, 2002 Nancy Cooke 31 Team Situation Awareness Assess accuracy and similarity of situation models of team members SPAM (Situation Present Assessment Method) queries--display not interrupted Durso, et al., 1998 Queries about future events How many targets are left to photograph? Team members queried in random order at designated point in scenario
within a 5-minute interval October 11, 2002 Nancy Cooke 32 Traditional Accuracy Metrics Team Referent .50 Team Member: 50% ACCURACY Air Vehicle Operator
October 11, 2002 Nancy Cooke 33 Heterogeneous Accuracy Metrics AVO Referent Team Referent .50 1.0 .33
PLO Referent DEMPC Referent 0 ACCURACY Overall: .50 Team Member: AVO Positional: 1.0 AVO= Air Vehicle Operator PLO = Payload Operator October 11, 2002 DEMPC = Navigator
Nancy Cooke Interpositional: .17 34 Results Across Studies Taskwork knowledge is predictive of team performance But True for psychological scaling, not factual tests Timing of knowledge test is critical October 11, 2002 Nancy Cooke 35
Knowledge Profiles of Two Tasks Knowledge Profile Knowledge Common metric (UAV) Knowledge profile characterizing effective teams depends on task (UAV vs. Navy) October 11, 2002 Overall accuracy
+ + + 0 36 Knowledge Profiles of Two Tasks Complementary Common UAV Task Navy Helicopter Task
Command-and-Control Planning and execution Interdependent Less interdependent Knowledge sharing Face-to-Face October 11, 2002 Nancy Cooke 37
Knowledge Acquisition Training Mission Experience Procedure: Knowledge Acquired: Taskwork Knowledge Teamwork Knowledge Teamwork knowledge is acquired through mission experience and its acquisition seems dependent on a foundation of taskwork knowledge acquired in training. October 11, 2002
Nancy Cooke 38 Results: Team Situation Awareness October 11, 2002 Situation Awareness and Performance 1.00 Proportion of Score Team SA mirrors the performance acquisition function and generally
improves with mission experience Team SA is generally good predictor of team performance (especially a repeated query) 0.80 0.60 SA Perf 0.40 0.20 0.00 1
2 3 4 5 6 7 8 9 10
Mission Number SA and Performance data from first UAV study. 39 Nancy Cooke Implications of Heterogeneous Metrics Can deal with apples and oranges issue Can assess knowledge underlying task performance Knowledge profiles of tasks can inform training and design interventions October 11, 2002 Nancy Cooke
40 Future Directions on Apples and Oranges Problem Apply metrics to fleeting knowledge Embed knowledge measures in task Need a taxonomy of tasks and additional profile work Need to connect the knowledge profile (symptoms) to diagnosis of team dysfunction or excellence October 11, 2002 Nancy Cooke 41 Questions or Comments?
October 11, 2002 Nancy Cooke 42 Holistic CTA for Teams October 11, 2002 Nancy Cooke 43 Team Cognition Framework Collective level +
+ Team Process Behaviors Holistic Level Individual knowledge Team Knowledge Team Performance October 11, 2002 Nancy Cooke
44 The Sum of All Team Members Problem Individual knowledge Collective level + + Team Process Behaviors Holistic Level Team Knowledge Team
Performance October 11, 2002 Nancy Cooke The Problem: Measures are taken at the individual level and aggregated, as opposed to being taken at the holistic level. 45 The Sum of All Team Members Problem Aggregating individual data is
problematic given the apples and oranges problem Team process behavior is missing from collective measures Cognition at the holistic level should be more directly related to team performance October 11, 2002 Nancy Cooke 46 Our Approach to the Sum of All Team Members Problem Consensus assessment tasks Consensus concept ratings Consensus teamwork
checklist Consensus SA queries Communication as a measure of team cognition October 11, 2002 Nancy Cooke 47 Consensus Assessment Tasks An Example: Concept Ratings 1) Step One: Individual Concept Ratings collected Present to each individual: airspeed altitude (1=related, 5=unrelated) Responses: AVO=4, PLO=1, DEMPC=5
2) Consensus Ratings Collected Present to the team: airspeed altitude (1=related, 5=unrelated) Prior responses: AVO=4, PLO=1, DEMPC=5 Team discussion: PLO: Well I said related since my camera settings for shutter speed and focus are dependent on each of these values DEMPC: OK, lets go with that 1 it is October 11, 2002 AVO= Air Vehicle Operator PLO = Payload Operator Nancy Cooke DEMPC = Navigator 48 Consensus Assessment Tasks
Results Consensus measures correlate moderately with performance compared to collective measures Perhaps consensus does not adequately tap in-mission process behavior Although collective measures and process behaviors predict team performance for co-located teams better than holistic measures, this is not true for distributed teams October 11, 2002 Nancy Cooke 49 Communication as a
Window to Team Cognition The Good Observable Team behavior diagnostic of team performance Think aloud in the wild Reflects team cognition at the holistic level Rich, multidimensional (amount, flow, speech acts, content) October 11, 2002 Nancy Cooke 50 Communication as a Window to Team Cognition The Bad Communication data
Analyses do not fully exploit data (e.g., dynamic, sequential aspect) Time spent talking October 11, 2002 Nancy Cooke 51 Communication as a Window to Team Cognition AND The Ugly Labor intensive transcription, coding, and
interpretation October 11, 2002 Nancy Cooke 52 Our Approach to Solving the Sum of All Team Members Problem Via Communication Analysis Communication Flow Analysis Content Analysis Using LSA October 11, 2002 Nancy Cooke 53
Analyzing Flow: CERTT Lab ComLog Data Team members use pushto-talk intercom buttons to communicate October 11, 2002 Nancy Cooke At regular intervals speaker and listener identity are logged 54 Analyzing Flow: ProNet-Procedural Networks Cooke, Neville, & Rowe, 1996 Nodes define events that occur in a sequence
An Example from UAV study: 6 nodes: Abeg, Aend, Pbeg, Pend, Dbeg, Dend ProNet: Find representative event sequences Quantitative: Chain lengths-->Performance Mission 2: R2 = .509, F(2, 8) = 4.144, p = .058 Mission 3: R2 = .275, F(1, 9) = 3.415, p = .098 Mission 5: R2 = .628, F(2, 8) = 5.074, p = .051 October 11, 2002 Nancy Cooke 55 Analyzing Flow: ProNet-Procedural Networks Qualitative: Communication patterns predictive of performance Pbeg Pbeg
Pend Aend Pend Aend Dbeg Abeg Dbeg Abeg Dend
Dend Team 2 before PLO-DEMPCs fight Team 2 after PLO-DEMPCs fight October 11, 2002 AVO= Air Vehicle Operator PLO = Payload Operator Nancy Cooke DEMPC = Navigator 56 Content Analysis with Latent Semantic Analysis (LSA) Landauer, Foltz, & Laham, 1998
A tool for measuring cognitive artifacts based on semantic information Provides measures of the semantic relatedness, quality, and quantity of information contained in discourse Automatic and fast We can derive the meaning of words through analyses of large corpora Large constraint satisfaction of estimating the meaning of many passages based on their contained words (like factor analysis) Method represents units of text (words, sentences, discourse, essays) as vectors in a high dimensional semantic space based on correlations of usage across text contexts Compute degree of semantic similarity between any two units of text October 11, 2002 Nancy Cooke 57
Content Analysis with Latent Semantic Analysis (LSA) An Example from UAV Study 1 67 Transcripts from missions 1-7 XML tagged with speaker and listener information ~2700 minutes of spoken dialogue 20,545 separate utterances (turns) 232,000 words (660 k bytes of text) Semantic Space: 22,802 documents Utterances from dialogues Training material Interviews with domain experts Derived several statistical measures of the quality of each transcript October 11, 2002 Nancy Cooke
58 Content Analysis with Latent Semantic Analysis (LSA) Team 1 Mission 3 Score: 750 Team 3 Mission 4 Score: 620 Team 8 Mission 6 Score 560 Team 7 Mission 3 Score 580 Team 8 Mission 3 Score ???? Team 6 Mission 3 Score 490
Team 5 Mission 4 Score 460 LSA-based communication score predicts performance (r =.79). October 11, 2002 Nancy Cooke 59 Other Communication Analysis Approaches Flow Analyses Measure of speaker dominance Deviations from ideal flow Clustering model-based patterns Content Analyses
Automatic transcript coding Coherence in team dialogue Measures of individual contributions October 11, 2002 Nancy Cooke 60 Conclusions October 11, 2002 Nancy Cooke 61 Summary Teams think
Understanding team cognition is critical for diagnosis of team dysfunction or excellence and later intervention Measuring team cognition is critical for understanding it There are challenges (e.g., apples and oranges, sum of all team members) October 11, 2002 Nancy Cooke 62 More to Do Further application of heterogeneous metrics Embedded, streamlined knowledge measures
Further validation Investigate generality across tasks Individual cognitive differences Beyond assessment to diagnosis October 11, 2002 Nancy Cooke 63 Contact Nancy J. Cooke New Mexico State University [email protected] http://psych.nmsu.edu/CERTT/ Moving to Arizona State University East January 2003 AZ
October 11, 2002 NM Nancy Cooke 64 Bibliography Methodological Reviews Cooke, N. J. (1999). Knowledge elicitation. In. F.T. Durso, (Ed.), Handbook of Applied Cognition, pp. 479-509. UK: Wiley. Cooke, N. J., Salas, E., Cannon-Bowers, J. A., & Stout, R. (2000). Measuring team knowledge. Human Factors, 42, 151-173. Cooke, N. J., Salas, E., Kiekel, P. A., & Bell, B. (in press). Advances in measuring team cognition. In E. Salas and S. M.
Fiore (Eds.), Team cognition: Process and performance at the inter- and intra-individual level. Washington, DC: American Psychological Association. Empirical Studies Cooke, N. J., Cannon-Bowers, J. A., Kiekel, P. A., Rivera, K., Stout, R., and Salas, E. (2000). Improving team's interpositional knowledge through cross training. Proceedings of the Human Factors and Ergonomics Society 44th Annual Meeting. Cooke, N. J., Kiekel, P. A., & Helm E. (2001). Measuring team knowledge during skill acquisition of a complex task. International Journal of Cognitive Ergonomics: Special Section on Knowledge Acquisition, 5, 297-315. CERTT Lab & UAV STE Cooke, N. J., Rivera, K., Shope, S.M., & Caukwell, S. (1999). A synthetic task environment for team cognition research.
Proceedings of the Human Factors and Ergonomics Society 43rd Annual Meeting, 303-307. Cooke, N. J., & Shope, S. M. (2002). The CERTT-UAV Task: A Synthetic Task Environment to Facilitate Team Research. Proceedings of the Advanced Simulation Technologies Conference: Military, Government, and Aerospace Simulation Symposium, pp. 25-30. San Diego, CA: The Society for Modeling and Simulation International. Cooke, N. J., & Shope, S. M. (in press), Designing a synthetic task environment. In S. G. Schiflett, L. R. Elliott, E. Salas, & M. D. Coovert, Scaled Worlds: Development, Validation, and Application. UK: Ashgate. Communication Analyses Kiekel, P. A., Cooke, N. J., Foltz, P. W., & Shope, S. M. (2001). Automating measurement of team cognition through analysis of communication data. In M. J. Smith, G. Salvendy, D. Harris, and R. J. Koubek (Eds.), Usability Evaluation and Interface Design, pp. 1382-1386, Mahwah, NJ: Lawrence Erlbaum Associates. Kiekel, P. A., Cooke, N.J., Foltz, P.W., Gorman, J. C., & Martin, M.J. (2002). Some promising results of communicationbased automatic measures of team cognition. Proceedings of the Human Factors and Ergonomics Society 46th Annual Meeting, 298-302. October 11, 2002 Nancy Cooke
65 References Cannon-Bowers, J. A., Salas, E. & Converse, S. (1993). Shared Mental Models in Expert Team Decision Making. In N. J. Castellan, Jr. (Ed.). Current issues in individual and group decision making (pp. 221-246). Hillsdale, NJ: Erlbaum. Hillsdale, NJ: Lawrence Erlbaum Associates.
Clark, H. H., & Schaefer, E. F. (1987). Collaborating on Contributions to Conversations. Language and Cognitive Processes, 2, 19-41. Cooke, N. J., Neville, K. J., & Rowe, A. L. (1996) Procedural network representations of sequential data. Human-Computer Interaction, 11, 29-68.Davis, J. H. (1973). Group decision and social interaction: A theory of social decision schemes. Psychological Review, 80, 97-125. Durso, F. T., Hackworth, C. A., Truitt, T. R., Crutchfield, J., & Nikolic, D. & Manning, C. A. (1998). Situation awareness as a predictor of performance in en route air traffic controllers. Air Traffic Control Quarterly, 5, 120. Fulk, J., Schmitz, J., & Ryu, D. (1995). Cognitive elements in the social construction of communication technology. Management Communication Quarterly, 8, 259-288. Hinsz, V. B. (1999). Group decision making with responses of a quantitative nature: The theory of social decision schemes for quantities. Organizational Behavior and Human Decision Processes, 80, 28-49. Hutchins, E. (1991). The social organization of distributed cognition. In L. B. Resnick, J. M. Levine, S. D. Teasley (Eds.), Perspectives on socially shared cognition (pp. 283-307). Washington, DC, USA: American Psychological Association. Janis, L. J. (1972). Victims of groupthink. Boston: Houghton Mifflin. Landauer, T. K, Foltz, P. W. & Laham, D. (1998). An introduction to Latent Semantic Analysis. Discourse Processes, 25, 259-284. Wegner, D. M. (1986). Transactive Memory: A contemporary analysis of the group mind. In B. Mullen and G. Goethals (Eds.), Theories of group behavior (pp. 185-208). New York: Springer-Verlag.
Wilkes-Gibbs, D., & Clark, H. H. (1992). Coordinating Beliefs in Conversation. Journal of Memory and Language, 31, 183-194. October 11, 2002 Nancy Cooke 66 Questions or Comments? October 11, 2002 Nancy Cooke 67
RETScreen® International Clean Energy Decision Support Centre Develops enabling tools that make it easier for planners, decision-makers and industry to consider energy efficient and renewable energy technologies (RETs) at the critically important initial planning stage Enabling tools significantly reduce the...
Investors across the country can invest in the plan and proceeds can be used to attend school in any state. Putnam has over 75 years of investment experience. The firm was founded in 1937 with one of the first mutual...
Type 2 Diabetes. Type 2 Diabetes. Body makes some insulin but usually do not start out on daily insulin injections. Blood sugars controlled with a combination of oral medications, healthy diet, exercise, and stress management. Ususally develop type 2 diabetes...
Most members of an executive board better understand and are more interested in financial terms than IT terms , so IT project managers must speak their language. Profits. are revenues minus expenditures. Profit margin . is the ratio of revenues...
Road Closures. Prior to closing a road you must notify your contractor coordinator. This includes both full or partial road closures. The Communication Center (757-380-4031) has to be notified prior to any road closure.
Get Tools and Resources. A brief history of the Evolution of encrypting viruses . How they do it. The biggies: Cryptolocker, Crypto Defense, Torrent Locker, CryptoWall 1, 2 and 3 ... "As soon as the email was opened, every single...