Delphi 650 BCE The Expert and validity of the Delphi Technique Phillip L. Davidson, PhD RESEARCH METHODOLOGY SIG APRIL 4, 2019 Agenda Before Delphi Origins of the Delphi Technique The Delphi Technique
The Problem What is an Expert? 1964 to 1972: The early years The Turning Point: 1974-1975 After 1975: No Definition for "Expert" Diversity of Opinion Summary
Discussion Conclusion Comments? Questions? Before Delphi It can be helpful to know the information that was valued by the original Delphi researchers to understand how and why Delphi started as it did. Key Germinal Writers:
Douglas Hadley (1938). McGregor from MIT (1938). Cantril from Princeton Origins of the Delphi Technique 1950s: RAND Corporation, under direction of the U.S. Air Force, developed the Delphi Technique. Purpose? To FORECAST how many atomic bombs Russia would have to deliver to completely destabilize the U.S. infrastructure. Qualifier: "The key purpose for using the Delphi method remains the collection of informed judgment on issues that are largely unexplored, difficult to define, highly context and expertise specific, [and] future-oriented"
(Fletcher & Marchildon, 2014, p. 3). Research schizophrenia? Kaplan, the philosopher. Helmer, Dalkey, Brown, and Gordon the scientists. Is Delphi qualitative or quantitative? Does the research methodology affect the definition of Expert? The Delphi Technique The centrality of the Delphi forecasting technique to the nascent discipline of futures research is unquestionable. No other forecasting method is so conspicuously a fixture of the contemporary effort to get prediction onto a scientific basis (Hill & Fowles, 1975, p. 179) Delphi is a set of procedures for eliciting [the] opinions of a group of people. In practice, the procedures would be used
with a group of experts or especially knowledgeable individuals (Dalkey, 1967, p. 1). The essential elements characterizing Delphi: Experts Anonymity Consensus Iteration through multiple rounds with controlled feedback.
Use of experts is a defining feature (Baker et al., 2006, p. 59) The Problem The biggest issue regarding the validity of the Delphi Technique is that all four of the defining characteristics have been ignored at one time or another in peer-reviewed journal articles. Without any firm definitions, the validity and credibility of the Delphi are questionable. The most critical design decision when conducting a rigorous Delphi-based study [is] expert panel design" (Donohoe & Needham, 2009, p. 416).
What is an Expert? Experienced though practice and education Widely recognized by peers Person with extensive knowledge or ability in a particular area of study Not necessarily a profession or academic (Wikipedia, 2019) 1964 to 1972 the early years
"Since the use of intuitive forecasting as a basis for longrange planning is unavoidable, we should at least make an effort to obtain this intuitive judgment as systematically as possible from persons who are recognized experts in the area of concern" (Gordon & Helmer-Hirschberg, 1964, p. 4). "The selection of experts is an intricate problem even when the category of expertise needed is well-defined (Brown, 1968, p. 4). Focus was on improving the process of arriving at efficient operating decisions (Brown & Helmer, 1964, pg. 1). It is assumed that experts are experts because they are objective (Weaver, 1972, p. 2). The Turning Point 1974-75
In 1974, Harold Sackman, one of the original researchers on the Delphi Technique, published an Assessment of the Delphi. Conventional Delphi is basically an unreliable and scientifically unvalidated technique in principle and probably in practice (Sackman, 1974, p. vi). Sackman recommended that, Delphi be dropped from institutional, corporate, and government use (p. 70). In 1975, Sackman published a text entitled Delphi technique. Until this time, much of the research was fragmented and exploratory. However, Sackmans comments started a discussion about the validity of the Delphi Technique that continues even to today. After 1975 No Definition for Expert
"The Delphi method places great emphasis upon the opinion of experts, yet the basic experiments were not conducted with experts, and there is no evidence that the opinions of experts are better than, or even different from, the opinions of nonexperts" (Fischer, 1978, p. 68). A serious problem is "the lack of an operational definition for the term expert that measures a person's ability to predict future events and the likelihood of selecting panelists who have a vested interest and/ or are biased in the area being studied by the Delphi method" (Fischer, 1978, pp. 69-70). Diversity of Opinion One should therefore choose experts whose combined knowledge and expertise reflects the full scope of the
problem domain. Heterogeneous experts are preferred to experts focused in a single specialty (Rowe & Wright, 2001, p. 128). Dalkey (2002) argues (and many researchers quote Dalkey), that one head is better than one (p. 234) The importance of defining the desired expert group is not contestable. However, there remains ambiguity in the ways in which definition is operationalised and reported. (Donohoe, 2009, p. 429). Simply because individuals have knowledge of a particular topic does not necessarily mean that they are experts" (Kenney, Hasson, & McKenna, 2011, p. 8). Summary part 1
The original researchers discussed the theory behind the Delphi Technique multiple times. Helmer and Kaplan were especially focused on the philosophical aspects of prediction in the area of social sciences. However, virtually all of the published research was quantitative in nature and focused on how one measures consensus and the process by which consensus is achieved. All of this left the concept of Expert undefined and problematical. Finally, is a somewhat angry response to this lack of definition, Sackman (1975) argued that the Delphi Technique should no longer be used. The results have been interesting. It seems because there were no specific definitions, virtually anything was OK.
Summary part 2 In 2002, Turoff introduced the idea of a Policy Delphi. The policy Delphi still used multiple rounds, but consensus was no longer relevant, and expertise was define as political expertise in the area under study. Paraskevas and Saunders (2012) elaborate on a normative or consensus Delphi. Other researchers introduce modified methods or e-Delphi (Donohoe et al., 2012; Cole et al., 2013; Shariff, 2015). Despite all the procedural variations, the lynchpin of the Delphi the expert still has no specific definition. One problem is that there are so many different disciplines and applications, one definition will not be adequate for all.
Discussion This short discussion and what constitutes an expert is part of a much bigger consideration still ongoing within the research community. Some rigorously stick to the classical academic definition of someone who has the education, who publishes in the field, and who has the respect of his or her peers. Others (Day, 2002) argue that anyone with experience, including housewives, even though they may not be an academic or a professional, may still be appropriate for a Delphi Study. At the same time, this lack of a clear definition has caused numerous issues with students. I had one student who
wanted to do a Delphi study with his fraternity brothers, which another wanted to survey students walking around campus, and a third who wanted to survey her coworkers. Conclusion As you can see, there is a huge variation in what is termed a Delphi Technique or study. It seems that almost anything goes. However, I ask you, can you have a research design that has no clear definition? In my opinion, regardless of who a student researcher decides to select for his or her expert panel, there must be a clear definition and validation of the panels expertise. Another side note, which often gets neglected, is that students often look to the Delphi Technique as an easy approach to doing research for a dissertation. The neglect the requirement for active involvement of the researcher during the research. Not only should the researcher be
collating data between rounds, but they should also be adding any new and relevant information. There is the likelihood of extensive bias in a Delphi study. Selection of experts is the first concern. Students need to clearly examine their own biases as part of their Comments? Questions?
The units are the mks units (meters-kilograms- seconds). There is a jetpack strapped to the mass, which fires with a force of 1 newton for 1 second and then is off for 1 second, and so on. We want to...
Software Myths (Developer Perspectives) Once the software is demonstrated, the job is done. Software Myths (Developer Perspectives) Until the software is coded and is available for testing, there is no way for assessing its quality. Software Myths (Developer Perspectives) The...
Noun Review: Common and Proper. Common nouns: a person, place, thing, or idea that is not specific or is named generally. Proper nouns: a specifically named person, place, thing, or idea. Proper nouns are capitalized . That painting is in...
Case Study: Porting an MFC Application to Unicode Harley Rosnow Software Development Lead Microsoft® Corporation Table of Contents Assumptions and Terminology Two-Executable Model FrontPage 10 Objectives Microsoft Foundation Classes FrontPage 10 Strategy Unicode Strings FrontPage 2000 User Interface FrontPage 10...
Expansion bus - connect external devices Local bus - a data bus that connect a peripheral device directly to the CPU Buses from a timing perspective: Synchronous buses - work off clock ticks - all devices using this bus type...
C8 sale debajo de la séptima vértebra cervical, y el resto de nervios espinales (T1 a Co) salen debajo de sus respectivas vértebras. Origen Fibras De la médula espinal salen 31 pares de nervios, cada uno tiene un par de...
Ready to download the document? Go ahead and hit continue!