Assessing the pragmatics of experiments with crowdsourcing: The case of scalar implicature Pranav Anand, Caroline Andrews, Matthew Wagers University of California, Santa Cruz Experiments & Pragmatic Processing Case Study: (Embedded) Implicatures Each of the critics reviewed some of the movies.
but not all ? Depending on the study: - no evidence of EIs evidence for EIs, with different response choices Worry: Are we adequately testing the influence of methodologies on our data?
Previous Limitation: Lack of Subjects and Money Crowd-sourcing addresses both problems Pragmatics of Experimental Situations Evaluation Apprehension subjects know they are being judged Teleological Curiosity - Subjects hypothesizing expected behavior, matching an ideal The experiment itself is part of the pragmatic context
See Rosenthal & Rosnow. (1975) The Volunteer Subject. Elements of Experimental Context Protocol Social Context / Task Specification Response Structure Response choices available to the subject e.g. True / False, Yes / No, 1-7 scale Prompt the Question directions for the Response Structure Immediate Linguistic/Visual Context
Our Goal: Explore variations of these elements in a systematic way Experimental Design Is this an accurate description? Some of the spices have red lids. Linguistic Contexts All Relevant, All Irrelevant, No Context
Protocol Experimental normal experiment instructions Annotation checking the work of unaffiliated annotators 4 Implicature Targets, 6 Some/All Controls, 20 Fillers Experiment 1: Social Context Focus on Protocol Annotation vs Experiment All Irrelevant No Story
All-Relevant Experiment Annotation Accuracy Prompt - Is this an accurate description? Response Categories - Yes, No, Dont Know Population: Undergraduates Experiment 1:
Social Context Finding: Social context even when linguistic context does not. Linguistic Context: No Effect Experiment 1: Social Context
Finding: Social context even when linguistic context does not. Lower SI rate for Annotation (p<0.05) Experiment 2 Prompt Type
Accuracy Prompt - Is this an accurate description? Response Categories - Yes, No, Dont Know Informativity Prompt - How Informative is this sentence? Response Categories - Not Informative Enough Informative Enough Too Much Information False Population: Mechanical Turk Workers Systematic Debriefing Survey
Experiment 2 Prompt Type Effect for Prompt Experiment 2 Prompt Type Effect for Prompt (p<0.001)
Effect for Context (p<0.001) Experiment 2 Prompt Type Effect for Prompt (p<0.001)
Effect for Context (p<0.001) Weak Interaction: Prompt x Context (p<0.06)
Experiment 2 Prompt Type No Effect for Protocol Experiment 2 Prompt Type Low SI rates overall But the debriefing survey
indicates that (roughly) 70% of participants were aware of some/all contrast Populations Turkers More sensitive to Linguistic Context Less sensitive to changes in changes in social context/ evaluation apprehension Undergraduates More sensitive to Protocol
Take Home Points Methodological variables should be explored alongside conventional linguistic variables Ideal: models of these processes (cf. Schutze 1996) Crowdsourcing allows for cheap/fast exploration of parameter spaces New Normal: Dont guess, test. Controls, norming, confounding all testable online
A potential check on exuberance Undergraduates may be WEIRD*, but crowdsourcing engenders its own weirdness High evaluation apprehension Uncontrolled backgrounds, skillsets, focus levels Unknown motivations Ignorance does not necessarily mean diversity This requires study if we rely on such participants more
* Heinrich et al. (2010) The Weirdest People in the World? BBS Acknowledgments Thanks Jaye Padgett and to the attendees of two Semantics Lab presentations and the XPRAG conference for their comments, to the HUGRA committee for their generous award and support, and thanks to Rosie Wilson-Briggs for stimuli construction.
I-VIII „Mihai Eminescu" director adjunct CIOBANU GABRIELA 7 Dej Şcoala „Mihai Eminescu" coordonator de programe şi proiecte educative BORA CARMEN 6 Dej Şcoala cu clasele I-VIII nr. 2 coordonator de programe şi proiecte educative BONŢIDEAN LILIANA 5 Dej Colegiul Naţional...
Factors related to the failure of the movie musical: 3. Failure of some big budget musicals. The Sound of Music (1965), a huge success, raised unrealistic expectations; musicals that followed did not give the same return on investment. . .
Also called classless routing or supernetting. Not exclusive of subnetting. Provides additional ways of arranging network and host information in an IP address. Conventional network class distinctions do not exist. Example: subdividing Class C network into six subnets of 30...
Protein Type of sugar unit Glycos- aminoglycan Proteoglycan Oligo- saccharide Glycoprotein Type of sugar unit Peptide Poly- saccharide Peptidoglycan 09/24/2013 Carbohydrates II; Lipids I p. * of 41 Proteoglycans: Glycosaminoglycans Unbranched heteroglycans of repeating disaccharides One component is GalN, GlcN,...
The hands - If you can, give the hands something useful to do. The edge of the hand usually looks better facing the camera as opposed to the back of the hand. Again, it is more slimming this way. Be...
Example. Marriages in the time prior to the Regency Era were easy to conduct and did not require anything more than stated vows by the couple. Marriage was seen as the most holy of unions, it is still viewed that...
The Age of Revolution The period between 1776-1800 was one of tremendous upheaval The old order, the time-honored arrangements of European hierarchies were crumbling under the burdens of maintaining colonial empires, waging wars
Soil Analysis What Is Soil? Mixture of organic and inorganic material May range from 100% inorganic (sand) to nearly 100% organic (peat) Inorganic part is minerals Organic part is decayed plant and animal material and is sometimes called humus Forensic...
Ready to download the document? Go ahead and hit continue!