Disability Services Research for Outcomes Monitoring and Outcomes
Disability Services Research for Outcomes Monitoring and Outcomes Improvement Data = Dollars Sheryl Ballenger, PhD Zerrin Ondin, PhD AMAC Accessibility Solutions and Research Center Started 2005, social entrepreneurship initiative
Complete, timely, efficient access services and research for students with printrelated disabilities. 1-stop shop: economies of scale, competitive prices December 2015: UN Award for
Leadership in Digital Accessibility for providing digital textbooks in accessible formats for over a decade. Who we are? Sheryl Ballenger, PhD Captioning and Described Media Services Manager AMAC Accessibility Solutions and Research Center College of Design, Georgia Tech Zerrin Ondin, PhD Research Scientist AMAC Accessibility Solutions and Research Center College of Design, Georgia Tech
Who are you? Now, it is your turn! Where do you work? What do you do? What do your administrators ask for?
What is your experience in conducting research? What are you expecting to get out of this session? Objectives of this session This session will provide an overview of research methods applicable for disability service providers, assistive technology, and accessibility specialists to monitor and improve outcomes of the services they provide. Upon attending this session, you will be able to: 1. Determine how to seek ideas for research projects that may benefit your
services, students, and your institution 2. Recognize ethics of studying people and when approval is needed from the Institutional Review Board and the steps to secure IRB approval 3. Develop a survey to collect data 4. Analyze and report the data you collected Why wait? Feel free to ask questions anytime... Defining the scene What Constitutes Research?
Questions get answers Investigate systematically What is outcomes monitoring and outcomes improvement research? Outcome monitoring and improvement research generally includes service providers collecting data from students via surveys or interviews to investigate how efficient their services were and gathering suggestions. Research Ethics
University cooperation with a for-profit company providing the service may only benefit one of these partners. Ensure research is answering gap question, not serving other purposes. Participants in research are volunteers and are permitted to not be a participant without retaliation.
Even in small sample sets, provide anonymity. Take great care to be honest and readily admit uncertainty. "...fakers and phonies who are attempting by what they say to manipulate...opinions." -H. Frankfort (2005) Introducing the case Say hi to John! John is working at: Georgia Tech He is an:
Information Technology Accessibility Manager He wants to know: What accommodations / AT are his students really using? What is your case? Please think about what you want to work on throughout this session. It might be something you are currently working on It might be something you would like to do for your office
You can work in groups You may work on Johns case. Writing Research Questions What is a Research Question? In order to find an answer, you must begin with a question.... Research question is the question your research project aims to answer. Research questions provide the key for planning and carrying out a successful
research project. May begin broad and then refine until it is concise and measurable. "I want world peace" may become I want to know if students "Have a quiet study place." The Value of Research Questions The value of research questions Define the project: a good set of questions summarizes in a few sentences what your project is concerned with Set boundaries: stops you from spending time on things not relevant to the questions
Give direction: helps you focus your efforts (searching literature, data gathering, method selection, analysis) Define success: has your project resulted in credible answers to the research questions? How to Write a Research Question You have considerable freedom in writing your research questions. It is always a good idea to remember three purposes of research: 1. to explore
2. to describe 3. to explain Pick one for your case. Research Question Examples Exploratory Study What are the experiences of students using the Kurzweil reading tool? Descriptive Study What are students' views about the Kurzweil reading tool? To what extent are teachers involved in and supportive of usage of the Kurzweil reading tool? Explanatory Study Do students read better with the Kurzweil reading tool?
Characteristics of Good Research Questions Good research questions: are clear and unambiguous; show the purpose(s) of your project (to explore, describe, explain); are answerable and point to the type of data needed to provide answers; are not trivial; form a coherent interconnected set (they are not an apparently random collection). Johns Research Questions He wants to know: What accommodations / AT are his students really using? His research questions are: What are the accommodations students used actively this
school year? What are the AT tools students used actively this school year? Your Turn Work individually or with your team and write a research question you want to answer with your research project then share it with us. Boring alert IRB Institutional Review Boards (IRBs) are responsible for reviewing the scientific, legal, and ethical merits of a human subjects research protocol before the
research starts and while the research is ongoing. An IRB is a federally-mandated panel that is charged with safeguarding the rights and welfare of human subjects in research. IRB approval is required in advance for all research projects that include human subjects. IRB approval must be obtained before data collection procedures start. Before applying for IRB approval, please make sure that your project is a human research study. IRB Research means a systematic investigation, including research development, testing, and evaluation, designed to develop or contribute to generalizable knowledge. Activities which meet this definition constitute research for purposes of this policy, whether or not they are conducted or supported under a program which is considered research for other purposes. For
example, some demonstration and service programs may include research activities. Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains (1) Data through intervention or interaction with the individual, or (2) Identifiable private information. IRB IRB IRB Office Office IRB "to do" List
1. Locate your institution's IRB website and review it 2. Learn how to get required human subjects training specific to your institution 3. Learn if there is any training opportunity to assist with IRB submission Workshop One-on-one consultation 4. Write a summary of your research idea and ask them if it requires an IRB approval Survey Development Survey & Questionnaire Development Typically, surveys gather data at a particular point in time with the intention of describing the nature of existing conditions, or identifying standards against which existing conditions can be compared, or determining the relationship that exist
between specific events (Cohen, Manion & Morrison, 2011). Examples of surveys are Opinion polls Students preferences for particular courses Estimate of time spent while using Dragon Dictate Some preliminary considerations* Goal of your survey Purpose of your research Target population
Who is going to fill out the survey Resources available Time, money, staff, tools Mode of data collection Researcher entering the data, or self-administered? Mail, telephone, internet, face-to-face interviews, online? *Note: If IRB approval is required, you will need this info for completing the IRB form. Challenges
Poor sampling (respondents) Sending survey to people who are not likely to approximate the characteristics of the target population Poor question design and wording Examples are provided in following slides Incorrect or biased responses
Due to poor question design and wording Low response or non-response Usually unavoidable You may consider providing incentives Sending reminders (up to 3) is a good idea Response Scale Options Two major question types: Closed and open-ended questions
Scales of data 1. Dichotomous 2. Multiple choice 3. Rating scales 4. Rank ordering 5. Open-ended questions Response Scale Options Explained 1. Dichotomous Two answer options
Are you a registered student? Yes / No 2. Multiple choice More than two answer options Please check computer software you are using Word / Excel / PowerPoint / One Note 3. Rating scales
Requires respondents to assign a value to the rated object How frequently do you use screen magnification tool? never / seldom / sometimes / often / always Response Scale Options Explained 4. Rank ordering Requires respondents to compare a list of different objects to one another Please rank the following features in order of importance, where 1 is the most important to you.
Location / comfort / service / value for money 5. Open-ended questions Allows respondents to give free-form answer Please describe your level of experience with screen readers. Empty space for respondents to write Avoiding Pitfalls in Question Writing 1. Leading question forces respondents to answer in a particular way Should concerned parents use infant car seats?
Do you think special car seats should be required for infant passengers? 2. Loaded question forces respondents into an answer that doesnt accurately reflect their opinion or situation Where do you like to party? What do you like to do on weekend evenings?
3. Double-Barreled question - force respondents to answer two questions at once How happy or unhappy are you with the rate of current school board funding and the common-core curriculum? How happy or unhappy are you with the rate of current school board funding? What do you think of the common core?
Avoiding Pitfalls in Question Writing 4. Absolute question using yes/no answers can keep respondents from leaving unbiased feedback Do you always eat breakfast? (Yes/No) How many days a week do you usually eat breakfast? (Every day/ 5-6 days/ 3-4 days/ 1-2 days/ I usually dont eat breakfast) 5. Unclear question - using tech jargon and acronyms
Do you own a tablet? Do you own a tablet (e.g. iPad, Android tablet)? Advice for Question Writing When creating questions keep in mind: Attempt to offer questions that do not require the respondent to contemplate their response very long. Your goal is to provide questions that respondents will be: 1. willing to answer
2. able to respond accurately 3. interpret in the way you intend Middle Response Dilemma When a range of options are offered, often respondents believe the middle response is the average- they may base their answer on average. Guess answer C on multiple choice test. Activity Let's work on below survey questions together.
Q1. Do you have any problem with your academic advisor? Q2. What is your experience using Kurzweil and Livescribe? Q3. Where do you go on campus when you feel stressed? Q4. How satisfied are you with the accommodations provided to you this current semester? The Most FAQ How many questions should I ask?
It depends on the purpose of your survey Always consider how long it will take to finish the survey Limit demographic questions if you are not going to use them Too many open-ended questions on self-administered surveys may cause noresponse Less is more Time to Begin! Survey Delivery and Recruitment First, choose a method to deliver your survey Online E-mail Online Survey Tool (Survey Monkey, Surveygizmo, Qualtrics) By snail mail
In-office handout Survey Delivery and Recruitment Advantages Disadvantages E-mail Low cost Potential high speed of returns
Need for comprehensive email list Potential risk of nonresponse Online Survey Tool Easy to administer Potential high speed of returns
Question skip patterns Data analysis tools Mail
Relatively low cost Can be provided in alternative format if necessary (like Braille) Response rates can be quite low Need for accurate mailing addresses In-office handout
Low cost High response rate Presence of the data collector Presence of the data collector Respondent may feel rushed
Need for comprehensive email list Potential risk of nonresponse Price Survey Delivery and Recruitment Telephone is not being used as it was in the past. Disruptive On researchers' time, not the participants' SurveyGizmo & Survey Monkey Please visit below links http://www.surveygizmo.com/s3/3922401/AHG2017
https://www.surveymonkey.com/r/abcolassistivetechstudent Accessibility of Surveys Provide multiple means of survey format Online survey tool, do the evaluation in advance Related guidelines https://www.w3.org/WAI/intro/wcag (WCAG)
https://www.access-board.gov/guidelines-and-standards/communic ations-and-it/about-the-section-508-standards/section-508-standard s (Section 508) Online accessibility information for online survey tools might not be sufficient and/or necessarily accurate. Check first! Survey Delivery and Recruitment Who to send? Depends on the purpose of the survey Depends on any specific regulation your instruction may have for accessing students for data collection Generally speaking, send it to all students registered to your office (your
population) Active vs inactive students Analyzing the Data How did you collect the data? via online survey tool Use tool's "results" feature via e-mail, mail, or in-office handout We suggest that you enter all data into Microsoft Excel
Analyzing the Data What type of data analyzing you need to use: It depends on your research question(s) and survey questions Most of the time descriptive statistics* is sufficient *Just describe in writing what the results show What is Descriptive Statistics Descriptive statistics are used to describe the basic features of the data in a
study. They provide simple summaries about the sample and the measures. Together with simple graphics analysis, they form the basis of virtually every quantitative analysis of data. With descriptive statistics you are simply describing what is or what the data shows. Measures of Descriptive Statistics Measures of distribution
Measures of central tendency The distribution is a summary of the frequency of individual values or ranges of values for a variable. The simplest distribution would list every value of a variable and the number of persons who had each value. Simple measure: frequency distribution. The central tendency of a distribution is an estimate of the "center" of a distribution of values. There are three major types of estimates of central
tendency: mean, median, mode. Measures of dispersion Dispersion refers to the spread of the values around the central tendency. There are two common measures of dispersion, the range and the standard deviation. Frequency Distribution Q1: Smartpen is helpful for my classes. Degree of agreement
Frequency Q2: How many hours you spend weekly for your assignments? Hours spent Number Strongly agree 20 Less than 5
5 More than 20 2 Mean, Median, Mode Question: What is your age? Answers: 20, 18, 21, 18, 21, 19, 21, 22, 21 Mean=sum of all the answers / number of answers Mean=(20+18+21+18+21+19+21+22+21)/9= 20.1 Median=median score 18, 18, 19, 20, 21, 21, 21, 21, 22 Mode=the most frequent value in your data set
21 Range Question: What is your age? Answers: 20, 18, 21, 18, 21, 19, 21, 22, 21 Range is the difference between the highest data value and the lowest data value 22-18=4 Presenting the Results Outline for a research report 1. Introduction 1. Problem Statement (NM*) 2. Purpose of the Study 3. Research Questions
4. Rationale for the Study / Significance of the Study (NM*) 2. Literature Review (NM*) 3. Methodology 1. Research Design 2. Participants 3. Survey Development 4. Data Collection 5. Data Analysis 4. Findings 5. Summary, Discussion, and Conclusion 6. References 7. Appendices *NM: Not Mandatory
Real Life Terminology a.k.a. Plain language Plain Writing Act of 2010 http://www.plainlanguage.gov/plLaw/index.cfm Use common words. If you must use unfamiliar words, define them for the audience. Avoid confusing words and phrases. Avoid or explain acronyms, jargon, and idioms. Presenting the Results
Everything depends on your purpose and your audience (to whom you will present this report) Usually 7-10 pages Include tables and graphs Explain tables and graphs shortly (one or two sentence) Make it accessible AMAC has a MOOC on ICT Accessibility (EdX platform) Q&A
H.B. 1927 (Bonnen, G.): Requires that the county clerk provide a list of voters that have submitted an Annual ABBM to the political subdivisions in their county that are holding an election. In turn, the early voting clerk of the...
Poetry and piety have begun to fail, ... Sibilance adds effect and emphasis. ... sees how man is destroying it, and these beliefs are portrayed in both the poems. Also Report to Wordsworth is a summary of all the problems...
Referred to as 'midnight clause' - lack of deliberation and thought on the drafting of the clause. Dismissed as 'boilerplate' and given standard wording. They should be customized depending on the needs of the parties and the specific agreements. Privileged...
65 y/o man presents with a 2 month history of fatigue and weight loss. FBC shows he has a haemoglobin of 60g/l (normal 130-180g/l) What does this tell you? The Peer Teaching Society is not liable for false or misleading...
Identifying positive heath benefits. Coping with negative symptoms. Areas of Interest (cont.) ... Information regarding schools with a health psychology emphasis. Information on which courses you should take to get started pursuing a career in the health psychology field.
Marko Porcije Katon. Rođen u Tuskulu (sabinsko područje) 234.g.pr.Kr. u plebejskoj obitelji . homo. novus. u Rimu kamo je došao s 20-ak godina. Borio se u II. punskom ratu, dio mladosti proveo radeći na farmi