Data-based Decision Making: Understanding, Assessing, and ...

Data-based Decision Making: Understanding, Assessing, and ...

Data-based Decision Making: Understanding, Assessing, and Using Data Slides for Module 10 Topic: Assessing Data Quality and Usefulness Assessing Data Quality and Usefulness 1 Assessing Data Quality and Usefulness Triangulating Data from Multiple Sources Data Reports and Formats Training Objectives Following the training, participants will be able to: 1. Identify and describe 4 factors to consider in

assessing the quality and usefulness of a study or data report 2. Describe the use of triangulation in comparing data from multiple sources and studies 3. Describe at least 5 characteristics of effective HIV data reports prepared for PC/PBs Training Objectives (cont.) 4. Identify at least 4 ways to make data presentations understandable and useful for all PC/PB members 5. Describe at least 3 ways to make data charts understandable and useful for PC/PB members, including those who are not data experts 6. Describe sound practices for receiving and discussing data Data Myths and Realities Consumers dont understand data Many people are uncomfortable using data: more people are

innumerate than illiterate People often think they wont understand HIV-related data and charts because they dont use them every day It takes a Ph.D. to understand RWHAP data People can learn to understand and use data Moderate training is sufficient to understand most RWHAP data if it is well presented Data will give you the answer Data can be spun to give you many different answers Data vary greatly in quality and need to be assessed Assessing Data Quality and Usefulness 2 All data are not created equal so PC/PBs need to assess the quality of the data and reports they receive Assessing Data Quality

PC/PB member roles: Review data from multiple sources Ask questions about how data were gathered, tabulated, and analyzed Compare and weigh data from different sources and studies Decide how much confidence to place in the data Give the greatest weight in decision making to the best data PC/PB Support Staff, Consultants, & Recipient Staff: Provide/present data from various sources Understand and share

information on data quality and limitations Critical Factors for Reviewing or Weighing Data 1. 2. 3. 4. Number of respondents/Size of study Representativeness/Sampling Content/Questions Quality Control Numbers and Representativeness: Who Was Included in the Survey or Study? 1 Numbers: Numbers of people or sample size more weight to data from larger numbers of people

Representativeness: More confidence in data when the individuals sampled were chosen to represent the entire HIV population, subpopulation, or the targeted portion of the community through: Probability sampling Purposive or representative sampling Numbers and Representativeness: Who Was Included in the Survey or Study? 2 Probability sampling: Using a random sampling method where each member of the population has an equal probability of being included, so that findings can be assumed to reflect the entire population from which the sample was drawn Purposive or representative sampling: Selecting people for the study so that they mirror the HIV population in your EMA or TGA or the subpopulation you are targeting Content/Questions and Quality Control

Content/Questions: Look at whether the questions asked were clear and understandable, so that they were likely to generate reliable data that really measure what they were supposed to be measuring Quality control: Look for evidence that the data were collected using appropriate methods and by appropriately trained individuals Data collection process carefully managed/monitored and agreedupon methods implemented Data reviewed for completeness, nonduplication, and data entry where relevant Questions to Ask in Assessing and Interpreting Surveys and Studies Who was responsible for the study? Were knowledgeable consumers and other PLWH involved in design? Does the tool use good questions? Are they clear and understandable? Do they seem likely to generate reliable data that really measure what the study is supposed to be

measuring? Was the tool pre-tested? What was the sample size? Is it representative? What evidence is there that the data were collected using appropriate methods and by trained individuals? Was there quality control to be sure the stated data gathering and analysis process was followed? Terms for Key Measures of Data Quality: Reliability and Validity Reliability involves consistency and repeatability of findings you would get the same results if: You did the study a second time You asked the same person the same question again Validity involves the credibility or believability of your findings they truly represent the phenomenon you are trying to measure The tools used measured what they were supposed to measure (internal validity)

The results can be generalized or assumed to apply to people beyond the sample in the study (external validity) Source: Introduction: Reliability and Validity, UC Davis, Quick Questions A: Assessing Surveys and Studies Your PC/PB had problems with data reliability in its last PLWH survey in 2016 a lot of questions seem to have been misunderstood, and findings were inconsistent with epi data and HIV care continuum data. The survey was done in a hurry, so that findings could be used in preparing your Integrated HIV Prevention and Care Plan. PC/PB staff and members have limited expertise with large-scale surveys, but you want to do a new survey next year. 1. What are some practical things you can do to improve data quality? 2. Who within the PC/PB should lead this effort? Quick Questions B:

Assessing Surveys and Studies Your EMA/TGA has several large subrecipients including a federally qualified health center (FQHC) and a clinic associated with a large teaching hospital. These subrecipients all receive research grants and seem to do a lot of internal assessment of client needs and service quality. You want to encourage them to share research findings with the PC/PB, but you also want to be able to review and assess the quality of their data, as you do for other data. 1. What might be a good process for inviting, receiving, and reviewing such studies? 2. What are some possible concerns? 3. Who within the PC/PB should lead this effort? Common Causes of Unreliable Data (Low Reliability) People dont know or cant remember the answer to the question Example: People may not accurately recall their CD4 count

so survey data on CD4 counts are often not reliable Example: Asked if they have been out of care for more than a year at any point since diagnosis, many PLWH dont remember Too much information is requested Example: A survey is so long or intake form questions are so complicated that people responding get tired and answer later questions with little thought Common Causes of Data Validity Issues (Low Validity) Poorly stated questions Example: A survey asks PLWH whether you have been diagnosed with HIV disease/not AIDS or with AIDS (stage 3 HIV); confusion around AIDS and diagnosed leads people who currently have a high CD4 count to say HIV/not AIDS responses are not valid The sample is small or unrepresentative Example: A focus group of 9 Latina PLWH provides new

information about this population, which the PC/PB wants to use to redesign a service model but the group were all of similar age, all English speakers and their needs are unlikely to represent the needs of all Latina PLWH in the EMA/TGA Quick Questions C: Data Reliability and Validity Issues The PC/PB wants to better understand issues facing recently incarcerated PLWH, who appear to have low retention, medication adherence, and viral suppression rates. When you look closely at RWHAP HIV care continuum data, PLWH survey data, and focus group data, you find that each one used a different definition of recently and incarcerated. You like the focus group definition best (continuously in jail or prison for at least 6 months during the past 3 years), but it included only 9 people, so you arent sure what weight to give the findings. 1. What kinds of data problems are these? 2. What might you do to avoid them?

Triangulating Data from Multiple Sources A PC/PB can have additional confidence in findings found in more than one data set, and can use multiple methods to increase understanding of a topic or issue that process is called triangulation Triangulation of Data The process of comparing data on the same topic from 2 or more sources or research studies to: Cross check or cross-validate the data-- See whether the different sources report similar findings Increase understanding of the topic PC/PBs can have greater confidence in findings that are reported from several different studies or sources or obtained through different methods Approaches to Triangulation

Can involve data from several types of needs assessment activities (such as PLWH survey, focus groups, provider survey) Can also involve data from the recipient or other sources (such as epi data, RSR data on client utilization, HIV care continuum data) Often involves comparing quantitative and qualitative data Comparisons should involve critical review of methods used to obtain data, to decide how much confidence to place in the data from each source Using Triangulation Identify an important question or issue Look at multiple data sources to see what information they provide on the question or issues Compare data to determine similarities and differences Explore possible reasons for differences Assess data sources to determine which should receive the most weight Where findings are similar across sources, especially the

better quality sources, use those findings in decision making Use findings to inform additional data gathering Examples of Triangulation 1 Issue: Treatment of Chronic Health Problems Data Sources Compared: PLWH Survey, Key Informant Interviews and Monitoring Data PLWH Survey: A very high percentage of PLWH from outlying counties in your new survey report serious health problems, especially diabetes and high blood pressure, that are not being adequately treated Interview and Monitoring Data: Key informant interviews and monitoring data confirm that these PLWH often have no primary care physician and infectious disease specialists are seriously overburdened Examples of Triangulation 2 Issue: Barriers to care/linguistic access Data Sources Compared: Focus Group and Provider Survey

Focus Group: A focus group of Latino immigrants reports that some providers have no Spanish-speaking clinical staff and are using receptionists as untrained interpreters Provider Survey: Provider surveys confirm a lack of language capacity and the need for funding for language services Sum Up: Assessing Data Quality and Usefulness All data are not created equal PC/PB members need to be able to ask appropriate questions and weigh data quality PC/PBs need to ask for and receive data in clear and useful formats, including reports and presentations that include user-friendly charts and explanations Triangulation helps in comparing various data sets and identifying similar findings PLWH play a valuable role in assessing data due to their familiarity with the system of care Data Reports and Formats

For maximum value in planning and decision making, even the most reliable and accurate data need to be presented in clear, understandable formats Importance of How Data are Presented Written reports, oral presentation, and charts have huge influence on understanding and use of data PC/PBs can become good users of data through: Use of well designed, clear reports and presentations Sound basic training on data sources and uses Mini-trainings provided along with presentations Allocation of the time necessary to discuss and clarify data PC/PB & recipient commitment to ensuring that all members feel

comfortable asking questions and know they will receive thoughtful, useful answers Preparing Data Reports for PC/PBs A data report prepared for the PC/PB should: Include narrative, tables, and charts Provide information on data sources, sample size and methods, and limitations Use plain language designed for your audience A plain language document looks good, is organized logically, and is understandable the first time you read it Use only necessary technical terms defined & explained Include an Executive Summary Provide comparisons to epi and other data and studies, local and national Be reviewed in draft by the responsible committee Preparing Oral Presentations

Data presentations to the PC/PB should: Use plain language Define new or unfamiliar terms Maximize use of charts with brief text to explain them Avoid data overload Include structured mini-training to maximize understanding of data presented Be previewed by the responsible committee and Executive Committee and revised as needed Be available electronically before the meeting, and copied and distributed at the meeting Preparing Data Charts Charts in reports or presentations to the PC/PB should: Be clear and easy to read color contrast, data labels, numbers large enough to read Provide data in a logical order Include brief text to highlight and explain content Use graphics to highlight key data

Use consistent formats Same type of chart used for same type of data throughout Same colors used for a population or other variable Specify total number and percent of responses/clients Making Data User-Friendly: Example The next two slides illustrate how data can be made easier to understand quickly they present the same data, but: One uses a table, with data presented clearly, but in no obvious order The other uses a bar chart, with data presented in order based on percent of people using each service Sample Data Table: PLWH Survey Services Received During the Past Year (Percent by Service Category) [N=486] Service Category Outpatient Ambulatory Health Services

% Receiving Service 83% Oral Health Mental Health Medical Case Management Medical Nutrition Therapy 55% 23% 92% 43% Housing Assistance Emergency Financial Assistance 19% 27%

Sample Data Bar Chart: Same PLWH Survey Data Services Received During the Past Year (Percent by Service Category) [N=486] Quick Questions D: Different Ways of Presenting Data Review the previous two slides, which present the same survey data in two different formats: as a data table and as a data bar chart. 1. Which one do you find easier to understand quickly? Why? 2. How much of the difference involves the format (table or bar chart), and how much involves other factors? 3. To what extent does your answer to Question 1 involve personal preferences? To what extent does it involve the level of data training you have had? Presenting and Discussing Data Summarize methods, numbers, sampling, limitations as

needed to assess data quality Refine/focus what you present based on your audience and the expected use of the data Highlight and explain key findings If you go beyond the data for interpretation or conclusions, clearly state your assumptions Highlight key issues or uncertainties Allow immediate clarifying questions and discussion at several points and at the end Discussing the Data The presenter and/or PC/PB leadership should: Allocate meaningful time for discussion whenever data are presented Encourage questions about data source and quality Identify and highlight findings that would benefit from diverse input and interpretation Be sure major points of discussion are summarized and included in minutes

Document and follow up on data questions and obtain/present answers Optional Slides for Activities Activity 10.5 Instructions Work with a small group, choosing a facilitator, recorder, and reporter. Be prepared to share your work with the full group. Activity 10.5: Using Triangulation Assume you are a committee of the PC/PB that just carried out the mental health data triangulation and received the data described in the handout, and answer the following questions: 1. What do you see as the key findings? 2. How much weight would you give the information from each source? Why? 3. What, if any, additional data might you want to review or collect before deciding on possible action?

4. How might you eventually use these data in planning? Activity 10.6: Assessing Data Charts and Tables Work in a small group, choosing a facilitator, recorder, and reporter. Review the situation and the data tables, charts, and/ or text summaries assigned to your group, and answer the related questions. Prepare your reporter to present the assigned materials and your review to the full group. Situation A: Sample Formats for Epidemiologic Data The formats presented here all involve tables and charts used to present data that come from Epidemiologic Profiles though they can of course be used for other types of data presentations

Format #1: Race/Ethnicity of People Living with HIV in the EMA by Jurisdiction, 2017 (Percent) N = 12,370 Race/Ethnicity Central City/ County N = 5,690 Suburban County #1 N = 3,831 Suburban County #2 N = 2,581 3 Rural Counties N = 258

Black, not Hispanic 65% 57% 35% 36% White, not Hispanic 16% 22% 37%

61% Hispanic 16% 7% 23% 2% Asian/Pacific Islander <1% 1% 4%

1% Mixed Race/ Other/Unknown 2% 3% 1% 0% Format #2: Proportion of Living HIV Cases, by Race/ Ethnicity, Sex and Mode of Transmission, EMA, 2017 (N=7,243) Format #3: Race/Ethnicity of People Living with HIV in

the TGA, 2017 Race/Ethnicity of People Living with HV in the TGA, December 2017 Format #4: People Living with HIV Disease in the EMA in 2017 A total of 5,828 people are living with HIV disease 47% Black 38% White 12% Hispanic/Latino 60% 30-49 years old 49% MSM 28% White MSM Out of 10 people living with HIV disease, approximately: 5 are Black 4 are White 1 is Hispanic/Latino

8 are male 6 are 30-49 years old 5 are MSM 3 are White MSM Format #5: Trends in New Diagnoses Sample Epi Profile Chart: 5-Year Trends in Newly Diagnosed HIV Cases by Year and Gender Identity, Washington DC Source: Annual Epidemiology and Surveillance Report: Data through 2016, Washington, DC Trend chart with data labels showing percentages of newly diagnosed PLWH by year and by gender identity, plus the total number of PLWH newly diagnosed each year. May seem more complicated but can be used each year. Situation B: Sample Formats for Needs Assessment Data The text slides, charts, and tables presented here come from needs assessment data reports based on PLWH and providers surveys, sometimes supplemented by

focus groups. Format #1: Mental Health & Psychosocial Services Uses: Mental health services are psychological and psychiatric treatment and counseling services offered to individuals with a diagnosed mental illness, conducted by a licensed mental health professional; psychosocial services include support & counseling services such as support groups that may be provided by peers or other non-licensed personnel (support service) Need/gap: 45% of consumers responding to the survey reported depression or other mental health issues in the past year; PLWH indicated a similar level of need for mental health services and peer support/support groups:

63% of consumers surveyed reported a need for mental health services, 52% received such services over the prior year, and 11% of those who needed services did not receive them 60% reported a need for peer support/support groups, 46% received them, and 14% of those needing them did not; concern that some may soon end Satisfaction: 72% of mental health clients and 65% of peer support/ support group clients were very satisfied; satisfaction with both services was somewhat higher in the central city than in other parts of the TGA Format #2: Sample Needs Assessment Data: PLWH Survey Co-Occurring Conditions Reported by PLWH, by Gender Identity (Percent) The most frequently reported co-occurring condition PLWH reported dealing with over the past year is depression and other mental health issues

reported more often by transgender and female PLWH than by men Format #3 Type of Medical Insurance, 2017 (N=3,657) Format #4 Medical-related Services Needed & Needed but Not Received by PLWH (Percent), Past 12 Months To determine percent who received a service, subtract Needed/Not Received from Total Needed Format #5: Most Important Services to Help PLWH Stay in Care As Reported by PLWH & Medical Case Managers Service Case Management Oral Health Services EFA - Rental Assistance EFA - Utilities

Medical care Medications (ADAP) Support Groups Groceries/Hot Meals Transportation Mental Health Services PLWH Ranking 1 2 3 4 5 6 7 8 9 10

PLWH N=265 216 144 123 111 104 103 73 70 70 60 MCM Ranking 1 9 7

10 2 3 6 11 4 5 MCM [N=20] 19 9 12 8 18 17 13 8 15

14 Situation C: Service Expenditures and Utilization Data The text slides, charts, and tables presented here all involve data on services expenditures, costs per client, and utilization data, all provided by the recipient. Framework #1: Targeted and Actual Service Utilization by Service Category, 2017 Service Category Target Actual % Attainment OAHS

1,437 1,608 112% Medical Case Management 1,064 2,098 197% Oral Health 386

302 78% Drug Assistance 339 486 143% 62 188 303% Mental Health

326 906 278% Medical Nutrition Therapy 137 126 92% EFA - Food Vouchers 279

293 105% Home Delivered Meals 215 198 92% Medical Transportation 250 364 146%

Linguistic Services 210 212 101% Professional Services - Legal 60 34 57% Health Insurance Assistance

132 98 74% Substance Abuse Treatment Framework #2: Most Used Service Categories, 2017 Rank Service Category Percentage 1 Outpatient Ambulatory Medical Care

79% 2 Medical Case Management 73% 3 Oral Health Care 29% 4 Medical Transportation 23%

5 Health Insurance Assistance 18% 6 Mental Health 17% 7 Food Bank/Home Delivered Meals 15%

8 Emergency Financial Assistance 14% 9 Medical Nutrition Therapy 8% 10 Substance Abuse Treatment 5% Framework #3: Expenditures, Clients Served, and

Costs per Client for Support Services, 2017 Support Service Category Child Care Emergency Financial Assistance Food Bank/ Home Delivered Meals Legal Services Linguistic Services Medical Transportation Services Psychosocial Support Services Non-Medical Case Management Expended $2,170 $168,197 # Clients Part A Cost Served

per Client 27 $80 323 $521 $215,252 337 $639 $68,066 $14,376 $38,676 $57,494 $286,328 30

73 805 26 348 $2,269 $197 $48 $2,211 $823 Framework #4: Most Used Services for Four Special Populations, 2017 MSM of color (n=934) # %

Outpatient/Ambulatory Health Services 652 70% Medical Case Management 594 Early Intervention Services Recently Diagnosed (n=397) # %

Outpatient/Ambulatory Health Services 266 67% 64% Medical Case Management 241 61% 148 16%

Early Intervention Services 82 21% Mental Health Services 132 14% Mental Health Services 60 15% Medical Transportation

215 23% Medical Transportation 87 22% Receiving Substance Abuse Treatment Services (n=354) # % Latina (n=98)

# % Medical Case Management 74 76% Outpatient/Ambulatory Health Services 326 92% 65

66% Medical Case Management 259 73% Outpatient Ambulatory Health Services Mental Health Services 195 55% Outreach

18 18% Oral Health Care 99 28% Food Bank/Home Del Meals 17 17% Medical Transportation

326 92% Medical Transportation 16 16% Framework #5: Percent of Funds Expended in Each Core Medical Services Category, 2017 One service category was overspent. Two were underspent by more than 20%.

Recently Viewed Presentations

  • Service Provider requirements for 802.11n

    Service Provider requirements for 802.11n

    Service Provider Requirements for 802.11n Detailed Date: 2005-03-16 Authors: Notice: This document has been prepared to assist IEEE 802.11. It is offered as a basis for discussion and is not binding on the contributing individual(s) or organization(s).
  • Risk assessment and control of risks

    Risk assessment and control of risks

    RiskAssess: Improve safety and save time! Phillip Crisp and Eva Crisp Plan Part 1 (P&E) • introduction to risk assessment Part 2 (P&E) • overview and features of RiskAssess website Part 3 (Individual) • hands-on using RiskAssess "Making hydrogen" Part...
  • Balancing State-Space Coverage in Planning with Dynamics

    Balancing State-Space Coverage in Planning with Dynamics

    Safe reciprocal motion coordination. Thank you for your attention! Kostas Bekris' research is supported by: the National Science Foundation (CNS 0932423), the Office of Naval Research, the Nevada NASA Space Grant Consortium and .
  • Public Private Partnership in Tourism

    Public Private Partnership in Tourism

    PUBLIC PRIVATE PARTNERSHIPS IN TOURISM Tourism in Madhya Pradesh: Pilgrimage Heritage Cultural (Recreation and Leisure) Adventure Economic (Nature and Wildlife) Rural Sustainable Tourism - Poverty Alleviation 7 mechanisms suggested by UNWTO: Employment of the poor in tourism enterprise; Supply of...
  • Microbiology: A Systems Approach - Valencia

    Microbiology: A Systems Approach - Valencia

    * Disinfection Use of physical or chemical agents (disinfectant) to kill vegetative bacteria and other microbes except endospores. Example: 10% Bleach solution on bench tops. * Antisepsis Use of physical or chemical agents to kill pathogens on living tissue.
  • Alkanes, Alkenes and Halogenoalkanes

    Alkanes, Alkenes and Halogenoalkanes

    Reactions of Alkanes. Alkanes are relatively unreactive. The only reactions of importance at normal temperatures are: with oxygen (combustion) the reaction is highly exothermic . and so alkanes are widely used as fuels. with the halogens chlorine and bromine
  • System Development Life Cycle (SDLC)

    System Development Life Cycle (SDLC)

    Objectives Describe the system development life cycle. List the benefits of structured programming. Identify the three basic control patterns. Identify the symbols used in creating a flowchart. Create a flowchart that describes a sequence control program. Create an algorithm that...
  • Money: Definitions, Measures, Time value + Introduction to ...

    Money: Definitions, Measures, Time value + Introduction to ...

    Step 2: Use the simple interest formula to calculate the future value of $1. v = ( 1 + r )n * p v = ( 1 + .01 ... ↑ the discount rate ↑ the cost of borrowing for...