2018 Surveys and Evaluations September 25, 2018 Denver,

2018 Surveys and Evaluations September 25, 2018 Denver,

2018 Surveys and Evaluations September 25, 2018 Denver, Colorado Agenda 3:00 pm 4:00 pm 2018 Stakeholder Surveys 4:00 pm 4:30 pm 2018 Excavator Outreach Evaluation API RP-1162 Objectives The primary purposes of the evaluation of the Public Awareness Program are to:

1. Assess whether the current program is effective in achieving the objectives for operator Public Awareness Programs as defined in Section 2.1 of the RP, and, 2. Provide the operator information on implementing improvements in its Public Awareness Program effectiveness based on findings from the evaluation(s) Scope of the Surveys Measure 1 Outreach Not addressed - is covered by separate evaluations Measure 2 - Understandability of the Message Is primarily covered by the first group of questions

Measure 3 - Desired Behaviors by Audience Is primarily covered by the questions addressing actions to take and stakeholder confidence ratings Measure 4 Bottom Line Results Not addressed in Stakeholder Surveys Survey Methodology 500 in-depth telephone interviews from each group Emergency Responder (ER) sample universe 27,000 Public Officials (PO) sample universe 60,000 Excavators (EX) sample universe 450,000 Simple random selection computer generated Conducted in first quarter of 2007, 2010, 2014, 2018 Approximately 40 states

Primarily in the western and mid-west states Geographic Area Changes to 2018 Surveys 1. Excavators (table 8) and Emergency Responders (table 10) - respondents in this study were presented a more comprehensive and descriptive list of conditions that may indicate a gas leak than in previous studies. This was intended to more closely match the list of conditions contained in the materials. Changes to 2018 Surveys 2. Excavator Survey question 13 shown in table 10 - the phrase close pipeline valves was revised to attempt

to operate pipeline valves in the list of actions to take in the event of a pipeline leak. This was intended to more closely match the wording in the messages contained in the materials. Changes to 2018 Surveys 3. All Surveys - the responses to the open-ended questions regarding the need for additional information and if they had suggestions for improving communications were listed in the reports and were not grouped into categories as in previous years. The purpose of this change was to provide additional detail about what stakeholders want and to gain further insight into their needs.

Observations 1. Excavators are much less likely to operate pipeline valves in the event of an emergency than they were in 2014. 2. Many areas indicated improvement. However, results varied more than the margin of error for some questions. This may be influenced by the number of new stakeholders included in the previous years mailings as those lists are used for sample selection (i.e. about 40% of the emergency responders were new to the program in 2017). Observations 3. The preference for electronic communications continues to increase.

4. The additional information needed by stakeholders is widely varied. 5. The percentage of stakeholders feeling like they need additional information is declining 6. The percentage of stakeholders offering suggestions for improving the communications is declining Board Recommendations 1. ER initiate email communications with CEMAs containing a link to an online form where stakeholders can submit specific requests for additional information 2. PO - Include postage paid business reply envelope and a reply card in the packet to facilitate specific requests for additional information 3. EX add a space or an item in the readership survey

in the back of the magazine where excavators can request additional specific information Additional Options Review Existing Survey Questions for Improvement Opportunities and Efficiencies Evaluate redundant questions and those that yield non-actionable results Review verbatims to determine any that are research-related or that point to trends that would benefit from inclusion in the survey Review the screening questions and process to prevent unqualified responses that may dilute the

data. Additional Options Enhance Survey Questions and Analysis to Better Assess Behavior Change and Program Impact Include additional behavioral-based questions in surveys (e.g., knowledge check, expected behavior, past behavior, engagement with communication, propensity to share information, etc.) Analyze data by those who recall receiving communication versus those who do not to understand program impact and set the stage for improvement

Additional Options Conduct Pulse Research Conduct more frequent and possibly smaller scale pulse research to better understand audience information needs, knowledge gaps, impact of communication, opportunities for communication enhancements, channel preferences, etc. Introduce qualitative methodologies (e.g., in-depth interviews, focus groups, etc.) to better understand target audience perspective and knowledge gaps Discussion Excavator Outreach Evaluation

Objectives and Scope 1. Determine API RP 1162 Measure 1 - Outreach: Percentage of Each Intended Audience Reached with Desired Messages California Other States 2. Compile information about: List quality Accuracy Completeness 3. Estimate program costs for various list options Methodology 1. Select Sample Counties:

Sutter County, California Mohave County, Arizona 2. Acquire Data: Info USA BB Direct (not used Info USA reseller) Acxiom National Data Group AZ State Licensed Contractors Listing CA State Licensed Contractors Listing MCH data was not used for this analysis Methodology 3. Normalize the Data: Correct and standardize addresses Modify company names

4. Assign Scope Determination: Out of Scope License is inactive Does not excavate (painting, roofing, etc.) In Scope Active license Construction involving excavation Methodology 5. Validate the Records: Considered valid if on two or more lists Unique records were manually validated Phone calls (1375) Web research 6. Determine Total Stakeholder Universe

Unique, valid and in scope records 7. Determine Percentages Current and possible combinations 8. Compare Costs Results: Reach Percent of valid excavators in 2017 mailing AZ Valid Excavators - 1307 InfoUSA Acxiom unique adds AZ Licensed unique adds

NDG unique adds CA Valid Excavators - 497 86 CA Licensed (active) InfoUSA unique adds Acxiom unique adds NDG unique adds

413 107 18 82 580 144 620 100% Cumulative % 38.0% 38.0%

6.6% 44.6% 44.4% 89.0% 11.0% Cumulative 100.0% 100% % 66.6% 66.6% 17.3% 83.9% 2.9% 86.8%

13.2% 100.0% Results: Reach Extrapolated reach for states without contractors lists InfoUSA and Acxiom Total Universe Percent Reach

Sutter County, CA 215 620 35% Mohave County, AZ 583 1,307

45% Results: List Quality Percent of invalid records (includes out of scope, unconfirmed and contractors with an inactive license) AZ List Source Total Invali d Rate Arizona

Contractors 757 5 0.6% Info USA 672 157 23%

1479 Total 436 550 863 Invalid 96 130 58% Rate 22% 24%

Info USA 259 63 24% NDG 654 365

56% Acxiom 125 28 22% NDG CA List Source Acxiom California Contractors

Results: Cost Analysis Quantity of valid and invalid when adding additional list sources AZ Sources Info USA + Acxiom + AZ List Info USA + Acxiom + AZ List + NDG Quantit y

Cost CA Sources Quantit y Cost 1,406 $1,546

CA List +Info USA 420 $462 CA List +Info USA +Acxiom 634

$697 CA List +Info USA +Acxiom +NDG 1,075 $1,182 2,406 $2,646

Board Recommendations 1. Eliminate California contractors with an inactive license status: approximately 21% of California mailing. 2. Add full list of SIC codes from InfoUSA and Acxiom to the California mailing: approximately 37% increase in quantity, increases reach to 87%. 3. Initiate new research effort to identify other data sources for consideration in upcoming mailings. Discussion

Recently Viewed Presentations

  • How to find essay literature - University of Southampton

    How to find essay literature - University of Southampton

    How to find essay literature Siobhan Dodds How to find essay literature Siobhan Dodds Google ,Google Scholar and Google Books Always a good place to start and to come back to!! To get to google scholar / books, get to...
  • The Truth about Bed Bugs - Massachusetts Library System

    The Truth about Bed Bugs - Massachusetts Library System

    The adult female bed bug will lay from 1 to 2 eggs a day if the temperature is between 70° and 80° more or less depending on the temperature. The eggs hatch quickly in 1 to 2 weeks. Bed bugs...
  • The Structure of Sport for People with Disabilities in South ...

    The Structure of Sport for People with Disabilities in South ...

    DISABILITY SPORT SOUTH AFRICA ... and SRSA to include disability as part of "Transformation Charter" SA Games provincial academies Development in SCSA Zone VI: will be working with ASCOD Zone VI MASS PARTICIPATION Rural development project funded by Transnet in...
  • STEP Annual Conference 2017 Graeme Logan, Strategic Director

    STEP Annual Conference 2017 Graeme Logan, Strategic Director

    Despite the plethora of programmes and actions, we wonder whether there is yet enough focus on the fundamental nature of the teaching and learning experience - described in OECD work as the "pedagogical core" - especially as experienced by those...
  • Starbucks and the Sirens - ea

    Starbucks and the Sirens - ea

    Starbucks and the Sirens I Wish I knew how to quit…. Starbucks Craze…. Everyday, hundreds, thousands, no wait; millions of people partake in the Starbucks craze. We love it, we crave it, and we get massive headaches without it!!! Each...
  • Primary Head teacher Seminars Summer 2013

    Primary Head teacher Seminars Summer 2013

    Section 1 to be completed by the social worker. Sections 2 to 8 to be completed by the DT. Section 9 is the joint responsibility of social worker and DT. After the meeting: Record details in each section of the...
  • Advanced Mathematical Methods COMP3006 Introduction to the course

    Advanced Mathematical Methods COMP3006 Introduction to the course

    Advanced Mathematical Methods COMP3006 Introduction to the course Introduction 2 sections Maths-Dr. Karen Page & Statistics -Dr. Simon Prince Maths until reading week Course contact details All communication concerning this course will be done via the email list.
  • Welcome! Parent & Family Session Defining New Roles

    Welcome! Parent & Family Session Defining New Roles

    Talk to your student TODAY about signing the parental release. ... Textbook Rental Program. Parking Permits $30 (Fall stickers available beginning July 5) Tools $$$ Art Supplies $$ Visit the VU Bookstore Today! All About the Money. All About the...