Using the National Survey of Student Engagement to

Using the National Survey of Student Engagement to

Using the National Survey of Student Engagement to Assess Educational Effectiveness at AICAD schools AICAD Consortium Meeting Pratt Institute, NY June 12, 2007 Agenda Introduction & NSSE overview What can you learn about your students and their experience from NSSE? NSSE Reports Benchmarking Consortium comparison

AICAD interests: marketing and institutional improvement NSSE details Timeline and Administration Questions What questions do you have right now? Introduction Activity: Assessment at your institution: What do you want to know about your students? Why do you want to know this? What is the purpose of your assessment initiative(s)? To what extent have you used NSSE data?

What is NSSE? Student Engagement The time and energy students devote to educationally purposeful activities and the extent to which institutions emphasize effective practice Engagement is a reliable predictor of student learning and personal development Institution can shape curriculum and resources for learning to promote engagement What Really Matters in College: Student Engagement Because individual effort and involvement are the critical determinants of impact, institutions should focus on the ways they can shape their academic, interpersonal, and extracurricular

offerings to encourage student engagement. Pascarella & Terenzini, How College Affects Students, 2005, p. 602 Foundations of Student Engagement Time on task (Tyler, 1930s) Quality of effort (Pace, 1960-70s) Student involvement (Astin, 1984) Social, academic integration (Tinto,1987, 1993) Good practices in undergraduate education (Chickering & Gamson, 1987) Outcomes (Pascarella, 1985) Student engagement (Kuh, 1991, 2005) Good Practices in Undergraduate Education (Chickering & Gamson, 1987; Pascarella & Terenzini, 2005)

Student-faculty contact Active learning Prompt feedback Time on task High expectations Respect for diverse learning styles Cooperation among students NSSE Survey & Results Survey offers an annual snapshot of student participation in programs and activities that institutions provide for their learning and personal development. Results provide an estimate of how undergraduates spend their time and what they gain from attending college. NSSE items represent empirically confirmed good practices; they reflect behaviors associated with desired outcomes of college. NSSE 2006 Participating Colleges &

Universities by Carnegie Classification 30% 25% 20% National 15% NSSE 2006 10% 5% 0% DRU-VH DRU-H DRU

Master-L Master- Master-S Bac-AS Bac-Div M Core Survey: NSSE Research based on effective educational practices Designed and tested for high validity and reliability Relatively stable over time High credibility of selfreported data Over 275,000 students at 600 institutions annually NSSE Survey Item Organization Q.1 Academic activities Q.2 Learning mental

activities Q.3 Reading & writing Q.4 Homework Q.5 Academic challenge Q.6 Co-curricular activities Q.7 Enriching educational experiences Q.8 Campus relationship Q.9 Time usage Q.10 Institutional emphasis Q. 11 Gains Q.12-14 Satisfaction NSSE Results Are diagnostic; to help institutions

look holistically at undergraduate experience Help pinpoint aspects not in line with mission, or what institution expects Identify weaknesses and strengths in educational program Help institutions know what to focus on to improve student learning and success Questions to answer with NSSE results How many hours per week do first-year students spend studying? Do women study more than men? What % of seniors work with faculty members on activities other than coursework (activities, committees)? Does this differ by major? What % of FY and SR spend 0 hours in cocurricular involvements? Is this more than at peer institutions? Do FY students work more frequently with

classmates on assignments outside of class than their counterparts at peer institutions? Questions to answer with NSSE results Do NSSE results match our mission and what we say about a [INSTITUTION] experience? Are we meeting our own expectations for having a supportive campus environment? Since implementing a new multicultural education initiative and expanding diversity programming, has our score on the diversity scale changed? Are FY who withdraw from the institution different in terms of engagement than students who are retained? How are we performing compared to our select peers (normative benchmarking) or to our institutionally identified standards (criterion benchmarking)? NSSE Deliverables

NSSE Institute Information Institutional Report (August) Using NSSE Data Comparison Reports Accreditation Toolkit Respondent characteristics Data Facilitators Guide (Demographic Information) Means and Frequencies (item averages and response percentages) Benchmarks of Effective Educational Practice Additional Reports (If Applicable) FSSE Report BCSSE Combined Report Data file (student-identified) Sample NSSE results: Frequency comparisons Frequency Comparisons: About how many hours do

you spend in a typical 7-day week participating in cocurricular activities (organizations, campus publications, student government, fraternity or sorority, intercollegiate or intramural sports, etc.) (1=0 hrs/wk, 2=1-5 hrs/wk, 3=6-10 hrs/wk, 4=11-15 hrs/wk, 5=16-20 hrs/wk, 6=21-25 hrs/wk, 7=26-30 hrs/wk, 8=more than 30 hrs/wk NSSEville State Selected Peers Carnegie Peers NSSE 2006 COCU 0 hr/wk 190 61% 2539 43% 3,440 52% # 43% RR01 NSSEville State Selected Peers Carnegie Peers 170 56% 2558 46% 3,341 52% NSSE 2006 50,704 47% 0 Hours on co-curricular activities = 61% FY vs. 56% seniors compared to 43% and 46% at Select Peer Institutions is this what NSSEville expects??

Sample NSSE results: Mean comparisons NSSEville State compared with: NSSEville State Selected Peers Benchmark 1. Effect Class Academic and Intellectual Experiences h. Worked with classmates outside of class to prepare class assignments Worked with faculty members on activities other s. than coursework (committees, orientation, student life activities, etc.)

ACL SFI a a b c Carnegie Peers Effect a b c

NSSE 2006 Effect a b Mean Mean Sig Sizecurrent school Mean year,Sigabout how Size often have Meanyou done Sig each Size In your experience at your institution during the of the following? 1=never, 2=sometimes, 3=often, 4=very often

c FY 2.11 2.34 *** -.27 2.33 *** -.24 2.36 *** -.28 SR 2.52 2.70 ** -.20 2.65 2.75 *** -.26

FY 1.47 1.55 1.59 ** -.14 1.56 SR 1.83 1.83 1.85 1.81 *

-.15 NSSEville State score on 1h. (working with peers outside of class) is significantly LESS than SELECT PEER institutions for FY and Seniors Benchmark Report Level of Academic Challenge Student Faculty Interaction Enriching Educational Experiences Active & Collaborative Learning

Supportive Campus Environment Sample NSSE Results: State Strength Benchmark Report NSSEville significantly higher score Supportive Campus Environment (SCE) for FY and SR on Supportive Campus Environment Benchmark Comparisons NSSEville State compared with: NSSEville State Selected Peers

Carnegie Peers Effect Class First-Year Senior Mean 100 First-Y ear Sig Size 57.8 ** .17 55.8

** .21 a Mean 60.9 59.7 a b c NSSE 2006 Effect Mean a

Sig b Size c 59.4 59.4 75 57.8 59.4 59.1 Selected P eers Carnegie P eers

NSSE 2006 50 25 0 NSSEville State Mean 59.1 56.6 First Year Students 60.9 Effect a Sig

* b Size c .16 AICAD Consortium Questions Value in developing consortium specific questions Compariso n options Potential Data sharing

Establish Core questions & others that rotate in NSSE Use The NSSE data is one among several pieces of information that is used to organize discussions about enrollment management, curricula, retention, and faculty development. Christopher Cyphers, Provost, School of Visual Arts Using NSSE Data Benchmarking longitudinal, criterion, normative Problem

Identification- results point to things institutions can do something about almost immediately Mobilize Action - to change/improve Helps inform decision- Context Setting paint a picture of the institution Evidence of outcomes & processes Refocus conversation about collegiate quality Provides lexicon for talking about collegiate quality in an understandable, meaningful way

Making Sense of Data: Benchmarking Three Approaches: Normative - compares your students responses to those of students at other colleges and universities. Criterion - compares your schools performance against a predetermined value or level appropriate for your students, given your institutional mission, size, curricular offerings, funding, etc. Longitudinal year to year comparison of your students to assess improvement Benchmarking withinExample: 1 AICAD consortium Writing Assessment Issue: Insure high quality writing experiences in the first year. Are we using writing center/tutors effectively? Relevant NSSE items: 1c, d; 3c,d,e; 11c. Provide student learning process & outcome indicators

NSSE results: First year students write short papers comparable to AICAD schools; but fewer med & long papers, are less likely to prepare 2+ drafts & also report lower gain in writing than AICAD peer schools. Interpretation: Benchmarking with AICAD schools indicates institution is underperforming; what other data might you gather to assess writing? What first year writing initiatives might help? What goals might you set for improvement? Example: 2 Benchmarking longitudinal (performance indicators & co-curricular improvements) Assessment Issue: Maintaining effectiveness and making targeted improvements with upper division students Relevant NSSE items: NSSE items identified as key performance indicators, gains items for seniors (11 a-p); and targeted improvements in co-curricular experiences (1h,s,t; 6a; 7b, 10f & diversity scale, 1e,u,v; 10c) NSSE results: Baseline NSSE = 2006, monitor indicators in

2008; assess impact of co-curricular enhancements & diversity initiatives started in 2006 by comparing NSSE 2006 SR to 2008 SR scores. Interpretation: Longitudinal benchmarking (2006-2008); could also benchmark with AICAD schools. Did you meet performance goals? Did the enhancements have an impact? Multi year comparison Using NSSE to market AICAD schools Demonstrate AICAD consortium and institutional strengths (items, NSSE benchmarks) in undergrad program Use results to show mission effectiveness i.e., gains items (11 a-p) & comparison peers show liberal education gains; use consortium results to focus on arts school mission

Provide results to prospective students and families Share results with current students, development office and alumni Institutional Example: NSSE and Enrollment Management The enrollment management area at Meredith has used NSSE results to help guide the enrollment marketing strategies. They look closely at trends and make adjustments to programs and campus visitation days to ensure that students are more cognizant of student involvement and engagement opportunities. An academic dean reports using NSSE information when speaking to parents at an admissions event. "Parents seemed impressed that there was data to support the points that I was making about what we say about the student/faculty relationships and educational

opportunities at Meredith." Institutional Example: Hanover College A detailed summary of NSSE is sent to the faculty as well as the Admission and Student Life staffs to ensure the results, both good and bad, are understood by key folks on campus. Last year, Admission requested an additional presentation and discussion of findings to help them better understand the strengths of the Hanover experience and how that impacts student fit. Helping Students and Families Focus on What Matters to Success Pocket Guide helps prospective students ask the right questions Good questions to ask of all schools, not just those that

participate in NSSE School counselors can request up to 1000 free pocket guides per year. Colleges and non-profit education organizations can request up to 300 copies free per year.* *Request via the NSSE Web site A Pocket Guide to Choosing a College: Are You Asking the Right Questions Connecting NSSE Data to Accreditation Standards Example Accreditation standard: Demonstrate effectiveness of student academic and social support services Evidence for institutional self study: Information about availability and student use of tutoring, writing support, peer study groups, counseling services NSSE indicates FY & SR believe institution emphasizes spending time studying and support for student success;

79% seniors tutored or taught peers; positive correlation between peer collaboration outside of class, satisfaction and first-year retention Positive student satisfaction data about support services Area for improvement - seniors indicate low gains in writing and completing drafts of papers; institution responds with examination of writing requirement in senior capstone and targets seniors for increased use of writing center NSSE and AICAD consortium Consider data sharing agreements Potential for additional comparison studies; prepare papers/presentations; examine shared concerns (retention, outcomes) Use consortium to explore common concerns Coordinate survey schedule Ideas to improve participation rate (incentives, persuasive to promote that survey is occurring at other AICAD schools??) Identify focus for additional questions (up to 20!)

Develop stable core of questions? or change focus? Or a mix? Rotate new questions in? Administration Details What challenges have you faced in your NSSE administrations? What concerns do you have about your next administration? Questions about the details? NSSE Timelines May NSSE/FSSE registration opens September NSSE/FSSE registration deadline NSSE materials due two weeks after

registration confirmation October NSSE pop. files, oversample, and consortium decisions due December FSSE materials and pop. files due Mid-January early February NSSE administrations open BCSSE registration begins (15 mos.) Mid-March early April FSSE administration opens June NSSE & FSSE administrations close BCSSE administration begins at many campuses

August Institutional Reports sent, including raw data and printed reports for NSSE, FSSE, and the prior summers BCSSE BCSSE administration continues September BCSSE data and reports sent to participating institutions NSSE Administration Administration Mode Paper: We need accurate mailing addresses, letterhead, signatures Web+: 4x the paper sample, we need e-mail and mailing addresses Web: 5x the paper sample, we need e-mail

addresses NSSE Administration Sample Size Numbers are based on mode and school size Oversampling can increase sample size or ensure adequate representation of populations of interest NSSE Administration Things that we need from you Contact persons Campus Project Manager (required) Campus Administrative Contact (required) Auxiliary Contact (optional) Population File All First-Year and Senior Students Accurate mailing and/or e-mail addresses Institutional letterhead and signature file (Paper mode only)

NSSE Administration Things for you to consider Broad buy-in from others at your institution (informal word-of mouth) Web-mode institutions: Good partnership with IT department Consortium NSSE Administration Things you to consider (cont.) Administration Plan Follow the IRB rules of Indiana University Bloomington Allowed up to 5 institutional contacts Promotion plan

Incentive programs Tips to boost response rates http:// NSSE: Only one step in assessment process Step #1: Survey Data Step Step#4: #4:Follow-up Follow-up Use Useresults resultsas as benchmarks benchmarkstoto monitor monitorprogress progress Faculty

& Faculty &student student focus groups focus groups Survey students Review results Develop preliminary list of strengths and opportunities for improvement Step #3: Action Plan Finalize plan Share plan with appropriate groups Link to strategic plan Implement action Step#2:

#2:Feedback Feedback Step Shareresults resultswith with Share faculty,administrators administrators faculty, students &&students Identifythemes themes&& Identify priorities priorities Designaction actionplan plan Design

NSSE in your assessment plan How often should I administer NSSE? Every Year: Gives you a snapshot of each class Every Three Years: Gives you a picture of a cohort at the beginning and the end of their college experiences Every Five Years: Works well with most accreditation cycles (Accreditation and Interim Reports) Other factors to consider Establishing a baseline Costs (using all core surveys) Additional Surveys/Sources of Data Time to take absorb results, make changes Updates for 2007 and 2008 No changes to survey content Select up to three customized comparison groups on your reports Electronic report delivery Executive Summary

Report Pocket Guide Report 45 Institutional Examples: Worcester Polytechnic Institute NSSE results showed FY students were less engaged than seniors New FY interdisciplinary, inquirybased seminars; better integration of disciplines; engaging introductory courses Associate Dean appointed to Office for the First Year Assessment plan in development with NSSE indicators as key component Institutional Example: NSSE & Assessing General Education goals

Used NSSE items in 11a-p to assess institutional impact on college-level competencies (a.k.a., indirect measures of student learning outcomes) Undergraduate seniors 2005 NSSE results confirmed findings from 2004 Most seniors (75%+) reported that KSU experience had substantial impact (VM+QAB) in 9 or 16 college-level competencies KSU rank ordered competencies, showing connection to mission, and compared to other masters instit where KSU was sig. higher, comparable, sig. lower on competencies Institutional Example: Program Development and Strategic Planning NSSE results framed a Sophomore Experience

2005 = Paces 5th year of participation Concern regarding SP- JR persistence; FY results offers context for understanding exp. as students enter SP year Established SP Experience Working Group to investigate if FY exp. carried over in SP year. Focused on low NSSE score items, conducted focus groups, created sophomore survey. Led to pilot of Pace Plan (mentoring), includes Career Exploration Course, Sophomore Kick-Off Day NSSE also used in strategic indicators, Accred, NCATE, AACSB, Faculty Development/Colloquia, items used by offices (Technology, Multicultural Affairs), studies performed by Enrollment Mngmt. NSSE suite The NSSElings The Faculty Survey of Student Engagement (2003) The Beginning College Survey of Student Engagement (2004) Additional Surveys The Law Student Survey of Student Engagement The College Student Experiences Questionnaire The College Student Expectations Questionnaire

The High School Survey of Student Engagement* The Community College Survey of Student Engagement* *Not administered by the Center for Postsecondary Research FSSE Faculty perceptions of how often their students engage in different activities Importance faculty place on various areas of learning and development Nature and frequency of interactions faculty have with students How faculty members organize class time FSSE Instrument: Survey Options Course-based (default) Responds to questions based on one particular undergraduate course section during taught during the current academic year

Typical-student Responds to questions based on the typical first-year student or senior taught during the current academic year New! Beginning College Survey of Student Engagement BCSSE Designed for entering firstyear students as a companion to NSSE Measures: pre-college academic and co-curricular experiences expectations for educationally purposeful activities during college BCSSE Instrument Three pilots: 04, 05, and 06 2004 pilot with 28 schools,

15,890 students 2005-2006 pilots with 80 institutions, 39,986 students Study effect of students background on NSSE scores Use to examine gap between expectations and engagement Registration now open Creative Campus initiative and Experimental questions developed by Arts Consortium in NSSE 2007 1 )In your experience at your institution during the current school year, about how often have you done each of the following? (Very Often, Often, Sometimes, Never) a) Attended an art exhibit, play, dance, music, theater, or other performance b) Talked about an art exhibit, play, dance, music, theater or other performance with other students, friends, or family

c) Participated as an artist, performer, or crew member in an art exhibit, play, dance, music, theater, or other performance d) Used your experiences and interest in the visual and performing arts in class discussions or assignments e) Explored a new subject area as a result of your attendance or participation in the arts. 2) To what extent has your experience at this institution contributed to your knowledge, skills, and personal development in the following areas? (Very much, Quite a bit, Some, Very little) a) Developing an understanding and enjoyment of an art exhibit, play, dance, music, theater, or other performance b) Developing a commitment to be involved in the arts (attendance or participation) 3) In a typical 7-day week, about how many hours do you spend in arts experiences on or off campus? [0,

1-5, 6-10, 11-15, 16-20, 21-25, 26-30, more than 30] 4) How many arts courses (art, music, theater, dance) have you taken since coming to college? [0, 1, 2, 3, 4 or more] 5) What motivates you to attend arts events (art, music, theatre, dance)? (Select all that apply) a) Class requirement b) Encouragement from friends c) Enthusiastic professor d) Personal interest e) Proximity f) Quality of the facility g) Opportunity to meet people h) To be involved on campus i) Promotion of the event on campus j) Quality of the event Creative Campus initiative and

Experimental questions developed by Arts Consortium in NSSE 2007 6) What keeps you from attending arts events (art, music, theatre, dance)? (Select all that apply): a) Expense b) Parking availability c) Lack of awareness of events d) Difficulty getting tickets e) Coursework demands f) Facility is rundown g) Location is not convenient h) Job demands i) No one to go with j) Extracurricular activities/meetings k) Lack of quality events l) Limited interest in arts events 7) Select the three (3) places you perceive to be the most active spaces on campus a) sports venues (football stadium, basketball arena) b) public plaza (main campus quadrangle or green)

c) arts venues (performing arts center, museums/galleries, rehearsal halls, art studios) d) student union e) dining hall f) residence hall g) library h) coffee shop or restaurant i) campus main street j) recreation area (fitness center, intramural fields) 8) Where have you attended arts events (art, music, theatre, dance) while attending this institution? Select all that apply. a) Theater buildings (performing arts center, recital hall) b) Museums/galleries c) Coffee shop or restaurant d) Large concert venues (stadiums, arenas, band shell, etc) e) Random places, such as street performances f)\ Residence halls g) Arts festivals

h) Off-campus; downtown or in the local community 9) To what extent have arts experiences at this institution contributed to your abilities in the following areas? [very much, quite a bit, some, very little] a) Thinking critically and analytically b) Thinking imaginatively or creatively c) Understanding fundamental concepts in my major d) Communicating clearly and effectively e) Working effectively with others f) Learning effectively on your own g) Taking intellectual risks 10) To what extent do you agree with the following statements? [strongly agree, agree, disagree, strongly disagree ] a) The arts are visible on my campus b) My institution encourages students to study the arts c) The arts curriculum at my institution is open to everyone d) My institution encourages students to participate in arts events e) The university community values the arts f) The arts contribute to the vitality of this campus

g) The presence of the arts enhances my collegiate experience Discussion and Questions Jillian Kinzie Associate Director, NSSE Institute Indiana University Center for Postsecondary Research 1900 East 10th Street Eigenmann Hall, Suite 419 Bloomington, IN 47406-7512 Ph: 812-856-5824 Fax: 812-856-5150 Web site: E-mail: [email protected]

Recently Viewed Presentations

  • Family Functions - Discrete vs. Continuous

    Family Functions - Discrete vs. Continuous

    Family Functions - Discrete vs. Continuous Discrete A discrete function/graph only consists of certain distinct points - points that can be counted or listed. Continuous A continuous function/graph consists of all points within a given interval (including up to infinity).
  • Tempokomandibularni Zglob

    Tempokomandibularni Zglob

    Kalcificirajući epitelijalni odontogeni tumor (Pindborg) benigni odontogeni tumor koji predstavlja svega oko 1% odontogenih tumora čeljusti. jednako je zastupljen u oba spola, češći je u mandibuli i to u području premolara/molara. često je (u preko 50% bolesnika) povezan s impaktiranim...
  • Animal Origins - Winchester

    Animal Origins - Winchester

    Animal Traits Multicellular Complex patterns of development Single-celled zygote to multicellular adult Heterotroph Internal processes Break down materials from the environment Ingest food into internal gut Most animals move Specialized muscle tissue & body plan Animal Body Plans Body plan...
  • Fixed Co2 Systems

    Fixed Co2 Systems

    sked for installation in main engineering spaces onboard lpd-17 / cvn-76 summary halon is a severe ozone depleting substance all production of halon stopped in 1994 / us navy has 1.6 million lbs of halon 1301 in reserve (30 year...
  • Welcome to 4SZ Class 4SZ Class Teacher Mrs

    Welcome to 4SZ Class 4SZ Class Teacher Mrs

    The PE kit is shorts and a t-shirt for indoor PE.For outdoor PE the P.E. kit is blue tracksuit/jogging bottoms, tracksuit top/sweatshirt, white T-shirt and trainers/plimsolls.. We encourage bare foot work for gymnastics for safety and high quality. No jewellery...


    Hello, this is Moroni Flake with English3. We recently partnered with TAMU and KUMC on their process for evaluating English proficiency for J-1 scholars. And in an effort to better understand the broader challenges among tier-1 research universities, we completed...
  • ASQ Kaizen Presentation Mar 24 2005

    ASQ Kaizen Presentation Mar 24 2005

    Introduction Kaizen Philosophy and Approach Kaizen Toolbox Results from Some Kaizen Events Lessons Learned ("The Bigger Picture") "KAI" - Take apart and make better "ZEN" - Think. Make good the actions of others. Do good deeds.
  • Monitoring Social Inequalities in Health: Integrating ...

    Monitoring Social Inequalities in Health: Integrating ...

    Measuring Social Inequalities in Health: Measurement and Value Judgments Sam Harper McGill University NAACCR Seminar 24 May 2011 This graph shows changes over time in geographic inequalities across the 50 US states in stomach cancer mortality from 1950 to 2000.