tmplate - NSSE Home

tmplate - NSSE Home

Session for NSSE Veterans Regional NSSE Users Workshop October 2005 Shimon Sarraf, Research Analyst Center for Postsecondary Research, Indiana University Bloomington Overview NESSIE A Quick Survey Reports Reviewed Benchmarks Explained 2005 NSSE Dataset Details Future NSSE Developments Discussion & Questions Goals Two-way conversation Share your experiences Good preparation for Shimons second and third session A Quick Survey to Get Things Started. Years of Participation in NSSE Background of attendees The NSSE Reports: Respondent Characteristics A quick snapshot of your institution Data quality: Confirming if sample is representative Response Rate and Sample Error 2005 NSSE Response Rates by Carnegie Classification 100 90 80 79 77 72 Response Rate (%) 70 58 60 53 49 50 40 39 36

33 32 30 20 17 16 14 10 7 3 0 Doc-Extensive Doc-Intensive Masters Mean Min Bac-Gen Max Bac-LA The NSSE Reports: Respondent Characteristics What is Sampling Error? Assumes random sampling An estimate of the margin likely to contain your "true" score, for example: If 60% of your students reply "very often" and the sampling error is 5%, it is likely that the true value is between 55% and 65%. More respondents --> smaller sampling error 2005 NSSE Sampling Error by Carnegie Classification 30 26 Sampling Error (%) 25 20 15 14 11 10 9 8 6 5 5 4

5 4 2 2 2 2 2 Doc-Extensive Doc-Intensive Masters Bac-Gen Bac-LA 0 Mean Min Max The NSSE Reports: Means Comparison NSSE 2005 Means Comparison Report NSSEville State University NSSEville Variable Means, statistical significanc e and effect sizes 1. Benchmark Asked questions in class or contributed to class discussions b. Made a class presentation c. Prepared two or more drafts of a paper or assignment before turning it in d. e. f. Worked on a paper or project that required integrating ideas or information from various sources Included diverse perspectives (different races, religions, genders, political beliefs, etc.) in class discussions or writing assignments Come to class without completing readings or assignments

Selected Peers Master's Effect Class Mean Mean Sig a Size b NSSE 2005 Effect Mean Sig a Size b Effect Mean Sig a Size b In your experience at your institution during the current school year, about how often have you done each of the following? 1=never, 2=sometimes, 3=often, 4=very often Academic and Intellectual Experiences a. NSSEville compared with: CLQUEST CLPRESEN ACL ACL REWROPAP INTEGRAT DIVCLASS CLUNPREP g. Worked with other students on projects during class h. Worked with classmates outside of class to prepare class assignments

OCCGRP i. Put together ideas or concepts from different courses when completing assignments or during class discussions INTIDEAS j. Tutored or taught other students (paid or voluntary) k. Participated in a community-based project (e.g. service learning) as part of a regular course CLASSGRP TUTOR COMMPROJ ACL ACL ACL ACL FY 2.71 2.55 *** .19 2.84 *** -.15 2.86 *** -.18 *** -.18 SR 3.02 2.82 *** .22 3.15 *** -.16 3.16

FY 2.24 1.95 *** .40 2.30 ** -.07 2.28 * .11 SR 2.66 2.57 2.93 *** -.32 2.88 *** -.26 FY 2.45 2.48 2.69 *** -.25 2.65 *** -.21 SR 2.32 2.32 2.55 *** -.24 2.51 *** -.20

FY 2.98 2.86 *** .15 3.08 *** -.13 3.08 *** -.12 SR 3.31 3.19 *** .15 3.36 3.37 * -.08 FY 2.75 2.65 *** .11 2.76 SR 2.74 2.64 * .11 2.83 ** -.11 2.83 ** -.11 FY 2.14

2.19 2.01 *** .17 2.03 *** .15 SR 2.31 2.28 2.05 *** .35 2.08 *** .31 FY 2.36 2.30 2.42 ** -.08 2.40 2.58 *** -.16 2.52 * -.09 SR 2.43 2.37 FY 2.41 2.34 SR 2.82 2.80

FY 2.60 2.49 * .07 * .08 *** .14 2.77 2.39 2.43 2.76 2.77 2.54 * .08 2.57 SR 2.89 2.88 2.91 2.93 FY 1.72 1.72 1.68 1.72 SR 1.86 1.87 1.90 1.94 * -.08 FY 1.39 1.40 1.56

*** -.21 1.54 *** -.19 SR 1.63 1.54 1.80 *** -.19 1.77 *** -.15 * .11 The NSSE Reports: Means Comparison What is Statistical Significance? Helps you answer the question, How likely is it that the difference between my average student and the average student at [comparison group] is due to chance? Significance determined by standard alpha values of p<.05, .01, or .001 The NSSE Reports: Means Comparison Potential problem: As N becomes large, almost everything becomes statistically significant How do we identify truly significant differences? This is a question of practical significance The NSSE Reports: Means Comparison What is Effect Size? Practical significance of the mean difference ES=mean difference/standard deviation .2 is often considered small, .5 moderate, and .8 large (but rare!) For example, while the difference in the means is statistically significant, the difference is so nominal that it doesnt

warrant further attention The NSSE Reports: Detailed Statistics NSSE 2005 Detailed Statistics NSSEville State University First-Year Students Standard Error of the Mean a NSSE 2005 NSSEville Selected Peers Master's NSSE 2005 NSSEville Selected Peers Selected Peers Master's NSSE 2005 Selected Peers Master's NSSE 2005 2.71 2.55 2.84 2.86 .02 .02 .01 .00 .82 .82 .84 .84 1,329 3,004 19,955 47,746 .000 .000 .000 .19 -.15

-.18 CLPRESEN 2.24 1.95 2.30 2.28 .02 .01 .01 .00 .77 .74 .78 .78 1,327 3,001 19,949 47,729 .000 .009 .133 .40 -.07 -.04 REWROPAP 2.45 2.48 2.69 2.65 .03 .02 .01 .00 .97 1.01 .97 .98 1,324 2,999

19,939 47,688 .371 .000 .000 -.03 -.25 -.21 INTEGRAT 2.98 2.86 3.08 3.08 .02 .02 .01 .00 .79 .84 .77 .78 1,324 2,999 19,933 47,691 .000 .000 .000 .15 -.13 -.12 DIVCLASS 2.75 2.65 2.76 2.77 .02 .02 .01 .00 .85

.89 .86 .87 1,326 2,998 19,932 47,678 .001 .841 .324 .11 -.01 -.03 CLUNPREP 2.14 2.19 2.01 2.03 .02 .01 .01 .00 .76 .79 .74 .74 1,324 2,994 19,911 47,644 .053 .000 .000 -.06 .17 .15 CLASSGRP 2.36 2.30 2.42

2.40 .02 .02 .01 .00 .81 .84 .81 .82 1,327 3,001 19,948 47,705 .031 .006 .071 .07 -.08 -.05 OCCGRP 2.41 2.34 2.39 2.43 .02 .02 .01 .00 .81 .86 .85 .85 1,327 3,002 19,948 47,718 .012 .304 .428 .08 .03

-.02 INTIDEAS 2.60 2.49 2.54 2.57 .02 .01 .01 .00 .79 .79 .79 .80 1,232 2,835 19,150 45,820 .000 .010 .191 .14 .08 .04 TUTOR 1.72 1.72 1.68 1.72 .02 .02 .01 .00 .82 .84 .83 .84 1,232 2,835

19,160 45,844 .991 .067 .787 .00 .05 .01 COMMPROJ 1.39 1.40 1.56 1.54 .02 .01 .01 .00 .73 .72 .82 .81 1,232 2,832 19,156 45,828 .573 .000 .000 -.02 -.21 -.19 ITACADEM 2.73 2.72 2.57 2.61 .03 .02 .01 .00 1.01

1.01 1.05 1.05 1,233 2,834 19,160 45,850 .846 .000 .000 .01 .15 .12 EMAIL 3.16 2.99 3.01 3.06 .02 .02 .01 .00 .80 .84 .85 .85 1,230 2,833 19,155 45,839 .000 .000 .000 .19 .17 .12 FACGRADE 2.54 2.43 2.62

2.62 .02 .02 .01 .00 .88 .87 .86 .86 1,232 2,835 19,158 45,834 .000 .004 .001 .13 -.09 -.09 FACPLANS 2.25 1.99 2.18 2.16 .02 .02 .01 .00 .85 .86 .89 .89 1,232 2,836 19,159 45,841 .000 .009 .001 .31 .08

.10 FACIDEAS 1.78 1.70 1.82 1.86 .02 .02 .01 .00 .82 .81 .85 .86 1,233 2,833 19,159 45,829 .004 .114 .003 .10 -.05 -.09 FACFEED 2.69 2.60 2.73 2.76 .02 .01 .01 .00 .75 .78 .80 .80 1,212 2,792

18,968 45,387 .000 .047 .001 .12 -.05 -.09 WORKHARD 2.55 2.46 2.64 2.63 .02 .02 .01 .00 .83 .86 .83 .84 1,210 2,795 18,964 45,391 .005 .000 .000 .10 -.11 -.10 FACOTHER 1.48 1.44 1.62 1.63 .02 .01 .01 .00 .76

.73 .84 .84 1,209 2,789 18,965 45,381 .150 .000 .000 .05 -.16 -.18 OOCIDEAS 2.62 2.68 2.69 2.73 .02 .02 .01 .00 .85 .87 .86 .86 1,211 2,792 18,960 45,380 .034 .006 .000 -.07 -.08 -.13 DIVRSTUD 2.52 2.71 2.52

2.60 .03 .02 .01 .00 1.03 1.00 1.01 1.02 1,212 2,790 18,956 45,366 .000 .969 .008 -.19 .00 -.08 DIFFSTU2 2.76 2.85 2.70 2.77 .03 .02 .01 .00 .96 .94 .97 .97 1,212 2,790 18,959 45,375 .009 .035 .726 -.09 .06

-.01 MEMORIZE 2.96 2.88 2.90 2.85 .02 .02 .01 .00 .83 .86 .87 .88 1,207 2,781 18,859 45,096 .013 .016 .000 .08 .07 .12 ANALYZE 3.12 3.05 3.04 3.09 .02 .01 .01 .00 .74 .76 .79 .78 1,207 2,781

18,849 45,068 .004 .000 .113 .10 .11 .05 SYNTHESZ 2.82 2.81 2.81 2.87 .02 .02 .01 .00 .80 .84 .84 .84 1,206 2,779 18,835 45,049 .714 .628 .065 .01 .01 -.05 EVALUATE 2.78 2.74 2.82 2.84 .02 .02 .01 .00 .83

.86 .85 .85 1,207 2,783 18,838 45,064 .136 .100 .020 .05 -.05 -.07 APPLYING 2.99 2.95 2.96 2.99 .02 .02 .01 .00 .81 .86 .85 .85 1,206 2,783 18,847 45,074 .198 .172 .873 .04 .04 .00 READASGN 3.47 3.32 3.23

3.31 .03 .02 .01 .00 .97 .89 .96 .97 1,197 2,771 18,754 44,895 .000 .000 .000 .16 .25 .16 NSSE 2005 Master's CLQUEST Master's Selected Peers NSSEville compared with: NSSEville Effect size d NSSEville compared with: NSSE 2005 Significance c Master's Number of respondents Selected Peers Mean, N, SEM, SD, p-value, effect size Standard deviation b NSSEville Mean

The NSSE Reports: Detailed Statistics What are Confidence Intervals? CI = Mean +/- 2SEM Multiplying the SEM by 2 creates a margin around the sample mean that is 95% likely to contain the true population mean. More respondents smaller standard error of the mean (SEM), more precise estimate Higher standard deviation greater SEM, less precise estimate The NSSE Reports: Frequency Distributions Counts and percentag es for each response option The NSSE Reports: Frequency Distributions Tip: Consider merging response options to create dichotomous variables (1/0) Frequently = often + very often Substantial = quite a bit + very much The NSSE Reports: New Features Selected Peer Group Including or Excluding your targeted oversample 5 Benchmarks of Effective Educational Practice Level of Academic Challenge Active and Collaborative Learning Student Faculty Interaction Enriching Educational Experiences Supportive Campus Environment Level of Academic Challenge Challenging intellectual and creative work is central to student learning and collegiate quality. Institutions promote high levels of achievement by setting high expectations for student performance. 11 items include: Preparing for class Reading and writing Using higher-order thinking skills Institutional environment emphasizes academic work Active and Collaborative Learning Students learn more when they are more

intensely involved in their education. Collaborating with others prepares students to handle practical, real-world problems. 7 items include: Asking questions in class Making presentations Working with other students on projects Discussing ideas from readings or classes with others Student Interactions with Faculty Interacting with faculty show students first-hand how to think about and solve practical problems. Teachers become role models and mentors for learning. 6 items include: Discussing assignments with a professor Talking about career plans with faculty member or advisor Getting prompt feedback on academic performance Working with a faculty member on a research project Enriching Educational Experiences Students need learning opportunities that complement the goals of the academic program. Provide opportunities to integrate and apply knowledge. 11 items include: Experiencing diversity Using technology Participating in internships Culminating senior experience Supportive Campus Environment

Students perform better and are more satisfied at colleges that are committed to their success. Does institution cultivate positive working and social relationships among different groups on campus? 6 items include: Helping students achieve academically Helping students cope with non-academic responsibilities Quality of relationship between student and peers, faculty, and administrative personnel Benchmarks of Effective Educational Practice How are benchmark scores calculated? 1. Items are converted to a 100-point scale: [(response value 1)/(total # of response values 1)]*100 2. Part-time students' scores are adjusted on four Academic Challenge items. 3. Student-level scores are created for each group of items by taking the mean, as long as 3/5ths of the items were answered. 4. Institutional benchmarks are the weighted averages of the student-level scores. Benchmark Report Level of Academic Challenge 75 Preparing for class (studying, reading, writing, rehearsing, etc. related to academic program) Number of assigned textbooks, books, or book-length packs of course readings 65 Benchmark Scores Challenging intellectual and creative work is central to student learning and collegiate quality. Colleges and universities promote high levels of student achievement by emphasizing the importance of academic effort and setting high expectations for student performance Level of Academic Challenge Items: Number of written papers or reports of 20 pages or more; number of written papers or reports of between 5 and 19 pages; and number of written papers or reports of fewer than 5 pages

55 Coursework emphasizing analysis of the basic elements of an idea, experience or theory 45 Nesseville Coursework emphasizing synthesis and organizing of ideas, Consortium Carnegie National 35First-Year 52.5 52.4 25Senior 56.3 55.6 First-Year Senior Nesseville 52.5 56.3 Consortium 52.4 55.6 Carnegie 51.8 54.9 National 53.4 57.0 information, or experiences into new, more complex interpretations and relationships 51.8 53.4 Coursework emphasizing the making of judgments about the value of information, arguments, or methods 57.0 Coursework54.9 emphasizing application of theories or concepts to practical problems or in new situations Working harder than you thought you could to meet an instructor's standards or expectations Campus environment emphasizing time studying and on academic work Benchmarks of Effective

Educational Practice Benchmark recalculation reports (04 & 05): Driven by new calculation process that began for the 04 administration Multi-year Comparisons EEE: not comparable because of response set change SFI: comparable by removing research item Benchmarks of Effective Educational Practice Institutions can use the student-level scores to: Investigate what groups are more engaged than others on your campus. Institutional subgroups (i.e., programs, departments) Student sub-groups (i.e., gender, race) Incorporate scale scores into predictive models of student outcomes (retention, g.p.a., satisfaction) Measurement Scales Satisfaction General Satisfaction Satisfaction plus Quality of Campus Relationships Campus Environment Environmental Emphases Quality of Campus Relationships Gains Factors Personal/Social General Education Practical Competence Deep Learning Activities Higher-Order Learning activities that require students to utilize higher levels of mental activity than those required for rote memorization (2b,c,d,e) Integrative Learning activities that require integrating acquired knowledge, skills, and competencies into a meaningful whole (1d,e,i,p,t) Reflective Learning activities that ask students to explore their experiences of learning to better understand how they learn NSSE Scalelets Course Challenge Varied Experiences Writing Information

Active Learning Technology Collaborative Learning Diversity Support for Student Course Interaction Out-of-Class Interaction Gains (academic, personal, vocational) Success Interpersonal Environment 2005 NSSE Dataset Details What do you need to know to match your Institutional Report numbers? smpl05 (sampling type) use 1 (base sample), 2 (standard oversample), 3 (requested oversample) values and 4 (targeted oversample) if targetos equals 1 inelig exclude all ineligible respondents use those with values of 1 for inelig Future Developments Customized Report Engine Archiving of reports and datasets online Integrating new Carnegie classification NSSE Knowledge Base Discussion and Comments Shimon Sarraf Research Analyst Indiana University Center for Postsecondary Research 1900 East 10th Street Eigenmann Hall, Suite 419 Bloomington, IN 47406 Ph: 812-856-2169 [email protected] www.nsse.iub.edu

Recently Viewed Presentations

  • Review of Skills Covered in the Summer Assignment

    Review of Skills Covered in the Summer Assignment

    How many meters is that? How many centimeters? How many megameters? A flash of light lasts 0.00056 seconds. How many milliseconds is that? How many microseconds? An object has a mass of 75kilograms. How many grams is that? A molecule...
  • Designing and Proportioning Concrete Mixtures

    Designing and Proportioning Concrete Mixtures

    Designing and Proportioning Concrete Mixtures. ... The data used for proportioning should also be from concrete with an c. that is within 7 MPa (1000 psi) of the strength required for the proposed work. Also, the data should represent at...
  • The Interaction between the Stock Market and Consumption: A ...

    The Interaction between the Stock Market and Consumption: A ...

    Literature Review;Stock Market Wealth Effect. Starr-Mc Cluer (2002)examined the impact of stock market wealth on consumer spending in the USA. The results were broadly consistent with the life cycle savings view that increases in wealth will result in small to...
  • Slide 4

    Slide 4

    Kensington Village Retirement/Nursing Home. AUGUST 2016. 2014Ai. Kensington Village Mission Statement: Providing "Resident First" Care. GREETINGS FROM
  • Indicator 7: Measuring Preschool Outcomes

    Indicator 7: Measuring Preschool Outcomes

    Indicator 7: Measuring Preschool OutcomesEntry Data Collection Using the COS Process. Hello and thank you for joining me for this overview of Indicator 7 and the child Outcomes Summary Process. This is one video in a series of 4. This...
  • Presentation and Validation of The Difi L2 Product

    Presentation and Validation of The Difi L2 Product

    Other groups (ITSG, AIUB) obtained similar results by using different inversion methods. Grace A/B:Institute of Geodesy (IfG),Graz University of Technology; Swarm monthly solutions: TU Delft. Validation . Cross-validation between comprehensive and dedicated models.
  • Gas Advisory Council EU-Russia Energy Dialogue Workstream on ...

    Gas Advisory Council EU-Russia Energy Dialogue Workstream on ...

    Third EU Energy Package in the context of EU-Russia energy cooperation. Presentation at the International Conference "Gas: Russia, Poland, Europe", Session on "Law & Business: Russian and the EU", Warsaw, 15-16 March 2012(Based on Walter Boltz's & Andrey A.Konoplyanik's joint...
  • William Stallings Data and Computer Communications

    William Stallings Data and Computer Communications

    Times New Roman Tahoma Arial Wingdings Blends CMPE 150 - Winter 2009 CMPE 150 -- Introduction to Computer Networks Text Reference Other Networking Courses Syllabus Grading Academic Integrity Some Current Topics NAS Network Attached Storage Computer Networking Computer Networking Computer...