Pedagogy and Instructional Design Part III: Courseware ...

Pedagogy and Instructional Design Part III: Courseware ...

5. Online Assessment and Evaluation Practices Dr. Curtis J. Bonk Indiana University and [email protected] Online Student Assessment Assessment Takes Center Stage in Online Learning

(Dan Carnevale, April 13, 2001, Chronicle of Higher Education) One difference between assessment in classrooms and in distance education is that distanceeducation programs are largely geared toward students who are already in the workforce, which often involves learning by doing. Focus of Assessment? 1.

2. 3. 4. Basic Knowledge, Concepts, Ideas Higher-Order Thinking Skills, Problem Solving, Communication, Teamwork Both of Above!!! Other Assessments Possible

Online Portfolios of Work Discussion/Forum Participation Online Mentoring Weekly Reflections Tasks Attempted or Completed, Usage, etc. More Possible

Assessments Quizzes and Tests Peer Feedback and Responsiveness Cases and Problems Group Work Web Resource Explorations &

Evaluations Sample Portfolio Scoring Dimensions (10 pts each) (see: 1. 2. 3. 4. 5. 6. 7.

8. Richness Coherence Elaboration Relevancy Timeliness Completeness Persuasivenes s Originality 1. 2.

3. 4. 5. 6. 7. 8. 9. Insightful Clear/Logical Original Learning Fdback/

Responsive Format Thorough Reflective Overall Holistic E-Peer Evaluation Form Peer Evaluation. Name: ____________________ Rate on Scale of 1 (low) to 5 (high): ___ 1. Insight: creative, offers analogies/ examples, relationships drawn, useful ideas and connections, fosters growth. ___ 2. Helpful/Positive: prompt

feedback, encouraging, informative, makes suggestions & advice, finds, shares info. ___ 3. Valuable Team Member: dependable, links group members, there for group, leader, participator, pushes group. ___ Total Recommended Contribution Pts (out of 15) E-Case Analysis Evaluation Peer Feedback Criteria (1 pt per item; 5 pts/peer feedback) (a) Provides additional points that may have been missed.

(b) Corrects a concept, asks for clarification where needed, debates issues, disagrees & explains why. (c) Ties concepts to another situation or refers to the text or coursepack. (d) Offer valuable insight based on personal experience. (e) Overall constructive feedback. Issues to Consider 1. 2. 3.

4. 5. Bonus pts for participation? Peer evaluation of work? Assess improvement? Is it timed? Allow retakes if lose connection? How many retakes? Give unlimited time to complete? Issues to Consider 6.

7. 8. 9. 10. Cheating? Is it really that student? Authenticity? Negotiating tasks and criteria? How measure competency? How do you demonstrate learning online?

Increasing Cheating Online ($7-$30/page, January, 2002, Phillip Long, Plagiarism: IT-Enabled Tools for Deceit?) (youll never buy Cliffnotes again) Reducing Cheating Online

Ask yourself, why are they cheating? Do they value the assignment? Are tasks relevant and challenging? What happens to the task after submittedreused, woven in, posted? Due at end of term? Real audience?

Look at pedagogy b4 calling plagiarism police! Reducing Cheating Online

Proctored exams Vary items in exam Make course too hard to cheat Try ($300) Use mastery learning for some tasks Random selection of items for item pool Use test passwords, rely on IP# screening Assign collaborative tasks Reducing Cheating Online ($7-$30/page, January, 2002, Phillip Long, Plagiarism: IT-Enabled Tools for

Deceit?) (resource) (software, $100,

free 30 day demo/trial) (software; essay verification engine, $19.95) (free database of 70,000 student term papers & cliff notes) (assoc.) (guide) Turnitin Testimonials "Many of my students believe that if they do not submit their essays, I will not discover their plagiarism. I will often type a paragraph or two of their work in myself if I suspect plagiarism.

Every time, there was a "hit." Many students were successful plagiarists in high school. A service like this is needed to teach them that such practices are no longer acceptable and certainly not ethical! Online Testing Tools Test Selection Criteria (Hezel, 1999)

Easy to Configure Items and Test Handle Symbols Scheduling of Feedback (immediate?) Provides Clear Input of Dates for Exam Easy to Pick Items for Randomizing Randomize Answers Within a Question Weighting of Answer Options More Test Selection

Criteria Recording of Multiple Submissions Timed Tests Comprehensive Statistics Summarize in Portfolio and/or Gradebook Confirmation of Test Submission

More Test Selection Criteria (Perry & Colon, 2001) Supports multiple items typesmultiple choice, true-false, essay, keyword

Can easily modify or delete items Incorporate graphic or audio elements? Control over number of times students can submit an activity or test Provides feedback for each response More Test Selection Criteria (Perry & Colon, 2001)

Flexible scoringscore first, last, or average submission Flexible reportingby individual or by item and cross tabulations. Outputs data for further analysis Provides item analysis statistics (e.g., Test Item Frequency Distributions). Web Resources on Assessment 1.

2. 3. best_suggested_links.shtml Rubric for evaluation technology projects: rubric.htm

Online Survey Tools for Assessment Sample Survey Tools

Zoomerang ( IOTA Solutions ( QuestionMark ( SurveyShare (; from Survey Solutions from Perseus (

Infopoll ( Web-Based Survey Advantages

Faster collection of data Standardized collection format Computer graphics may reduce fatigue Computer controlled branching and skip sections Easy to answer clicking Wider distribution of respondents Web-Based Survey Problems: Why Lower Response Rates?

Low response rate Lack of time Unclear instructions Too lengthy Too many steps Cant find URL Survey Tool Features

Support different types of items Maintain email lists and email invitations Conduct polls Adaptive branching and cross tabulations Modifiable templates & library of past surveys Publish reports Different types of accountshosted, corporate, professional, etc.

(Likert, multiple choice, forced ranking, paired comparisons, etc.) Web-Based Survey Solutions: Some Tips

Send second request Make URL link prominent Offer incentives near top of request Shorten survey, make attractive, easy to read

Credible sponsorshipe.g., university Disclose purpose, use, and privacy E-mail cover letters Prenotify of intent to survey Tips on Authentification Check e-mail access against list Use password access Provide keycode, PIN, or ID #

(Futuristic Other: Palm Print, fingerprint, voice recognition, iris scanning, facial scanning, handwriting recognition, picture ID) Evaluation Champagne & Wisher (in press) Simply put, an evaluation is concerned with judging the worth of a program and is essentially conducted to aid in

the making of decisions by stakeholders. (e.g., does it work as effectively as the standard instructional approach). Evaluation Purposes Cost Savings Improved Efficiency/Effectiveness Learner Performance/Competency Improvement/Progress

What did they learn? Assessing learning impact How well do learners use what they learned? How much do learners use what they learn? Kirkpatricks

Levels Reaction Learning Behavior Results 4 Percent of Respondents Figure 26. How Respondent Organizations Measure Success of Web-Based Learning According to the

Kirkpatrick Model 90 80 70 60 50 40 30 20 10 0 Learner satisfaction

Change in knowledge, skill, atttitude Job performance Kirkpatrick's Evaluation Level ROI My Evaluation Plan Considerations in Evaluation Plan 8. University or

Organization 7. Program 6. Course 5. Tech Tool 1. Student 2. Instructor 3. Training 4. Task What to Evaluate? Studentattitudes, learning, jobs. 2. Instructorpopularity, survival.

3. Trainingeffectiveness, integratedness. 4. Task--relevance, interactivity, collab. 5. Tool--usable, learner-centered, friendly, supportive. 6. Courseinteractivity, completion. 7. Programgrowth, model(s), time to build. 8. Universitycost-benefit, policies, vision. 1. 1. Measures of Student Success (Focus groups, interviews, observations, surveys, exams, records)

Positive Feedback, Recommendations Increased Comprehension, Achievement High Retention in Program Completion Rates or Course Attrition Jobs Obtained, Internships Enrollment Trends for Next Semester

1. Student Basic Quantitative Grades, Achievement Number of Posts Participated Computer Log Activitypeak usage,

messages/day, time of task or in system Attitude Surveys 1. Student High-End Success

Message complexity, depth, interactivity, qing Collaboration skills Problem finding/solving and critical thinking Challenging and debating others Case-based reasoning, critical thinking measures Portfolios, performances, PBL activities 2. Instructor Success Technology training programs Funding adequate Utilize Web to share teaching

Positive attitudes, more signing up Course recognized in tenure decisions Understands how to coach 3. Training Outside Support

Training ( Courses and Certificates (JIU, e-education) Reports, Newsletter, & Pubs (e.g., surveys) Aggregators of Info (CourseShare, Merlot) Global Forums (; GEN: http:// Resources, Guides/Tips, Link Collections, Online Journals, Library Resources (e-global Library) Courses:

1. DWeb: Training the TrainerDesigning, Developing, and Delivering Web-Based Training ($1,200 Canadian) (8 weeks: Technology, design, learning, moderating, assessment, course development, 2. 3. Techniques for Online Teaching and Moderation

Writing Multimedia Messages for Training Certified Online Instructor Program Walden Institute12 Week Online Certification (Cost = $995) 2 tracks: one for higher ed and one for online corporate trainer

Online tools and purpose Instructional design theory & techniques Distance ed evaluation Quality assurance Collab learning communities

Distance Ed Certificate Program (Univ of WisconsinMadison) 12-18 month self-paced certificate program, 20 CEUs, $2,500-$3,185 Integrate into practical experiences Combines distance learning formats

to cater to busy working professionals Open enrollment and self-paced Support services lecture/ 3. Training Inside Support

Instructional Consulting Mentoring (strategic planning $) Small Pots of Funding Facilities Summer and Year Round Workshops Office of Distributed Learning Colloquiums, Tech Showcases, Guest Speakers

Newsletters, guides, active learning grants, annual reports, faculty development, brown bags Technology and Professional Dev: Ten Tips to Make it Better (Rogers, 2000) 1. Offer training 2. Give technology to take home 3. Provide on-site technical support 4. Encourage collegial collaboration 5. Send to prof development conference 6. Stretch the day 7. Encourage research 8. Provide online resources

9. Lunch bytes, faculty institutes 10. Celebrate success RIDIC5-ULO3US Model of Technology Use 4. Tasks (RIDIC): Relevance

Individualization Depth of Discussion Interactivity Collaboration-Control-ChoiceConstructivistic-Community RIDIC5-ULO3US Model of Technology Use 5. Tech Tools (ULOUS): Utility/Usable Learner-Centeredness Opportunities with Outsiders Online Ultra Friendly

Supportive 6. Course Success Few technological glitches/bugs Adequate online support

Increasing enrollment trends Course quality (interactivity rating) Monies paid Accepted by other programs 7. Online Program or Course Budget [email protected] (asks how pay, how large is course, tech fees charged, # of courses, tuition rate, etc.)

Indirect Costs: learner disk space, coordination, phone, admin training, creating student criteria, accreditation, integration with existing technology and procedures, library resources, on site orientation & tech training, faculty training, office space, supplies Direct Costs: courseware, instructor, business manager, help desk, books, seat time, bandwidth and data communications, server, server back-up, course developers, postage

7. Program: Online Content Considerations Live mentors? Beyond content dumping? Interactivity? Collaboration? Individual or cohort groups? Lecture or problem-based learning? Record keeping and assessment? 8. Institutional Success

E-Enrollments from new students, alumni, existing students Additional grants Press, publication, partners, attention Cost-Benefit model Faculty attitudes Acceptable policies

8. Increase Accessibility Make Web material ADA compliant (Bobby) Embed interactivity in lessons

Determine student learning preferences Conduct usability testing Consider slowest speed systems Orientations, training, support materials e.g., CD-ROM 8. Initial Lessons to Learn

Start small, be clear, flexible Create standards and policies Consider Instructor Compensation: online teaching is not the same Look at obstacles and support structures Mixed or blended may dominate 8. What steps in getting it work?

Institutional support/White Paper Identify goals, policies, assess plans, resources (hardware, software, support, people) Faculty qualifications & compensation

Audience Needs: student or corporate Finding Funding & Partnering Test software usability testing, system compatibility, fits tech plans 8. How long to build a program? Year 1: Experimental Stage Year 2: Development Stage Hire people, creating marketing materials, assess, etc.

Year 3: Revision Stage Year 4: Move On Stage Final advicewhatever you do Ok, How and What Do You Assess and Evaluate?

Recently Viewed Presentations

  • The Federal Reserve

    The Federal Reserve

    The First Bank of the United States (BUS) Only bank with federal charter. Mercantilist movement behind banks. Fed owns 20%, deposits funds here. BUS buys government debt; issues notes.
  • Darwin's Voyage on the Beagle - CLAS Users

    Darwin's Voyage on the Beagle - CLAS Users

    Darwin's Voyage on the Beagle 1831-1836 - Darwin was Fitzroy's fourth choice - Purposes of voyage: Provide maps of South America useful to the British Navy. Britain viewed South America as a market for its manufactured products and a source...
  • Monday, 10/24/2011

    Monday, 10/24/2011

    The Bunsen burner was attached to the gas valve, lit, and adjusted to produce a hot, blue cone of flame. One end of a clean cotton swab was dipped in the first known solution until saturated and then was placed...
  • Punctuation - Quia

    Punctuation - Quia

    all punctuation . is. correct. Since Grandma wasnt looking, Julie filled the dogs bowls with leftovers from the meal. Since Grandma wasn't looking, Julie filled the dogs' bowls with leftover's from the meal. Since Grandma wasn't looking, Julie filled the...
  • Chapter 17: Stress and Health

    Chapter 17: Stress and Health

    PSYCHOLOGICAL FACTORS AND HEALTH Headaches Heart Disease Cancer Question: In what ways can people cope with stress? COPING WITH STRESS - DEFENSE COPING Defense coping - one way to reduce the immediate effects of a stressor Substance Abuse - use...
  • The National Context: How Is Principal Preparation Changing

    The National Context: How Is Principal Preparation Changing

    INSPIRE-LP)—enables documentation of leadership practices and school improvement and organizational indicators from the perspective of program graduates who are working as school principals. ... Accreditation /State Program Approval. Program Review & Redesign. Licensure.
  • The magnitude of H&N IMRT dose delivery errors

    The magnitude of H&N IMRT dose delivery errors

    CROSS-PLANE. For our symmetry analysis, up to 3 and 4% dose differences were seen in the angular in-plane and cross-plane deliveries, respectfully, with more TLD showing significant differences from the baseline delivery in both cross-plane deliveries.
  • CHAPTER 2: Visual Description of Data

    CHAPTER 2: Visual Description of Data

    Times New Roman Book Antiqua Monotype Sorts movnglnc.ppt - Moving Line Equation CHAPTER 2: Visual Description of Data - Part 1 Chapter 2 - Learning Objectives Chapter 2 - Key Terms Chapter 2 - Key Terms Sturges' rule Alternatives to...