Copyright 2012 Pearson Education, Inc. or its affiliate(s).

Copyright  2012 Pearson Education, Inc. or its affiliate(s).

Copyright 2012 Pearson Education, Inc. or its affiliate(s). All rights reserved. 800 837 8969 Admissions testing: A journey in finding successful candidates Belinda Brunner Test Development Strategist Pearson VUE 2 The why, what and how of admissions testing 3 Why use standardised testing in selection? The purpose of an admissions test

Underlying premise: The higher education admissions selection process should be meritocratic and fair. MERITOCRACY. a system in which the talented are chosen and moved ahead on the basis of their achievement; leadership selected on the basis of intellectual criteria. meritocratic \mer--t-kra-tik\ adjective. FAIRNESS. Having or exhibiting a disposition that is free from favouritism or bias; impartial. fr \ adjective. The role for standardised testing What is a standardised test? An assessment in which the administration and

scoring procedures are consistent: Test takers receive same set of questions or questions are selected for administration based upon a set of balancing rules Administration conditions are consistent and standardised Scoring procedures enable comparisons across time and administrations 6 The goal of an admission selection process is to capture as clear a picture of the applicant as possible. Other factors in admissions decisions

Previous grades Personal statements Writing samples Interviews Recommendations Source: NACAC Admissions Trends Survey, 2013. As reported in State of College Admissions Report (http://www.nxtbook.com/ygsreprints/NACAC/2014SoCA_nxtbk/#/28) 9 Trends in admissions decisions

http://www.pbs.org/newshour/updates/new-toolcolleges-using-admissions-decisions-big-data/ http://www.onlineschools.org/beyond-the-sat/ Case study (Hypothetical) ABC University Large university offering a range of undergraduate degree programmes in the arts and sciences International students Current admissions process: Secondary school grades Application with personal statement Interview

11 Case study ABC University Why introduce an admissions test to the undergraduate admissions process? Common, standardised assessment Broaden access to higher education Controlled environment to complete the writing sample Targeted interviewing

12 Predictors of success: Aptitude vs achievement measures Aptitude: the degree of readiness to learn and to perform well in a particular situation or in a fixed domain ~ Corno, L., Cronbach, L.J., Kupermintz, H., Lohman, D.F., Mandinach, E.B., Porteus, A.W., & Talbert, J.E. (2002, p. 3). Remaking the concept of aptitude: Extending the legacy of Richard E. Snow. Mahwah, NJ: Lawrence Erlbaum Associates. Achievement: planned changes in cognitive behavior of students resulting from instructions or trainingdefined in terms of content ~ Haladyna, T.M. & Rodriguez, M.C. (2013, p. 5) Developing and validating test items. New

York: Routledge. 13 Predictors of success: Aptitude vs achievement measures Aptitude Reasoning tests Achievement Grades averaged across subjects

Course grades Subject-specific knowledge tests Adapted from Lohman (2004). Aptitude for college: The important of reasoning tests for minority admissions. In Zwick, R. (Ed.), Rethinking the SAT: The future of standardized testing in university admissions (pp. 41-54). New York: Routledge Palmer. 14 A comparison of medical school admissions tests Test Country of Origin Aptitude/

Reasoning Achievement/ Knowledge Writing Task(s) Website Target Population Secondary School Students Undergraduate Students BMAT UKCAT UMAT MCAT GAMSAT United United Australia

United States Australia Kingdom Kingdom Components www.admissionst estingservice.org

www.ukcat.ac.uk www.umat.acer.edu.a u www.aamc.org www.gamsat.acer.ed u.au Case study ABC University Process for determining the test constructs 16

Case study ABC University Admissions test components: University-wide reasoning test Verbal reasoning Quantitative reasoning Spatial ability Subject tests for selected degree programs Writing sample 17 Summary Why use a standardised test? Standardised admissions tests are particularly useful when there

is a lack of a standardised curriculum in the pre-university academic settings. Reasoning tests can be useful in identifying applicants who have not had the same opportunities to learn, but who have the ability to perform well if given the opportunity. Reasoning tests can also be useful in identifying students with a particular talent. 18 Reasoning tests thus have a place at the admissions table. It is not at the head of the table, as some once thought; rather, such tests provide a way to supplement grades and other measures of past achievement In other words, prior achievement is often an important aptitude for future learning. But it is never the only

aptitude, and sometimes not even the most important aptitude. ~ Lohman, D.F. (2004, p. 54). Aptitude for college: The importance of reasoning tests for minority admissions. In Zwick, R. (Ed.), Rethinking the SAT: The future of standardized testing in university admissions (pp.41-54). New York: Routledge Palmer. 19 What role should standardised testing play in selection? Using test scores Using admissions test scores 21

What part do admissions tests play in the selection process? 22 How are admissions test scores considered? Profiles of US students, grades 3-12, on the Verbal, Quantitative & Nonverbal batteries of the Cognitive Abilities Test, Form 6 (Lohman & Hagen, 2001) Lohman, D.F., Gambrell, J., & Lakin, J. (2008). The commonality of extreme discrepancies in the ability profiles of academically gifted students. Psychological Science, 50, 269-282. 23 What does this mean for how admissions test

scores are considered in the selection process? Consider 3 hypothetical students on a hypothetical admissions test: 3 sectional scores reported Verbal Quantitative Spatial reasoning Scores reported on a 300-900 scale 24 Options for how scores are considered (Lohman, 2012) And rule: Must score above the cut score on all sections to be considered for admission Avg rule: Average score must be above the cut

score to be considered for admission Or rule: Must score above the cut score on one of the sections to be considered for admission Lohman, D.F. (3 Dec 2012). An aptitude framework for college admissions testing. Paper presented at the First International Conference on Assessment and Evaluation. National Center for Assessment in Higher Education, Riyadh, Saudi Arabia. 25 Using admissions test scores Admitted No No And rule: Must score above the cut score on all sections

26 Using admissions test scores Admitted No Admitted Avg rule: Average score must be above the cut score 27 Using admissions test scores Admitted Admitted

Admitted Or rule: Must score above the cut score on one of the sections 28 Using admission test scores Student Major Scores Decision rule

Verbal Quant Spatial Avg And Avg Or 1 History

703 645 635 661 Y Y Y 2

Maths 440 675 640 585 N N Y 3

English 835 620 650 702 N Y Y

29 Using admission test scores Student Major Scores Decision rule Verbal Quant Spatial

Avg And Avg Or 1 History 703 645 635

661 Y Y Y 2 Maths 440 675

640 585 N N Y 3 English 835 620

650 702 N Y Y 30 Using admission test scores Student Major

Scores Decision rule Verbal Quant Spatial Avg And Avg

Or 1 History 703 645 635 661 Y Y

Y 2 Maths 440 675 640 585 N

N Y 3 English 835 620 650 702 N

Y Y 31 Using admission test scores (Lohman, 2012) The And rule The most restrictive Administratively convenient The same effect occurs if both admissions test scores and grades in previous coursework must exceed a minimum value. The Avg rule Separate scores can receive different weights Higher score in one area compensates for lower scores in others

Ignores distinct information if different scores reflect disparate test constructs The Or rule Most defensible of the three methods if the sectional test scores measure disparate test constructs May assist in identifying talent in a particular area 32 Case study ABC University Liberally-set minimum acceptable score or Minimum cut score;

scores used for placement 33 Case study ABC University or 34 Case study ABC University Common, standardised assessment All applicants complete the reasoning tests Broaden access to higher education

Or rule for secondary grades/reasoning test scores Controlled environment to complete the writing sample Writing sample completed as part of the standardised assessment Targeted interviewing Interviewing occurs after standardised testing 35 Summary What role should standardised testing play in the selection process? How test scores should be used depends upon: The purpose for testing

The context in which further learning occurs The other factors considered in the selection process Different ways of incorporating multiple measures into selection can lead to different decisions as to who is admitted and who is not. 36 How is a fit-for-purpose admissions test created? Test development and administration Meritocracy in testing 38 Validity: Predictive

Key question: How well does the admissions test predict subsequent academic performance? Most commonly studied using first-year grades Generally speaking, studies support the incremental validity of standardised admissions tests* Complicating factors: Restriction of range Criterion inaccuracies *See Zwick, R. (2006). Higher education admissions testing. In Brennan, R.L. (Ed.), Educational measurement (pp. 647-680). Westport, CT: Praeger and American Council on Education. 39 Validity: Construct/content

Research/expert opinion to support the included constructs Creation of a test plan/blueprint Linking of test items to the test plan SMEs involved in the item writing & review processes Review of items include content & statistical review Item bias and sensitivity review 40 Test factors influencing reliability The test itself Test length Item format & quality

Test administration Standardised & consistent testing experience Test construction & equating procedures Homogeneity in the test taker population a reliability coefficient tells us about a quality of a test (and one that we usually value) but not about the quality of a test ~ Traub, R.E. & Rowley, G.L. (1991, p. 44). Understanding reliability. Educational measurement: Issues and practice, 10(1), pp. 37-45. Retrieved 15 July 2013 from http://ncme.org/linkservid/65F3B451-1320-5CAE-6E5A1C4257CFDA23/showMeta/0/ 41 Fairness in testing The Standards for Educational and Psychological Testing (AERA, APA

& NCME, 2014) view fairness according to various perspectives: In treatment during the test process As a lack of measurement bias In access to the construct(s) measured As validity of individual test score interpretations for the intended uses 42 Use of computer-based testing (CBT) for admissions testing security Professional, standardised, consistent Increased testing environment Increased test centre coverage &

accessibility More frequent testing opportunities: windows or on-demand Immediate results reporting Advanced item types Measurement accuracy and efficiency Data-rich results Brand extension & customisation 43 Case study ABC University Undergraduate admissions test Test administration Computer-based test administered in a 4-month test window

Test design 3 sections: Verbal reasoning, quantitative reasoning, spatial ability Subsections (specific item types) defined for each section Test structure Designed to promote fairness and prevent item overexposure 44 Case study ABC University Test structure Verbal Quantitative Spatial

45 Case study ABC University Test structure Verbal Quantitative Spatial Testlet Testlet Testlet Testlet Testlet Testlet Testlet Testlet Testlet 1 2 3 1 2 3

1 2 3 Testlet: A group of items that will be administered together All testlets within a section are comparable according to content and statistical properties Items within a testlet are administered in randomised order 46 Case study ABC University Test structure Verbal Quantitative

Spatial Testlet Testlet Testlet Testlet Testlet Testlet Testlet Testlet Testlet 1 2 3 1 2 3 1 2 3 Pre-test pool (Select n from n) Pre-test pool

(Select n from n) Pre-test pool (Select n from n) Testlet: A group of items that will be administered together All testlets within a section are comparable according to content and statistical properties Items within a testlet are administered in randomised order Pre-test pool: Pre-test items administered for trialing/field testing purposes; they do not count toward the test-takers scores Items randomly selected from the pool for administration Each test taker receives the same number of pre-test items 47

Case study ABC University Scoring & results reporting IRT equating Scaled scores on a 300-900 scale Post scoring for early test takers, immediate scoring thereafter Test structure allows for building an item bank of pre-tested items to allow for immediate scoring in subsequent years 48 Summary Standardised tests can provide a reliable

measure to be considered in selection decisions. Test design, content, development, administration and use (and how these relate to the tests purpose) are all validity considerations. To be fair, testing procedures should treat all equitably, safeguard against bias, and the results should reflect testtaker ability on the desired test construct. 49 Summary Computer-based testing is state-of-the-art admissions testing. Uses computer technology that

applicants use in their private lives and for study Greater access and flexibility Supports a wider variety of question formats Supports the use of modern psychometric methods which strengthen the credibility and defensibility of the test 50 A selection system should always be considered a work in progress. ~ Lohman, D.F. (3 Dec 2012, p. 16). An aptitude framework for college admissions testing. Paper presented at the First International Conference on Assessment and Evaluation. National Center for Assessment in Higher Education, Riyadh, Saudi Arabia.

51 Thank you Belinda Brunner [email protected] 52

Recently Viewed Presentations

  • ISAIAH - d2y1pz2y630308.cloudfront.net

    ISAIAH - d2y1pz2y630308.cloudfront.net

    Isaiah was called to the prophetic office during the year King Uzziah died. 2. Isaiah saw the Lord in the temple. Above him were seraphim each with 6 wings. 3. One of them shouted, "Holy, holy, holy is the Lord...
  • FOIL What is a FOIL in literature? A

    FOIL What is a FOIL in literature? A

    Who is the Foil? Usually the foil contrasts the protagonist; however, this is not always the case. In the classic good-guy vs bad-guy scenario, both the hero and the villain can each be considered the other's foil, in that each...
  • Delphi Presentation - University of Arizona

    Delphi Presentation - University of Arizona

    Reliability Presentation Test-Retest James Blackwood - AED 615 Fall Semester 2006 Test-Retest Reliability Test-Retest method of determining reliability is accomplished by administering a test to a group After a period of time has passed, the same test is re-administered to...
  • Project 2061: Student Assessment Aligning Science Assessment to

    Project 2061: Student Assessment Aligning Science Assessment to

    An iron railing developing an orange, powdery surface after standing in air. Applying the Necessity Criterion: The knowledge in the learning goal is not needed. Answer choice D, the correct answer, is a specific instance of a general principle (SIGP)....
  • Antony Bounds Deputy Director, Recruitment and Admissions Our

    Antony Bounds Deputy Director, Recruitment and Admissions Our

    He is currently Deputy Vice Chancellor at the University of Derby where Professor Plowden has delivered a step-change in academic quality, with the institution's league table rising from 91st to 48th. ... which produces an individualised report for each student...
  • Client's Logo/Name

    Client's Logo/Name

    Boire Analytics. Leverage . Data to Optimize your . Business and to ultimately drive customer profitability . Industry Pioneer who began career in 1983 in data science. Established . Track . Record, Loyal . Client Base. Solving the right problem...
  • 12 - rcc.edu

    12 - rcc.edu

    Lateral horn. Ventral horn. Ventral funiculus. Lateral funiculus. White. columns. Dorsal root. ganglion. Dorsal root (fans out into . dorsal rootlets) ... Medial lemniscus (tract) (axons of second-order neurons) Nucleus gracilis. Nucleus cuneatus (a) Spinocerebellar. pathway . Dorsal column ...
  • Chapter 9

    Chapter 9

    Chapter 10. Molecular Geometries and Bonding Theories. Lewis structures do not indicate the molecular architecture - the shape of the molecule. The shape and structure of a molecule determines much of its physical and chemical characteristics.