Send the link below via email or IMCopy
Present to your audienceStart remote presentation
- Invited audience members will follow you as you navigate and present
- People invited to a presentation do not need a Prezi account
- This link expires 10 minutes after you close the presentation
- A maximum of 30 users can follow your presentation
- Learn more about this feature in our knowledge base article
Do you really want to delete this prezi?
Neither you, nor the coeditors you shared it with will be able to recover it again.
Make your likes visible on Facebook?
You can change this under Settings & Account at any time.
Measuring higher education learning outcomes
Transcript of Measuring higher education learning outcomes
...and whom do we tell? Measuring student
learning outcomes The Holy Grail? Andreas Schleicher AHELO Conference, Paris, 11 March 2013 Or the Alchemists Stone? Some difficult questions Know why you are looking Know what you
are looking for Know how you'll recognise it when you find it The Holy Grail was a well-described object, and there was only one true grail The Alchemists’ stone was to be recognised by transforming ordinary metal into gold The medieval Alchemists’ followed the dictates of a well-established science but that science was built on wrong foundations. The search for the Holy Grail was overburdened by false clues and cryptic symbols We can do better It's hard to improve what isn't measured The cost of action is significant... Information feeding peer pressure and public accountability has become more powerful than legislation and regulation… Comparing student learning outcomes can help…
Individuals to make better informed choices and employers to assess the value of qualifications
Universities to understand their comparative strengths and weaknesses
Policy makers to quantify stocks and flows of high level skills and to assess value for money Major challenges need to be overcome
Defining and operationalising higher education learning outcomes in ways that are valid across programmes, institutions, sub-systems and cultures
Comparing “like with like” institutions and mounting large-scale assessments ...but so is the cost of inaction Without such data, judgements about higher education outcomes will continue to be made on the basis idiosyncratic rankings derived from higher education inputs There was a time when the public turned to universities to judge the quality of education... There is pressure to increase access with limited financing and international competitiveness matters more than ever Learning outcomes
are the bottom line ...there is no longer a need to go to college... Comparative data at system levels beyond the scope
Too much variation in institutional structures across countries
Nationally representative samples unrealistic (at least in the short term)
Mandated assessment, even if it were possible, will not be effective as a tool for improvement at the level of service provision
Large cross-country differences in enrolment rates raise questions about interpretation of comparative performance measures Focus on measures at the level of institutions, departments and faculty
The idea of AHELO is to combine the definition of an OECD measure of quality with reliable assessment methods to which institutions can voluntarily subscribe and which might progressively find wider acceptance Assessing disciplines
Strengths: Easily interpretable in the context of departments and faculties
Challenges: Requires highly differentiated instruments, excludes competency areas that are not amenable to large-scale assessment or not sufficiently invariant across cultures and languages, less value added, as measures already exist at national levels Transversal competencies
Strengths: Less dependent on occupational and cultural contexts, applicable across HEIs, departments and faculties, powerful drivers for improving the quality of teaching in the disciplines
Challenges: Reflect cumulative learning outcomes, need to be related to prior learning, do not relate to the kind of subject-matter competencies that many HEIs, departments or faculties would consider their province Systems vs. institutions Disciplinary vs. transversal competencies Individuals, whether prospective students or employers, would want to know the “bottom line” of the performance of institutions, departments or faculties Institutions and policy makers wishing to assess the quality of services provided would be interested in the “value added” by the institutions Stakeholders and information needs Measures that...
Reflect central and enduring parts of higher education teaching that relate to quality of outcomes
Reflect aspects that can be improved
Are cross-culturally appropriate and valid across institutions and systems Balance between breadth and depths
Avoid tunnel vision but give educators the depth needed to stimulate improvement Balance between outcomes and process
Not just the design and implementation of an assessment, but also a process of communication with key stakeholders on the nature and value of the assessment and the information gains Measures that are as comparable as possible…
…but as specific for institutions as necessary Understanding what the assessment reveals about students’ thinking to shape better opportunities for student learning
Responding to assessments can enhance student learning if tasks are well crafted to incorporate principles of learning
Capitalise on improved data handling tools and technology connectivity to combine formative and summative assessment interpretations What we expect from assessments Some criteria Some methodological challenges What we expect from measures of student learning outcomes Continuity Comprehensiveness Coherence Build on a well-structured conceptual base—an expected learning progression—as the foundation for assessments
Consistency and complementarity across administrative levels of the system Using a range of assessment methods to ensure adequate measurement of intended constructs and measures of different grain size to serve different decision-making needs
Provide productive feedback, at appropriate levels of detail, to fuel accountability and improvement decisions at multiple levels A continuous stream of evidence that tracks the progress The validity challenge Can we drink from the firehose of increasing data streams that arise from new assessment modes? Can we sufficiently distinguish the role of context from that of the underlying cognitive construct ? Can we utilise new technologies to gain more information from students without overwhelming students with more assessments? What is the right mix of crowd wisdom and traditional validity information ? How can we create assessments that are activators of students’ own learning ? Direct and indirect measures of research outcomes Alumni ratings or consumer perceptions about quality Labour-market and social outcomes Institutional factors and non-cognitive characteristics that are known to be tied to successful study and achievement Measures of institutional efficacy How do assessments link to things like... The test of truth is how assessments link to other things like... Support improvement of learning at all levels of the education system Are largely performance-based Make students’ thinking visible and allow for divergent thinking Are adaptable and responsive to new developments Add value for teaching and learning by providing information that can be acted on by students, teachers, and administrators Are part of a comprehensive and well-aligned continuum, communicate what is expected In conclusion Student learning outcomes must be in the critical path of assessing the outcomes of higher education We now know we can test some of them internationally We don't yet know what part of the bigger picture on learning outcomes tests can and should be The search for the Holy Grail and the Alchemists stone motivated many generations In the dark all higher education institutions look the same Email: Andreas.Schleicher@OECD.org Thank you! Find out more about our work at: www.oecd.org/education ...and remember: Without data, you are just another person with an opinion With some light we can see important differences (cc) photo by medhead on Flickr ...there is no need to go to another country to pursue international studies. ...to study... ... or to get a degree... ...or to meet with a professor... ...now the public looks for data to judge the quality of universities