Information feeding peer pressure and public accountability has become more powerful than legislation and regulation…
It's hard to improve what isn't measured
Comparing student learning outcomes can help…
- Individuals to make better informed choices and employers to assess the value of qualifications
- Universities to understand their comparative strengths and weaknesses
- Policy makers to quantify stocks and flows of high level skills and to assess value for money
...but so is the cost of inaction
Know why you are looking
(cc) photo by medhead on Flickr
...now the public looks for data to judge the quality of universities
There was a time when the public turned to universities to judge the quality of education...
There is pressure to increase access with limited financing and international competitiveness matters more than ever
The cost of action is significant...
Major challenges need to be overcome
- Defining and operationalising higher education learning outcomes in ways that are valid across programmes, institutions, sub-systems and cultures
- Comparing “like with like” institutions and mounting large-scale assessments
Without such data, judgements about higher education outcomes will continue to be made on the basis idiosyncratic rankings derived from higher education inputs
Learning outcomes
are the bottom line
...there is no longer a need to go to college...
...to study...
... or to get a degree...
...or to meet with a professor...
...there is no need to go to another country to pursue international studies.
Measuring higher
education learning outcomes
In the dark all higher education institutions look the same
With some light we can see important differences
Michael Stevenson
Beijing 23 September 2014
Some difficult questions
Know why you are looking
The search for the Holy Grail and the Alchemists stone motivated many generations
The medieval Alchemists’ followed the dictates of a well-established science but that science was built on wrong foundations. The search for the Holy Grail was overburdened by false clues and cryptic symbols
We can do better
The Alchemists’ stone was to be recognised by transforming ordinary metal into gold
Know how you'll recognise it when you find it
In conclusion
Tertiary
Student learning outcomes must be in the critical path of assessing the outcomes of higher education
Upper secondary
We now know we can test some of them internationally
Below upper
secondary
We don't yet know what part of the bigger picture on learning outcomes tests can and should be
Thank you!
Find out more about our work at:
www.oecd.org/education
www.pisa.oecd.org
Andreas.Schleicher@OECD.org
SchleicherEDU
...and remember: Without data, you are just another person with an opinion
Know what you are looking for
Know how you'll recognise it when you'll find it
What we expect from assessments
- Understanding what the assessment reveals about students’ thinking to shape better opportunities for student learning
- Responding to assessments can enhance student learning if tasks are well crafted to incorporate principles of learning
- Capitalise on improved data handling tools and technology connectivity to combine formative and summative assessment interpretations
Systems vs. institutions
Are largely performance-based
Support improvement of learning at all levels of the education system
Comparative data at system levels beyond the scope
- Too much variation in institutional structures across countries
- Nationally representative samples unrealistic (at least in the short term)
- Mandated assessment, even if it were possible, will not be effective as a tool for improvement at the level of service provision
- Large cross-country differences in enrolment rates raise questions about interpretation of comparative performance measures
What do we assess...
...and whom do we tell?
Are part of a comprehensive and well-aligned continuum, communicate what is expected
Make students’ thinking visible and allow for divergent thinking
What we expect from measures of student learning outcomes
Balance between breadth and depths
- Avoid tunnel vision but give educators the depth needed to stimulate improvement
Add value for teaching and learning by providing information that can be acted on by students, teachers, and administrators
Are adaptable and responsive to new developments
Focus on measures at the level of institutions, departments and faculty
- The idea of AHELO is to combine the definition of an OECD measure of quality with reliable assessment methods to which institutions can voluntarily subscribe and which might progressively find wider acceptance
Measures that...
- Reflect central and enduring parts of higher education teaching that relate to quality of outcomes
- Reflect aspects that can be improved
- Are cross-culturally appropriate and valid across institutions and systems
The validity challenge
Measures of institutional efficacy
The test of truth is how assessments link to other things like...
Alumni ratings or consumer perceptions about quality
Direct and indirect measures of research outcomes
Measures that are as comparable as possible…
…but as specific for institutions as necessary
How do assessments link to things like...
Disciplinary vs. transversal competencies
Balance between outcomes and process
- Not just the design and implementation of an assessment, but also a process of communication with key stakeholders on the nature and value of the assessment and the information gains
Labour-market and social outcomes
Institutional factors and non-cognitive characteristics that are known to be tied to successful study and achievement
Assessing disciplines
- Strengths: Easily interpretable in the context of departments and faculties
- Challenges: Requires highly differentiated instruments, excludes competency areas that are not amenable to large-scale assessment or not sufficiently invariant across cultures and languages, less value added, as measures already exist at national levels
Stakeholders and information needs
Individuals, whether prospective students or employers, would want to know the “bottom line” of the performance of institutions, departments or faculties
Some criteria
Some methodological challenges
Can we sufficiently distinguish the role of context from that of the underlying cognitive construct ?
Can we drink from the firehose of increasing data streams that arise from new assessment modes?
- Using a range of assessment methods to ensure adequate measurement of intended constructs and measures of different grain size to serve different decision-making needs
- Provide productive feedback, at appropriate levels of detail, to fuel accountability and improvement decisions at multiple levels
Comprehensiveness
How can we create assessments that are activators of students’ own learning ?
Transversal competencies
- Strengths: Less dependent on occupational and cultural contexts, applicable across HEIs, departments and faculties, powerful drivers for improving the quality of teaching in the disciplines
- Challenges: Reflect cumulative learning outcomes, need to be related to prior learning, do not relate to the kind of subject-matter competencies that many HEIs, departments or faculties would consider their province
Institutions and policy makers wishing to assess the quality of services provided would be interested in the “value added” by the institutions
What is the right mix of crowd wisdom and traditional validity information ?
- Build on a well-structured conceptual base—an expected learning progression—as the foundation for assessments
- Consistency and complementarity across administrative levels of the system
Coherence
Can we utilise new technologies to gain more information from students without overwhelming students with more assessments?
A continuous stream of evidence that tracks the progress