Loading presentation...

Present Remotely

Send the link below via email or IM


Present to your audience

Start remote presentation

  • Invited audience members will follow you as you navigate and present
  • People invited to a presentation do not need a Prezi account
  • This link expires 10 minutes after you close the presentation
  • A maximum of 30 users can follow your presentation
  • Learn more about this feature in our knowledge base article

Do you really want to delete this prezi?

Neither you, nor the coeditors you shared it with will be able to recover it again.


ACUMEN: Academic Careers Understood through Measurement and Norms

Presentation on ACUMEN to the Open Access Coordinating Workshop, EU, Brussels, 4 May 2011

Paul Wouters

on 24 May 2011

Comments (0)

Please log in to add your comment.

Report abuse

Transcript of ACUMEN: Academic Careers Understood through Measurement and Norms

A cademic C areers U nderstood through M asurement and N orms E Centre for Science and Technology Studies (Leiden University) Department of Information Science (Bar-Ilan University) Cybermetric Lab (CSIC Madrid) Statistical Cybermetrics Research Group (Universit of Wolverhampton) Sihtasutus Archimedes (Estonia) Institut f. Bibliotheks- und Informationswissenschaft (Humboldt University) Wirtschaftsinformatik (Wildau University) Royal School of Library and InformationScience (Copenhagen) e-Humanities Group (Royal Netherlands Academy of Arts and Science) Source: http://www.flickr.com/photos/buhsnarf/2534278930/sizes/m/in/photostream/ Key problems:
produce excellent research but keep space for creativity
how does output evaluation work at individual level?
can we combine traditional indicators with new web based and open access measures? ACUMEN approach:
analyze peer review
use scientometric indicators
explore webometric indicators
ethnographic research of career paths
analyze gendered patterns ifferent ? First, evaluation will be analysed from the perspective of the individual researcher's career development. Evaluation is a more complex interaction than simply the measurement of the performance of the researcher. It is a communication process in which both evaluators and the researcher under evaluation define what the proper evaluation criteria and materials should be. The key outcome of evaluation systems is not only the conclusion with respect to the future prospects of the researcher and her manuscripts. At least as important, and sometimes even more important, are the intermediate effects of the process of evaluation on the researcher, on the evaluator, and on the future instances of evaluation. ACUMEN aims to provide state-of-the-art tools to the research community that can be used both by evaluators and individual researchers in the form of an ACUMEN Portfolio of evidence which is based on a set of ACUMEN criteria for Good Evaluation Practices. The Portfolio will enable a researcher to propose an extended set of materials and criteria for evaluation in relation to the relevant scientific and social mission of her research. Second, ACUMEN puts the constructive effects of evaluation central in order to assess the implications of new evaluation criteria and guidelines on individual careers and on the scientific system as a whole. Evaluation systems inevitably produce quality and relevance as much as they measure it. This holds both for indicator based evaluation and for qualitative peer review evaluation systems. Evaluation systems have these effects because they shape the career paths of researchers and because they form the quality and relevance criteria that researchers entertain. These feedback processes also produce strategic behaviour on the side of the researchers which potentially undermines the validity of the evaluation criteria. ACUMEN will therefore put central how current and new forms of peer review and indicator systems as main elements of the evaluation process will define different quality and relevance criteria in the evaluation of individual researchers, on the short term as well as on the longer term. Third, ACUMEN will analyse the diversity of current evaluation practices in a comparative research design. The existing evaluation practices and cultures vary by nation, by institution and by discipline. Although virtually all evaluations aim to ascertain excellence at the international level, how this is operationalized varies greatly. In some cases, citation analysis is very influential, in other cases evaluators and researchers tend to frow upon these quantitative indicators or claim that they are not applicable to their discipline or institution. In some countries, traditional peer review systems are still dominant at the national level, whereas in other countries these criteria have been supplanted with a large set of requirements based on the economic and social effects of research. The Acumen Portfolio At the level of the individual researcher it would be unfair to rely on one or a small set of indicators with which to evaluate their research because research contributions are too varied. Nevertheless, researchers need to be evaluated for many reasons and therefore need to be able to provide fairly standardised evidence so that they can be compared with other researchers. In response, the proposed ACUMEN Portfolio is a combination of sources of evidence: CV, bibliometrics, webometrics and peer review, together with anarrative that uses the evidence to make a case for the value of the researcher's contribution. The Portfolio format will be designed to be flexible enough to represent a wide range of different types of contribution but standardised enough to allow comparisons between portfolios. The project will aim to demonstrate that the evidence-based portfolio is flexible enough to reasonably represent a far wider range of researchers than previous methods. The Portfolio will have a modular structure, which should enable researchers to turn specific components on or off depending on their context and disciplinary background. Source: A. Zuccala, “The layperson and open access”, ARIST, 2009: 359 - 396 Source: Celina Ramjoué, "FP7, Science in Society, and Open Access:putting ACUMEN in context", CWTS, Leiden, 11 March 2011
Full transcript