Meetings with research evaluators

In a second phase, the working group on defining new research evaluation methods for the FWB/CoARA inter-university project met with those involved in evaluation within the universities to explore their practices, needs and constraints. For methodological reasons, the decision was taken to survey the evaluators. In total, one hundred interviews were conducted across the five universities in the FWB. The evaluators were selected on the basis of their experience on evaluation committees:

• research projects;
• academic careers (recruitment and promotion).

What are the key criteria they would like to see abandoned or given greater weight in research evaluation? Are metrics still commonly used, and if so, which ones? Is there recognition of Open Science practices in evaluations? Are they for or against the use of narrative CVs? Evaluators from scientific project evaluation committees (doctoral and post-doctoral grants, larger-scale projects such as Concerted Research Actions (ARC), etc.) and academic career evaluation committees (recruitment and promotion) have shared their views with us.

Three significant observations can be highlighted:

1. As expected, the criteria used in the evaluation vary greatly depending on the scientific discipline or research culture, the stage of the candidate’s career, the programme or funding scheme to which the candidate is applying, and the type of evaluation panel.
2. Criteria relating to the research profile appear to take precedence over those relating to the teaching profile and service aspects. Among the most frequently cited are: the quality of the candidate’s CV, the originality of the research topic, publication metrics, and, finally, the candidate’s performance during the interview.
3. The evaluators highlight a number of difficulties, including:

• • Difficulties in establishing common criteria for assessing different disciplines, as well as in comparing a variety of profiles;
  • A lack of calibration benchmarks prior to assessments;
  • A sense of lacking legitimacy: not being ‘specialists’ in all fields;
  • A heavy workload.

To view all the findings and opinions expressed during the meetings with the research evaluators, please refer to the summary report available below: