Our University’s REF steering group last met two weeks ago and Professor Andrew Church mentioned that a number of HEIs are adopting specific initiatives to assess research.  We will consider these initiatives more closely over the coming months but I would like to share these with you now.

Firstly, the San Francisco Declaration on Research Assessment (usually referred to as San Francisco DORA) is one that has been referred to fairly frequently.  I first heard of DORA in 2015 when I read ‘The Metric Tide’, the report of the independent review of the role of metrics in research assessment and management. This report recommended that institutions sign up to DORA and, since then, the uptake among UK institutions has been growing: DORA now has over 12,000 signatories worldwide of which 10 or so are from the UK and include Sussex, Manchester, Imperial, Brunel, Edinburgh and UCL.

DORA was initiated by the American Society of Cell Biology and a group of publishers and journal editors back in 2012 in order to “improve the ways in which the outputs of scientific research are evaluated”.  It centres on the belief that there is an over-reliance on bibliometrics, such as the Journal Impact Factor, which is seen by many as flawed.  Other key themes are that research needs to be evaluated on its own merit (not on which journal it’s published in) and that we need to make the best of the flexibility afforded by online publishing.

DORA makes a general recommendation that bibliometrics should not be used as a “surrogate measure” for the quality of research and then makes a number of specific recommendations for publishers, academic institutions, research funders, organisations that supply metrics and individual researchers.  All the aforementioned are invited to sign DORA.

For a university, signing DORA would mean it is obliged to:

• Be explicit about the criteria it uses for the assessment of research and researchers
• Reinforce that the content of research is what is important (rather than what metric scores it has or what journal it is published in)
• Consider the value and impact of all research outputs
• Consider using a variety of measures to assess research and researchers

So, why have only a handful of UK universities signed it?  Some would say DORA feels rather negative in tone and I have even heard some referring to it as an ‘anti-journal metric triade’.

Subsequent to the publication of DORA, the bibliometric experts at the Centre for Science and Technology Studies in Leiden, in collaboration with Professor Diana Hicks (Georgia Tech), published the Leiden Manifesto in April 2015.  This, too, is set against the “Impact Factor obsession” and offers “best practice in metrics-based research assessment so that researchers can hold evaluators to account, and evaluators can hold their indicators to account”.  The manifesto recommends 10 principles that are derived from the best practices of (quantitative) bibliometric exercises.

The following is a very brief summary of the principles:

1. Quantitative evaluation should support qualitative, expert assessment
2. Measure the performance against the research objectives of the institution, group or researcher
3. Protect excellence in locally relevant research
4. Keep data collection and analytical processes open, transparent and simple
5. Allow those evaluated to verify data and analysis
6. Account for variation by field in publication and citation practices
7. Base assessment of individual researchers on a qualitative judgement of their portfolio
8. Avoid misplaced concreteness and false precision
9. Recognise the systemic effect of the assessment and indicators
10. Scrutinise indicators regularly and update them

In an effort to make the manifesto as accessible as possible, there is a really neat video version which is well worth watching: .

I must admit, I feel a lot more comfortable with the Leiden Manifesto, which takes a more positive approach: a balanced call for the sensible, contextualised and transparent use of all publication metrics.  There is no option to sign up to this but, earlier this year, both the University of Bath and Loughborough University announced that they have developed a set of principles to assess their research which draws upon Leiden Manifesto.  At our next REF steering group, we will discuss both these initiatives at length and deciding whether they can inform our approach.