Nick Stern produced another review, this time about the Research Excellence Framework (REF).

In REF2014, and in the preceeding Research Assessment Exercises, research output was evaluated by an individual's 4 best papers in the last 6 years. This emphasizes quality over quantity, which is a good thing, but punishes people who do decent applied work. Not everybody can be a top researcher. Not all students need to be educated by top researchers. The exclusive focus on quality has lead to many applied economists moving to business or geography, where standards are lower. Although 66 universities offer an economics degree, only 28 submitted their economists to the economics panel in REF2014.

The Stern Review recognizes this problem. It notes that only 1/3 of faculty were submitted to REF2014. Strangely, the right-wing tabloids have failed to pick this up. There are 100,000 lecturers and professors employed by UK universities who are paid for 2 days a week to do research -- but do not actually produce any research of note.

The Stern Review also proposes a solution: All shall be included. The threat of exclusion from REF2021 spurs people on. Stern suggests to take away an incentive to do well.

There is a second proposal in the Stern Review. Instead of assessing individuals, collectives will be assessed. The number of papers for individuals will range between 0 and 6, with the average number at 2. As top researchers compensate for the poor performance of their weaker colleagues, the latter have even less of a reason to do anything.

The third proposal is to go from the best 4 papers to the best 2 papers in 6 years. The bias towards quality over quantity grows.

The fourth proposal is to end portability. At the moment, publications are assigned to the university that employs the author at the census date. This leads to substantial mobility of top researchers towards the better endowed institutions in the year leading up to the REF -- and a wage premium for the best. In the future, publications will be assigned to the affiliation listed on the paper. This will change rather than end the game. Mobility will be at the start of the assessment period rather than at the end, and it will be based on the promise to deliver rather than the delivery. Ex ante mobility leads to more mismatches than ex post mobility. The labour market becomes less efficient.

The end of portability brings another game. Talented researchers will delay submitting the final revisions of their papers until they have negotiated a move. Footloose ones will hop from university to university, selling their conditional acceptances to the highest bidder.

Combined with Stern's second proposal, it is conceivable that a department submits only papers by people no longer on the payroll.


Stern also missed an opportunity. The REF is a large effort and therefore done only every six years or so. The REF determines reputation and research funds for half a decade, so universities understandably invest considerable effort into their submissions. The Stern Review does not change this. In fact, the number of papers to be read by the panels increases as 4 papers by 1/3 of all faculty is less than 2 papers by all faculty.

The Stern Review rejects an assessment based on metrics. Peer review may be superior if done well, but as the panels are overwhelmed -- one panel member bragged about "reading" 10 papers before breakfast -- papers are likely to be judged by their cover and people and institutions by their reputation.

A scientometric assessment is more objective, and can be done more frequently. A less frequent peer-review could then focus on the more qualitative aspects.

UPDATE 11 August: A colleague points to the dynamics of deciding whether the wonderful paper by Professor A should be included in lieu of the magnificent paper by Dr B.

UPDATE 12 August: Over at EJMR, someone points out that it no longer pays to hire a foreign big-shot on a part-time contract (as intended by Stern). Instead, you want to hire the bigshot on a visiting contract.

UPDATE 12 August (2): EJMR is at its charming best. Two further things occurred to me. Under the proposed rules, hiring will be based on uncertain future returns. The uncertainty about the average is, of course, smaller in larger departments. Stern thus favours the bigger departments.

On the other hand, a smaller, poorer department can now better afford to take a punt on someone. Under the current rules, you would hire someone promising, see her flourish, and disappear just before the REF. Under the proposed rules, she would be snapped up later, and her papers count towards your submission.