Experts sceptical about Polish research assessment proposals

A proposed overhaul of Poland’s research assessment system has been met with scepticism, with experts raising concerns about cost, bias and a potential negative impact on universities.

Produced by the Polish Academy of Sciences, the proposals encourage a move away from bibliometrics in favour of peer review and expert assessment. Funding, the academy advised, should comprise a “basic subsidy” based on the number of employees and the “cost-effectiveness” of the research unit, as well as supplementary funding that matches a percentage of external grants received and an “excellence subsidy” determined by expert assessment.

Because of cost and organisational limitations, only “units that have already demonstrated their high quality” should receive expert assessment, the academy said. Overall, the academy said, the proposed system would be “relatively simple and not susceptible to manipulation”.

Marcin Pałys, a former rector of the University of Warsaw and chair of Poland’s Central Council of Science and Higher Education, welcomed the academy’s emphasis on expert assessment over bibliometrics. A reliance on bibliometrics, he said, can incentivise researchers to churn out papers, regardless of quality.

Marta Natalia Wróblewska, a higher education scholar at SWPS University, noted the susceptibility of Poland’s current research assessment system to external interference. “We don’t use metrics like impact factor or journal impact factor but rather a list of journals and publishing houses that is more or less manually compiled by a committee appointed by the minister,” she said.

“Part of the dissatisfaction with the current system comes from the fact that the list doesn’t reflect the actual standing of journals or publishing houses at all.”

The former minister of education and science, Przemysław Czarnek, was “notorious for hand-steering the list”, Dr Wróblewska said, with small theological journals receiving similar weighting to some more renowned titles.

Nevertheless, both Dr Wróblewska and Professor Pałys said a move towards expert assessment could be hindered by a lack of trust in experts in Poland. “How do you guarantee that the ‘international peer reviewers’ will be unbiased, objective and resistant to manipulation? Who will appoint them?” asked Dr Wróblewska. The sole focus of expert assessment on high-performing research units, she added, would see weaker units lose out on valuable critical feedback.

Such reforms, both noted, would also come at considerable cost. “[The authors] acknowledge that their proposals make sense only when combined with a significant increase of research funding,” Professor Pałys said. “The prospects for increasing [government] investment in science are not optimistic at the moment.”

Stanisław Kistryn, coordinator of the Polish chapter of the Coalition for Advancing Research Assessment and a professor at Jagiellonian University, said the academy proposals were “not good for universities”, with the proposed funding structure omitting any financial incentives for teaching excellence.

Professor Kistryn also raised concerns about the academy’s plan to fund research units rather than institutions as a whole. “In many universities, faculties are semi-autonomous, but if there is separate funding and separate assessment then we do not have a university any more – we have a federation of faculties, and no common research policy is possible,” he said.

Dr Wróblewska suggested that the academy’s proposals for a new assessment system were premature, noting that Poland’s current assessment regulations were established in 2018, with only one round of evaluation, in 2022, carried out to date.

“There are details of the regulations that require tweaking for sure, but overhauling the entire system would amount to throwing out the baby with the bathwater,” Dr Wróblewska said. “To my mind, what would be helpful is an independent review of the evaluation exercise to assess its influence on the field.”

emily.dixon@timeshighereducation.com