EU to ‘improve and maximise’ impact of research

The next framework programme for European research funding should have “a more sophisticated approach” to impact, according to Carlos Moedas, the European Commissioner for research, science and innovation.

Speaking at a European Research Area conference in Berlin last month, he said that the commission had an obligation to understand and communicate the impact of its work to finance ministers and the general public.

It is not yet clear how he hopes to develop the impact agenda in Europe. But experts are concerned about the direction it could push science and question whether the actions of the European Research Council, which funds blue-skies research solely on the basis of excellence, will be protected.

Mark Reed, professor of social innovation at Newcastle University, fears the commission may be considering a post hoc assessment exercise to evaluate the impact of research grants that have previously been awarded.

That could raise the prospect of such a project being similar to the Higher Education Funding Council for England’s controversial research excellence framework, which uses case studies of the impact generated by research to help influence how much funding universities secure in the future.

“To do that at the scale of the EU would be fairly monumental,” Professor Reed told Times Higher Education.

Academics currently applying for European research funding do have to describe the impact that they expect their work to have. Impact is one of three criteria, alongside excellence, and quality and efficiency, used to decide which proposals secure funding.

But Professor Reed said that simply beefing up this part of European grant applications would not be sufficient to achieve what Mr Moedas appears to want. “He may have set himself a task that is rather challenging to achieve. If he is serious about this, as he sounds, then he will need to think about looking back in time,” he added.

He added that there was a “need to change” the pre-award assessment of impact at the European level because often EU-funded researchers have a “huge focus on communicating their outcomes” with the idea that disseminating their findings will lead to impact.

This results in “very costly events for policymakers…to which very few people turn up”, he said. Changing this culture so that researchers start to think about impact in a structured and planned way ahead of starting their project could be “low-hanging fruit” for Mr Moedas and would be “very easy” to introduce, he said.

Professor Reed was not the only academic to have concerns about where Mr Moedas’ comments would lead European research.

Peter Strohschneider, president of the German Research Foundation, said that the speech “implies a rather weak concept of what science and research is about. It takes research as a means of economic growth and innovation,” he said.

Introducing a REF-type exercise in Europe would be “very expensive” and lead to “scientific and intellectual shortcomings”, he said. “Questions of metrics are questions of power…This is all about political decisions on what we take as relevant in research. This is not illegitimate but it is not the wisest way to steer the entire EU research system,” he added.

Katrien Maes, chief policy officer at the European League of Research Universities, said that how the commission decides to measure impact will depend on the structure of the next framework programme.

The commission has done much work to simplify the complicated process of applying for European research funding, but there is still work to be done in this area, she said. “This potentially complicates things again enormously,” she added.

But she urged the commission not to define impact as “a narrow economic act. Impact can take a very long time to materialise and it is not just a question of patents, spin-offs and licences”, she said.

Lidia Borrell-Damian, director of research and innovation at the European Universities Association, said that the ERC should be protected against the introduction of any measures of impact.

“The ERC is a [red] line. If they do introduce impact there, then I think they risk the wrath of the sector,” she said, adding that the EUA and others have sent the commission “very strong signals” about this.

In 2012, Helga Nowotny, then president of the ERC, said that adopting an impact agenda would destroy the organisation.

Dr Borrell-Damian added that the European Innovation Council, a new proposition to support the commercialisation of technologies, could create the impact that Mr Moedas is looking for.

Whether he is hoping to improve research impact through the EIC, or pre- or post-award assessment, remains unclear, but there is other evidence that he is clearly looking at options. Earlier this year, the commission put out a call for experts to sit on a high-level group to advise them on maximising the impact of European research and innovation.

In a statement for THE, the commission spokeswoman for research, science and innovation said: “Improving the impacts of EU-funded research and innovation is a very high priority.”

She added that “clear progress” had been made on impact during Horizon 2020, the current framework programme. “[W]e will seek to further improve and maximise the impact of future EU funding for research and innovation, knowing that not all research will have a concrete and immediate impact,” she said.

Lessons on impact

The impact agenda has been gaining momentum in recent years as policymakers worldwide grapple with the need to prove the economic value of research.

The idea of measuring the impact of research in a serious and systematic way originated in Australia, with the aborted research quality framework. In the mid-2000s, this looked at the use of case studies to describe the impact of completed research – a system eventually adopted in the UK’s 2014 research excellence framework.

Impact is again gaining ground in Australia. In 2012, the Australian Research Council established a working group with other government research agencies to look at the possible ways of measuring engagement and impact for their next research assessment exercise.

Attila Brungs, vice-chancellor and president of the University of Technology Sydney, said: “Having an acknowledgement of impact is how we change the world and make it a better place.”

He added that Carlos Moedas, the European Commissioner for research, science and innovation, should “stay away” from using a “few simple quantitative measures around impact” as these tend to measure engagement not impact. Using the wrong metrics can also “be very distorting and drive the wrong behaviour” among researchers, he added.

He favours the case study approach to measuring impact but added that Mr Moedas should not rush into impact assessment before learning the lessons that other research systems have.

Kathrin Möslein, chair for information systems innovation and value creation at the University of Erlangen-Nuremberg, said that Mr Moedas should set up a comparative project that looks at the most promising practices in impact assessment.

In Germany, researchers are usually asked to produce impact strategies in the early phase of grant proposals, and previous achievements of impact are usually assessed alongside the project proposal. At the end of a project, the evidence of impact is again considered. “[Impact] is usually not an issue discussed separately from research, but an integral part of research practice,” she added.

In the US, a project is under way that looks at the economic impact of research in a very different way.

The UMETRICS project at the Institute for Research on Innovation and Science at the University of Michigan uses national statistics – such as the number of businesses started and job creation – to look at the impact individual researchers have compared with those not working on research projects.

Julia Lane, professor in the Robert F. Wagner Graduate School of Public Service at New York University, thinks that Mr Moedas should take heed of the US system, adding that the way research impact was measured in the UK was “a bureaucratic nightmare” that was “a waste of time and energy and has been pretty unconvincing”, she said.

“It is not serious, it is not scientific. So for the scientific community to have such an unscientific approach is an embarrassment,” she added.

In other parts of the world, the impact agenda is less developed.

In China and Hong Kong, measures of impact are not required for research funding. Walter Ho, director of the Office of Research and Knowledge Transfer services at the Chinese University of Hong Kong, said: “At the moment, more focus is being directed to technology transfer rather than impact. Some metrics such as [the] number of patents granted, companies formed and licensing income are evaluated.”

But he is anticipating that this will change in the years ahead. “The Research Grants Council in Hong Kong will probably require universities in Hong Kong to report impact cases in the next round of assessment exercise in 2018,” he added.

holly.else@tesglobal.com