The purpose of this blog is to provide a platform for commentary on science matters. The Office of the Prime Minister’s Science Advisory Committee is politically independent and will refrain from political debate.


Impact – Early attention and a broader lens on potential impacts can pay off

Several months ago, I wrote an essay discussing the different meanings that various stakeholders have applied to the concept of ‘impact’ in research. That paper discussed how varying perspectives on the meaning of impact influence how research policy is developed and applied. Increasingly governments and taxpayers want to understand the impacts of their investment in public R&D. Because of this, there has been increasing commentary, including my own, on the need for more inclusive ways to think about – and especially to measure – the full breadth of research impact.

Understanding the broad range of impacts of research is important because ultimately governments have to weigh their investments in public R&D against other potential demands on national coffers, and in my previous discussion I argued that it is therefore important to have a comprehensive view of the range of impacts that benefit a nation and justify the increasingly large investments that societies make in public research. This is not easily done as there is no accepted taxonomy of impact and the range of potential impacts can extend from academic and reputational value through to smarter public policies and practices, and to direct economic benefits.

The assessment of impact, as it is often employed in academic circles, refers to the semi-objective measurement of publication quality. This is increasingly understood to be an inappropriate use of both the metric and the term, yet it has become embedded in the academic lexicon, not least because of the sector’s persistent emphasis on journal ‘impact factors’. This understanding of the term ‘impact’ has been carried through to the development of funding tools in tertiary education based on various measures of academic performance.

Now however, novel assessment tools are beginning to appear which try and address the need for a fuller appreciation of impact. Perhaps most well developed is the UK’s use of ‘impact stories’ in its Research Excellence Framework, which recognises that a broad range of stories may usefully describe the spectrum of potential impacts.

Evaluation specialists at the science-policy interface are indeed pushing the conventional boundaries, with some now also considering behavioural changes in both the science and policy communities as hitherto unaccounted impacts that research – and especially new types of research structures – can have.

This week, we can add to this list of endeavours a piece of work that has been developed over the past two years by the Small Advanced Economies Initiative (SAEI), for which New Zealand provides the international secretariat The new SAEI framework document, Broadening the Scope of Impact, which was jointly led by the SAEI secretariat and Science Foundation Ireland and supported by input from agencies across the SAEI network, provides a taxonomy for considering the broad range of impacts for significantly sized research efforts such as centres of excellence. It also discusses the notion that impacts should be considered ex-ante to increase the potential for a given research program to produce them. It argues for a mixed methods approach to assessment, whereby the funder and research group jointly explore the potential impacts of research ex- ante and agree how to measure them. Not only is this approach likely a better way of collecting relevant information, it can also generate important positive change in the attitudes and practices of both researchers and funders. Of the six economies that co-developed this framework (Denmark, Finland, Ireland, Israel, Singapore and New Zealand), some are already experimenting with its application. Hopefully such initiatives and concepts will themselves impact on the way we think about, structure and ultimately measure our publicly funded research.

Comments are closed.