What happened in Reykjavik…
As Richard Harvey remarked, working groups need to go through the phases of storming, forming, norming and performing. The working group on Societal Impact & Relevance of Research (SIRR for short) had started its storming discussion in October in Amsterdam, but needed some follow up on that in Reykjavik, focusing on the key question “Why does my university want to engage in SIRR?”
This applied in particular to the topic of Assessment mechanisms for SIRR; for Open Access & Fair Data, the next step of ‘forming’ seems closer at hand. For a third topic: ‘disruptive research’, the ‘storming phase’ is still in its early moments.
Assessment mechanisms for SIRR
We formulated as our vision that the Aurora network can clarify the university’s role in the society and the university role regarding the global challenges. In order to do this we need to demonstrate the societal value and relevance of research and education. We want to use the network for benchmarking and exchange of experience (and maybe peers).
Although for systems/mechanisms for the evaluation/assessment of SIRR Aurora members are dependent on national policies that are beyond their control, it is still important that we formulate our own ideas on what these systems/mechanisms should look like to serve our universities’ mission.
We want to use the Aurora SIRR group to articulate these ideas, also in terms of what the design of evaluation/assessment systems should look like and how we can avoid ‘gaming’ and perverted application of these systems. We aim for SIRR that respects the core academic values ánd helps us to improve our interaction with society and government.
We already identified some design elements:
- Following the LERU document we conceptualize impact as an outcome of a non-linear, collaborative co-creation process with focus on productive interactions as impact indicators
- We would like to explicitly add the importance of public participation/engagement in realizing societal impact
- We acknowledge and respect that not all academics may have to engage in SIRR; we recognize the importance of ‘blue sky research’. Tension between the focus on SIRR and the importance of Blue Sky research is acknowledged.
We underlined that it is important not to reinvent the wheel and to build on already existing valuable elements for evaluation/assessment of SIRR. We noted two current Societal impact assessment systems for research specifically:
- The Standard Evaluation Protocol (SEP) is used in the Netherlands and (slightly differently) in Flanders, with a focus on (1) products, (2) use of products and (3) marks of recognition (relatively quantitative – listing of items). Assessment in SEP takes place at the department level for recognition.
- The Research Excellence Framework (REF 14) is used in the UK. It looks more qualitatively at: (1) research environment (e.g. department strategies to encourage SI) and (2) case studies which includes theory of change (mechanisms) besides outcomes/ changes (significance & reach). Assessment is done at university level for recognition AND budget allocation.
We recognize the need for mixed methods approach, using the best elements of various existing systems. We also need to take heed of the considerable investment in time and energy that the SEP and REF require.
In a wider perspective, it is important to see SIRR in conjunction with the societal impact and relevance of education (e.g. through community service learning) and the socio-economic impact of a research university on its immediate surroundings.
Open Access and Fair Data
We discussed how we can profit from Aurora in our universities’ aspirations in this field on the basis of an excellent presentation by Arjan Schalken from VU.
We noted that the “Amsterdam Call for Action on Open Science” (2016) provides a good framework and action list for Aurora members and the network as a whole to stimulate Open Science & Fair Data.
The survey taken by Arjan among Aurora members serves as a good starting point.
… and how we will take it further
Over the next months, the members of the SIRR group will focus on:
- the evaluation/ assessment mechanisms: formulating our own ideas on an ‘ideal’ system to better respond to the systems to which we are subjected;
- the Open Access & Fair Data agenda set by the Amsterdam Call for Action.
We will work closely together with the Bibliometrics group in both action lines