SDG Analysis: Bibliometrics of Relevance

The mission of the Aurora-network is to tackle global societal challenges in areas like the Sustainable Development Goals of the United Nations.

This project creates a dashboard which demonstrates the Societal Relevance and Societal Impact of Research of Aurora Universities. This dashboard shows the research contributions in these societal challenges, and how policymakers have used the research available to tackle these challenges.

The data dashboard you see above has the status of a proof of principle. Researchers helped us improve the classification model behind this dashboard. We are currently processing the survey data. The survey results and the new classification model will be released in the second half of this year.

The dashboard is interactive, you can click on the SDG Icons, the University names, and the data points in the graphs to change the perspectives or highlight parts of the data.

A Look Inside

The animation above (click to view) shows several interactive diagrams slicing different ways through the data.

SDG Contributions: shows the publications found per SDG for each university.
Aurora SDG Map: Also shows the publications found but placed on the map locating each Aurora University. (Click on the SDGs to see the bubbles change)
SDG influence indicators: Shows the publications found, plus the scientific citations, the level of Open Access, and the use of the SDG papers in policy document from (Non-) Governmental Organisations.
The Universities Profile: shows the ‘profile’ of a university-based on the scientific excellence and the societal policy uptake of the papers. (Click on the universities to see the different profiles.)
The Universities Graph: shows the difference between each university regarding scientific excellence and the societal policy uptake of the papers. (Click on the SDGs to see how the universities compare per SDG)
Societal Attention: Policymakers and Journal readership; shows the (Non) Governmental Organisations that use the SDG 13- Climate Action Papers in their policy documents, it also shows the journals that these policymakers read. (Select the NGO’s to see the journals they read and maybe researchers might publish in to reach their audience. But also select the journals, to see in reverse, what policymakers read in particular that particular journals)

Explanatory example:

The bubble diagram above shows what SDG’s the Universities in the Aurora Consortium contribute the most (size of the bubble), have most research articles in the top 10% percentile of most cited Journals (horizontal axis), and have been mentioned the most in policy documents at national governments and non-governmental organisations (vertical axis).

Method

SDG queries are created and reviewed by bibliometricians in the Aurora network. The bibliometric instrument performs per SDG a query on a bibliometric database, in this case, Scopus. The result is a collection of papers per SDG. Additional information is added to these papers, top 10% percentiles as a measure for research excellence (using SciVal), open access status as a measurement for availability of those papers to society (using ImpactStory), and the mentions in references to those papers in public policy documents from (N)GO’s (using Altmetric).

Read more about the concept in the Whitepaper 2017 [1] and the presentation at the Liber congress in 2019 [2], software available on Github [3]

[1] Whitepaper | Maurice Vanderfeesten, & René Otten. (2017, November). Societal Relevant Impact: Potential analysis for Aurora-Network university leaders to strengthen collaboration on societal challenges. Zenodo. http://doi.org/10.5281/zenodo.1041405

[2] Presentation | Vanderfeesten, Maurice, Otten, René, Both, Joeri, Schmidt, Felix, Spielberg, Eike, Kullman, Lars, & Farar, Jaqui. (2019, June). How does Our Research Influence Policy on Global Societal Changes? A Bibliometric Proof of Concept Targeting the Sustainable Development Goals of the United Nations. Zenodo. https://doi.org/10.5281/zenodo.3260279

[3] Software | https://github.com/Aurora-Network-Global

PROJECT 2: SDG CLASSIFICATION IMPROVEMENT [2019-2020]

Objective (And Relation To Aurora Mission And Vision)

Strategic objective:

To provide Aurora university leaders with a tool to strengthen the focus on societal challenges within their university and in the research collaboration outside and within Aurora.

Specific objective:

To show the distribution of research-output that the Aurora universities and the Aurora network as a whole are contributing to each of the 17 sustainable development goals. (There is overlap with the 7 EU societal challenges).

Intended effect:

To enhance a ‘problem/challenge’ based approach – in contrast to a disciplinary approach – to assessing quality and impact of research as a mechanism to strengthen to focus on (and appreciation for) problem-oriented research, both within Aurora and among policy makers at national and European level.
To increase insight in research strengths and complementarities within the aurora network, as a tool to help find current and potential collaboration in multi-disciplinary research areas, and forge strategic alliances within societal relevant themes across the universities in the network.

Main Deliverables

All Aurora universities are represented in the SDG tool v1. (As showcase for “Macron Initiative”)
An IMPROVED/MATURE v2 bibliometric tool to survey and analyse the research output and impact of a unit (university, universities network) in terms of its societal relevance, academic quality & impact, societal attention passed on policy documents (possibly also news and patents).
Visualisations based on the collected data, that help users (policy-, grant-, and communication-advisors) answer their questions for demonstrating / stimulating societal impact.
Tested the effectiveness of the SDG tool on Use cases

Main Activities / Work Packages

On-boarding & Macron Initiative showcase: the other remaining universities need to be brought up to the speed of this project, and add their data to the existing v1 model.
- Online instruction for bibliometricians from UEA, Iceland, Antwerp and Bergen to enter their baseline data in a shared space, and add entries to the “SDG-data-entry” file.
- Run Power BI on new data for dashboard with all Aurora partners.
- Instruct Aurora institutional coordinators about the validity of the data.

Query crafting: For each of the 17 SDG’s a new v2 search query is crafted, reviewed and tested.

- Describe the new query crafting method, that maximises the recall and precision. (incl. searching for literature, to re-use proven methods)
- Setup collaborative environment (with features like commenting, version control and ) for working on the queries, and sharing test/validation results. Google Drive / Docs
- Each institution has to identify one or more key researchers per SDG, who are available for advisory work.
- Bibliometricians need to ask these key researchers what keywords are missing from the current query, and what publications and journals should appear in the query results that reflect that SDG. They need this information later on for creating an SDG Corpus, and to craft a new SDG query that outperforms the previous one.
- Agree on a common working and notation procedure (how to write readable query syntaxes, to make it easier to check for errors.
- New SDG queries are crafted, reviewed and tested/validated on sample size, recall and precision.
  - 4 SDG’s are divided among the aurora universities. The VU will make the final review.

Data collection: Data points are entered in a common sdg-data-entry file or database.

- Preparation
  - Find developer for automation
  - Decide on how and where the data is being collected. (file on a shared drive, database, etc.) And what runtime environment to use, under what campus IP-range. (An Aurora university who has access to Scopus, Scival-, Altmetric- and Unpaywall-API’s)

- Setup a shared data space, and runtime environment.

- Development:
  - Write and test scripts
- Data-acquisition:
  - publications-data [sept-oct]: SDG queries are run on Scopus to get the publication metadata per SDG for each institution.
  - performance- and impact-data [oct-nov]: Publication identifiers are parsed through Scival, Altmetric and unpaywall-data to get the data on Top 10% percentiles, Policy mentions and Open Access status.

Data-Visualisation: data points are used in business intelligence tool.
- Connect the database to PowerBI
- Update existing Visualisation
- Workshop PowerBI / Elastic Search / Kibana
- Discuss with users for additional visuals
- Investigate co-authorship network graphs (eg. VOSviewer)

Publish & intellectual property: Publish results on method and utilisation of SDG Universities Profile.
- Determine what parts of the project should be protected, but still can be made open as possible.
- Publish about the non-restricted information, in a peer-reviewed open-access journal.

Outreach & communication: Present Aurora’s SDG Universities profile v2 to UN and EC DG research.
- Announce project start
- Announce v1 results for macron initiative showcase
- Announce v2 results
- Announce publication
- Announce evaluation results

Evaluate: determine the effectiveness of the SDG tool v2 based on Use cases defined in the workshop in Essen
- Create a research-/test-plan, and determine the definition of ‘success’ for each use case.
- Instruct grant-officers/ communication officers/policy advisers on how to use the tool.
- Gather information on the success rate.

PROJECT 1: SDG PROOF-OF-PRINCIPLE [FINISHED] [2017-2018]

Main Deliverable

A bibliometric tool to survey and analyse the research output and impact of a unit (department, faculty, university, universities network) in terms of its *) societal relevance, *) academic quality & impact, *) societal impact.

If possible, the tool will offer a mechanism to combine and reconcile the societal categorisations for research of UNESCO´s Sustainable Development Goals and the European Union´s Societal Challenges.

Intended Effect

To enhance a ‘problem/challenge’ based approach – in contrast to a disciplinary approach – to assessing quality and impact of research as a mechanism to strengthen to focus on (and appreciation for) problem-oriented research, both within Aurora and among policymakers at national and European level.

To increase insight into research strengths and complementarities within the aurora network, as a tool to help forge strategic alliances within societal relevant themes across the universities in the network.

Method

The bibliometric instrument performs per SDG a query on a bibliometric database eg. Scopus or Web of Science. (SDG queries are created and reviewed by bibliometricians in the Aurora network.) For each SDG the publication skew is determined (based on a number of papers published), for those papers the excellence is determined (top percentiles of journals), the availability of those papers to society (open access), and the references to those papers in public policy documents from (N)GO’s. The interactive diagram below shows a proof of concept how this could look like for a limited selection of SDG’s.

Read more about the concept in the Whitepaper [1]

The illustration below shows the hypothetical feedback-loop between policy and research. Here we illustrate the possible influence policy from societal partners like (N)GO bodies can have on the research output on policy-topics. And on the other hand, that open and excellent research can be fed-back into re-evaluation or creating a new policy. Thus research influencing society in an in-direct level.

[1] Maurice Vanderfeesten, & René Otten. (2017, November). Societal Relevant Impact: Potential analysis for Aurora-Network university leaders to strengthen collaboration on societal challenges. Zenodo. http://doi.org/10.5281/zenodo.1041405