The mission of the Aurora-network is to tackle global societal challenges in areas like the Sustainable Development Goals of the United Nations.
This project creates a dashboard which demonstrates the Societal Relevance and Societal Impact of Research of Aurora Universities. This dashboard shows the research contributions in these societal challenges, and how policymakers have used the research available to tackle these challenges.
The data dashboard you see above has the status of a proof of principle. Researchers helped us improve the classification model behind this dashboard. We are currently processing the survey data. The survey results and the new classification model will be released in the second half of this year.
The dashboard is interactive, you can click on the SDG Icons, the University names, and the data points in the graphs to change the perspectives or highlight parts of the data.
A Look Inside
The animation above (click to view) shows several interactive diagrams slicing different ways through the data.
- SDG Contributions: shows the publications found per SDG for each university.
- Aurora SDG Map: Also shows the publications found but placed on the map locating each Aurora University. (Click on the SDGs to see the bubbles change)
- SDG influence indicators: Shows the publications found, plus the scientific citations, the level of Open Access, and the use of the SDG papers in policy document from (Non-) Governmental Organisations.
- The Universities Profile: shows the ‘profile’ of a university-based on the scientific excellence and the societal policy uptake of the papers. (Click on the universities to see the different profiles.)
- The Universities Graph: shows the difference between each university regarding scientific excellence and the societal policy uptake of the papers. (Click on the SDGs to see how the universities compare per SDG)
- Societal Attention: Policymakers and Journal readership; shows the (Non) Governmental Organisations that use the SDG 13- Climate Action Papers in their policy documents, it also shows the journals that these policymakers read. (Select the NGO’s to see the journals they read and maybe researchers might publish in to reach their audience. But also select the journals, to see in reverse, what policymakers read in particular that particular journals)
The bubble diagram above shows what SDG’s the Universities in the Aurora Consortium contribute the most (size of the bubble), have most research articles in the top 10% percentile of most cited Journals (horizontal axis), and have been mentioned the most in policy documents at national governments and non-governmental organisations (vertical axis).
SDG queries are created and reviewed by bibliometricians in the Aurora network. The bibliometric instrument performs per SDG a query on a bibliometric database, in this case, Scopus. The result is a collection of papers per SDG. Additional information is added to these papers, top 10% percentiles as a measure for research excellence (using SciVal), open access status as a measurement for availability of those papers to society (using ImpactStory), and the mentions in references to those papers in public policy documents from (N)GO’s (using Altmetric).
Read more about the concept in the Whitepaper 2017  and the presentation at the Liber congress in 2019 , software available on Github 
 Whitepaper | Maurice Vanderfeesten, & René Otten. (2017, November). Societal Relevant Impact: Potential analysis for Aurora-Network university leaders to strengthen collaboration on societal challenges. Zenodo. http://doi.org/10.5281/zenodo.1041405
 Presentation | Vanderfeesten, Maurice, Otten, René, Both, Joeri, Schmidt, Felix, Spielberg, Eike, Kullman, Lars, & Farar, Jaqui. (2019, June). How does Our Research Influence Policy on Global Societal Changes? A Bibliometric Proof of Concept Targeting the Sustainable Development Goals of the United Nations. Zenodo. https://doi.org/10.5281/zenodo.3260279
 Software | https://github.com/Aurora-Network-Global
PROJECT 2: SDG CLASSIFICATION IMPROVEMENT [2019-2020]
To provide Aurora university leaders with a tool to strengthen the focus on societal challenges within their university and in the research collaboration outside and within Aurora.
To show the distribution of research-output that the Aurora universities and the Aurora network as a whole are contributing to each of the 17 sustainable development goals. (There is overlap with the 7 EU societal challenges).
- To enhance a ‘problem/challenge’ based approach – in contrast to a disciplinary approach – to assessing quality and impact of research as a mechanism to strengthen to focus on (and appreciation for) problem-oriented research, both within Aurora and among policy makers at national and European level.
- To increase insight in research strengths and complementarities within the aurora network, as a tool to help find current and potential collaboration in multi-disciplinary research areas, and forge strategic alliances within societal relevant themes across the universities in the network.
- All Aurora universities are represented in the SDG tool v1. (As showcased for “Macron Initiative”)
- An IMPROVED/MATURE v2 bibliometric tool to survey and analyse the research output and impact of a unit (university, universities network) in terms of its societal relevance, academic quality & impact, societal attention passed on policy documents (possibly also news and patents).
- Visualisations based on the collected data, that help users (policy-, grant-, and communication-advisors) answer their questions for demonstrating / stimulating societal impact.
- Tested the effectiveness of the SDG tool on Use cases
1. All Aurora universities are represented in the SDG tool v1. (As showcased for “Macron Initiative”)
The Interactive SDG dashboard, as shown above
2. An IMPROVED/MATURE v2 bibliometric tool to survey and analyze the research output and impact of a unit (university, universities network) in terms of its societal relevance, academic quality & impact, societal attention passed on policy documents (possibly also news and patents).
New SDG queries (rigorously improved SDG classification model). This is the new classification model for categorizing research output related to one of the SDG’s. Qualifications for improvements were to increase recall (completeness) and precision (soundness), to represent better the universities’ output and the targets within the goals.
These are the by-products that we necessary to produce the Primary deliverable.
Aurora SDG Data Collection (including the “golden set”) All the data of the by-products, including
- The old SDG classification models
- The SDG survey data (“golden set”)
- The SDG text analysis data
- The new SDG classification model
To point out the SDG survey data contains the “golden set”; validated research papers hand-picked by the expert researcher in a particular SDG field, that can be used by other SDG classification projects to evaluate their own models.
This interactive platform shows the live data of the different text analysis reports from CorTexT for each SDG based on the SDG survey outcomes.
- Term Extraction: after text normalisation (stemming, etc) we extracted 2 terms in bigrams and trigrams that co-occurred the most per document, in the title, abstract and keyword
- Contrast analysis: the co-occurring terms in publications (title, abstract, keywords), of the papers that respondents have indicated, relate to this SDG (y-axis: True), and that have been rejected (x-axis: False). In the top left, you’ll see term co-occurrences that a clearly relate to this SDG. The bottom-right are terms that appear in papers that have been rejected for this SDG. The top-right terms appear frequently in both and cannot be used to discriminate between the two groups.
- Network map: This diagram shows the cluster-network of terms co-occurring in the publications related to this SDG, selected by the respondents (accepted publications only).
- Topic model: This diagram shows the topics and the related terms that make up that topic. The number of topics is related to the number of targets of this SDG.
- Contingency matrix: This diagram shows the top 10 of co-occurring terms that correlate the most.
SDG queries Co-creation platform on GitHub
This model is put on GitHub, a code-community platform that is highly facilitating co-creation. We are aware this project will end, and have it on a platform where it can be shared an improved by an open-source community is in the spirit of open science.
3. Visualizations based on the collected data, that help users (policy-, grant-, and communication-advisors) answer their questions for demonstrating / stimulating societal impact.
[to be delivered]
4. Tested the effectiveness of the SDG tool on Use cases
[to be delivered]
- On-boarding & Macron Initiative showcase: the other remaining universities need to be brought up to the speed of this project, and add their data to the existing v1 model.
- Online instruction for bibliometricians from UEA, Iceland, Antwerp and Bergen to enter their baseline data in a shared space, and add entries to the “SDG-data-entry” file.
- Run Power BI on new data for dashboard with all Aurora partners.
- Instruct Aurora institutional coordinators about the validity of the data.
- Query crafting: For each of the 17 SDG’s a new v2 search query is crafted, reviewed and tested.
- Describe the new query crafting method, that maximises the recall and precision. (incl. searching for literature, to re-use proven methods)
- Setup collaborative environment (with features like commenting, version control and ) for working on the queries, and sharing test/validation results. Google Drive / Docs
- Each institution has to identify one or more key researchers per SDG, who are available for advisory work.
- Bibliometricians need to ask these key researchers what keywords are missing from the current query, and what publications and journals should appear in the query results that reflect that SDG. They need this information later on for creating an SDG Corpus, and to craft a new SDG query that outperforms the previous one.
- Agree on a common working and notation procedure (how to write readable query syntaxes, to make it easier to check for errors.
- New SDG queries are crafted, reviewed and tested/validated on sample size, recall and precision.
- 4 SDG’s are divided among the aurora universities. The VU will make the final review.
- Data collection: Data points are entered in a common sdg-data-entry file or database.
- Find developer for automation
- Decide on how and where the data is being collected. (file on a shared drive, database, etc.) And what runtime environment to use, under what campus IP-range. (An Aurora university who has access to Scopus, Scival-, Altmetric- and Unpaywall-API’s)
- Setup a shared data space, and runtime environment.
- Write and test scripts
- publications-data [sept-oct]: SDG queries are run on Scopus to get the publication metadata per SDG for each institution.
- performance- and impact-data [oct-nov]: Publication identifiers are parsed through Scival, Altmetric and unpaywall-data to get the data on Top 10% percentiles, Policy mentions and Open Access status.
- Data-Visualisation: data points are used in business intelligence tool.
- Connect the database to PowerBI
- Update existing Visualisation
- Workshop PowerBI / Elastic Search / Kibana
- Discuss with users for additional visuals
- Investigate co-authorship network graphs (eg. VOSviewer)
- Publish & intellectual property: Publish results on method and utilisation of SDG Universities Profile.
- Determine what parts of the project should be protected, but still can be made open as possible.
- Publish about the non-restricted information, in a peer-reviewed open-access journal.
- Outreach & communication: Present Aurora’s SDG Universities profile v2 to UN and EC DG research.
- Announce project start
- Announce v1 results for macron initiative showcase
- Announce v2 results
- Announce publication
- Announce evaluation results
- Evaluate: determine the effectiveness of the SDG tool v2 based on Use cases defined in the workshop in Essen
- Create a research-/test-plan, and determine the definition of ‘success’ for each use case.
- Instruct grant-officers/ communication officers/policy advisers on how to use the tool.
- Gather information on the success rate.
PROJECT 1: SDG PROOF-OF-PRINCIPLE [FINISHED] [2017-2018]
A bibliometric tool to survey and analyse the research output and impact of a unit (department, faculty, university, universities network) in terms of its *) societal relevance, *) academic quality & impact, *) societal impact.
If possible, the tool will offer a mechanism to combine and reconcile the societal categorisations for research of UNESCO´s Sustainable Development Goals and the European Union´s Societal Challenges.
To enhance a ‘problem/challenge’ based approach – in contrast to a disciplinary approach – to assessing quality and impact of research as a mechanism to strengthen to focus on (and appreciation for) problem-oriented research, both within Aurora and among policymakers at national and European level.
To increase insight into research strengths and complementarities within the aurora network, as a tool to help forge strategic alliances within societal relevant themes across the universities in the network.
The bibliometric instrument performs per SDG a query on a bibliometric database eg. Scopus or Web of Science. (SDG queries are created and reviewed by bibliometricians in the Aurora network.) For each SDG the publication skew is determined (based on a number of papers published), for those papers the excellence is determined (top percentiles of journals), the availability of those papers to society (open access), and the references to those papers in public policy documents from (N)GO’s. The interactive diagram below shows a proof of concept how this could look like for a limited selection of SDG’s.
Read more about the concept in the Whitepaper 
The illustration below shows the hypothetical feedback-loop between policy and research. Here we illustrate the possible influence policy from societal partners like (N)GO bodies can have on the research output on policy-topics. And on the other hand, that open and excellent research can be fed-back into re-evaluation or creating a new policy. Thus research influencing society in an in-direct level.
 Maurice Vanderfeesten, & René Otten. (2017, November). Societal Relevant Impact: Potential analysis for Aurora-Network university leaders to strengthen collaboration on societal challenges. Zenodo. http://doi.org/10.5281/zenodo.1041405
- Jan 2018: Project Kick-off
- Feb 2018: Initial version of 17 SDG queries
- March 2018: Reviewed version of 17 SDG queries
- April 2018: data collection & first analysis based on 17 SDG queries
- May 2018: publication of Aurora SDG university profiles
- June 2018: evaluation on utalisation of SDG university profiles
- Sept 2017: Presentation of Proof of Concept for Dies Natalis Speech from Prof Pieter vd Beukering about Sustainability
- Nov 2017: White paper with proof of concept on Potential analysis for Aurora-Network university leaders to strengthen collaboration on societal challenges
- Dec 2017: Team Site for Aurora Bibliometricians
- Feb 2018: Seminar on Societal impact.
- Feb 2018: Initial DRAFT version of 17 SDG queries [final version in available for project members]
- April 2018: Interactive diagram