SDG Analysis: Bibliometrics of Relevance

Strategic objective:

To provide Aurora university leaders with a tool to strengthen the focus on societal challenges within their university and in the research collaboration outside and within Aurora.

Specific objective:

To show the distribution of research-output that the Aurora universities and the Aurora network as a whole are contributing to each of the 17 sustainable development goals. (There is overlap with the 7 EU societal challenges).

Intended effect:

1. To enhance a ‘problem/challenge’ based approach – in contrast to a disciplinary approach – to assessing quality and impact of research as a mechanism to strengthen to focus on (and appreciation for) problem-oriented research, both within Aurora and among policy makers at national and European level.
2. To increase insight in research strengths and complementarities within the aurora network, as a tool to help find current and potential collaboration in multi-disciplinary research areas, and forge strategic alliances within societal relevant themes across the universities in the network.

1. All Aurora universities are represented in the SDG tool v1. (As showcased for “Macron Initiative”)
2. An IMPROVED/MATURE v2 bibliometric tool to survey and analyse the research output and impact of a unit (university, universities network) in terms of its societal relevance, academic quality & impact, societal attention passed on policy documents (possibly also news and patents).
3. Visualisations based on the collected data, that help users (policy-, grant-, and communication-advisors) answer their questions for demonstrating / stimulating societal impact.
4. Tested the effectiveness of the SDG tool on Use cases

1. All Aurora universities are represented in the SDG tool v1. (As showcased for “Macron Initiative”)

The Interactive SDG dashboard, as shown above

2. An IMPROVED/MATURE v2 bibliometric tool to survey and analyze the research output and impact of a unit (university, universities network) in terms of its societal relevance, academic quality & impact, societal attention passed on policy documents (possibly also news and patents).

Primary deliverable

New SDG queries (rigorously improved SDG classification model). This is the new classification model for categorizing research output related to one of the SDG’s. Qualifications for improvements were to increase recall (completeness) and precision (soundness), to represent better the universities’ output and the targets within the goals.

This model is both human and machine-readable, freely and openly available in a way it can be shared and improved in the future and can track the provenance of those changes.

Secondary Deliverables

These are the by-products that we necessary to produce the Primary deliverable.

Aurora SDG Data Collection (including the “golden set”) All the data of the by-products, including

1. • The old SDG classification models
   • The SDG survey data (“golden set”)
   • The SDG text analysis data
   • The new SDG classification model

To point out the SDG survey data contains the “golden set”; validated research papers hand-picked by the expert researcher in a particular SDG field, that can be used by other SDG classification projects to evaluate their own models.

SDG Text Analysis dashboard

This interactive platform shows the live data of the different text analysis reports from CorTexT for each SDG based on the SDG survey outcomes.

• Term Extraction: after text normalisation (stemming, etc) we extracted 2 terms in bigrams and trigrams that co-occurred the most per document, in the title, abstract and keyword
• Contrast analysis: the co-occurring terms in publications (title, abstract, keywords), of the papers that respondents have indicated, relate to this SDG (y-axis: True), and that have been rejected (x-axis: False). In the top left, you’ll see term co-occurrences that a clearly relate to this SDG. The bottom-right are terms that appear in papers that have been rejected for this SDG. The top-right terms appear frequently in both and cannot be used to discriminate between the two groups.
• Network map: This diagram shows the cluster-network of terms co-occurring in the publications related to this SDG, selected by the respondents (accepted publications only).
• Topic model: This diagram shows the topics and the related terms that make up that topic. The number of topics is related to the number of targets of this SDG.
• Contingency matrix: This diagram shows the top 10 of co-occurring terms that correlate the most.

This can be used by other SDG classification projects to enrich their models with our findings.

SDG queries Co-creation platform on GitHub

This model is put on GitHub, a code-community platform that is highly facilitating co-creation. We are aware this project will end, and have it on a platform where it can be shared an improved by an open-source community is in the spirit of open science.

The model is both human and machine-readable, freely and openly available in a way it can be shared and improved in the future and is able to track the provenance of those changes.

3. Visualizations based on the collected data, that help users (policy-, grant-, and communication-advisors) answer their questions for demonstrating / stimulating societal impact.

[to be delivered]

4. Tested the effectiveness of the SDG tool on Use cases

[to be delivered]

1. On-boarding & Macron Initiative showcase: the other remaining universities need to be brought up to the speed of this project, and add their data to the existing v1 model.
   • Online instruction for bibliometricians from UEA, Iceland, Antwerp and Bergen to enter their baseline data in a shared space, and add entries to the “SDG-data-entry” file.
   • Run Power BI on new data for dashboard with all Aurora partners.
   • Instruct Aurora institutional coordinators about the validity of the data.

2. Query crafting: For each of the 17 SDG’s a new v2 search query is crafted, reviewed and tested.

• • Describe the new query crafting method, that maximises the recall and precision. (incl. searching for literature, to re-use proven methods)
  • Setup collaborative environment (with features like commenting, version control and ) for working on the queries, and sharing test/validation results. Google Drive / Docs
  • Each institution has to identify one or more key researchers per SDG, who are available for advisory work.
  • Bibliometricians need to ask these key researchers what keywords are missing from the current query, and what publications and journals should appear in the query results that reflect that SDG. They need this information later on for creating an SDG Corpus, and to craft a new SDG query that outperforms the previous one.
  • Agree on a common working and notation procedure (how to write readable query syntaxes, to make it easier to check for errors.
  • New SDG queries are crafted, reviewed and tested/validated on sample size, recall and precision.
    • 4 SDG’s are divided among the aurora universities. The VU will make the final review.

3. Data collection: Data points are entered in a common sdg-data-entry file or database.

• • Preparation
    • Find developer for automation
    • Decide on how and where the data is being collected. (file on a shared drive, database, etc.) And what runtime environment to use, under what campus IP-range. (An Aurora university who has access to Scopus, Scival-, Altmetric- and Unpaywall-API’s)

• • Setup a shared data space, and runtime environment.

• • Development:
    • Write and test scripts
  • Data-acquisition:
    • publications-data [sept-oct]: SDG queries are run on Scopus to get the publication metadata per SDG for each institution.
    • performance- and impact-data [oct-nov]: Publication identifiers are parsed through Scival, Altmetric and unpaywall-data to get the data on Top 10% percentiles, Policy mentions and Open Access status.

4. Data-Visualisation: data points are used in business intelligence tool.
   • Connect the database to PowerBI
   • Update existing Visualisation
   • Workshop PowerBI / Elastic Search / Kibana
   • Discuss with users for additional visuals
   • Investigate co-authorship network graphs (eg. VOSviewer)

5. Publish & intellectual property: Publish results on method and utilisation of SDG Universities Profile.
   • Determine what parts of the project should be protected, but still can be made open as possible.
   • Publish about the non-restricted information, in a peer-reviewed open-access journal.

6. Outreach & communication: Present Aurora’s SDG Universities profile v2 to UN and EC DG research.
   • Announce project start
   • Announce v1 results for macron initiative showcase
   • Announce v2 results
   • Announce publication
   • Announce evaluation results

7. Evaluate: determine the effectiveness of the SDG tool v2 based on Use cases defined in the workshop in Essen
   • Create a research-/test-plan, and determine the definition of ‘success’ for each use case.
   • Instruct grant-officers/ communication officers/policy advisers on how to use the tool.
   • Gather information on the success rate.

A bibliometric tool to survey and analyse the research output and impact of a unit (department, faculty, university, universities network) in terms of its *) societal relevance, *) academic quality & impact, *) societal impact.

If possible, the tool will offer a mechanism to combine and reconcile the societal categorisations for research of UNESCO´s Sustainable Development Goals and the European Union´s Societal Challenges.

To enhance a ‘problem/challenge’ based approach – in contrast to a disciplinary approach – to assessing quality and impact of research as a mechanism to strengthen to focus on (and appreciation for) problem-oriented research, both within Aurora and among policymakers at national and European level.

To increase insight into research strengths and complementarities within the aurora network,  as a tool to help forge strategic alliances within societal relevant themes across the universities in the network.

The bibliometric instrument performs per SDG a query on a bibliometric database eg. Scopus or Web of Science. (SDG queries are created and reviewed by bibliometricians in the Aurora network.) For each SDG the publication skew is determined (based on a number of papers published), for those papers the excellence is determined (top percentiles of journals), the availability of those papers to society (open access), and the references to those papers in public policy documents from (N)GO’s. The interactive diagram below shows a proof of concept how this could look like for a limited selection of SDG’s.

Read more about the concept in the Whitepaper  [1]

The illustration below shows the hypothetical feedback-loop between policy and research. Here we illustrate the possible influence policy from societal partners like (N)GO bodies can have on the research output on policy-topics. And on the other hand, that open and excellent research can be fed-back into re-evaluation or creating a new policy. Thus research influencing society in an in-direct level.

[1] Maurice Vanderfeesten, & René Otten. (2017, November). Societal Relevant Impact: Potential analysis for Aurora-Network university leaders to strengthen collaboration on societal challenges. Zenodo. http://doi.org/10.5281/zenodo.1041405

• Jan 2018: Project Kick-off
• Feb 2018: Initial version of 17 SDG queries
• March 2018: Reviewed version of 17 SDG queries
• April 2018: data collection & first analysis based on 17 SDG queries
• May 2018: publication of Aurora SDG university profiles
• June 2018: evaluation on utalisation of SDG university profiles

• Sept 2017: Presentation of Proof of Concept for Dies Natalis Speech from Prof Pieter vd Beukering about Sustainability
• Nov 2017: White paper with proof of concept on Potential analysis for Aurora-Network university leaders to strengthen collaboration on societal challenges
• Dec 2017: Team Site for Aurora Bibliometricians
• Feb 2018: Seminar on Societal impact.
• Feb 2018: Initial DRAFT version of 17 SDG queries [final version in available for project members]
• April 2018: Interactive diagram