SDG Analysis: Bibliometrics of Relevance

Strategic objective:

To provide Aurora university leaders with a tool to strengthen the focus on societal challenges within their university and in the research collaboration outside and within Aurora.

Specific objective:

To show the distribution of research-output that the Aurora universities and the Aurora network as a whole are contributing to each of the 17 sustainable development goals. (There is overlap with the 7 EU societal challenges).

Intended effect:

1. To enhance a ‘problem/challenge’ based approach – in contrast to a disciplinary approach – to assessing quality and impact of research as a mechanism to strengthen to focus on (and appreciation for) problem-oriented research, both within Aurora and among policy makers at national and European level.
2. To increase insight in research strengths and complementarities within the aurora network, as a tool to help find current and potential collaboration in multi-disciplinary research areas, and forge strategic alliances within societal relevant themes across the universities in the network.

1. All Aurora universities are represented in the SDG tool v1. (As showcase for “Macron Initiative”)
2. An IMPROVED/MATURE v2 bibliometric tool to survey and analyse the research output and impact of a unit (university, universities network) in terms of its societal relevance, academic quality & impact, societal attention passed on policy documents (possibly also news and patents).
3. Visualisations based on the collected data, that help users (policy-, grant-, and communication-advisors) answer their questions for demonstrating / stimulating societal impact.
4. Tested the effectiveness of the SDG tool on Use cases

1. On-boarding & Macron Initiative showcase: the other remaining universities need to be brought up to the speed of this project, and add their data to the existing v1 model.
   • Online instruction for bibliometricians from UEA, Iceland, Antwerp and Bergen to enter their baseline data in a shared space, and add entries to the “SDG-data-entry” file.
   • Run Power BI on new data for dashboard with all Aurora partners.
   • Instruct Aurora institutional coordinators about the validity of the data.

2. Query crafting: For each of the 17 SDG’s a new v2 search query is crafted, reviewed and tested.

• • Describe the new query crafting method, that maximises the recall and precision. (incl. searching for literature, to re-use proven methods)
  • Setup collaborative environment (with features like commenting, version control and ) for working on the queries, and sharing test/validation results. Google Drive / Docs
  • Each institution has to identify one or more key researchers per SDG, who are available for advisory work.
  • Bibliometricians need to ask these key researchers what keywords are missing from the current query, and what publications and journals should appear in the query results that reflect that SDG. They need this information later on for creating an SDG Corpus, and to craft a new SDG query that outperforms the previous one.
  • Agree on a common working and notation procedure (how to write readable query syntaxes, to make it easier to check for errors.
  • New SDG queries are crafted, reviewed and tested/validated on sample size, recall and precision.
    • 4 SDG’s are divided among the aurora universities. The VU will make the final review.

3. Data collection: Data points are entered in a common sdg-data-entry file or database.

• • Preparation
    • Find developer for automation
    • Decide on how and where the data is being collected. (file on a shared drive, database, etc.) And what runtime environment to use, under what campus IP-range. (An Aurora university who has access to Scopus, Scival-, Altmetric- and Unpaywall-API’s)

• • Setup a shared data space, and runtime environment.

• • Development:
    • Write and test scripts
  • Data-acquisition:
    • publications-data [sept-oct]: SDG queries are run on Scopus to get the publication metadata per SDG for each institution.
    • performance- and impact-data [oct-nov]: Publication identifiers are parsed through Scival, Altmetric and unpaywall-data to get the data on Top 10% percentiles, Policy mentions and Open Access status.

4. Data-Visualisation: data points are used in business intelligence tool.
   • Connect the database to PowerBI
   • Update existing Visualisation
   • Workshop PowerBI / Elastic Search / Kibana
   • Discuss with users for additional visuals
   • Investigate co-authorship network graphs (eg. VOSviewer)

5. Publish & intellectual property: Publish results on method and utilisation of SDG Universities Profile.
   • Determine what parts of the project should be protected, but still can be made open as possible.
   • Publish about the non-restricted information, in a peer-reviewed open-access journal.

6. Outreach & communication: Present Aurora’s SDG Universities profile v2 to UN and EC DG research.
   • Announce project start
   • Announce v1 results for macron initiative showcase
   • Announce v2 results
   • Announce publication
   • Announce evaluation results

7. Evaluate: determine the effectiveness of the SDG tool v2 based on Use cases defined in the workshop in Essen
   • Create a research-/test-plan, and determine the definition of ‘success’ for each use case.
   • Instruct grant-officers/ communication officers/policy advisers on how to use the tool.
   • Gather information on the success rate.

A bibliometric tool to survey and analyse the research output and impact of a unit (department, faculty, university, universities network) in terms of its *) societal relevance, *) academic quality & impact, *) societal impact.

If possible, the tool will offer a mechanism to combine and reconcile the societal categorisations for research of UNESCO´s Sustainable Development Goals and the European Union´s Societal Challenges.

To enhance a ‘problem/challenge’ based approach – in contrast to a disciplinary approach – to assessing quality and impact of research as a mechanism to strengthen to focus on (and appreciation for) problem-oriented research, both within Aurora and among policymakers at national and European level.

To increase insight into research strengths and complementarities within the aurora network,  as a tool to help forge strategic alliances within societal relevant themes across the universities in the network.

The bibliometric instrument performs per SDG a query on a bibliometric database eg. Scopus or Web of Science. (SDG queries are created and reviewed by bibliometricians in the Aurora network.) For each SDG the publication skew is determined (based on a number of papers published), for those papers the excellence is determined (top percentiles of journals), the availability of those papers to society (open access), and the references to those papers in public policy documents from (N)GO’s. The interactive diagram below shows a proof of concept how this could look like for a limited selection of SDG’s.

Read more about the concept in the Whitepaper  [1]

The illustration below shows the hypothetical feedback-loop between policy and research. Here we illustrate the possible influence policy from societal partners like (N)GO bodies can have on the research output on policy-topics. And on the other hand, that open and excellent research can be fed-back into re-evaluation or creating a new policy. Thus research influencing society in an in-direct level.

[1] Maurice Vanderfeesten, & René Otten. (2017, November). Societal Relevant Impact: Potential analysis for Aurora-Network university leaders to strengthen collaboration on societal challenges. Zenodo. http://doi.org/10.5281/zenodo.1041405

• Jan 2018: Project Kick-off
• Feb 2018: Initial version of 17 SDG queries
• March 2018: Reviewed version of 17 SDG queries
• April 2018: data collection & first analysis based on 17 SDG queries
• May 2018: publication of Aurora SDG university profiles
• June 2018: evaluation on utalisation of SDG university profiles

• Sept 2017: Presentation of Proof of Concept for Dies Natalis Speech from Prof Pieter vd Beukering about Sustainability
• Nov 2017: White paper with proof of concept on Potential analysis for Aurora-Network university leaders to strengthen collaboration on societal challenges
• Dec 2017: Team Site for Aurora Bibliometricians
• Feb 2018: Seminar on Societal impact.
• Feb 2018: Initial DRAFT version of 17 SDG queries [final version in available for project members]
• April 2018: Interactive diagram