This dashboard provides analytics on the SDG relatedness of REF Impact cases. In this dashboard we present multiple analyses regarding SDG relatedness of REF Impact cases, both the REF 2014 cases as the REF 2021 cases. Below we give an introduction of the methodology and the definition of metrics used in the dashboard.

For questions on how to use the SDG mapper or the results of our analyses in a policy context, please contact Wilfred Mijnhardt of Erasmus Univesity/Rotterdam School of Management (wmijnhardt@rsm.nl)

For technical questions please contact Guido de Moor of Dialogic (demoor@dialogic.nl)

For the development of the RSM SDG mapper, our custom-made Machine Learning model, we used open data and open source technology where possible.

The model is based on a State-of-the-Art algorithm developed by Google (namely, Google's BERT). We fine tune and validated this model on the Springer Nature SciGraph (2019), an open dataset with over 7 million publication abstracts. We label this dataset with search queries.

The search queries (1 for each SDG) were developed through an eclectic meta research method. We incorporate multiple sets of keywords developed by different SDG mapper research groups around the world (the Aurora network, University of Bergen, the Open SDG-ontology by TechnoteAI and the RLEX SDG resources), all have published open data sets. By incorporating multiple data sources, we ensure a robustly labeled dataset with multiple rounds of validation.

We update this procedure periodically with new and updates datasets. The RSM SDG mapper is now applied to all kinds of results by RSM, to demonstrate the SDG relatedness and relevance of our portfolio in research, education and engagement.

For the analyses in this dashboard we use the following information from the cases to estimate the SDG relevance: Title, summary of the impact, underpinning research, and the details of the impact.

Our model is open source and published on Github.

The data underlying the analyses can be dowloaded here.

We present a metrics-based analyses of UK institutions participating the REF 2014 and REF 2021 regarding the relationship between the results of the REF impact cases (both for REF 2014 and REF 2021) and sustainable development, as represented by the UN SDG’s.

Like REF, we define “reach” and “significance” of REF Impact cases. We use “SDG reach” and “SDG significance” in the spirit of REF21 logic but see the global Sustainable Development challenges (as defind by the 17 UN SDG’s) as the beneficiaries of impact, not the societal stakeholders as used by REF.

We assume that the higher the SDG relatedness, the higher to potential reach for these sustainable development challenges. The composition and nature of the SDG portfolio of an institution represents the potential significance for the societal challenges.

We relate our SDG mappings to institutional context through relating them to “research power” and “investment power” of UK universities.

Below we give the definition of metrics used in the dashboard. The aim is to develop meaningful input to make the next edition of REF more focused on sustainable development as represented by the Sustainable development goals (SDG’s). We aim to contribute to the strategic need to embed SDG relatedness and impact logics into the impact narratives of universities and business schools.

We use different perspectives in the subpages of this dashboard. See the navigation in the left column of this dashboard.

We need to start by managing expectations. Please note that SDG relatedness is a calculated proxy of how closely the content of a case relates to the content of one or more SDG’s and in which degree. The proxy is produced by a trained algorithm, not a human expert. The metrics we produce an indication of impact potential. We assume that a high levels of relatedness of university results with the SDG’s also raises the potential value for users of the knowledge in the context of the societal challenges of each SDG. SDG relatedness is NOT a measure of actual SDG impact, which needs different methods to assess. It serves to support impact processes and serve as input for a meaningful dialogue to develop the institutional impact narrative.

We have developed 3 metrics, both on the level of cases, as on the level of institutions and we apply these to build analytics for institutions to understand their performance and benchmark themselves with other institutions and averages of relevant groups of institutions (like Russel group universities)

SDG Ratio (%).

This metric is calculated at the individual SDG level. We map each impact case on SDG relatedness. One impact case can relate to multiple SDGs. The SDG Ratio is aggregated to a unit of analysis level for each SDG, can be a school, a university or a group of institutions.

The SDG Ratio is calculated by digviding the number of cases related to a specific SDG by the total number of impact cases of the institution. 

The AI SDG Mapper gives a score (proxy of relatedness) between 0 and 1 for each SDG. For the SDG ratio aggregation we have defined a cutoff point. We only include cases with a score higher than 0,5. As it is fair to say that there is a relatedness when the proxy indicated this with a confidence of 50% or more. This is simply an arbitrary cutoff point. We have decided that counting cases with a <50% ratio is simply too liberal (also see below how we calculate SDG significance as a weighted proxy).

SDG Profile 

The SDG profile is about giving an overall reach for the portfolio of 17 SDGs for specific unit of analysis (e.g. an institute, year, group or unit of assessment) defines its overall SDG footprint for the results that are mapped by the SDG mapper. This can be different source documents, like impact cases, articles, PhD/MSc theses, or other written documents in the mapped collection. In this dashboard we only used REF impact cases. SDG profiles are very useful for benchmarking institutions and find comparable or complementary SDG footprints. This SDG-Metric is valuable for the development of the impact narrative of an institution, in the same way as bibliometrics provide benchmark value for research assessments.

SDG Significance

As REF impact casesare qualitative narratives, each impact case can relate to a single SDG or multiple SDG’s, and each case can have multiple levels of relatedness to the SDGs. 

We decided to mirror the REF GPA logic for the design of the SDG significance metric.  The REF GPA is defined by the overall quality of an institutes submission. In REF, submissions are scored by experts between 1 and 4 where 1 is a submission recognised nationally in terms of originality, significance and rigour and 4, where a submission is considered world-leading in terms of originality, significance and rigour. REF GPA is calculated by multiplying the % at each quality level by its rating (e.g % of 4*-rated research multiplied by 4), adding the four together and dividing by 100.

For the design of the SDG Significance metric we do a comparable calculation. We use all the results produced by the SDG mapper, not only the results with a >50% proxy as applied in the SDG ratio metric. We mirror the REF GPA logic by calculating the percentage of cases that fall in different proxy quartiles (excluding impact cases that have a zero SDG relatedness score, as “unclassified”).

The SDG significance is calculated by multiplying the % at each quartile by a weight. Next, the weighted % is summed and divided by 100 to determine the final SDG Significance. As we do not have the availability of many experts and panels to decide on the scores, we need to rely on the proxies provided by the algorithm and the distribution of all the proxies on an institution’s portfolio of impact cases over the SDG’s. SDG significance, Like REF GPA, gives an indication of the overall power in SDG relatedness of an institution. This SDG significance can be ranked or to describe universities or groups of universities that have high SDG promise for their work.

The following table summarizes the SDG GPA logic and the 5 SDG significance metric levels.

To make the SDG mapping and the SDG significance metric more meaningful for institutional strategic dialogues on impact, we have added institutional context metrics. For this we also are inspired by metrics used by UK universities to demonstrate their REF power.

We have selected 2 metrics that we could calculate based on the REF open data results, as published by the REF or by others, like universities themselves or by the UK government (through HESA).

An example of a research question in this contextualization effort is: Do institutions with high research power and/or high investment power also have a higher SDG reach and significance?

Currently we only did the context analysis for the REF2021, as demonstrator for the value of contextualized benchmarking.

Research power

Research power is defined as the REF GPA multiplied by the full-time equivalent (FTE) number of researchers submitted. We apply the calculation used by some institutions, such as UCL, in UK using this metric.

Investment power

We use the most recent available overall Income figures as published by HESA on institute level (£) for the year 2020-21. These data are publicly available through HESA.