The Sustainable Development Goals: joining-up new standards in a disconnected world
Discussion Paper No. 1 | February 10, 2016|Beata Lisowska | Data Scientist, Development Initiatives
Data has immense potential to help drive poverty eradication and sustainable development. While more and more data is becoming available, much of it cannot be turned into useful information because it is being published in different formats or to incompatible standards. Through the Joined-up Data Standards (JUDS) project, Publish What You Fund and Development Initiatives, with support from the Omidyar Network, are working to identify priority areas and technical approaches to join up data standards.
This paper is the first in a series of discussion documents that aims to shine a light on the dilemmas and challenges facing standards setters and to suggest approaches and methodologies that can improve the comparability and interoperability of data.
In September 2015 world leaders pledged to improve the world through the Sustainable Development Goals (SDGs). Led by the United Nations (UN) Inter-Agency Expert Group, experts have been focusing on the technical aspects of this ambitious framework. Why? Because the success of the new goals relies heavily on their measuring and monitoring systems.
Data standards are documented agreements on representations, formats, definitions and rules by which data are recorded. Creating standards poses many challenges, from high-level architectures down to specific methodologies. In that sense, the SDG framework is both a new standard and also a composite of existing standards. The Joined-up Data Standards approach that this project is exploring is a useful lens through which to analyse the gaps, overlaps and potential clashes that are inherent in such a large undertaking.
The Millennium Development Goals (MDGs) served as a starting point for the new SDG framework, and indicators developed by other standards bodies helped shape its various components. All eight MDGs were brought forward to the new standard and only six of the new goals are truly original. New goals and targets nevertheless mean new monitoring elements are needed. From a statistical viewpoint such a task is challenging – it brings the need to establish the availability of historical data that must be in place to create a timeline for new indicators: a rearview mirror to see where we have come from.
More than half of the MDG indicators (37 out of 60) were included in the new SDG standard. However, of these only 11 were incorporated ‘as is’ with their original methodology; 11 were marginally altered and 15 were adopted with a completely revised methodology. While subject matter experts have good reasons for changing indicators, or methodologies in indicators, these changes pose challenges for the producers and users of the data alike.
A case in point is the way in which child mortality is measured. In the shift from the MDG to SDG monitoring framework, infant mortality rate (IMR) has been replaced by the neonatal mortality rate (NMR). While there are strong clinical reasons for this change, replacing IMR with NMR involves a trade-off with the efforts put in over the past 15 years to create a consistent dataset (particularly given that the causes of neonatal mortality are different from those afflicting older infants) and the additional challenges involved in collecting real NMR data rather than deriving it through statistical estimations.
Adopting methodologies from other tried and tested standards can counterbalance the problems associated with creating new indicators. The World Health Organization Indicator Monitoring Registry (WHO-IMR) provides a comprehensive and credible collection of well-documented and transparent indicators. Yet looking at SDG 3, which relates to health, we find that only 14 of the 24 health-related indicators have been chosen from the WHO registry. Furthermore, 9 of the 24 proposed indicators currently have no metadata on source or methodology associated with them.
Although the monitoring of the SDGs has been at the forefront of international campaigning around a revolution for sustainable development data, it is the meeting of these targets that is of much greater importance. The data needed to plan and provide resources for action extend well beyond monitoring. This raises a further challenge for data standards. Will it be possible to compare inputs, particularly international and domestic resource flows, against impacts measured by the indicators?
Most external financial resources for development are reported to the Organisation for Economic Co-operation and Development (OECD) Development Assistance Committee (DAC)’s Creditor Reporting System (CRS), with has its own system of sector classification. Domestic resource flows are generally classified in ways that are broadly compatible with the UN maintained Classifications of the Functions of Government (COFOG). There is no easily compatible mapping between the SDGs and either the CRS or COFOG classifications. Indicators dealing with social protection are a case in point: there is no equivalent sector in the CRS, yet the subject is at the heart of four SDGs.
Achieving sustainable development is not about pursuing 169 targets independently. It will involve combining resources across many crosscutting initiatives. Similarly, the data that is required to describe, meet and monitor these goals needs to be comparable and interoperable. This paper is aimed at opening up discussion on some of these challenges.
The SDGs are a collection of goals and targets that require measurement, resulting in a collation of data points on a variety of defined subjects. They form a new standard: not one that has been created from scratch but rather an amalgam of existing standards. This hybrid is in fact a complex hierarchy of standards.
- The goals and targets are political standards that encapsulate global aspirations – they define anticipated outcomes.
- The indicators are evaluation standards and represent the best efforts of experts to quantify the intentions of the political standards – they define what should be measured.
- The methodologies associated with each indicator are the real data standards – they define how measurements should be made. They in turn consist of two distinct elements: defining both the data that needs collecting as well as the statistical manipulation that is required to turn the data (or estimates) into globally compatible statistics.
This is a multifaceted array. It raises the question of whether the SDGs will go forward as a loose (and slightly random) collection of hundreds of standalone standards, or whether a commitment to coherence and consistency will result in a unified, collaborative approach across a range of data communities.
The SDG framework has been highly influenced by and incorporates a number of existing standards. Much has naturally been inherited from the MDGs but elements from a range of other contributing standards have been incorporated. In this paper we limit our discussion to the WHO-IMR. Contributing standards like these are the building blocks of the SDG framework.
The MDGs came into being on 8 September 2000, at the end of UN Millennium Summit, when world leaders unanimously adopted the ‘United Nations Millennium Declaration‘. This identified key objectives organised into 8 goals and 19 targets relating to poverty, education, health, environment, gender equality and development.
The MDG framework defined 60 indicators to monitor the goals. This clearly defined, transparent framework of the MDGs has mobilised international efforts to allocate necessary funding and policy change to address global development priorities. However, as a UN review of the contributions of the MDG agenda to foster development pointed out, one of the biggest weakness of the MDG framework was the perception that the agenda was donor-centric and lacked the sufficient consultation at the drafting stage. The review also pointed out that the MDGs lacked focus on environmental sustainability, productive employment and decent work, and inequality.
The Sustainable Development Agenda adopted in September 2015 took notice of these weaknesses. The new framework took cognisance of the Rio Declaration on Environment and Development, which ensured that humans are at the centre of sustainable development (Principle 1) and that the least developed countries and those most environmentally vulnerable shall be given a special priority (Principle 6).
This is why the SDGs are so different from the MDGs in both scope and content. The political goalposts – as evidenced in the SDG goals and targets – were shifted substantially. The challenge facing the standards setters is to find sustainable data sources and statistical methodologies to embrace these changes.
In every field covered by the SDGs, a community of experts and a collection of standards already exists. One such standard is the World Health Organization Indicator and Measurement Registry / WHO-IMR, the global authority on health-related indicators. It is a database curated by the WHO and contributed to by a range of reputable institutions – a central source of detailed information on health-related indicators. The metadata includes information on definitions, data sources and methods of estimation. The indicators are systematically reviewed and provide a transparent and consistent source of metadata. As the pre-eminent standard setting body for health indicators, it would make sense for the SDG 3 – ‘ensure healthy lives and promote well-being for all at all ages’ – to look to the WHO-IMR.
For the SDG indicators – most of which measure outcomes or impacts – to have relevance, they need to be assessed in the context of the activities that lead to their desired results. One such area is resource flows. Both domestic and international resource allocations will have significant impacts on indicator performance and it is therefore important that assessments can be made using comparable data from these two ecosystems.
External funding, in particular official development assistance, continues to plays an important role in the development of low income countries. The OECD DAC CRS is a database for donor countries and multilateral agencies to account for their aid spending. Spending is classified in 26 ‘sectors’ (or purpose codes) and a total of 151 sub-sectors, to a large extent reflecting donor countries’ priorities. These sectors have also been adopted by the publishing standard of the International Aid Transparency Initiative (IATI), which provides more timely and forward-looking data on development activities.
Mapping the SDG targets to CRS codes can provide an important insight into the international financial assistance framework in place to support the least developed countries in achieving the SDGs. In February 2015 the OECD DAC Working Party on Statistics published a review of purpose codes and policy markers in light of the SDGs and post-2015 agenda. This discussed a preliminary mapping between SDG targets and OECD DAC sectors and highlighted that although the mapping of SDG goals to top-level CRS sectors is relatively good, mapping of targets is problematic due to difficulty in one-to-one mapping. The target–sector relationship is difficult to establish in the many instances where sectors are either too specific or too broad to match neatly to a particular SDG target.
The monitoring of domestic resources can be assessed from data on government expenditure. The governments of developing countries are encouraged by the International Monetary Fund (IMF) and World Bank to present their expenditure using a Medium Term Expenditure Framework, which classifies flows by ‘function group’ and ‘budget group’. These groups relate to socio-economic activity and are based, as recommended by the IMF and World Bank, on the UN COFOG.
To be able to view the impacts of combined domestic and external resource allocations on socio-economic development requires a large number of disparate datasets being able to talk the same language. This is a huge challenge. The comparability of CRS, COFOG and SDG definitions and classifications is core to its solution.
Defining relationships between incompatible standards has become a lot easier over the past decade as a result of the development of SKOS – the Simple Knowledge Organisation System. SKOS provides a standard language for defining non-exact relationships such as ‘broader’, ‘narrower’, ‘close’ and ‘related’ matches between concepts (definitions) across taxonomies (standards). It is also a component of the semantic web – a set of standards that promote common linked data formats and exchange protocols across the internet.
If one taxonomy contains a single classification for ‘health’ while another contains separate classifications for public health, primary health and so on, SKOS allows for the codification of the relationship – ’primary health’ in taxonomy B is a narrower match of ‘health’ in taxonomy A’ – in a way that allows databases, web pages and other information systems to seamlessly understand and represent this link. Similarly, the ability to describe a relationship as a ‘close’ rather than ‘exact’ match allows for greater accuracy in mapping.
Research has been facilitated through the use of PoolParty software, a thesaurus server developed by the Semantic Web Company. This platform stores and maps the data standards analysed in this paper and outputs cross-referenced data to build visualisations (chord diagrams) using Circos software.
The following section represents some of the preliminary findings that have emerged from the research carried out to date.
Figure 1 provides a visual representation of the coverage between the goals. New goals introduced in the SDGs relate to access to modern energy (SDG 7), building resilient infrastructure and promoting sustainable industrialisation (SDG 9), reducing inequality within and among countries (SDG 10), making cities inclusive, safe and resilient (SDG 11), climate change (SDG 13) and promoting peaceful, inclusive societies (SDG 16). The SDGs have assimilated all the goals of the Millennium Declaration.
The SDGs consist of 17 goals and 159 corresponding targets. At the time of writing, 223 indicators have been selected: 161 of these are indicated as green and 62 as grey. The UN Inter-Agency Expert Group is still in the process of resolving the remaining grey indicators.
Out of 60 MDG indicators, 37 have been incorporated into the SDGs framework. Of these:
- 11 indicators were incorporated verbatim into the SDG monitoring framework
- 11 have been marginally modified
- 12 have broader indicators
- 3 were incorporated in a narrower capacity.
Figure 2 provides a visual summary of indicator relationships between the MDGs and the SDGs. It shows the exact matches, and a substantial proportion where no match exists.
Figure 3 expands the mapping in Figure B to include close matches – indicators that have been modified in transition from MDG to SDG. While some of these changes are trivial – a change, for example, from ‘proportion’ to ‘percentage’ – others are not and will require new data.
Monitoring child mortality has been a focus of both the MDGs and SDGs. The under-five mortality rate (U5MR 0–5 years) was the first indicator introduced globally to monitor child mortality and is included in both frameworks. The MDGs also included the infant mortality rate (IMR 0–1 year) as an indicator but in the SDGs this has been replaced by the neonatal mortality rate (NMR 0–28 days).
Clinically NMR and IMR are very different indicators. Among neonates the major cause of death is sepsis and complications during birth. The major killers between 28 days and 1 year are malaria, diarrhea, pneumonia, infectious disease or malnutrition. So why the change?
Analysis of child mortality over the past 15 years purportedly shows that although all three rates have been decreasing, it is the NMR that exhibits the slowest rate of decrease. The improvement of the NMR can be connected with the quality of hospital care and antenatal care that both mother and a child have received. This problem has been addressed by initiatives such as ‘Every Newborn’, run by UNICEF and the WHO, which points out that the MDGs did not focus on the issue of the newborn mortality. This argument would thus appear to have merit, notwithstanding the fact that the IMR is an indicator associated not only with health but also with social care, nutrition and access to basic sanitation and clean water.
From the standpoint of the data, replacing IMR with NMR, rather than incorporating both, has potentially serious consequences. Data on the IMR is used to feed into the U5MR, which in turn is the foundation for the estimation of NMR. IMR was initially incorporated into the SDG monitoring framework along with NMR but has now been dropped. As a result, a dataset with a 15-year timeline is being lost and the way in which NMR is estimated is corrupted.
Estimates are, unfortunately, critically important. Mortality should be measured using civil registration data. However, civil registration data on neonatal death exists for only 38 out of 139 developing countries for between 1990 and 2009.
The UN Inter-agency Group for Child Mortality Estimation estimates the IMR and U5MR using model regression methodology, which incorporates observation from either of the two indicators (in most cases derived from survey estimates) and the model life table is used to estimate the remaining one. NMR, on the other hand, is estimated based on the already estimated U5MR through a multilevel statistical model.
This is a good example of the delicate balance between politics, science and data. It is by no means unique
SDG 3 – ‘ensure healthy lives and promote well-being for all at all ages’ – includes 24 indicators, which monitor 13 targets. The methodology for all these indicators bar three has nominated the WHO as the entity responsible for monitoring the indicators. As the UN agency responsible for health, and an institution respected for its standard-setting, this is to be expected. However, although the WHO is named as the responsible organisation the indicators are not necessarily present in its registry. For example:
- The indicator ‘number of new HIV infections per 1,000 uninfected population’ is not registered in the WHO-IMR. The WHO-IMR indicators closest to the SDG’s ‘estimated number of new hepatitis B infections per 100,000 population in a given year’ are ‘hepatitis B immunisation coverage among 1-year-olds’ and ‘percentage of one-year-old children immunised with three doses of hepatitis B vaccine’ (WHO Western Pacific Region indicator). No methodology with an explanation as to how the proposed indicator will be measured is currently available.
- ‘Number of people requiring interventions against neglected tropical diseases’ is also absent from the WHO-IMR and the detailed methodology cannot be found.
- ‘Coverage of treatment interventions for substance use disorders’ is monitored by the UN Department of Economic and Social Affairs and the closest indicator found within WHO-IMR resource is ‘screening and brief interventions for substance use and substance use disorders’.
- ‘Proportion of population with access to affordable essential medicines on a sustainable basis’ can be matched to the WHO Health Systems Strengthening (WHO-HSS) indicator ‘Right of access to essential medicines’.
To assess the impact of external resource flows on SDG performance, it is going to be necessary to join up sector classifications in systems such as the CRS with SDG targets. A relationship needs to be established between inputs and outcomes. The visualisation of the mapping between SDG targets and CRS sectors (Figure 5) is indicative of just how complex the relationship between these two standards is.
Notwithstanding that most of the mappings are based on inexact ‘related to’ or ‘broader’ relationships, there are still SDG targets that could not be assigned to any CRS sector.
This paper provides first steps in analysing the standards that are the building blocks of the new Sustainable Development Goals framework. Two lessons stand out.
The problem of historically inconsistent and incomplete indicators highlighted by this paper has also been acknowledged by the UN Inter-Agency Expert Group. Assigning indicators into three tiers has for now, solved the lack of methodology and data for a large proportion of the indicators:
- Tier I: indicators with established methodology
- Tier II: indicators with established methodology but no available data
- Tier III: indicators with no official methodology
The need for this Inter-Agency Expert Group classification is a direct result of the challenges inherent in setting ambitious goals that cannot be properly satisfied by existing standards.
Both the choice of indicator and its associated methodology pose a number of dilemmas to the data expert.
- Do you design a ‘pure’ indicator that is theoretically precise in its accurate interpretation of the target?
- Do you make a pragmatic choice based on the availability of data?
- Do you make a choice based on the potential interoperability of the data?
In the ideal world all three conditions should be met, but this is rarely possible. What should at least be recognised is that it is critical for the designers of indicators to be fully aware of the importance of getting this balance as ‘right’ as possible.
Designing a new standard, particularly one as complicated as this, is a mammoth undertaking in a world that already contains a plethora of development data standards. How, then, should it be assessed?
The common thread running through this paper is that very few statistics have any relevance on their own. Without history, context and comparison they are often meaningless. Inputs, outputs, outcomes and impacts are all connected. We live in a joined-up world and the data we use is no different.
Data connectivity is like language: if we want to understand each other we have two options:
- We can speak the same language. We can design interoperability into standards from the outset.
- Or we can employ an interpreter. We can build tools that create logical relationships between standards.
The Joined-up Data Standards project is pursuing solutions in both of these directions.
 The indicators have been categorised during consultations as green (complete agreement), yellow (needing further agreement) and grey (‘more in-depth discussion and/or methodological development’).