• UNDG
  • UNDG
  • UNDG
Silofighters: the UNDG blog

Natural Language Processing to align national plans in Serbia with Global Goals

BY Karla Robin Hershey, Dennis Schleppi, Vuk Batanović | 17 October 2018

The Republic of Serbia is on a mission to map the work that is underway to implement the 17 Sustainable Development Goals (SDGs). As a first step, the UN Country Team in Serbia reviewed the compliance of the country’s policy framework with the 169 targets and 230 indicators of the 17 SDGs and assessed the country’s readiness to proceed with their implementation. The fact that Serbia is a candidate country for the accession to the EU, the Republic of Serbia called for a special review of the compliance and complementarities between 35 EU accession negotiation chapters that are implemented through a series of ongoing reform processes and linkages to Agenda 2030. We conducted a real time comprehensive analysis in late 2017 to early 2018 using UNDP’s developed methodology called RIA – Rapid Integrated Assessment (RIA). The objective of the assessment tool is to support countries in mainstreaming the SDGs into national and subnational planning, by helping assess their readiness for SDG implementation. We engaged a dozen of national experts who successfully reviewed over 100 of national policy documents, including those setting the targets relevant for the EU accession. Most of the documents were in Serbian, with a few available in English. The exercise outlined those areas that are well covered with the existing policy instruments, identifies areas where more attention is needed by policy makers, detects bottlenecks and accelerators and reviews institutional capacities in place to implement the SDGs. Scaling up and gaining efficiency with artificial intelligence We know it is important work, so we were on the lookout for innovative ways to make this process and other similar policy mapping exercises easier and more efficient. In January 2018, we heard about a pilot initiative between UNDP and IBM research which demonstrated that an artificial intelligence (AI) approach could be time saving and provide accurate mapping information. Using AI based on natural language processing (NLP) techniques could be successful in automating the rapid integrated assessment process that provides a baseline to measure future progress. The assessment, which looks at defining a roadmap for a country to implement the SDGs, was our starting point. They piloted the assessment in five countries where policy documents were available in English. We got satisfactory results from the pilot. We teamed up with local policy experts from the SeConS Development Initiative Group, an independent organization which aims at contributing to the long-term socio-economic development and improving the living conditions of individuals and social groups in Serbia and the region. Our team at the UN also met with natural language processing experts from the School of Electrical Engineering at the University of Belgrade to take this initial pilot to the next level – and research how the assessments could be translated from English into another language, thus for the first time facilitating an automated mapping of policy documents in Serbian. The development of the methodology and testing of the automated policy mapping exercise in Serbia is being implemented between August and November 2018. Talking to a computer in Serbian is not as easy as Siri makes it look Thanks to an abundance of language tools, resources, and algorithmic NLP models available in English, the initial pilot allowed for an automation in countries where English is the predominant language for official documents. In the attempt to translate the automated text processing to Serbian, our team noticed several linguistic traits that make this work particularly challenging: Unlike English, Serbian changes in form according to grammatical functions such as tense, mood, number and gender. Serbian is a fully digraph language, meaning that it can be written using two different alphabets (Serbian Latin and Serbian Cyrillic script). Latin characters often appear in Cyrillic texts, especially where foreign terms (usually from one of the European languages) are presented verbatim. Although Serbian grammar often uses the same default Subject-Verb-Object word order as English, the very nature of the language makes word ordering more flexible. In addition to the language-related challenges mentioned above, we also identified the following specific context related advantages: The automated policy mapping will focus on specific sectors – social protection, health, education. In this area we have adequate data both in quality and in quantity. Given the specific focus of the automated analysis, we will be able not only to compare automated versus manual policy mapping results, but also to get a more specific idea of the data gaps in the social, health and education sectors, which is very important for localizing Agenda 2030 in Serbia. By closing sectoral data gaps for nationalization process for the global goals, the pilot project in Serbia will also create a baseline to support the country’s SDG reporting obligations. This is particularly relevant given that Serbia will provide its first voluntary national review at the High-Level Political Forum in New York in 2019 on its SDG progress to date. The voluntary national reviews aim to facilitate the sharing of experiences, including successes, challenges and lessons learned, with a view to accelerating the implementation of the 2030 Agenda. These reviews also seek to strengthen policies and institutions of governments and to mobilize multi-stakeholder support and partnerships for the implementation of the SDGs. The Republic of Serbia will present the results produced by the automated mapping on achievements in the area of reducing inequalities in the country. Getting started, getting technical Our first step was to choose a sample of the 17 SDGs to be analyzed, limiting the dataset. Taking into consideration the quality and format of data available, and keeping in mind that next year’s voluntary national review discussion will focus on inequality, the team selected five SDGs that are clustered under the heading People, including: SDG 1: No Poverty: end poverty in all its forms everywhere SDG 2: Zero Hunger SDG 3: Good Health and Well Being: ensure healthy lives and promote well-being for all at all ages SDG 4: Quality Education SDG 5: Gender Equality: achieve gender equality and empower all women and girls. The second step was to consolidate the document database previously used in the manual assessment process to ensure that documents were available in a machine-readable format. This presented our team with a significant technical problem, since most documents were available in PDF format, which is not great for precise text extraction. Initial tests indicate that a combination of Adobe Acrobat Pro’s text extraction mechanism and a replacement procedure through which particularly problematic PDF files would be replaced with an easier to read alternative (e.g. Word files) could prove to be successful in tackling this problem. The months ahead We expect a number of technical innovations to surface from the process of adapting the proposed AI approach to texts in Serbian. The complexity of texts in Serbian will be decreased through the use of stemmers, tools that reduce each word to its stem (a stem is similar to a word’s root form). Such tools have been found to increase natural language processing model performance on several semantic tasks in Serbian, so there is good reason to believe these tools may be effective with the similar, albeit more complex, rapid integrated assessment exercise. Our initial efforts show that flexible word ordering is not likely to be a major issue in terms of transferring the (English-centric) automated pilot exercise to Serbian, since the AI method focuses on sentence or paragraph-level semantics, where the exact word ordering becomes less important. Finally, we will work around the lack of available data from the manually-conducted rapid integrated assessment in Serbian by setting up a simulation, dividing the available Serbian document collection into two groups: a training set and test set. By conducting a manual rapid integrated assessment for the training set, a foundation will be established for the automated assessment for the test set in Serbian. After these technical and algorithmic adaptations have been completed, the School of Electrical Engineering at the University of Belgrade and SeConS will measure the effectiveness of the AI method using the data from the manual exercise conducted in Serbia earlier this year and will submit a report showing the comparison between the two report, more importantly we be looking to see if the accuracy of the AI driven report the same or superior to the manually produced rapid integrated assessment report.   Despite all of the linguistic and technical challenges, this project could prove to be beneficial for data collection and analysis processes not only in Serbia, but also for neighboring countries, due to close linguistic ties within the sub-region. We will discuss the results of this pilot exercise extensively with data holders, producers and users, including the Government and civil society partners, to obtain their valuable input to inform the way forward. The UN Country Team will use the additional feedback to see if and how this automated policy data search could be used to save time and improve the accuracy of data analysis. Lessons learned will be applied to other activities in Serbia aimed at supporting Government efforts toward fulfilling their priorities towards Agenda 2030 in Serbia. The questions that we hope to answer  in the follow up consultations include: Can we use automated policy mapping for other processes beyond the initial SDG data mapping? How can we use it to map the progress towards SDG achievement and its linkages to the EU Acquis? Whatever the answers to these questions may be, we will keep you updated. Watch this space and follow our progress on social media.     Photo by: Nathaniel Shuman

Silofighters: the UNDG blog

Innovation scaling: It’s not replication. It’s seeing in 3D

BY Gina Lucarelli

My brother is a mathematician and on family vacations, he talks about data in multi-dimensions. (Commence eyes-glazing over). But as the family genius, he’s probably on to something. Lately, in my own world where I try to scale innovation in the UN to advance sustainable development, I am also thinking in 3D, or, if properly caffeinated,  multi-dimensionally. As new methods, instruments, actors, mutants and data are starting to transform how the UN advances sustainable development, the engaged manager asks: when and how will this scale?  To scale, we need to know what we are aiming for.  This blog explores the idea that innovation scaling is more about connecting experiments than the pursuit of homogeneous replications. Moving on from industrial models of scaling innovation In the social sector, the scaling question makes us nervous because the image of scaling is often a one dimensional, industrial one: let’s replicate the use of this technology, tool or method in a different place and that means we’ve scaled. This gives us social development people pause not only because we can’t ever fully replicate [anything] across multiple moving  elements across economic, social and culture. Even if we could replicate, it would dooms us to measuring scaling by counting the repeated application of one innovation in many places.   Thankfully, people like Gord Tulloch have given us a thoughtful scaling series that questions the idea that scaling social innovation is about replicating single big ideas many times over. [Hint: he says scaling innovation in the public sector is less about copy-pasting big ideas and more about legitimizing and cultivating many “small” solutions and focusing on transforming cultures.]  Apolitical’s spotlight series on scaling social impact includes a related insightful conclusion: when looking at Bangladesh’s Graduation Approach as one of the few proven ways out of poverty, they suggest that while the personalized solutions work best, they might be replicable, but too bespoke to scale. So if scaling ≠ only replication, how do we strategize for scale? I’ve got a proposal:  what if we frame the innovation scaling question more about doing deep than broad? The scaling question becomes: How will we move from distinct prototypes managed by different teams at the frontier of our work to a coherent, connected use of emergent  experiments in programme operations? Scaling also means moving from fringe to core Scaling innovation in a large organization like the UN has a glorious serendipity to it. Did you hear that we are looking into impact bonds in Armenia? What about the food security predictor in Indonesia? Nice collective intelligence approach in Lesotho. Blockchain is being used for cash transfers in Pakistan and Jordan. Check out the foresight in Mauritius. UNICEF is using Machine learning to track rights enshrined in constitutions. UNHCR is using it to predict migration in Somalia. UNDP is testing out social impact bonds for road safety in Montenegro. These organic innovations are beautiful and varied and keep us learning, but we as a UN system are not yet scaling in 3D. These days, I’ve been talking to people (my brother’s eyes glaze over at this point) about how to see various methods of innovations not as distinct categories of experiments, but rather as connected elements of an emergent way of doing development. Towards a connected kind of 3D.  Yes innovation is more of an evolving set of disruptions than a fixed taxonomy of new methods, but if we narrow our scope for a moment to the subset of innovations which have passed the proof of concept stage, can we start thinking seriously about how they connect? [As an important side note, thinking in terms of taxonomies of innovations is not a panacea. Check out @gquagiotto’s slides for a more thorough story on how classification is trouble for public sector innovation because it means we limit our vision and don’t see unexpected futures where they are already among us.] Projectizing innovation without keeping an eye on the links among the new stuff won’t get us far, and might even be counter-productive.  Instead, what would it be like if innovations were deployed in an integrated way? A bit like Armenia’s SDG innovation lab where behavioral insights, innovative finance, crowd-sourced solutions and predictive analytics [among others] are seen as a package deal.  I am looking for collaborators to learn more about how are all these methods and tools related. Do they help or hinder each other? Are there lessons that can be learned from one area and applied to others? Should some new tech and methods not be combined with others? 9 elements of next practices in development work A few of us UN experimenters came together in Beirut in July to pool what we know on this.  We had a pretty awesome team of mentors and UN innovators from 22 countries. We framed our reflections around the 9 elements of innovation which I see as approaching critical mass in the field. This is by no means exhaustive, but it’s a start to moving these methods from fringe tests led by various teams to core, connected operations. Here are the “nine elements of next practice UN” we are working with: Tapping into ethnography, citizen science and amped up participation for collective intelligence to increase the accuracy, creativity, responsiveness and accountability of investments for sustainable development. Using art, data, technology, science fiction and participatory foresight methods to overcome short-termism and make sustainable futures tangible. Complementing household survey methods with real time data and predictive analytics to see emerging risks and opportunities and design programmes and policies based on preparedness and prevention. Building on the utility of “superman dashboards”  for decision makers to helping real people use their own data for empowerment, entrepreneurship and accountability. Leveraging finance beyond ODA and public budgets by finding ways to attract private capital to sustainable development. Evolving the way we do things and even what services we offer by managing operations through new technologies Applying psychology and neuroscience for behavioral insights to question assumptions, design better campaigns and programmes and to generate evidence of impact when it comes to people’s behavior. Carving out space for science and technology partnerships within the UN’s sustainable development work Improving how we support our national partners in managing privacy and ethical risks Moving from “that’s cool” to “aha it’s all connected” We need to start thinking of these 9 elements as connected. It might be that they reinforce each other - whereby focusing on data empowerment gives meaning, context and legitimacy to the use of big data to understand behaviors and online activity. Or that they undermine each other - in the way that citizen science can undermine innovative finance pay-outs, or behavioral insights are helping companies get around privacy regulations. Looking for the practical connections, here’s what we’ve got so far: Collective intelligence methods that listen to people organically can help determine whether your behavioral campaigns are resonating.  Because people’s intell is often more granular than statistics, they could also be used to test whether new forms of finance are making an impact on health, education and other development issues. Small scale and/or internal experiments in the UN to manage operations with new technology help us know what the next generation privacy and ethics risks are. Experiments in gray zones can then inform future-oriented regulatory frameworks. Keeping a focus on helping people use data for empowerment is a good northstar when using new data and predictive analytics to ensure that cultivating realtime sources of data isn’t deepening the digital or data privacy divide. Using foresight methods or predictive analytics can point to signals of where to invest with innovative finance instruments [Follow Ramya from IFRC innovations for more on this. Hence some early connections form a budding conspiracy theory! If you are thinking multi-dimensionally too, or using a few of these methods and see where this line of thinking can be improved, help me draw more lines on the innovation conspiracy board! [Or tell me why this is the wrong tree to be barking towards… That’s always helpful too.]   We’re working on a playbook to codify what we know so far in terms of principles and methods for each of these 9 elements. Stay tuned for that... and please do get in touch to throw your own knowledge in!


You Tube
Mainstreaming, Acceleration and Policy Support: The UNDG MAPS approach to achieve the 2030 Agenda
5 July 2016
UNDG members on Twitter
UNDG on Twitter


Note: The boundaries and names shown and the designations used on the map do not imply official endorsement or acceptance by the United Nations.