Education Consultancy – Evaluation of DHIS2 for Education

Terms of Reference: Evaluation of DHIS2 for Education

This is a tender for evaluation of the implementation of a digital platform ecosystem approach to education management information strengthening in The Gambia, Togo, Uganda, Mozambique, and Sri Lanka. The key aim of the evaluation is to assess the implementation and results of the DHIS2 for Education pilot project. It will also provide insights on DHIS2’s potential as an EMIS solution, with a focus on sustainability, local capacity building, and adaptability to country-specific needs in the education sector.

Background

There is growing support for a reconceptualization of education management information systems (EMIS) to address the importance of statistics in supporting a government’s focus on access, equity and quality strategies. The shift towards SDG4 priority areas and the coronavirus pandemic has highlighted the need for a flexible EMIS that can respond to changing education sector needs. Yet, the sector faces two broad data challenges: availability of data and how data is used. In addition, the sector grapples with how to leverage a comprehensive national digital infrastructure for education. This infrastructure can yield improved planning and integration between ministry partners and the empowerment of the professionals working in the space between policymaking, classrooms, and schools.

A number of countries across the globe have developed their own EMIS data systems. Some have in-house technical skills to successfully adapt and scale their systems to meet changing needs, goals, and priorities. Others struggle to adapt their data systems as they may lack those skills and are locked into vendors they cannot afford. Previous attempts at digitizing education data have generally not succeeded because they have primarily focused on the technology itself, and not on building the long-term local capacity required to sustain the system over time.

DHIS2, originally developed for health information management, has demonstrated potential as a versatile platform for EMIS across multiple countries, particularly in Sub-Saharan Africa. The Norwegian Agency for Development Cooperation (Norad), has supported the development of DHIS2 since the 1990s, and since 2018 has piloted its use in the education sector. Five Ministries of Education have been involved in the project funded by Norad: Mozambique, The Gambia, Sri Lanka, Togo, and Uganda.

DHIS2 for Education takes a modular approach to system scaling; allowing countries to upscale as they develop capacity as well as sharing of source code for developers to modify, refine and customize. The approach focuses on facilitating local ownership and building local capacity through the HISP model. The approach hinges on a model where action research and participatory design, software development implementation support and building of national core competencies are equally important and interrelated, carried out in collaboration with local partners. The model promotes learning across country borders through networks of action. This way of working has emerged through the recognition of the relative failure to institutionalize and make changes sustainable. The emphasis on scale through a focus on networks is therefore not so much about size or geographic coverage, as about facilitating the necessary learning processes for sustainability.

Project to be evaluated

The objective has been the development of a robust knowledge base for how to configure DHIS2 for the education sector and develop an out-of-the-box configuration based on international standards and best practices. The project engages stakeholders to find ways to collect, report, visualize, and disseminate district and school education data, and supports ministries to locally customize the system to fit their needs. Furthermore, HISP seeks to build in-country capacity to mold and scale the system and procedures over time. This means deepening the reach of the digital system to day-to-day school operations where feasible, as well as widening the circle of stakeholders providing and using education data, such as school managers, district planners and community leaders. To achieve this, technical capacity must be built at regional and district level. DHIS2 being a Digital Public Good means that the system is owned and can be customized by the ministry with the aim to have little reliance on foreign technical assistance and builds on open standards. A global community of ministries, implementers and developers means a sharing of challenges and innovations across country borders.

Expected outcomes of the project to be evaluated:

1. Build a knowledge base to assess if DHIS2 is a feasible option to support sustainable EMIS strengthening through implementation research and participatory innovation.
2. Critical mass of human capacity built for running and maintaining DHIS2 for EMIS with a focus on key ministry staff and district levels.
3. DHIS2 platform expanded to fully support the shift to learner-centered EMIS.

The countries involved in the project have been driven by differing operational models: some working at central level with the national EMIS unit before moving down to regional and district levels. Others have taken a regional or district approach from the start. In addition, depending on population size, coverage and funds some ministries have used DHIS2 an aggregate (numerical) data approach while others focused on individual level unit records.

Objectives of the evaluation

The evaluation will explore the different approaches in the five countries, how they were implemented and explore to what extent the planned results and outcomes have been achieved. Additionally, the evaluation will explore the added value and potential benefits of using digital public goods like DHIS2 in education at national and global scale. The evaluation should summarize findings, lessons learnt and provide recommendations.

Review questions

To what extent were the outcomes achieved?

• To what extent were the outcomes of the intervention achieved?
• What were the main successes and challenges of the intervention?
• What are the contributing factors to the success and challenges of the intervention?

How did the different operational models and contexts in the five countries influence the results achieved?

• To what extent has the operational model (HISP model) been effective in the five countries?
• What are the main differences in the operational models in the five countries?
• To what extent was analysis done of the context before initiating interventions in the various countries?
• How did the context shape the interventions and influence results and outcomes achieved?
• What were the main success factors and challenges with the operational models?
• To what extent has human capacity been built to implement the intervention at national and/or sub-national/district level?
• To what extent have the interventions been sustainable?
• To what extent have the interventions been cost-effective and are there differences between the different countries?

Alignment and complementarity of DHIS for Education

• To what extent is DHIS2 for Education complementing and aligned with other platforms and systems at national and global level?
• To what extent have synergies and connections been leveraged in terms of human capacity, knowledge, experience and innovation in particular between Ministries across both education and health sectors and country borders?

Methodology

• Comparative case studies of DHIS2 for education implementations in two countries, with desk review and remote interviews with key stakeholders in three countries
• Interviews with education ministry officials, subnational education stakeholders, DHIS2 implementers (HISPs), partners and national core teams in the ministry
• Analysis of DHIS2 usage data and system performance metrics
• Review of capacity building initiatives and their long-term impact on national and local DHIS2 expertise (government officials and HISP)
• Assessment of innovations developed using the DHIS2 platform in education contexts.
• Analysis of global DHIS2 for Education network and its impact on individual country implementations
• Review of international collaboration and knowledge sharing across project countries
• Review of project documents (proposal, reports etc.)

Deliverables

1. Evaluation report addressing evaluation questions
2. Highlight best practices and challenges for implementing and scaling DHIS2 for education in Sub-Saharan African contexts.
3. Short advocacy brief highlighting key findings and recommendations.

Timeline

A 2-month (8 week) evaluation period:

• Week 1-2: Desk review and evaluation design
• Week 3-5: Data collection (interviews, case studies, cost analysis, global network analysis)
• Week 6-7: Data analysis and report drafting.
• Week 8: Report finalization, including global impact analysis and strategic roadmap, and presentation.

Required qualifications

The consultant or evaluation team should possess:

• Expertise in education technology and information systems in Sub-Saharan African contexts
• Strong understanding of DHIS2 and its applications in education
• Experience with comparative analysis and cost-effectiveness studies of digital solutions.
• Knowledge of capacity building approaches for sustainable digital innovations
• Experience in evaluating digital public goods and their global impact.
• Understanding of international development and global education initiatives
• Ability to work in French and/or Portuguese is positive but not a requirement.
• Demonstrated ability to communicate effectively.