working paper

Reshaping Monitoring, Evaluation, and Learning for Locally Led Adaptation

Tamara Coger Sarah Corry Robbie Gregorowski
Partners:
Download PDF

3. MEL Approaches, Methods, and Tools to Support LLA

Drawing on examples and case studies, this section reviews approaches, methods, and tools specific to the four stages of the MEL cycle: design and planning, monitoring, evaluation, and learning. It includes discussion of lessons learned and benefits and challenges to implementing these practices in support of LLA.

3.1. Design and Planning

The methods and tools presented below support design and planning of a MEL system and have implications for the planning and implementation of adaptation interventions and the extent to which they are locally led.

Hiring local experts in senior MEL roles is an opportunity to support LLA. MEL systems that are designed and led by external experts can discount local knowledge and expertise. Such a practice perpetuates power imbalance and is less likely to result in comprehensive MEL systems (Estrella and Gaventa 1998; Van Zyl and Claeyé 2019). Locally available skills and expertise can be appraised in the planning phase of the MEL cycle. Additional resources may be required to engage local or national knowledge brokers and, where MEL experts are not available locally, to strengthen the capacity of local actors to lead MEL processes.

Theories of change can be used as a tool to ensure that adaptation interventions reflect local priorities and realities. Developing a theory of change is a process rather than a static product and can support LLA design by drawing attention to the assumptions and evolving understanding linking intervention activities and outcomes (Valters 2015). Local input to the theory of change process is most consequential if integrated from intervention inception. This may require a structural change to adaptation programming, however, as theories of change are often defined and designed before local partners are engaged. Routinely revisiting a theory of change throughout an intervention to test progress against locally determined outcomes and to integrate local knowledge and learning provides a deeper understanding of the link between climate change and the results of the intervention (NEF 2017).

Tools to build a detailed understanding of local stakeholder and contextual dynamics include political economy analysis, stakeholder mapping, and community-based climate vulnerability assessment. These three common tools can support assessment of relevant stakeholders, the nature of their vulnerabilities, and sociopolitical, economic, and environmental factors at play in a particular context. Political economy analysis (PEA) is concerned with how power and resources are distributed and contested. A useful PEA informs where locally driven opportunities for change may emerge and where constraints may need to be addressed, including why institutions matter (Teskey 2020). Stakeholder analysis and mapping is a way to identify an intervention’s key stakeholders, assess their interests, needs, and incentives and clarify how these may affect and inform an intervention’s design and delivery (Rai et al. 2015). Early engagement with diverse stakeholders creates space for them to influence the intervention design and delivery process (Leventon et al. 2016). Community-based climate vulnerability assessment integrates local knowledge and engages communities in the formulation of LLA plans. It provides a clear starting point for the definition of appropriate intervention and wider climatic indicators (ADB 2018).

Creating a climatic baseline and climatic monitoring system is key to understanding the outcomes of LLA. Once a detailed understanding of local stakeholder dynamics has been established, a corresponding and locally specific understanding of the climatic conditions and dynamics affecting those local stakeholders also needs to be developed where feasible. This is often described as a “declining climatic baseline.” Defining and measuring LLA results in the context of these climate dynamics helps actors plan, understand, and learn from an LLA intervention. This may require external expertise and can be constructed in collaboration with local stakeholders. Establishing a climatic baseline and then monitoring the climatic context in which an LLA intervention is delivered, in parallel to monitoring the results of the intervention itself, informs learning and adjustments required to continue the intervention in the face of climate changes. This is not a simple undertaking due to the temporal and spatial unpredictability of climatic shocks and stresses. For example, flood events are hard to predict and have locally specific impacts. Historical weather and climate records can provide useful climatic baseline data. Geographic information systems (GISs) and other technologies are increasingly being used to overlay and illustrate climatic data baseline and subsequent climatic changes.

3.2. Monitoring

Monitoring can support LLA by providing information for agile course correction, mutual accountability, and learning about the process, context, and expected outcomes (Warner 2017). Some monitoring methods and technologies described below can reduce the burden on participants and ensure that information collected is available and useful to populations who have limited access to climate information. While these examples of technological solutions can support more equitable access to information and influence in the monitoring process, it is important to recognize that not all populations have equal access to technology, and that technology alone cannot address challenges such as local agency in adaptation that are rooted in an imbalance of power.

The concept of adaptive capacity is a useful starting point for defining appropriate indicators and adaptation metrics. Adaptation is often assessed through concepts such as risk, vulnerability, and resilience, or through proxies that are expected to lead to adaptation, such as adaptive capacity (Leiter and Pringle 2018). The UK-funded Building Resilience and Adaptation to Climate Extremes and Disasters (BRACED) program created a set of locally defined and context-specific subindicators based on adaptive capacity, including indicators related to new knowledge and skills, new attitudes and behaviors, and shifting institutional relationships, which ultimately supported new, locally driven policies and practices (BRACED 2015).

Monitoring frameworks offer potential to enable adaptive management and prioritize learning. Adaptive management enables experimentation and innovation, and sometimes expects that interventions fail. Monitoring systems that support adaptive management serve multiple functions: they generate evidence on an intervention’s management and quality of delivery, allow for comparison between different types of interventions, focus on near-real-time learning, monitor unintended consequences and failures as well as successes, and ensure shared understanding among MEL participants of how evidence will be applied to support positive change. The Global Learning for Adaptive Management initiative is supporting the wider uptake of adaptive management principles and practices, including in climate change adaptation. This includes a focus on learning about effective MEL for adaptive management. The key is to design LLA monitoring frameworks not just for data collection but also for intentional and collective reflection on the data to support ongoing and real-time learning, course correction, and decision-making (IDS 2020).

Creating open data governance structures can support locally led decision-making. Climate science, earth observation data, and lessons learned about LLA are more likely to be used by local decision-makers if they are accessible through appropriate and demand-driven platforms (Acclimatise 2018). Solutions to data accessibility and application challenges involve open climate data, collaboration, and investments in education and long-term alliances for the codesign and coproduction of knowledge. The principles of open access, privacy, and appropriately packaged and accessible data apply across scales from national and international GIS initiatives down to climate data generated at the local level.

Sharing information about LLA also supports local ownership, decision-making, and social capital. The Mercy Corps Pastoralist Areas Resilience Improvement through Market Expansion (PRIME) project in Ethiopia uses a technology-driven approach to inclusive decision-making and bottom-up data collection. Its online platform, Kiprojects, allows local NGO and program staff to suggest and submit new activities for rapid approval. Staff can also make real-time learning about project progress easily accessible for other team members across Ethiopia (Desai et al. 2018).

Free, prior, and informed consent (FPIC) is another data governance structure that specifically supports the protection of local and indigenous knowledge shared in the context of MEL. FPIC is a principle that informs the right to consultation and is enshrined in the UN Declaration on the Rights of Indigenous Peoples and the International Labour Organization Indigenous and Tribal Peoples Convention 169 (FAO 2016).

Locally led data collection, data analysis, and learning can be enabled and enhanced through appropriate information and communications technologies (ICT). Although technology cannot solve all problems related to accessibility and inclusion, data collection, analysis, and visualization science and technology innovations can support MEL of LLA by making it more feasible to engage populations who are traditionally hard-to-reach or not given opportunities to express their views, as well as to directly engage local actors in large numbers. Remote sensing, use of social media for data analysis and engagement, electronic financial transactions, and mobile applications and devices are some examples that can support MEL. Locally driven demands for accountability and social activism by civil society encourage uptake of locally accessible ICT tools and platforms, such as through mobile phones. A prominent example is the Slum/Shack Dwellers International initiative Know Your City, described in Box 2.

Box 2 | Case Study: Slum Dwellers International’s Know Your City Campaign

Participatory, pro-poor, people-centered urban governance

Know Your City (KYC) unites organized slum dwellers and local governments in partnerships anchored by community-led slum profiling, enumeration, and mapping. The KYC initiative demonstrates citizen-generated data collection not simply to support external monitoring processes but also, and more important, as a form of social and political capital that defies traditional power dynamics for slum dwellers in cities. Slum/Shack Dwellers International (SDI) used this extensive data collection to bring perspectives and requirements from the ground up to decision-makers, like local government officials and city planners, in order to inform decisions about resources and adequate responses.

Local ownership throughout the MEL cycle

Young people in informal settlements owned their role in MEL activities (digital data capture and peer-to-peer exchanges) and collaborated in the generation of learning and knowledge-sharing, including media and films centered on living in slums and informal settlements. The KYC campaign created data-sharing and information systems that were transparent and built trust between campaign administrators and local communities. People who live in informal settlements were trained to create a community profile through learning-by-doing approaches to monitoring, gaining data collection and management skills.

Tech-enabled, citizen-generated data

In-person and widespread monitoring ensured that local perspectives and voices were anchored the campaign outcomes. The Ugandan SDI Alliance and partners built the capacity of enumerators living in slums to use open-source software and handheld GPS devices to identify community assets and risks. Enumerators were also taught how to record and analyze data and produce reports. After local participants validated mapped data, initial digital maps were produced. Local participants then used this information to negotiate with local authorities on potential community development initiatives. The locally led data collection methods applied in the KYC case demonstrate the value not only of the data collected but also of the process to broaden understanding and accountability within the community (horizontal), and on multiple scales among the community, local government, state level actors, intermediaries, and the private sector (vertical).

Sources: Bolnick et al. (2018); Antonio et al. (2012).

3.3. Evaluation

There is increasing recognition that conventional evaluations (often characterized as project-focused, ex-post, and designed and delivered by external international evaluation teams) are not fit for purpose or supportive of LLA (IPCC 2014). LLA is inherently subjective and rooted in local context. There are clear benefits for both local participants and funders in tailoring evaluations to ensure that resources are aligned with local needs and desired outcomes. This often requires flexible evaluations and cocreation of evaluation processes. Imposing externally constructed outcomes for adaptation interventions risks misaligning resources with what is needed and valued at a local level, and evaluation methods that will not effectively measure LLA outcomes.

The approaches outlined below draw primarily from influences of participatory action research, developmental evaluation, and realist evaluation. They describe methods that are tailored-for-purpose, context-specific, and use evidence generated from evaluations for the benefit of local stakeholders (Pawson and Tilley 1997; Patton 2008).

Several conventional evaluation methods are closely aligned with the principles of local agency and prioritize learning.

  • Tracking Adaptation and Measuring Development (TAMD) is a standard methodology employed where participatory evaluation processes are the priority, including participatory data collection and indicator development that ensures local agency in the evaluation process (Brooks et al. 2013).
  • Most Significant Change (MSC) is a flexible participatory monitoring or evaluation technique in which evaluation practitioners and local stakeholders collect narratives of change following an intervention to understand and highlight the most significant changes resulting from an intervention. MSC narratives and the value attributed to them are validated by other participant stakeholders in the community (Davies and Dart 2005).
  • Appreciative inquiry is a change management process used as an evaluation tool focused on the strengths of an intervention, learning what is working well and investing in the factors that sustain successful outcomes. The inquiry itself is a collaborative effort, capturing the positive features of an intervention and encouraging continuous positive change (Acosta and Douthwaite 2005).
  • Climate Resilience Evaluation for Adaptation through Empowerment (CREATE) is an evaluation method developed by the International Union for Conservation of Nature for policymakers, field practitioners, or local actors. It is based on community-based vulnerability assessments but has been adapted as a tool for community self-evaluation. CREATE is flexible, designed for learning and experimentation, and aims to assess vulnerabilities and adaptive capacities, with the added value of supporting local actors in identifying pilot activities and strategic interventions as well as short-, medium-, and long-term adaptation activities (Shott and Mather 2012).

The climate-smart villages approach described in Box 3 provides a case of participatory, locally driven evaluation tailored to hyperlocal (village) contexts and scaled up for global learning through an accessible database.

Box 3 | Case Study: Evaluating Adaptive Interventions across Climate-Smart Villages

Collaborative evaluation approach

The climate-smart villages (CSV) approach to agricultural research for development aims to help agricultural communities across Asia, Africa, and Latin America adapt to climate impacts. A CSV project has evaluated technological and institutional responses to coping with climate variability and impacts in agriculture through the use of climate-smart agriculture (CSA). Following the rigorous, collaborative evaluation activity, lessons learned were shared through social learning platforms.

The CSV approach was designed to bridge the gap between climate projections and practical agriculture techniques that local actors could use to adapt to climate shocks and stressors. It involved farmer-led assessments and iterative learning and feedback, applied at different scales from local plots to farms, households, and communities.

Collaborative generation of evidence and data collection

CSA techniques are evaluated using a variety of methods, including surveys, farmer group evaluations, and ICT-based feedback tools such as crowdsourcing, to coproduce evidence. The CGIAR research program Climate Change, Agriculture, and Food Security and its partners designed a multiscale evaluation that was accessible to local actors through a digital platform, reflecting local experiences. In Tanzania, one evaluation tool employed was the 5Q method, which was cost-effective, adaptable, and straightforward, reflecting the farmer’s experience and progress at the local level in real time. 5Q uses “feedback rounds” of short, simple surveys based on five tailored questions and automated voice surveys and local technicians speaking directly with farmers.

Lessons learned are widely shared and accessible

Local stakeholders provide input into the design of CSV techniques based on their knowledge of risk management and then assess other applied CSV techniques. Local actors learn about the benefits and barriers of CSV through farmer-to-farmer exchanges, ICT-based tools, farming fairs, women’s organizations, and sharing videos of successful technologies, practices often supported by local government engagement. The CSV approach is valuable for stakeholders who benefit from the real-time feedback and access to the lessons and evaluations from other CSVs through an online platform.

Sources: Aggarwal et al. (2018); Jarvis et al. (2015).

Subjective evaluations and self-assessments integrate local stakeholders’ values, perspectives, and perceptions of risk. Béné et al. (2019) found that subjective understanding of resilience and a sense of self-efficacy have a strong positive impact on the households’ ability to effectively recover from shocks. These results suggest that local actors who have a clearer understanding of their resilience are better able to cope with effects of climate change (Béné et al. 2019).

Use of subjective measures of climate resilience is an emerging approach that offers a simple, efficient, and locally driven alternative to predefined resilience indicators or indices. Subjective measures are used in contexts where capturing real experiences from the bottom up is a priority, when local stakeholder experiences are assessed and compared over time, and where additional methods of measuring resilience complement subjective definitions (Jones and Tanner 2017). The Subjectively Evaluated Resilience Score was used for household resilience surveys in Uganda to understand the resilience capacity of local actors and establish a baseline for an ongoing impact evaluation (Jones 2019). Similarly, Reckien et al. (2013) developed vulnerability maps based on interviews with five socioeconomic groups in South New Delhi. Local actors’ subjective experiences of physical climate impacts and abstract climate concepts were used to determine potential causes of vulnerability and future adaptation options. The vulnerability maps indicated that lower-income groups were more affected by climate-related impacts and would benefit most from infrastructure investments. Subjective measures of resilience and vulnerability create space for a context-specific and localized assessment of an adaptation intervention according to the needs and logic of the community whose resilience is in question.

Culturally responsive evaluations can amplify local voice and support equitable outcomes. Hood et al. (2015) provide comprehensive guidelines for conducting culturally responsive evaluation. The guidelines recognize the centrality of culturally defined values and beliefs to any evaluation. In culturally responsive evaluation, outcomes are valid if evidence generated is supported by the cultural and local context, as opposed to validity defined by conventional definitions of research rigor. Culturally responsive evaluation specifically aims to address bringing equity to evaluations through its focus on historically marginalized groups (Hood et al. 2015).

Johnston-Goodstar (2012) suggests that evaluation advisory groups (EAGs) are ideally suited to bringing evaluators and indigenous communities together to discuss the evaluation process. Using language and tools acceptable to indigenous or context-specific worldviews, and centering indigenous voices, EAGs present an opportunity to examine differences in values, norms, and assumptions, and reflect on power relations that might influence the outcome of an evaluation or the evaluation process. For example, the Swinomish Indian Tribal Community in the U.S. state of Washington informed the design of evaluation metrics for a community planning intervention following a process similar to EAGs. After consultations with tribal elders had established a mutual understanding of the evaluation process, “climate change health” metrics were integrated into a conventional evaluation and planning process. Cocreating an evaluative process was also a valuable decision-support and learning tool for the Swinomish community and in other evaluations based on indigenous knowledge (Donatuto et al. 2020; Kerr 2012).

Although not uniquely tailored to locally led action and participation, the UN Children’s Fund guide to evaluation for equitable development draws on decolonial and feminist approaches to evaluation, indigenous knowledge, and human rights practice and could be adapted to the LLA context (Segone 2012). Another resource is the Equitable Evaluation Project, a peer-to-peer platform used to encourage evaluators to reflect on the cultural considerations and racial bias in evaluation processes, tools, methods, and offer approaches that can create positive social change and equitable outcomes (Equitable Evaluation 2017).

3.4. Learning

Common methods of integrating learning into adaptation interventions include regular learning reviews, use of online knowledge-sharing platforms, facilitated learning events or dialogues, and critical reflection through formal evaluation (Harvey et al. 2017). Below we assess a selection of less common approaches, methods, and tools for learning that are especially relevant to the LLA context.

Distinct processes for accountability and learning can address tension between these objectives. Reporting on target outputs for upward accountability purposes may discourage a culture of learning from failure and flexibility to adapt, and can consume intervention resources like team member time (Spearman and McGray 2011). If integrated from the start of an intervention, a two-track model with dedicated workstreams and resources for separate accountability and learning functions can encourage learning by removing the disincentive to report failure (Fisher and Anderson 2018; Smith 2020). The BRACED program has a dedicated “Fund Manager” who accounts for the effectiveness of the development assistance funding that supports the program, as well as a dedicated “Knowledge Manager” who is responsible for evidence-generation and learning about resilience (BRACED and Bond 2019). Donors can also allocate specific resources for learning or innovation funds to ensure adequate resources and commitment to learning objectives (Harvey et al. 2017). For example, the Collaborative Adaptation Research Initiative in Africa and Asia (CARIAA) had a designated “Opportunities and Synergies Fund,” which supports learning and research-into-use for adaptation interventions.

Social learning supports collaborative decision-making for adaptation contexts (Cundill and Rodela 2012). Social learning can support LLA as a process that entails active engagement and agency, recognizing that information, consultation, and participation are insufficient for effective adaptation (Collins and Ison 2009; Cundill et al. 2014).

Social learning is still somewhat ambiguously defined, posing challenges for implementation (Medema et al. 2014). CARIAA addressed this challenge by focusing on creating an enabling environment for social learning, rather than developing a prescriptive learning approach. The enabling conditions the initiative emphasized align with others identified in this paper, including flexibility in resource allocation and scope and separating MEL’s learning and accountability functions (Ensor and Harvey 2015). Initial findings from community adaptation interventions indicate that social learning can bring economic benefits and community cohesion if they are based on sustained engagement with relevant local stakeholders, capacity development for learning, reflection and cocreation of new knowledge, and challenging institutional barriers (Van Epp and Garside 2019).

Prioritizing self-directed learning and creating a learning culture can promote sustained local ownership of LLA (Smith 2020). If not intentionally locally driven, learning can be top-down and not necessarily meet the learning needs of local actors (Valters 2015). One of many examples of self-directed learning is the practice by farmers in Western Kenya of integrated soil fertility management. Multiple years of group learning led to scaling-out of farming practices and expansion of scope to include additional income-generating activities, suggesting that the advantages of locally owned learning may be worth trade-offs such as additional investment of time, resources, and capacity building (Ramisch et al. 2006).

In South Sulawesi Province, Indonesia, the Climate Adaptation through Sustainable Urban Development research project tested various qualitative and quantitative participatory evaluation methods to understand how well its water resilience project was creating a learning culture. The “Factors of Success” method, a collaborative group exercise that identifies and maps ideas of project success over time, and the “Obstacles and Enablers” method, in which participants reflect on potential “obstacles” to project success as well as potential “enablers” to overcoming these obstacles, were two qualitative methods that supported critical reflection and anticipatory learning. They could be integrated into a MEL system to promote a learning culture and participatory learning about LLA (Larson et al. 2016).

The case study in Box 4 uses an example from the Devolved Climate Finance mechanism piloted in Mali, Tanzania, Senegal, and Kenya to illustrate how bottom-up learning can support locally led adaptation investments and decision-making.

Box 4 | Case Study: Local-Level Climate Fund Decentralization

Context

The Devolved Climate Finance (DCF) intervention places local actors at the heart of the adaptation management and financing process and seeks to improve the responsiveness of national decentralization policies to local impacts of climate change. The program supports community adaptation initiatives and local governments through prioritized investment in public goods that have a high socioeconomic impact. These public-good investments are identified and prioritized by local representatives against a devolved climate finance budget managed by the local government.

Adaptation planning committees complement learning-oriented MEL systems

The DCF MEL system is based on the TAMD framework and centered on adaptive and flexible management across local and national levels. The DCF mechanism is intended to strengthen existing monitoring, reporting, and verification processes in devolved government financing and planning processes. TAMD assesses the quality and scope of climate risk management investments and practices, then evaluates the local adaptation outcomes and impacts. In Tanzania in 2014, adaptation planning committees (APCs) were established to build trust between donors and local stakeholders. APCs at the ward or communal level had the autonomy to establish budget priorities, while regional APCs would improve recommendations without vetoing local budget priorities. This mechanism improved the rigor of the resilience planning and ensured that local actors’ needs were addressed while reassuring donors through a collaborative investment oversight process.

Bottom-up learning to inform decision-making

Lessons learned from testing adaptation techniques at the village level are collated through wide community consultations designed to represent diverse social groups and people often marginalized in decision-making, particularly women and young people. APCs and existing village assemblies are used to inform adaptation intervention strategies and spending. The combination of participatory planning tools used in the DCF planning process with local government authorities enabled local actors to articulate their livelihood strategies and led to subjective explanations of community resilience and well-being, as well as local resource use. The lessons learned from these participatory processes generated knowledge and evidence that local government authorities used to inform decisions about local investments and spending and enhance the awareness of climate change impacts in their local area.

Source: DCF Alliance (2019).

Learning through games can facilitate the communication of complex systems and support learning and dialogue. This is a method employed in support of climate resilience and disaster risk reduction by the International Federation of the Red Cross and Red Crescent Societies Climate Centre. The Climate Centre’s games use storytelling, active learning, emotional engagement, and problem-solving for learning about and managing climate risk and natural disasters (Solinska-Nowak et al. 2018). This unique method requires unique resources, including game facilitators, designers, and willing participants. Although there is insufficient evidence available to link learning through games to adaptation outcomes in the long term, anecdotal evidence suggests strong learning outcomes, especially when participants are involved in the game’s design (Bachofen et al. 2012).

Peer-to-peer learning supports local agency and social learning for LLA. One example of peer-to-peer learning for adaptation was an exchange the Adaptation Fund hosted in 2019 focused on resilient water and agriculture. Representatives of various national implementing entities of the Adaptation Fund learned directly from one of their peers in Chile about agricultural resilience to drought, soil erosion, wildfire, and unpredictable precipitation. Following the exchange, they applied lessons learned to enhance agricultural resilience in their respective contexts, three within two months of the exchange (Adaptation Fund 2019).

Knowledge exchange platforms are a vehicle for long-term, cross-scale learning and the building of an evidence base around effective and equitable LLA. Documentation, including through collaboration with knowledge partners such as local universities and dedicated platforms for sharing learning, is a simple but important method to support learning for LLA (Harvey and Fisher 2013). Global platforms appropriate for vertical knowledge exchange about LLA include the annual Community-Based Adaptation conference, the annual UN Framework Convention on Climate Change (UNFCCC) Conference of the Parties, and the annual Gobeshona conference in Bangladesh.

Alternative methods of disseminating best practice for local adaptation horizontally among local actors can include the use of digital media. For example, the North East Network—a knowledge broker for adaptation interventions in the Indian states of Assam, Nagaland, and Meghalaya—uses participatory video to disseminate local knowledge on sustainable natural resource management practices (CDKN 2018).

Start reading