Globally, protected areas are the most commonly used tools to halt biodiversity loss. Yet, some are failing to adequately conserve the biodiversity they contain. There is an urgent need for knowledge on how to make them function more effectively. Impact evaluation methods provide a set of tools that could yield this knowledge. However, rigorous outcome-focused impact evaluation is not yet used as extensively as it could be in protected area management. We examine the role of international protected area funding agencies in facilitating the use of impact evaluation. These agencies are influential stakeholders as they allocate hundreds of millions of dollars annually to support protected areas, creating a unique opportunity to shape how the conservation funds are spent globally. We identify key barriers to the use of impact evaluation, detail how large funders are uniquely placed to overcome many of these, and highlight the potential benefits if impact evaluation is used more extensively.
The need to provide a credible evidence base for conservation actions is increasingly recognized as pivotal by conservation scientists [1–3]. Additionally, protected area managers often desire better information to inform their management decisions [4,5]. However, the use of outcome-focused impact evaluation for generating evidence in conservation management has seen limited implementation to date, especially in relation to protected areas. This absence appears surprising given that impact evaluations using carefully selected comparison groups and empirically measured baselines are one of the most effective means of generating evidence for which conservation actions work and which fail to meet their goals .
The international funding organizations that provide substantial financial support to protected areas are influential stakeholders, especially in the developing world. The policies and priorities of international funding agencies can dictate the locations, actions and, importantly, the methods used for monitoring, evaluation and reporting of each project they fund. Despite having a diversity of individual goals, these funding organizations are united by their stated objective to achieve maximum impact for every dollar spent. It seems peculiar therefore that international conservation organizations are relative latecomers to the use of rigorous impact evaluation  compared with organizations working in health, education and development, where impact evaluation is both more common and increasing . Existing assessments have generally been done by academics [9,10], rather than funding institutions, and generally test the impact of protection per se rather than improved management or altered support to protected areas .
Three major groups of donors dominate the landscape of international protected area funding. The largest total funding for protected areas comes from the World Bank and the Global Environment Facility (GEF) who have both spent billions of dollars establishing and supporting protected areas over the past 25 years . The next largest international sources of funds for protected areas are bilateral agencies. These channel funds from developed to mainly developing countries, e.g. the United States Agency for International Development . The last group of significant funders are international non-governmental organizations (NGOs) such as the Global Conservation Fund, World Wildlife Fund (WWF) and Conservation International (CI). The broad range of above-described agencies demonstrates a diverse range of approaches to project assessment and the use of rigorous impact evaluation. It is difficult to usefully generalize about each of the three groups with respect to their use of impact evaluation, but examples of excellent practice occur in all groups. For example, WWF and CI have an ongoing large-scale marine protected area impact evaluation in Indonesia , and the World Bank uses impact evaluation extensively for many of its programmes ; indeed, it has even established the Strategic Impact Evaluation Fund to invest in impact evaluation. However, we are not aware of any impact evaluations by the World Bank that address its investments in protected areas.
2. Reasons for low utilization of impact evaluation
The set of issues described below are not unique to conservation agencies and do apply, to some extent, in other fields such as health and development policy. However, conservation agencies appear to have been slower to overcome these issues. Conservation projects have a number of additional challenges (recently described by Baylis et al. ) that may be causing slow uptake of impact evaluation. These challenges include projects with diverse outcomes operating at multiple scales, plus spatial spillovers that are more severe than in other fields.
(a) Fear of exposing failures
Conservation actions can be expensive, and funds are limited . Expectations of a positive return on investment generate pressure for programmes to claim success and ‘bury failure’ . This pressure on implementing agencies to appear successful is strong, as costs of failure are both direct (e.g. negative press or professional censure) and indirect (diversion of funding from unsuccessful programmes) .
However, project failures (and partial failures) often provide the most instructive lessons for future projects and an unwillingness to scrutinize failure is a missed learning opportunity. Conservation management involves substantial uncertainty; rapidly changing social and ecological landscapes mean there are bound to be conservation actions that fail. Arguably, the most worthwhile outcome of an unsuccessful project is sufficient knowledge to avoid repetition.
(b) Costs of impact evaluation
High-quality impact evaluations carry a substantial upfront cost. The average cost of World Bank impact evaluations is around $500 000, which is about 1.4% of the total cost of the evaluated interventions and 0.5% of the cost of the projects of which those interventions are a part . At the Investment Finance Corporation, the median cost of an evaluation relative to the project budget is about 7% of a median project costing $564 000. In fields such as business, health and justice between 5% and 30% of total project budget is routinely allocated to outcomes evaluation [19,20]. The UN is moving from budgeting on the basis of 1% of the programming budget of the agency to 3%. This trend reflects the operational role of evaluation as a means to ensure accountability and to contribute to learning .
The cost of an impact evaluation should be judged relative to the value of the information it will produce [22,23]. A $2 million evaluation of a $500 000 programme might be extremely cost effective if the study helps policymakers decide whether or not to scale up into a billion dollar national programme. Costs also mean that impact evaluations should not be required of every programme; rather they should be commissioned strategically to assess those programmes that are unproven and are either widely used or are new and promising. Being selective in this way makes the overall budget for impact evaluation manageable relative to the overall budget for operations.
(c) Misaligned incentives across agencies
Many conservation interventions are short-term projects. The benefits of a careful evaluation, however, will largely be realized after the project ends and will accrue to the funder and global conservation community. Field personnel are thus better off investing their time and resources in actions that will yield benefits to them rather than to the larger conservation community .
(d) Perceptions of mission creep
Collecting, storing and analysing the data required for meaningful impact evaluation is often seen by managers of funding agencies as mission creep: the spending of resources outside essential areas. Protected area managers are often reluctant to divert scarce resources away from management actions to monitoring and evaluation . In this environment, it is challenging to spend resources on monitoring design and implementation, and database systems. It is also challenging to spend resources monitoring outside the project area, so that counterfactuals can be established. To address this concern in the GEF 5 replenishment, money was made available in the Global and Regional Exclusion Fund pool (for biodiversity projects) specifically to monitor control sites in projects that wanted to design and implement an impact evaluation ; however, uptake was very low. In many environmental management agencies, the monitoring and database systems are the first to be cut during budget contraction cycles, as agencies prioritize field management. However, in the long term, the dearth of information leads to inefficiency and poorer decision-making, as managers are not fully aware of the status and trends of the environments they manage and so cannot respond accordingly.
(e) Perceptions that existing knowledge of how programmes work is sufficient
It would indeed be wasteful to conduct impact evaluations if existing knowledge were sufficient to achieve consistently excellent project outcomes. However, this is rarely the case given the complexity and contextual sensitivity of protected-area-focused projects. Recent reviews assessing the social and biological effectiveness of protected areas [26,27] have highlighted that surprisingly little is known about when and how protected areas function optimally. In addition, the context of protected areas is changing with new global conservation targets, and new threats such as climate change .
(f) Lack of technical capacity
Implementing agencies and projects may not have personnel with the technical training to carry out high-quality evaluations. Impact evaluations can be complex, requiring careful study design and analytical tools. Solutions include funders providing resources to recruit/train the required expertise or to provide in-house support and facilitation to the implementing agency and project staff. For instance, the World Bank has extensive technical expertise to support and carry out evaluations , but that expertise appears focused on sectors away from environmental projects.
(g) Technically challenging circumstances
Many protected area evaluations are challenging owing to confounding factors, small samples, lack of counterfactuals, multi-scale project outcomes, slow rates of change of outcomes, diverse/intangible objectives, etc. . Not all outcomes for all projects will be cost-effective candidates for impact evaluations; however, most projects will have some aspects that can be evaluated. The outcomes to prioritize for evaluation are those that are known to have unexplained spatial and temporal variability and relate to key project objectives. Planning for the evaluation in the project design will increase the number of project objectives that can be usefully evaluated. Progress in project design and statistical analysis techniques is increasing the range of outcomes and policies that can be usefully evaluated .
(h) Complex institutional arrangements
The diversity of management agencies and practitioners often leads to a plethora of project objectives (e.g. scientific, biological, humanitarian) where success in one objective might even be linked to failure in another. Encouraging participants, including local actors, to agree on a set of explicit project objectives to enable impact evaluation may be difficult in many conservation contexts . There may also be overlapping responsibilities for different aspects of projects, making any evaluation more logistically and politically difficult.
3. Global Environment Facility impact evaluation case study
In 2012, the GEF Independent Evaluation Office (IEO) and United Nations Development Programme (UNDP) Independent Evaluation Office commissioned a project to measure the impact on biodiversity of past GEF support to protected areas. The challenges encountered by the authors during this impact evaluation highlight some of the practical difficulties associated with retroactive impact evaluation at broad scales. This case study is not exhaustive, but does illustrate many key issues in applying impact evaluation to protected areas. The insights are in the context of the enormous importance of the GEF in supporting protected areas and the conservation of biodiversity, while operating in a complex institutional environment.
(a) Background to global environment facility evaluations
The GEF is one of the world's largest conservation funders, having committed over $5 billion  to biodiversity projects since 1991, and providing a total environmental commitment including non-biodiversity funding of $13.5 billion plus $65 billion in leveraged co-financing for 3900 projects in more than 165 developing countries . At least $2 billion has been directed to the implementation and management of protected areas ; though project documents record total expenditure accurately, they rarely record specific estimates of the amount spent supporting protected areas. GEF funds are disbursed in replenishment cycles that have occurred approximately every 4 years since 1991. GEF-6 is the current cycle, which commenced in 2014 and ends in 2018. Types of support to protected areas vary widely and include a wide range of on-ground actions, from establishing new protected areas to reducing human–wildlife conflict. Further, many projects not specifically targeted at protected areas still support them in various ways, often through capacity building of relevant biodiversity management institutions.
The budget dedicated to impact evaluations by the GEF IEO has been approximately $300 000 per year (table 1), excluding the costs of IEO staff, of which three are focused on impact evaluation. There are also evaluation resources in the offices of some GEF implementing agencies that can perform evaluations (e.g. UNDP Independent Evaluation Office). The current level of evaluation resourcing (at less than 1% of funds committed) appears insufficient to rigorously evaluate even a small proportion of projects undertaken. Indeed, the GEF IEO planned to complete only five impact evaluations between 2012 and 2015 .
(b) Description of the impact evaluation attempted
The evaluation of the impact of GEF support to protected areas used quasi-experimental design. It focused on two biological outcome metrics, deforestation rates and changes in vertebrate abundances, and one metric of enabling conditions, management effectiveness scores. These outcomes were chosen because these data existed in freely available databases and they were relevant to the majority of protected-area-focused projects. The treatment being evaluated was the presence of GEF-funded support to individual protected areas. The ideal control group would have been identical protected areas that did not receive GEF-funded support.
(c) Data challenges encountered during impact evaluation
The documents that track and record GEF projects (i.e. mid-term reviews and terminal evaluations), while extensive, were not designed to enable impact evaluation for protected areas. The information they contain is collected at the project scale and not at the scale of individual protected areas. This means they did not capture several pieces of information about the projects necessary to enable their impact to be evaluated. To effectively evaluate the impact of projects on protected areas, it is necessary to know (i) what the specific objectives of the projects were, (ii) which protected areas were being targeted and where they were located, (iii) over what time-span interventions took place, (iv) how and on what were resources allocated within the protected areas, (v) what specific actions and activities were carried out in individual protected areas, and (vi) how objective-related biodiversity outcome metrics are changing in the protected areas. For the majority of projects, these pieces of information were absent from the available records. Of particular difficulty for impact evaluation was the lack of certainty about whether or not individual protected areas were included in projects, which compromised the selection of suitable comparison sites to represent the counterfactual outcomes. Some of the sites selected for comparison, which were believed to be unsupported by the GEF, may in fact have received unrecorded GEF support. Where projects had highly non-specific objectives, it was difficult to assess whether a valid theory of change existed between GEF support to protected areas and the biological outcomes assessed (deforestation rates and vertebrate abundance changes).
The tools currently used by the GEF to evaluate projects (terminal evaluations and the Management Effectiveness Tracking Tool, METT) are useful to assess the individual projects processes. However, they are often constructed in a way that makes cross-cutting analysis of outcomes difficult. For example, where names and general locations of protected areas might be sufficient at a project level to understand where conservation efforts were targeted, a more standardized, systematic and accurate list of completed actions is needed when comparing interventions across thousands of protected areas.
(d) Case study findings
The GEF project analysis we attempted was carried out after the completion of the projects, using data collected for purposes other than impact evaluation. This presented a number of challenges that were difficult to overcome. Many of these challenges could have been avoided if impact evaluation had been designed into the projects from the beginning. Most significantly, despite the GEF having a thorough system of project monitoring and evaluation, there was an absence of measured baselines and selected comparison sites prior to project initiation. While it is recognized that real-world evaluation must often be adaptive and inventive, there is no substitute for designing the evaluation prior to project initiation.
The GEF does systematically collect one type of data at the level of individual protected areas which is standardized across projects and thus holds potential for broader large-scale analysis: the METT. The METT is a questionnaire that collects information on the management inputs and processes of individual protected areas. It is a formal requirement for all GEF projects in protected areas to conduct METT assessments before, during and after each project. However, the GEF does not have a formal mechanism to capture METT data nor a quality-controlled database to keep these data. For this project, it was necessary to create a new database and gather METTs from a range of sources within implementing agencies.
(e) Lessons learned from the case study
Most importantly, impact evaluation needs to be considered in the design phase of projects and not only after project completion. Most agency evaluation systems are retroactive, and generally only collect information on project processes and outputs rather than outcomes and impacts. The benefits of considering evaluation when designing projects are that it will clarify programme objectives, identify theories of change, and define performance indicators and data collection protocols . To facilitate the link between project design and evaluation, it may help if there were closer collaboration between the evaluation and implementation departments within agencies to enable the evaluator's expertise to inform project design. The high degree of independence of the evaluation offices in the GEF and UNDP is necessary for evaluating some organizational functions and where a paucity of data requires subjective assessments to be reported. However, where rigorous impact evaluation is implemented, then project outcomes are measured with empirical data and a high level of independence is not necessary to ensure unbiased reporting.
Data management and the measurement of empirical project outcomes could be improved by coordination with suitable specialist institutions to enable collection and curation of relevant biodiversity monitoring data. These institutions include UNEP's World Conservation Monitoring Centre (protected area attributes and locations, and management effectiveness data), IUCN (Red List of Species including species' range maps), Global Forest Watch (forest change data) and the Living Planet Index at the Zoological Society of London (population time-series). But data management also starts at the project level, where transparent and thoroughly tested protocols for how to collect empirical information need to be established across all projects. Outcome-focused data collection should be strategically targeted to answer the most pressing questions of the agency, reduce the costs to a manageable level and increase the feasibility of producing high-quality evaluations. Outcome metrics need to be selected to allow inferences on time-scales relevant for action. Where biodiversity data are too noisy temporally and spatially, then intermediate indicators (e.g. fire frequency, poaching incidents) may be more suitable for drawing credible inferences about project impacts.
4. Funders as potential leaders driving the uptake of protected area impact evaluation
There are many reasons why international funding agencies are well positioned to drive the use of impact evaluation in protected area management:
(1) Funding bodies, especially those using trust funds, have long institutional lifespans and outlooks . Impact evaluation needs long-term support to build a strategic global evidence base for conservation policies . Funders that have repeated funding rounds can incorporate learning and evidence from earlier impact evaluations into projects of subsequent funding rounds, creating a feedback cycle that improves institutional performance. Additionally, some project and management impacts will not manifest within the usual 3–5 year project cycles; it will require reassessment of impacts long after projects are completed to fully understand longer-term changes and funding bodies are well placed to do this.
(2) Funders have existing project evaluation infrastructure and capacity. This includes systems for collating and storing information, and the expertise to design and access these data. Only a marginal increase in effort or a refocusing of data collection may be needed to allow rigorous impact evaluations of selected priority projects.
(3) Funders have substantial power and influence over the project implementers. If protected area funders take steps to facilitate and encourage impact evaluation then implementing agencies will be inclined to comply, especially if funds to carry out the evaluations and other incentives are provided.
(4) There are opportunities for experimental impact evaluations to be carried out, especially with the larger funders. Experimental project designs can make drawing inferences about project effects easier and more credible .
(5) Impact evaluation combined with project budgetary data allows the estimation of cost effectiveness of actions and projects. Typically, data on budgets are difficult to access and collect, but the existing systems of the funders already have these data. An understanding of cost effectiveness provides the key information to allow the optimum allocation of limited funds .
(6) Funders have a degree of independence from project implementers. This makes them well placed to bring learning from impact evaluations to the fore even if it involves unsuccessful projects, as there is limited reputational risk to the funder.
Funding bodies of all types, including funder-coordinating bodies (e.g. the Consultative Group on Biological Diversity), NGOs and grant-making government agencies, have the opportunity to lead the way in the application of impact evaluation in protected area projects. There is an opportunity to fill significant knowledge gaps in issues surrounding protected areas by application of rigorous impact evaluation. Increased use of impact evaluation will benefit the global conservation community by increasing the likelihood that projects achieve their objectives and by reducing the repetition of unsuccessful projects. Increased use of impact evaluation will also help funders improve their own performance by being able to allocate funds to maximize impact based on experience. Fully achieving these benefits will require high quality evidence generated from rigorous impact evaluation using control groups and baseline measurements focused on tangible outcome measures. For many agencies, even those with extensive evaluation experience this will be a substantial change from their current practice. Evaluations focused mostly on project processes and those that are carried out retroactively using monitoring data cannot provide the same high quality of evidence as well-conducted impact evaluations.
The process of increasing the use of impact evaluation does not need to be centrally controlled and directed by funding agencies. Instead, small changes to project selection criteria may nudge the implementing agencies to increase its use. At present, the misalignment of incentives between implementers and funders, i.e. the lack of reward and increased risk for implementers that evaluate, is a clear barrier. But if the quality of a project's impact evaluation plan is made part of the selection criteria for receiving project funding, then implementers will be more motivated to consider using well-designed impact evaluation. Large international funders can provide leadership by embracing impact evaluation. In turn, this could catalyse a cascade of impact evaluation as nations and project proponents seek to meet the required standards; then domestic donors, and central and state governments may follow suit.
Such a change would necessarily require a visionary and fundamental shift, but is possible. One challenge to overcome is to encourage data collection in control sites where projects are not taking place. One way to reduce this challenge is for funders to strategically identify projects in their portfolios that would be suitable for evaluation. The selection of projects could be done effectively using structured decision making (SDM, ) a method used successfully in several fields including conservation . SDM would ensure the best possible selection of metrics to monitor and evaluate each project by identifying learning objectives, and using these and their feasibility to then select candidate projects in their portfolios. An SDM process could also guide decisions on how much to spend for each evaluation, including measurement of control sites. Using the SDM approach would avoid enforcing a fixed percentage expenditure on monitoring and evaluation, would avoid spending funds on projects and sites at which learning is too difficult, and would avoid spending resources on collecting data at non-project sites unless it was definitely going to be used for evaluation.
Conservation at the scale envisaged by international policy initiatives, such as REDD+ and the Convention on Biodiversity, clearly stands to benefit from a solid body of evidence on what works, what does not, where and why. From the Asian Development Bank to the US Agency for International Development, donors across the board are beginning to embed impact evaluations into their practices; it is time the same is done for projects targeting protected areas. At a minimum, funding agencies must permit funds to be used for impact evaluation as a routine part of project grants and not require special permissions for allocating funds to monitoring and evaluation. Ideally, they would incentivize impact evaluation by awarding increased funding to agencies and projects that demonstrate an ability and willingness to carry out rigorous impact evaluations.
All authors conceived and wrote the paper.
We declare we have no competing interests.
We received no funding for this study.
One contribution of 16 to a theme issue ‘Measuring the difference made by protected areas: methods, applications and implications for policy and practice’.
- Accepted August 10, 2015.
- © 2015 The Author(s)