CVLLP 24: Monitoring and Evaluation

Project Objective

The CVLLP will deliver a programme of projects to support the conservation, enhancement and celebration of the Churnet Valley – a special landscape fringing the Peak District in the Staffordshire Moorlands and northern parishes of East Staffordshire.  The monitoring of progress and the evaluation of impacts is a key part of the process.  As a product of the development phase, this document establishes the proposed approach, reports on initial actions linked to evaluation, and sets out preliminary proposals for implementation once the programme commences in April 2012.

Context

The Churnet Valley is a unique area that is suffering from pressures associated with low farming incomes, a decline in the condition of attractive features such as old bridges and dry stone walls, and a shortage of young people entering land management amongst a range of economic, community and environmental challenges.  Projects will be focused on four areas - protecting heritage and wildlife habitats by conserving, enhancing and connecting built and natural features; encouraging community participation and volunteer involvement in maintaining the landscape; increasing community awareness, knowledge of and access to the landscape; and offering training in traditional skills and opportunities for economic diversification.  Clearly, the range of actions involved will have a major impact on the sustainability of the area.  The careful monitoring of delivered projects and an evaluation approach that feeds back into the process in order to maximise the benefits and, in sending alerts, minimise any constraints linked to partnership activity, is essential.

In establishing a tailored approach towards the monitoring and evaluation of activities, actions and impacts within the CVLLP, Heritage Lottery Fund guidance on best practice has been taken into account.  In this respect, the approach, objectives and processes involved are related to monitoring and evaluation that is embedded in the partnership and thus helps to focus activities and act as a motivational element.  It is also firmly focused on assessing both short and long term benefits and thus outputs and outcomes, some of which will be long term.

Prior to setting out an approach to CVLLP monitoring and evaluation, it is important to set objectives for the activity.

Key objectives for CVLLP monitoring and evaluation

The CVLLP monitoring and evaluation plan is designed to:

• effectively monitor progress and evaluate the effects of outputs and wider outcomes achieved with regard to the cluster of actions funded through the CVLLP programme on an ongoing basis over the 2012-2016 period; within this primary context to;
• initiate and develop high quality monitoring and evaluation approaches and actions tailored to the context of the CVLLP but transferable to other comparable project environments, and with feedback loops to help support the maximisation of project benefits to the area; and to;
• design, develop and implement an approach that meets and exceeds minimum monitoring and evaluation requirements as specified by the Heritage Lottery Fund by fostering an innovative best practice approach.

Working definitions of key activities

In order to ensure clarity of purpose and approach it is useful to define terms at the outset, where appropriate building from HLF guidance.

Monitoring and evaluation

Monitoring is the process that collects and collates information about project outputs.  Unlike evaluation, it is non-judgemental, focusing on the actual progress made with regard to agreed (funded) activities.  To be most effective in supporting the delivery of activity proposed, it needs to be carried out systematically over the lifetime of the project.  It also needs to build in feedback mechanisms so that any delays or constraints impeding delivery can be identified and acted upon as soon as possible.  In this respect, an independent monitoring and evaluation team – especially where it comprises socio-economic and ecological project expertise - should be able to observe and advise on the steps that might be taken to get back on course.  This is particularly the case where a suite of projects being delivered within one landscape partnership programme is generating similar, related and, to a degree, generic impacts.

Evaluation is the process of judging project outputs in terms of their impact, effects and, ultimately, outcomes.  In this way, the effects of project activities are viewed as initially hard outputs such as length of walls rebuilt, habitat area improved, or volunteer days completed. However most importantly, longer term, and sometimes softer outcomes, such as the uses of completed structures, the benefits through visitor attitudes or farmers benefitting from restored boundaries are enduring, but less tangible and thus softer benefits (such as the improved understanding of the wildlife benefits of woodland management).

Although evaluation is often completed on an ex post basis – that is, following the completion of projects – and which can identify the stream of effects generated by project actions, there is much to be gained from developing an ongoing approach to evaluation.  This involves an ex ante assessment of the type initiated in the present document and planned during the early stages ofthe CVLLP delivery period together with ongoing evaluation activity at the appropriate stages of each project’s delivery cycle, and a final ex post assessment.  There is also the matter of encouraging outcomes evaluation to continue on a sustainable basis after the completion of the funding period.  Ongoing evaluation should involve a series of interim evaluation activities with capacity for feeding back results to delivery teams.  As with monitoring, good evaluation will involve feedback loops so that any constraints may be addressed.

Outputs and outcomes

The monitoring and evaluation (MaE) approach recognises the key distinction between outputs and outcomes, both of which are important elements of impact and effect with regard to CVLLP.  Outputs relate to quantifiable delivery targets whilst outcomes are the wider impacts, often developing over the longer term, of project actions.  Each output will connect to a series of outcomes.  For example, a completed improvement to a length of dry stone wall or a canal towpath is an output, but that output will generate outcomes in terms of the acquisition of skills as well as improved boundary management or towpath usage by visitors.  In each case, the latter (outcomes) are more difficult to assess, often requiring qualitative research (via surveys for example) to generate evidence.

Monitoring and evaluation challenges and needs

It is important to review and assess any challenges confronting the monitoring and evaluation (MaE) of the programme and, in response, to specify how these will be dealt with in the approach adopted.

In general, the array of actions and activities funded through CVLLP will involve a mix of environmental-ecological and socio-economic impacts.  There is thus a need for a dual but integrated approach.  The solution proposed is the formation of an integrated MaE team with a range of appropriate expertise who will work closely together.  Given the scope of the partnership activities, there will be a range of interrelated projects and a mix of partners.  This is productive as it will lead to closer working between groups for the collective benefit of the area and its formal and community institutions.  However, it will also create potential problems for the MaE activity which will need to monitor and assess a collective effect as well as individual project achievements.  Assessing partnership level benefits (outcomes) as well as individual project outcomes will require a carefully focused MaE approach, again supporting the need for a team-based approach.

Of course, the effects of CVLLP will pan out within and beyond the funding period and this is a challenge for the evaluation in particular.  An ongoing MaE team involvement through the lifetime of the partnership’s funding activity, together with an approach that involves regular feedback, will ensure that not only outputs but also longer-term outcome expectations are assessed.  The MaE will be focused on assessing achievements and supporting their maximisation and sustainability.  Project leads will be encouraged to continue assessing outcomes beyond the funding period and, where necessary, the team will train participants, including local community members, in appropriate (survey and related) techniques.

The regular monitoring of project outputs, with feedback, alerts on any constraints, and suggestions for improvement where necessary, is a key task for the MaE.  In order to ensure a smooth process, the MaE team will be established and formalised to work with the ProjectCo-ordinator.  As part of work completed in the development phase, liaison with project leads has been occurring to check and, if necessary, adjust indicators and output/outcome measurement.  Once the programme is operational, systematic monitoring and reporting of project outputs on a regular (normally quarterly) basis will ensure that any shortfalls are evident at an early stage.

Evaluating landscape partnership projects as a genre offers both challenges and opportunities.  Given the range of projects supported by CVLLP – all with long-term aims as well as short-medium-term objectives – it will be vital to assess the impact of these as the programme is delivered.  Benchmarking has been encouraged at the outset and is underway in several projects but the MaE team will work closely with project leads to firm up pre-funding positions prior to the commencement of CVLLP funded actions.  It will also complete an area socio-economic assessment at the outset of the programme.  Integrating a mix of quantitative and qualitative data, this will be used as startup benchmark position – with regard to the rural economy, visitor issues and so on, as well as a basis for assessing wider area results towards the end of the funding period.

Within the project array there are inevitably some ‘hard to measure’ outcomes.  In this regard, the MaE will work with projects to develop appropriate (often qualitative) methodologies and to support data capture.  As set out in Appendix 1, the team has, in advance of the programme start-up, completed a review of data collection challenges at the project level.

There will be a need to address social and economic benefits as well as environmental and landscape effects.  As a result, the MaE team will comprise a socio-economic impact consultant and an ecological consultant supported by data collection volunteers working for the entire project period.

The monitoring and evaluation approach

Given its objectives, the monitoring and evaluation approach will be conducted in partnership with project lead organisations and the local community.  This will be designed to establish monitoring and especially evaluation as a core element within the project and to help focus activities and act as a stimulus/motivator.  There will be a formal process of ongoing review and reporting with close attention to outputs-outcomes links and the building of impact over the funding period.  There will be periodic and final reviews.  The analysis based on the interim (periodic) work will identify any challenges and constraints and, via alerts to the delivery team, will establish feedback loops.  This will involve early results being fed back to enable adjustments in delivery if necessary and appropriate.

In addition to close working with project leads – where needed, advising on measurement and assessment, offering training for monitoring approaches and evaluation data needs – the core team will work with volunteers and members of the community associated with the various projects.  These groups will support data collection in both monitoring and evaluation.

The monitoring and evaluation will be focused on hard outputs and the stream of outcomes that will occur.  There will be quantitative recording of outputs, together with regular monitoring visits.  The MaE team will co-ordinate and, where appropriate, initiate the tracking of beneficiaries and qualitative research to directly assess outcomes, particularly where socio-economic outputs such as an improved attitude and commitment by land managers, volunteers involved in conservation, or canal visitors or users with a better understanding of wildlife issues in the canal environment are core to a specific project.  The socio-economic outcomes of projects that are primarily driven by ecological or environmental objectives (and outputs) will also be assessed.  The outcomes element of the evaluation approach will be primarily based on qualitative methods, including the use of purpose designed questionnaires focused on the social and economic benefits of each project’s activities.

Given the fact that all projects are focused, either directly or indirectly, to some degree on Churnet Valley wide benefits, some broad-based evaluation work will be appropriate.  In that respect, a survey of generic outcomes will be developed at the outset with data collection started after eighteen months of the delivery period.  This will be designed by the monitoring and evaluation team who will coordinate activity, train and support community volunteers and analyse and report on results.  The results of this major survey activity will be combined with desk-based analysis of transformation and change in the area’s landscape and local economy through the follow-up to the initial area socio-economic assessment.

The ecological and environmental projects and actions, as well as any effects ensuing from socio-economic projects, will be evaluated using benchmarking and post project impact testing by experts from within the team.  This will be additional to the range of biodiversity research activities included within the overall programme.

It is important that the benefits of CVLLP are sustained beyond the funding period and there are evaluation implications in that regard.  There will be support from the team for post-funding evaluation to be sustained and, where appropriate, guidance and training will be given.

The celebration of programme success is essential.  As such, the output and outcome performance will be publicised and disseminated through a mix of reports, web-site alerts and dissemination events.

Throughout the project funding period the monitoring and evaluation team will adopt a flexible approach, responding to the needs of assessment and opportunities to elicit evidence on, information capture and evidence of sustained impacts as they arise.

Monitoring and evaluation tasks and provisional timetable

At this preliminary stage, it is envisaged that the following array of tasks will be completed.

Prior to the start of the funding period in April 2012

There are four pre-startup monitoring and evaluation tasks to be completed.  These are, in sequence, as follows:

• confirmation and formation of the independent CVLLP monitoring and evaluation team
• the final review of outcome measurement methods and the development of a draft guidance document for programme participants together with the design of reporting templates
• confirmation of participating projects and output/outcome targets
• production of the final agreed monitoring and evaluation plan

Socio-economic Monitoring & Evaluation

Activity during the CVLLP funding period: April 2012-March 2016

Clearly, the major monitoring and evaluation activity will take place during, and to some degree, following, the four year CVLLP funding period.  The provisional stages, set out here as tasks for completion with a broad indication of deadlines, are as follows.  All tasks assigned to the MaE team will be completed on an independent basis but in liaison with the CVLLP Co-ordinator.

Clearly, the detailed nature of these tasks will be developed once the programme of activities is underway.

Following confirmation of the full CVLLP funding from the Heritage Lottery Fund, this monitoring and evaluation plan will be further developed in terms of specific reporting points and timescales, with all projects confirming a clearly developed mechanism for data capture.

Ecological monitoring & evaluation

The Partnership has agreed that we require separate, independent, ecological monitoring Churnet Valley Living Landscape Partnership (CVLLP) Projects:

• Sustainable Woodland Project (CVLLP 1);
• Caldon Canal Vegetation Project (CVLLP 5).

This is because both projects have put a significant emphasis upon ecosystem services through linking real socio-economic benefits from holistic woodland management. These projects will endeavour to maximise the potential market for woodland based products and create and economic driver to sustain woodland management into the future. However, there is concern that creating an economic driver for woodland management may encourage land managers to exploit woodland holdings, leading to a reduction in, rather than the proposed improvement in, the biodiversity value of these undermanaged woodlands.

The Royal Society for the Preservation of Birds (RSPB) will be undertaking baseline surveys and continue to monitor birds within woodlands, and have identified the training of a number of volunteers to help with this work within their project (CVLLP 1). However, birds are not the only, nor the most ideal, monitoring tool for evaluating changes in woodland management over such a short period of time (4 years).

In order to support this work CVLLP will engage an independent ecological consultant to undertake baseline and ongoing invertebrate studies of woodland brought into management during the first 2 years of the project, including British Waterways woodlands from project (CVLLP 5). The survey techniques will be for invertebrates populations and follow Natural England methodology ‘Surveying freshwater and terrestrial invertebrates’