The International Year of Evaluation: Finding a role for evaluation in programme design

Graeme Ramshaw, WFD’s Director of Research and Evaluation, reflects on the M&E team trip to the American Evaluation Association conference last month.

2015 is the International Year of Evaluation. By year’s end, well over 80 evaluation-related conferences and events will have been held across the world. One of the largest of these events, the annual American Evaluation Association (AEA) conference, was held in Chicago earlier this month, and WFD was in attendance.

We were presenting a mixed method evaluation of our parliamentary programme in the Democratic Republic of Congo under the new Democracy and Governance stream within AEA. The presentation illustrated the need to look beyond conventional methodologies when evaluating performance in complex programmatic and contextual circumstances. It also emphasised the inherent challenge in applying evaluation frameworks retroactively, in the absence of clear logic and theories of change.

In many ways WFD’s experience echoed core themes of a conference where participants were both celebrating the increased profile the International Year had provided to evaluation and lamenting the persistent tension between programme implementers and evaluators. The assumption that evaluation exists merely for accountability at the end of the programme cycle clearly remains pervasive among many programme managers in a variety of organisations and fields. At the other extreme, a number of conference sessions documented efforts to run impact evaluations whose conception preceded the awarding of an implementation contract. Clearly, a balance needs to be struck.

At WFD, we are adapting the way we do evaluation to integrate better with our programme design processes. Beyond reviewing logframes, we are working with programme teams to identify their intended outcomes through a more iterative process involving their stakeholders as much as possible. By developing strong theories of change at an early stage, we can make our monitoring, and ultimately our evaluation more meaningful, testing assumptions and interrogating instances where the programme logic has deviated from its expected path.

It is too early to assess the impact of the International Year of Evaluation. But we hope that it includes greater recognition among policymakers and implementers of the role of good programme and evaluation design at the outset as enablers of good evaluation at the end of the programme.  WFD and our colleagues in Europe look forward to working with our North American counterparts in the AEA Democracy and Governance stream to move this agenda forward and ensure that the advances of 2015 are not forgotten in 2016 and beyond.

Continue Reading

Interview: Richard Youngs on the puzzle of non-western democracy

Many politicians, diplomats, and experts today argue in favour of non-Western models of democracy.

Yet it remains unclear what such models should look like. It may be more useful, Dr Richard Youngs argues in his new book The Puzzle Of Non-Western Democracy, to think in terms of specific areas of democratic variation that can encourage democratic renewal—outside, but also within, the West.

As part of WFD’s expert engagement series, our London staff were given the opportunity to discuss the issues raised in the book with Dr Youngs, who is a Senior Associate with the Carnegie Endowment for International Peace. Before the group conversation, we had a chance to sit down with Dr Youngs to explore his ideas. Below is the transcript of our interview.

Why did you decide to write this book?

RY: It had become clear to me that this issue of non-western democracy was becoming much more important and powerful at the policy level. It’s been a debate that’s been around for a long time, but it’s been becoming much more prominent in recent years – civil societies around the world, political leaders in rising powers, want to feel they’re contributing new ideas to what democracy means and how different forms of democratic institutions work. All western governments and agencies like WFD realise we need to take this search for democratic variation more seriously. The challenge is to understand what that means in practice.

puzzle of non-western democracyEveryone who works in democracy and governance appreciates context is absolutely critical – and that in different countries you will get different kinds of democracies as a result. But isn’t that something that is already understood by everyone?

RY: It is understood. In a way this is why the book relates to a very live policy debate, because all western governments and their leaders recognise that in trying to support democratic reform abroad, it’s not about supporting one particular model. It’s about understanding what citizens want in different regions of the world, what their understandings of democracy are. But, that’s easier said than done. We do also have to be a little bit cautious in thinking that there is a single non-western model of democracy that is completely different to what exists currently in other countries. We should be able to support forms of democracy that are right for local contexts without undermining core norms of liberal democracy.

That sounds like the key tension really between those two points. Have you figured out a way of resolving that tension?

RY: It’s not easy, and it’s about striking the right, delicate balance between supporting different models of democracy without ending up unwittingly supporting forms of democracy that verge on being almost soft forms of authoritarianism. We need to explore different understandings of rights, what we mean by liberal rights, in different countries around the world. People tend to want a more communitarian form of identity. They believe in democracy but they want to feel democracy is respecting their local understanding of morality and religion, of communal identity. With that, agencies and donors need to do more to change the narrative, in a sense – to convince people and show people that the core standards of liberal democracy don’t need to threaten local, traditional identities.

Was there anything that you weren’t expecting when you started your work on this book that emerged during its preparation?

RY: One thing that surprised me is how vibrant the debates are in countries around the world about this question, and just how sharp the differences are within different developing countries and rising powers. Some people in these countries feel they do want something that is radically different from western style liberal democracy. Other people find it quite condescending, where people argue their countries should have a different form of democracy. They insist they want the same basic rights as those that exist in established democracies. There’s a great deal of divergent opinion over this. It’s very difficult for agencies working in the field to get the balance right between understanding local contexts, supporting different varieties of democracy, but at the same time realising there are core democratic standards and institutions that do have universal applicability.

That leads to a final question about what we in Britain we should make of this. What should we take away from what you’re saying about these different kinds of democracies?

RY: The UK, as with other countries interested in doing what they can to foster democratic reform, needs to be a little bit more open to experimentation, to realising that a lot of the interesting innovations in democratic representation are today coming from other places in the world like Brazil and India. They need to be open to learning about those innovations and taking them on board. That doesn’t mean reversing everything the Westminster Foundation does; its traditional approach is to parliamentary strengthening and political party strengthening has a lot of validity, but somehow they need to be linked in with very interesting new forms of representation and protest and social movements that are emerging around the world.

Continue Reading

Monitoring democratic governance programmes: Evolution over revolution

Graeme Ramshaw, WFD’s Director of Research and Evaluation, describes how our new approach to M&E finds space for flexibility within the confines of the DFID logframe.

Desire for change

The current fashion in monitoring and evaluation, particularly within the governance sector, is to argue the inadequacies of our current tools and approaches. Consistent across all of these critiques is that existing methods inhibit flexibility and adaptability and reward simple accountancy over complex change. The necessity of meeting targets creates incentives for programmes to select indicators that are easy to measure rather than those that demonstrate the full extent of a programme’s impact. These proxy indicators give an artificial specificity to processes of change that are far more nuanced than a small handful of tangentially relevant numbers could convey.

And for the most part, these criticisms are spot on. Almost ten years after the 2006 White Paper that put the concept of governance at the forefront of DFID and other donors’ agendas, we are still using inadequate measures to assess progress. But the problem with many of these critiques is that they offer no useful alternatives to the current mainstream models. Evidence drawn from numerous programmes is distilled down to rather generic recommendations about context, local ownership, and iterative design. Relying on these almost universally accepted concepts, however, means they often don’t engage with the fundamental rationales behind the dominance of current monitoring and evaluation approaches and why donors are so wedded to them.

Tyranny of the Logical Framework?

Logical frameworks have their roots in social psychology literature of the 60s and the 70s (Nakamura and Russell-Einhorn 2015). And they serve two critical functions for institutional donors, particularly governments. They provide clarity on both the deliverables and attribution. The importance of these two concepts in the context of public sector budgeting and spending cannot be overestimated.

Indeed, the calls for adopting less restrictive approaches to monitoring and evaluation could not have picked a worse time. The austerity pervading the British government and others around the world has heightened the demand for clarity about what donors are getting for their money and how the VFM for one project compares to any other possible use of those funds. Results that are difficult to deliver and for which both success and attribution are unclear are at a big disadvantage.

Likewise, DFID’s ring-fenced budget has brought with it increased scrutiny as domestic cuts to services and benefits begin to impact UK citizens at home. Yet, while its funding increases, its staffing relative to that funding declines, putting fewer and fewer people in charge of greater and greater amounts of programming budget. This means less time to engage in the kind of responsive programming that DFID’s own evidence base suggests is the best approach to achieving sustainable reform. Rather, staff must fall back on tools that enable them to track progress in the most time-efficient way, generally through quantifiable indicators.

Any approach adopted by a programme funded by an institutional donor therefore has to find a way to engage with the logical framework, rather than rebel against it. This means finding innovative ways to insert flexibility into the overall rigidity of the model and to attempt to have the logical framework serve the programme rather than the other way around. At WFD, we have worked with DFID to test a number of new approaches that merge old and new forms of monitoring and evaluation that will hopefully allow us to better illustrate the contribution we make to democratic strengthening worldwide.

Contextualising ‘Impact’

To do this, we took the three main elements of the logframe in turn. At Impact level, there’s been an increasing move toward using high level indices (WGI, Freedom House, etc) to measure achievement. This is highly problematic because these indices are designed to capture trends not incremental change. Over the typical two- to three-year time horizon of WFD’s programmes, the data from these indices are mostly useless. Moreover, for smaller organisations like WFD, it is unrealistic to expect that our relatively small levels of investment are going to tilt broad governance indicators at national level.

While the preference was to dispose of indices all together, we ultimately settled on a compromise whereby the Impact level of a programme would be measured by two indicators, one of which would be index-based. However, programmes are encouraged to look beyond the traditional global indices to explore other options that might be better suited to the programme objective. For instance, in our corporate level logframe, WFD is using the Varieties of Democracy project’s Liberal Democracy Index as its components most closely align with our core objectives.

The second indicator relies on qualitative stories of change written using a consistent methodology across programmes. These contextualise the impact that WFD is making, demonstrating how changes at institutional level can benefit individuals and organisations through policy advocacy and implementation. Together these two indicators present a picture of how the context is trending over the period and how WFD is specifically making a contribution to or against that trend depending on the context, enabling us to situate our work more effectively.

Capturing processes as outcomes

At Outcome level, the big issue was indicators that are black/white, pass/fail. While this may work (emphasis on may) with other development projects in terms of service delivery, it is manifestly inappropriate for institutional development questions we deal with at WFD in terms of parliaments and parties. Measuring democracy, or even changes in democratic processes, is often nuanced and context-specific.

Aggregating this across a varied portfolio of programmes, spanning parliamentary representatives, staff, civil society, and political parties, is a challenge. Setting absolute standards masks achievement at the lower end of the scale and over-incentivises selecting partners or problems that are at or near the intended targets at the baseline. Likewise, allowing too much flexibility leaves the programme as a whole unable to demonstrate its cumulative impact, undermining its relationship with donors.

So, we wanted something that would enable us to measure along a spectrum but at the same time give DFID something they could assess systematically. In response, we’ve developed a hybrid solution that seeks to meld a variety of evaluation approaches. This will produce data that both captures changes at programme-level across all WFD interventions and aggregates sufficiently well that DFID can see and assess WFD’s broader achievement as an organisation. We are calling this approach the outcome matrix approach.

The outcome matrix approach draws on existing achievement rating scale approaches and combines it with elements of the outcome mapping methodology to create a measurement tool that enables each programme to identify its progress markers and assess themselves against them over time. DFID used to have its Achievement Rating Scale whereby it judged projects’ progress against its stated outcomes. It allowed for movement over time within projects and gave some means of comparing across projects as well. The problem was it was never clearly defined as to what each score meant. We’ve tried to mitigate that by mixing in the outcome mapping approach where projects establish their expectations in the beginning with their stakeholders.

The outcome matrix approach will encourage the teams use outcome mapping methodologies to think through what they need to see, expect to see, would like to see, and would love to see and develop a basket of progress markers for each. This way each programme is working toward targets that are appropriate for them and we are judging them on the merits of their own progress rather than their progress relative to others. These four baskets will then be assigned a score from 1 (lowest) to 4 (highest) to enable aggregation across programmes akin to the old achievement rating scale.

Throughout implementation, the outcome matrix will be monitored and updated, if necessary, based on events. At the end of each year, each programme will report progress as a score, providing evidence in their annual report to support their self-assessment. This will be reviewed by central M&E staff and by external evaluators as part of the Annual Review process.

Defining the contribution

Lastly at Output level we wanted to switch the conversation away from counting activities and focus on what parliaments and parties actually gain from interacting with us. This enables us to have a more direct theory of change about how we create the right conditions for democracy to be consolidated and practiced. We determined that WFD has three principal approaches to catalysing change through its programmes.

First, WFD provides its partners and direct beneficiaries with relevant, in-depth knowledge and technical expertise on parliamentary democracy drawn from practitioners and politicians across the UK and global political spectrum. Second, WFD links its partners and direct beneficiaries to networks and international platforms that promote inclusive democratic policies and practice. Lastly, but perhaps most importantly, WFD brokers relationships within political spheres, recognising that in many cases, the impediments to democratic change lie not within the structure or rules of institutions but in the relationships between political institutions and how these create informal ‘rules of the game.’

Adding this layer to our monitoring work takes us beyond merely counting number of participants or number of products produced. It makes us think more deeply about exactly how we’re expecting our partners to benefit from our programmes and how they can translate it into their parliamentary and political party work. It also allows us to have much more meaningful conversations around value for money and why, in a crowded international development field, our programmes offer a unique contribution.

Going forwards

There is general recognition that monitoring and evaluation for governance work remains an inexact science. It seems, however, that rather than collaborating to adapt and evolve existing approaches to meet the needs of practitioners in the governance field, different segments are proceeding in opposite directions. On one hand, the critiques mentioned earlier in this piece have called for abandonment of logframes and renewed emphasis on qualitative methodologies. On the other, cadres within DFID have pressed on with ever more quasi-scientific, quantitative tools that seek certainty where, perhaps, there is none.

A better approach may be to agree that, as with so many other things in this area of work, adaptability is the critical factor. Where objectives lend themselves to being measured quantitatively, let us do so. Where objectives lend themselves to being measured qualitatively, let us do so. And where a mixed approach is needed, develop one. Let the nature of the programme guide us in how we monitor it rather than our own preconceptions of the evidence we expect to see. But let us also not lose sight of a sense of realism that reminds us that we are all ultimately beholden to the political economy of our donors whose need for predictability and accountability is ever present. As appealing as a revolution may be, evolution is more likely to take us where we want to go.

Featured image: Kate Dollarhyde

Continue Reading

Community of Practice launch: Appetite for better evidence base emerges in parliamentary strengthening debate

“Getting a parliament encased within a culture of parliamentarianism is the desirable goal,” constitutionalist Lord Norton of Louth told yesterday’s Community of Practice launch. “The question is how you achieve it.”

Britain’s parliamentary strengthening community has been striving to answer that question for many years.

But there’s always more that can be done to improve coordination and collaboration to help identify what works best.

That’s why we’ve helped set up the Community of Practice, launched yesterday in Parliament at a meeting chaired by Stephen Twigg MP.


Stephen Twigg MP chaired the launch event

The idea – as WFD’s Graeme Ramshaw outlined in a blog yesterday – is to provide the “hub” called for by MPs on the international development select committee.

Its establishment has been welcomed by those in government.

Stefan Kossoff of DFID said the Community of Practice was a “brilliant initiative” which will “help advance thinking and practice in this area”.

He said DFID would look to engage as much as it could, sharing research and contributing to the UK’s shared offer overseas, as the Community of Practice develops its work.

“It is going to provide a really important forum for the sharing of lessons and learning amongst UK organisations and researchers in this field,” he said.

“We still lack a systematic evidence base so if we can advance that it would be good. It provides us with an opportunity to strengthen coordination in how we work overseas, building on respective strengths of organisations in this room.”


Stefan Kossoff speaks at the event

The discussion was opened by Greg Power, director of Global Partners Governance, who gave an update on the work that he and Tom Carothers of the Carnegie Institute are doing to revise DFID’s guidance note on parliamentary strengthening.

His remarks emphasised the importance of political analysis and discussed the need for key objectives including self-sustaining reforms and effective measurement, before touching on the big-picture challenges about governance work.

“There needs to be a degree of realism about what you’re going to get from a parliamentary project,” he said. “The point about working with a parliament is to change the political culture.” When that’s achieved, “you’re much more likely to see those benefits repeated time after time”.

The debate about the benefits of improving parliaments’ capabilities is far from settled, however. Dr Tim Kelsall of the Overseas Development Institute think-tank (pictured below) offered a sceptical response to Greg Power’s comments, suggesting that “parliamentary oversight is just one piece of the development puzzle”. He warned against “getting carried away with the centrality of parliaments to development” and pointed out that “the evidence base is still rather fragile”.

The Community of Practice, it’s hoped, will help fix that. Westminster Foundation for Democracy has signed a research partnership with the Department of Politics and International Relations at Oxford University which can translate the expertise of our implementers into substantive research. The aim is to address precisely that lack of a robust evidence base being pointed to now.

As Wednesday evening’s event showed, a big part of the continuing debate is focusing on how to design the most effective programmes and how to measure the impact in an area as complex as political reform.

Lord Norton, highlighting this, spoke of the need to “mould” the approach to specific contexts. “If you go with the checklist approach, the danger is you’re imposing a straitjacket that may not fit with a particular society,” he said.

Stefan Kossoff, too, underlined the “critical” need for avoiding the “blueprint approach”. It’s in working out the right way to apply flexible methods where our collaborative work going forwards will be so important. As he put it: “The Community of Practice is about disseminating good practice as opposed to best practice.”

In the coming weeks and months we’ll be looking to work with the CPA, the overseas offices of the Commons and Lords and our other partners to build on last night’s successful launch. If you would like to contribute, please contact our Director of Research and Evaluation at Graeme dot Ramshaw @wfd.org.

Continue Reading

Community of Practice – a new hub for parliamentary strengthening

Graeme Ramshaw, WFD’s director of research and evaluation, writes about the thinking behind this week’s Community of Practice launch:

Parliamentary strengthening is receiving more and more attention these days.

The Commons’ international development select committee carried out an inquiry into the topic last year which found much to praise and celebrate.

But it also noted that coordinating and sharing of practices has, thus far, been erratic.

The committee’s final report strongly recommended that “consideration be given to the establishment of a stronger Westminster ‘hub’ which would bring together UK institutions with different kinds of expertise to enable them to cooperate rather than compete”.

They’ve got a point. Existing, fragmented dissemination strategies, while effective, haven’t done enough to convincingly inform the policy and strategic decision-making of the British government and other donors in parliamentary strengthening.

So later today organisations and individuals interested in UK parliamentary strengthening will meet in the CPA Room in the Houses of Parliament to launch the Community of Practice we hope will deliver the ‘hub’ sought by MPs..

The Westminster Foundation for Democracy (WFD) and our partners the Commonwealth Parliamentary Association (CPA) UK and the House of Commons and House of Lords Overseas Offices all see this as an opportunity to improve coordination.

But it’s about more than that. We want the Community of Practice to be a place for organisations to feed in new research, new opportunities and, hopefully, interesting questions for all of us to debate.

Launching the Community of Practice

Purpose:
•         To share new information and practices on parliamentary strengthening, especially between different communities (academic, policy making, assistance providers etc).
•         To gain greater insight into what has worked and what hasn’t in parliamentary strengthening and democracy assistance programmes.
•         To meet regularly to shared views on parliamentary strengthening.
•         To be a potential platform for joint undertakings (research, workshops, policy analysis etc)

Proposed Form:
•         Informal group, to which any organization that works on parliamentary strengthening can sign up by contacting WFD, CPA UK of the Overseas Offices.
•         Core group that meets at least twice-yearly to discuss new initiatives for community.
•         Links with other Communities of Practice in related fields, such as the PPPeer Network (a “global Political Party Peer Network, consisting of leading political party organizations and donors from all over the world”); and the IDEA (The International Institute for Democracy and Electoral Assistance, based in Sweden) community of practice in political finance.

Activities:
•         Participation in talks, conferences, workshops etc., whenever organized by members of the community.
•         Establish working-groups on specific topics.
•         Establish email group that anyone can use to distribute notifications of events, new publications, questions etc.
•         Twitter hashtag to allow sharing of articles or news with Community of Practice.

Duration:
•         Indefinite, but WFD, Overseas offices and CPA UK can play a leading role in 2015-16 to ensure continuity and progress in the initial phase.

If you would like to contribute, please contact me at Graeme dot Ramshaw @wfd.org.

Continue Reading