Prof Nagel und Prof Terberger
Social cohesion

Social cohesion

Evaluation: Independent, but not detached

It’s not always flattering when development projects are rated, but it’s always valuable – because evaluating helps to avoid mistakes in the future. In a lively discussion, the head of the Evaluation Unit, Professor Dr Eva Terberger, and member of the KfW Executive Board Professor Dr Joachim Nagel analyse the benefits of evaluations, their significance for Financial Cooperation (FC) and the world of tomorrow.

About Mr Nagel
Prof Dr Nagel

Prof. Dr Joachim Nagel is Member of KfW Group’s Executive Board, responsible for KfW’s international business.

How important are evaluations for FC, and what in particular do you appreciate about them?

Prof. Dr Joachim Nagel: Evaluations are at the core of what we do at the development bank because, with regard to FC interventions or FC projects, it is ultimately all about what effects they have. We use the evaluations to check whether what we imagined at the beginning of a project was actually achieved. In so doing we measure ourselves against our own claims and goals – and of course, against those of the Federal Government who commissions our work. At the same time, we document for the public how we work and how we use the funds entrusted to us. In this respect, the evaluations are very important.

Evaluations always sound good for reasons of transparency, objectivity and integrity. Are you not just paying lip service?

Prof. Dr Eva Terberger: No. Our evaluation results are taken seriously. They are reflected upon and used for ongoing work at the development bank. Sometimes the operational departments even ask if we could take a look at a project, and we’re particularly proud of that. From the outside though we receive far fewer reactions to or inquiries about our reports. I regret that.

Critics consider evaluations of this kind to be tokenism. What would you say to them?

Nagel: Such criticism annoys me because it shows their ignorance. The evaluation reports are not courtesy reports. They have substance and can sometimes be painful to read. By no means do the results always turn out as desired; but evaluations are about the willingness to be self-critical and continue developing. We are ready to do that.

Terberger: Evaluations are a very effective learning tool. We give feedback to the bank, and time and time again we see evaluation impact: the designs of follow-up projects have been adapted and made more incisive.

About Mrs Terberger
Prof Dr Eva Terberger

Prof. Dr Eva Terberger heads the FC Evaluation Unit at KfW Development Bank.

Can you give some examples?

Terberger: In my early years here, the evaluations of sewage treatment plants often bemoaned the lack of concepts for the disposal of sewage sludge, because sewage sludge stored under unprofessional conditions can be a risk. Today, the planning of every sewage treatment plant is accompanied by a concept for sewage sludge disposal. Evaluation results have also helped to tailor “green finance” projects: small, standardised energy-efficiency measures require different promotional schemes than larger investments in renewable energies. And I could go on.

Mr Nagel, you have now been a member of the KfW Executive Board for around one and a half years. What have you learnt yourself from the evaluations?

Nagel: First and foremost I have learnt that development cooperation is a process and that you need a lot of patience. This is demonstrated quite clearly by the reports. Projects often run well initially, but then occasionally drift in the wrong direction: this can be due to the local partners, but circumstances can change as well. Achieving the intended effects is always a new challenge.

Terberger: Especially since KfW often works in fragile contexts. There you have to take higher risks and you achieve your objectives less often, but it is still extremely important to remain committed.

Refugees in Jordan
Impact

In a troubled region of the world, the evaluation attaches importance to making results visible quickly, for example in supporting Syrian refugees in Jordan.

Nagel: I would go even further: failure is part of our work, especially in fragile states. That obviously doesn’t mean burning through taxpayers’ money, but not every measure ends successfully. Yet if we draw the right conclusions from a poor evaluation, we have learnt something for the future. We know that almost one in every four people lives in a fragile environment today, and that this proportion will increase in the coming years. In terms of what awaits us, evaluations play a very important role. Findings that we make today about our work, our strengths and our weaknesses can make our projects of tomorrow more stable and effective, especially in a turbulent world.

Under such circumstances, how important is it to show results quickly?

Nagel: It’s extremely important. Speed is a key factor in stabilizing the situation. People have to see and feel that they are supported and that their living conditions are improving. We have to be even faster in such contexts and avoid the delays that we see again and again, even with projects in troubled regions. Such delays are sometimes caused by external factors, other times we are responsible, it’s our own processes. We’re working on this.

Terberger: I can only agree. This is why results quickly being visible is an important evaluation indicator for assessing the effectiveness of interventions in fragile contexts.

Read more under the image gallery.

Prof Dr Nagel in Jordan
On the spot

Prof. Dr Nagel at a refugee camp in Jordan. Commitment in a fragile context, a key pillar of the work at KfW Development Bank.

The world is unsettled and it is becoming more complex and diverse. Nevertheless, the FC Evaluation Unit continues to work with simple yardsticks such as ratings and a general success rate. Does this help you, Mr Nagel?

Nagel: A kind of rating scale, a label, always helps. The first thing I look at is how a project is rated. You read more intensively if a project has received a very good or a very bad rating.

Terberger: A distinction must be made between the classifications of individual projects, which are based on differentiated data and observations. And then you have the success rate, which we estimate using statistical standards based on the evaluated random sample of projects. I don’t like to focus on the success rate because it is easily misunderstood. As I said, risk and sometimes failure are part and parcel of development cooperation (DC). Nevertheless, a success rate of 77 per cent is often interpreted as meaning that we “made a mess” of 23 per cent of all projects, as a headline in the press once read. Anyone who writes something like this overlooks the risks that FC has to take in order to remain innovative and successful.

Nagel: Development cooperation can be compared to venture capital, except our success rates are better.

Read more under the illustration.

Evaluation overview
On behalf of the Federal Government

KfW has been supporting the German Federal Government in implementing its development-policy goals since 1960 within the scope of Financial Cooperation (FC). We combine financing know-how with development-policy expertise. On behalf of the German Federal Government, and primarily the German Federal Ministry for Economic Cooperation and Development (BMZ), we promote and support programmes and projects that mainly involve public-sector players in developing and emerging economies.

More information on KfW's evaluation work

The evaluations usually take place quite a long time after the end of a project. Might it not be necessary to start earlier, possibly with interim evaluations, to increase the effectiveness of projects?

Nagel: I’m a big fan of this because you can make adjustments and adapt projects in time. It is not possible on a large scale with the existing capacities, but we should try to do this more often than before.

Terberger: However, we must be careful that the evaluations do not monitor construction progress or even become “high-level enforcers” of controls. We need the confidence of our colleagues so we can work. We need to be independent, but not detached. An evaluation unit of know-it-alls is incompatible with our approach of learning together.

Read more under the interactive graphic.

How do you guarantee the independence of the evaluation unit if you are also part of the organisation?

Terberger: Independent evaluation for us means never having had responsibility for the project being evaluated. As a special feature of FC we have put a peer evaluation system in place: someone who has operative responsibility for the electricity sector in Asia for example evaluates an energy project in Africa. We call this internal delegation to the evaluation unit. This means we have independent experts with a great deal of knowledge about FC, and the great thing is that the evaluation experience flows back into their operative work. They are essentially able to refocus on impacts as the core objective of FC.

Nagel: For me, the evaluations being independent is a great asset. I would never consider interfering. The evaluations and the results are sacrosanct for me. I take note of them, but I would never insist on changing a rating, turning a four into a three.

Terberger: I’ve never experienced this in all these years, not once.

Source
Cover evaluation report

This article is taken from the 15th Evaluation Report, published in June 2019. The report documents the work of Financial Cooperation for the years 2017-2018.

See the issue

How could the evaluation work be made more attractive?

Terberger: I would want the evaluations and their “lessons learnt” to be even more visible to the outside world. We could do better here.

Nagel: Absolutely. The issues are global, they are important, they affect us all. We should reconsider our formats and forms of communication. Perhaps we should become more visual, make even greater use of digitalisation potential – for the evaluations themselves, but also for their publication.

Ms Terberger, you will be retiring in the summer of 2019. What else would you like to hand on to your successor?

Terberger: A plea to maintain and further develop evaluations as a learning tool, and the hope that the new head of evaluation will experience the task as enriching as I do.

Published on KfW Stories: Thursday, 6 June 2019