How effective is disaster relief?

When disaster strikes, non-governmental organizations (NGOs) are among the first on the scene. The United Nations estimates that there are now more than 37,000 international NGOs, with major donors relying on them more and more.

Inevitably, there are problems. Both the Rwandan genocide of 1994 and the 2004 Indian Ocean tsunami saw chaotic competition among NGOs. Yet there have also been landmark successes. More than 1,400 NGOs operating in 90 countries helped to get 123 countries to ratify the treaty banning landmines. But the sheer scale of the disaster relief "industry" — plus the longer-term development efforts of NGOs — is raising serious concerns about how to measure their performance.

Flexibility allows NGOs to be innovative in ways that organizations like the UN often cannot. But there are few international rules on what an NGO actually is, and the lack of control can lead to unpredictable consequences. In Chad recently, the French NGO L'Arche de Zoé tried to smuggle children out of the country without obtaining permission from either parents or the government.

Among the questions being asked by NGOs, the UN, and national donors is how to prevent the recurrence of past mistakes. The wake-up call for most NGOs came after the Rwandan genocide, when hundreds of small organizations tried to set up ad hoc operations in refugee camps in the Democratic Republic of Congo and Tanzania. Some camps turned into staging posts for armed factions. In the ensuing chaos, more than 50,000 refugees died from cholera.

There was also mayhem following the Indian Ocean tsunami. At one point, more than 400 NGOs were on the ground in Aceh, Indonesia, competing for resources, personnel, and funding. Many of the lessons learned in Rwanda were forgotten or ignored as smaller NGOs with little or no experience in dealing with disasters caused much of the confusion.

The situation in Indonesia led the UN to adopt a new "cluster" system to improve coordination. And, after a review of the Rwanda debacle, 400 NGOs and UN organizations working in 80 countries got together in the Sphere Project to develop a common humanitarian mandate and handbook of standards outlining the minimum performance required of any NGO working in a disaster zone.

As the number of post-intervention reviews increases, a rudimentary framework for evaluating the impact of NGOs has appeared. Rather than simply looking at project inputs and outputs, the emphasis has turned towards measuring the overall impact of an operation.

The idea is to find out if the lives of the people on the receiving end were changed for the better in any sustained way. More and more donors are also insisting that NGOs provide measurable proof that they make a difference.

That sounds fine in theory, but in practice there are drawbacks. By demanding quantifiable results, donors may force program managers to choose easily achieved targets over less measurable actions that accord with sound humanitarian principles. Or reporting about aid programs may be skewed to keep donor funds flowing. The greatest danger is that humanitarian relief will be tailored to meet donors' demands, rather than actual needs.

Until recently, the record on evaluating responses to humanitarian emergencies has been patchy at best. CARE, as both a relief and development agency, can take a long-term approach to disasters, matching emergency relief with a rehabilitation and recovery phase. But this is not an option for NGOs that focus only on emergency responses.

Once their allotted time is up — or their funds run out — they tend to pack up and leave. Even for NGOs that stick around, determining their relief efforts' impact in the middle of a crisis is difficult, if not impossible. Emergencies are chaotic: staff and resources are stretched, the local population is very unlikely to be able to provide meaningful feedback, and pre-crisis baseline data are largely unavailable, so comparisons are complicated. Moreover, all too often, events move too quickly to be measured accurately. And, until recently, donors who were willing to pay for relief were less likely to finance follow-up evaluations.

As a result, emergency relief evaluations often rely on little more than guesswork and assumptions. A 2004 report by the Humanitarian Policy Group cited a survey carried out in Ethiopia after UN agencies said that humanitarian efforts had averted widespread famine in 2000. The claim sounded credible until the subsequent survey showed that the area's crude mortality rate had actually risen to six times the normal base rate. Most of the deaths were from communicable diseases, which malnourished people may well have contracted after crowding into feeding centers.

The HPG therefore recommended long-term monitoring of future humanitarian responses, and said that success or failure should be judged in a broad context rather than by a narrow focus on a specific project. Many people who survive an earthquake or a flood, for instance, may soon face another crisis if the disaster also destroys their only means of earning a living.

New and more sophisticated analytical tools are needed to understand these long-term effects, along with sufficient training to ensure that new methods are applied properly in the field. A recent innovation has been the Coping Strategy Index, devised by the World Food Program and CARE, which analyzes how people cope with short-term food crises while also taking into account their future vulnerability to hunger.

NGOs do the lion's share of the world's humanitarian work, and some mistakes are inevitable. But as we deepen our experience of humanitarian relief and development, we must learn the lessons of the past and understand how much more there is to know.

Robert Glasser is Secretary General of CARE International. Copyright: Project Syndicate/Europe's World, 2008. Exclusive to The Sunday Times