The Fallacy of Financial Ratios - Part 2
By Tony Poderis
October 27, 2003
Click here for part one of this article.
It's Up To The Development Professional
If you're a development professional in a non-profit organization, you're the person who needs take the reins on this one. Just as your organization should not wait for funders to set the measurement scale, you should not wait for those who manage your programs to address the issue of evaluation. You're the one charged with bringing in the grants, and evaluation is likely to become a make or break issue with foundations and other grant-making organizations.
So, to get you started, here are five actions your organization should take in regard to evaluation:
- Include a statement of organization and program evaluation in the strategic
plan.
- Get buy-in on the part of everyone in the organization for the need to
conduct meaningful evaluation.
- Build an evaluation component into every program activity.
- Include an evaluation component in every grant request and ask for funding
to cover it.
- Work with other non-profits and with funders to address the issue of evaluation.
Evaluation of a non-profit program comes in two flavors. First, there is the traditional model as performed by a neutral external evaluator. Its advantage is the inherent objectivity of an evaluation performed without bias. Then there is the participatory model in which staff, users/clients, and other stakeholders participate in a process that produces an evaluation of a project or program.
In general, external evaluation has been perceived to have greater credibility. However, most non-profit organizations lack the funding to constantly engage external evaluators. The advantages to participatory evaluation include greater staff satisfaction with the process and as a result easier implementation of recommendations.
There are a number of different types of program evaluation, but the one most likely to resonate with funders is outcome evaluation. The goal of outcome evaluation is to determine whether the actions taken delivered the desired results, and were they cost efficient.
When conducting an outcome evaluation, you must first identify the outcomes you wish to achieve. Then determine what criteria will be observed to gauge success or failure. Next develop a process for making those observations, and finally report the results of your analysis and findings.
I'm not going to address here the identification of outcomes and determination of criteria for a participatory outcome evaluation. This is a planning process. Nor am I going to get into how to write a final report. However, I think a few suggestions on how to collect data are in order.
Surveys are a good way to collect a lot of information quickly. Unsigned
questionnaires guarantee anonymity. They're easy to manage, and multiple-choice
responses can be easy to quantify. But you have to be careful not to write questions
that bias responses. Questionnaires lack a personal touch, and both survey design
and sample selection require a high level of expertise. At the very least, a
professional should be involved in question creation.
Focus groups give you a chance to explore issues in depth. Putting six
or seven people in a room with a video camera running and asking questions of
the group as a whole can yield valuable information. Try it with users/clients,
contributors, or just about any constituency. However, it is tough to get the
people to commit to giving the time and then show up when expected. The group
facilitator needs to be able to establish instant comfort for participants and
keep them both engaged and on track. You will probably need a professional communicator
as group facilitator. Focus groups should be scheduled on a continuing basis
to establish benchmarks and measure change. Since the responses are freeform,
it can be hard to analyze results, and that analysis can be quite time consuming.
Interviews give you a chance to talk with users/clients one-on-one. They
can yield some great information due to the give and take of the conversational
process. Maybe you should be conducting an "exit interview" with each user/client
or a cross section of them. However, the interview process is time consuming.
The information acquired is often anecdotal in nature and can be very hard to
quantify. It is easy for a less skillful interviewer to bias responses without
intending to.
Observation is the most direct way to gather information about how a
program is functioning if the observer is well trained and understands the nuances
of what he or she is observing. The downside is that an observer is an intrusion
in the normal process and therefore changes it. What is observed can be hard
to categorize and interpret, and it is an expensive way to judge outcomes.
Case reviews are the most complete form of assessment of a given user's
or client's experience. They are a great way to show exactly what the organization
does, how it does it, and how an outcome is achieved. The problem with them
is that they are very time consuming and really only show what happens for one
person. They don't provide quantifiable data. They don't show breadth, and they
only work for certain types of organizations.
A great deal more could be said about the process of evaluation and outcome measurement. However that is a topic unto itself, and there are people better qualified than I to address it. My interest here is to encourage you to motivate your organization to implement a process of ongoing evaluation of its programs. The data yielded will go a long way towards heading off the mounting pressure for one-size-fits-all evaluation ratios.
Greater evaluation of organizations and their programs is coming. It is not going away. The question is: are the non-profit organizations going to define the process or are the funders? I know which answer I prefer.
Tony Poderis is a development consultant, speaker and author of It's a Great Day to Fund-Raise. You can reach him through his web site at www.raise-funds.com
