Outcome Evaluation

The benefits or changes that occur for individuals or populations during or after they participate in program activities are the program outcomes. Outcomes may relate to behavior, skills, knowledge, attitudes, values, condition, or other attributes. They are what participants know, think, or can do; how they behave; or what their condition is; that is different following the program.¹

Outcome evaluations can be viewed as short term or longer term, and are important to making major decisions about program continuation, expansion, reduction, and funding.² In fact, to qualify for funding, programs need to include outcome evaluation into their program design. The only differences in purpose between short-term and long-term outcome evaluation are how soon you can do the evaluation and the program effect that is measured.

Click on each for more:

• Short-term Outcome Evaluation
  
  - The Basics of Short-term Outcome Evaluation
  - Types of Information Produced by Short-term Outcome Evaluation
  - Short-term Outcome Evaluation Methods and Tools
  - Using Short-term Outcome Evaluation Results
  - Scenarios
  - Getting Started
  - Tips for Best Results
   
• Long-term Outcome Evaluation
  
  - The Basics of Long-term Outcome Evaluation
  - Types of Information Produced by Long-term Outcome Evaluation
  - Long-term Outcome Evaluation Methods and Tools
  - Using Long-term Outcome Evaluation Results
  - Scenarios
  - Getting Started
  - Tips for Best Results

Short-term Outcome Evaluation

The Basics of Short-term Outcome Evaluation

Description

Short-term outcome evaluation is the process of assessing indicators of the program’s progress toward its ultimate goal(s). It typically measures changes in outcomes such as availability of materials, knowledge, attitudes, beliefs, and skills.

Purpose

The purpose of short-term outcome evaluation is to determine whether the program creates changes in short-term outcomes for the target population (e.g., access to information, tobacco knowledge, or refusal skills) that lead to changes in longer-term outcomes or the ultimate goal (e.g., a reduction in smoking behavior, or in tobacco-related morbidity and mortality).

For example, suppose one of your program activities was to provide legislators with an information session on public attitudes toward second-hand smoke. Short-term outcome evaluation would assess the more immediate results of the program, such as the degree to which knowledge, or attitudes, about smoke-free environments and related legislation changed among legislators who attended the session.

At this point, evaluators usually are not measuring changes in behavior (e.g., voting for smoke-free environment legislation). Although information about this behavior could be used as a measure of outcome, it may take a while to change, so it might better be used as a measure of longer-term outcome.

When To Conduct

Measurements of the target population’s knowledge, attitudes, and beliefs are taken immediately before the first encounter with the target population (e.g., before the first training class or before any materials are distributed). These are the “baseline” measurements. Begin measuring changes in knowledge, attitudes, and beliefs immediately after the first encounter; and continue to measure changes after the program has made contact with each person or group who participates in the program.

Target Population

For short-term outcome evaluation, the target population usually consists of participants, (i.e., people, workplaces, or communities that received the program service.) Sometimes comparison or control groups are also used in short-term outcome evaluations. The members of the comparison group should not have received the program service, but otherwise be as similar as possible to those who did receive it.

Top

Types of Information Produced by Short-term Outcome Evaluation

The types of information you can expect from short-term outcome evaluation include:

• Amount of change in resources (e.g., number of culturally appropriate materials available)
• Amount of change in the target population’s knowledge, attitudes, and beliefs
• The degree to which a program is meeting its short-range objectives (e.g., a change in awareness about the dangers of second-hand smoke among program participants)
• An indication of whether the program is moving toward achieving its ultimate goals (e.g., a reduction in lung cancer)
• Evidence of which materials and activities appear to be effective and which do not
• Indications of how resources might best be moved from areas of the program that do not appear to be productive to those that do

Top

Short-term Outcome Evaluation Methods and Tools

Generally, short-term outcome evaluation uses methods that count resources available, or measure the target population’s knowledge, attitudes, beliefs, and behaviors before any program activities occur (baseline measurement), and again after each person or group receives the activity.

Surveys

Knowledge, attitudes, and beliefs are almost always measured by a survey instrument, such as a questionnaire, containing closed-ended items (e.g., multiple-choice questions). For example, you could ask each person attending a training class to complete a questionnaire before and after the class to find out how much their knowledge, attitudes, and beliefs changed as a result of the training program. For more about survey methods, go to the Questionnaires section of Data Collection in The Power of Proof series.

Observation

Occasionally, knowledge, attitudes, and beliefs are assessed by direct observation. For example, an observer might watch a volunteer talk to a legislator about smoke-free environments, both before and after training the volunteer, to see whether the volunteer is correctly using the skills he was provided. Evaluators might also observe group discussions, to watch and listen for signs of participants’ attitudes or beliefs. Observation is often more costly, less efficient, and less feasible than administering a survey instrument. For more about observation methods, go to the Observation section of Data Collection in The Power of Proof series.

Top

Using Short-term Outcome Evaluation Results

As we discussed in the section on Logic Models in the Setting the Stage section, each activity is tied to a short-term outcome. If the short-term outcome evaluation results are positive for a particular set of activities, you can use these results to justify continuing this set of activities. If the results are negative, you can use them to suggest revisions, or new activities.

Obviously, if short-term outcome evaluation shows that none of the activities are effective, long-term outcome evaluation of the program is not necessary. Programs with positive short-term outcome results are more likely to receive further funding than those whose results are negative. However, if an evaluator can show why the program was ineffective and how it is being modified to make it effective, the program may be able to justify receiving further funds.

Top

Scenarios

Listed below are three examples of short-term outcomes, taken from Defining your Program Using Logic Models in Setting the Stage, and scenarios of how they might be accomplished.

Top

Getting Started

Here are some tips for getting started with short-term outcome evaluation.

1. Think small. Focus on one program or one aspect of your services (e.g., one activity) at a time. This will help you to gather data more effectively and will provide more meaningful results. Example: Select “make presentation to funders” as the activity of interest.
    
2. Identify each short-term and intermediate-term outcome from your logic model. Example (continued): The short-term outcome associated with our selected activity is “funders are aware of the need of funding of intervention for high risk populations.” The intermediate-term outcome is “increased funding of interventions for high risk populations.”
    
3. Decide upon a measure for each outcome. Example (continued): We can measure funders’ awareness with a measure of perceived importance of funding specific high risk populations. We can measure increased funding by comparing dollars expended for high risk populations from the target funders before and after the intervention.
    
4. Decide whether or not a comparison group will be used. If so, select the comparison group. Example (continued): Let’s decide to use a comparison group of funders we have not yet approached.
    
5. Develop a set of guidelines for administering the measures (e.g., where and when to administer, who should administer, order of administration). Example (continued): Trained interviewers will administer the measure of perceived importance to the designated contact person in each funding group over the telephone at a time convenient to the designated contact person.
    
6. Administer each measure before starting any program activities associated with that outcome. Example (continued): The designated contact person can be asked about perceived importance and about dollars expended immediately before the presentation is made.
    
7. Plan for administering each measure again after each participant completes the activity associated with that measure. Decide when to re-administer the measures to the comparison group. Example (continued): In our example, it is probably best to re-assess the funding outcome after the next funding cycle has occurred. Perceived importance can be re-assessed at any time after the presentation has taken place.

Top

Tips for Best Results

The following list includes important tips for beginning short-term outcome evaluation.

• Start with the program’s logic model.
   
• Be sure to assess the most critical short-term outcomes.
   
• Allow a reasonable amount of time between the pretest and the posttest for the outcome you are interested in changing.
   
• Use a comparison group if at all feasible.

Top

Long-term Outcome Evaluation

The Basics of Long-term Outcome Evaluation

Description

Long term outcome evaluation is the process of measuring whether your program met its ultimate goal (e.g., reducing morbidity and mortality due tobacco use) or the outcomes that immediately precede that goal (e.g., a reduction in smoking behavior). Some evaluators also refer to evaluation of whether the ultimate goal was met as Impact Evaluation. Because there has been some confusion in the use of this language, however, we have avoided the use of that term.

Purpose

The purpose of long-term outcome evaluation is to measure changes in preventive behaviors that are directly linked to tobacco-related morbidity and death, and measure changes in the morbidity and death itself.

When To Conduct

Preparation for long term outcome evaluation begins when the program is being designed. The design of the program affects the quality of the data you will have for long-term outcome evaluation. Furthermore, an initial measure of the outcomes of interest (i.e., baseline data) must be collected immediately before participants have their first encounters with the program.

For ongoing programs (e.g., a series of tobacco use prevention classes given each year in elementary schools), conduct long-term outcome evaluation as soon as enough participants have completed the program activities to make long term outcome evaluation results meaningful. Depending on the number of children in tobacco prevention classes, you could conduct long term outcome evaluation, for example, every year, every three years, or every five years.

For one-time programs (e.g., a six-month program to lobby legislators to support smoke-free environment legislation), begin long-term outcome evaluation as soon as the program is complete. Consider also conducting a follow-up of long-term outcome evaluation a year or three years after the program is complete, to find out how well the program’s effects are sustained over time.

Target Population

For long-term outcome evaluation, the target population consists of all the participants (i.e., people, workplaces, communities) that received the program service. Usually comparison or control groups are also used in long-term outcome evaluations. The members of the comparison group should not have received the program service, but otherwise be as similar as possible to those who did receive it.

Top

Types of Information Produced by Long-term Outcome Evaluation

Service providers expect clear evidence that the resources they expend actually produce benefits for people, and want to know that programs to which they devote their time really make a difference. Through long-term outcome evaluation, they can determine if programs really make a difference in the lives of people. Some other things long-term outcome evaluation can show include:

• Improvements in accountability.
• Feedback information back into programs on how well they are doing.
• Findings programs can use to adapt, improve, and become more effective.
• Data about why the program is doing what it's doing and how it thinks participants will be better off.
• A clearer picture of the purpose of their efforts for program mangers and staff. That clarification alone frequently leads to more focused and productive service delivery.

Being able to demonstrate that their efforts are making a difference for people pays important dividends for programs long term. It can, for example, help programs:³

• Recruit and retain talented staff
• Enlist and motivate able volunteers
• Attract new participants
• Engage collaborators
• Garner support for innovative efforts
• Win designation as a model or demonstration site for other programs
• Retain or increase funding
• Gain favorable public recognition

Top

Long-term Outcome Evaluation Methods and Tools

The Basic Guide to Program Evaluation, written by Carter McNamara, MBA, PhD, suggests some of these steps for accomplishing long-term outcome evaluation⁴:

1. Go to your logic model and identify the major long-term outcomes that you want to examine or verify for the program under evaluation. To review logic model development and use, see Defining your Program Using Logic Models in Setting the Stage.
    
2. Prioritize the outcomes and, if your time and resources are limited, pick the top two to four most important outcomes to examine for now.
    
3. Specify a "target" goal for the outcome, e.g., what number or percent of high risk youth you commit to prevent from smoking throughout their high school years.
    
4. For each long-term outcome, specify what observable measures, or indicators, will suggest that you're achieving that outcome with your target population. This is often the most important and enlightening step in outcomes-based evaluation.
    
5. Identify the information you need for these indicators.
    
6. Decide how that information can be efficiently and realistically gathered. Consider program documentation (e.g., correspondence, minutes, etc.), observation of program personnel and participants in the program, questionnaires and interviews about participants’ perceived or actual benefits from the program, case studies of program failures and successes, etc.
    
7. Collect the information.
    
8. Analyze the information.
    
9. Report the findings.

Long-term Behavioral Outcomes versus Morbidity and Mortality

The methods used for measuring changes in behavior are essentially the same, relatively-easy methods as those used to measure changes in knowledge, attitudes, and beliefs during short-term outcome evaluation. In general, however, measuring changes in morbidity and mortality is not so easy. For example, you can measure the change in smoking behavior of minority youth who participated in a tobacco use prevention class soon after the class is over. Measuring the reduction in morbidity and mortality as a result of those same youths' changes in behavior is much more difficult.

A major cause of this difficulty is that, while every youth has a certain attitude and behaves in a certain way with regard to tobacco, the number of them who will die or suffer serious morbidity as a result of their tobacco use is small. Furthermore, in most instances it will take 20 or more years of tobacco use for sickness or death to occur. Therefore, documenting changes in morbidity and mortality that are directly the result of your program to reduce tobacco use requires a vastly larger study population than does documenting changes in attitudes, beliefs, and behaviors. And, in addition to a large study population, documenting changes in morbidity and mortality requires a long-term study, which is both expensive and time consuming.

In spite of the difficulty of demonstrating a program’s effect upon morbidity and mortality, many funders, especially government agencies, want this information. So, what can you do to provide it? Our recommendation is to convert data on behavior change into estimates of changes in morbidity and mortality.

When a long-term study of morbidity and mortality is not feasible, you can convert the more readily obtained information on changes in behavior into estimates of changes in morbidity or mortality. To do this, you must have three items of information:

• Data showing the prevalence of the behavior (e.g., smoking) before the program began (i.e, baseline data on the pre-program behavior of the target population)
• Data showing the prevalence of the behavior after the program is complete (i.e., data on the post-program behavior of the target population)
• Data showing how much reducing the behavior reduces the associated morbidity or mortality

Once you have this information, it is possible to turn data on behavior change into reductions in morbidity and mortality. At this point, you may want the help of a statistician. For those of you who are mathematically inclined, you can click here for an example of the calculation.

Top

Using Long-term Outcome Evaluation Results

The main use of long-term outcome evaluation results is to demonstrate the effectiveness of your program with data that are meaningful in terms of health. Positive results of long-term outcome evaluation provide even stronger evidence than those of short-term outcome evaluation to justify continued use of funding, volunteers, and other resources for your program. Furthermore, you are more likely to be able to publish long-term outcome results in professional or scientific journals than you are the results of any other type of evaluation, if you believe these results (positive or negative) can be of value to researchers or other tobacco prevention programs.

Top

Scenarios

The following is an example of a long-term outcome, taken from Defining your Program Using Logic Models in Setting the Stage, with a scenario for how to assess its accomplishment.

Long-term outcome:
Reduce population disparities in tobacco use.

Scenario:
Suppose your program has been working at the state level in Georgia since 1999. You are fortunate in that you can use existing data from the Centers for Disease Control and Prevention (CDC) to measure your outcome.

The Behavioral Risk Factor Surveillance System (BRFSS) provides state-level data about persons age 18 or older for the following question: Do you now smoke cigarettes every day, some days, or not at all? It also includes information on ethnicity and race. These identical questions have been asked since 1996. The entire English-language versions of the questionnaire can be found at http://www.cdc.gov/brfss/questionnaires/english.htm

To assess the effect of your program, you would look at the disparity in tobacco use in 1999, and then compare it to the disparity in tobacco use in the most recent year available. You might define disparity as the difference in smoking rate between whites and those of other race/ethnicities. If you were interested in specific other race/ethnicities, you could compare these specific groups to whites.

Top

Getting Started

To get started with your long-term outcome evaluation, follow these steps⁵.

1. Elicit Input. While it is important to involve staff and stakeholders in each type of evaluation, it is especially important to involve them in the long-term outcome evaluation design process. Be sure that everyone who might be impacted by the evaluation has input into the long-term outcomes and goals of the project and understands their role in the evaluation process.
    
2. Review the logic model. Identify each long-term outcome and goal from your logic model. For more about logic models, review Defining your Program Using Logic Models in Setting the Stage.
    
3. Identify barriers and constraints. Brainstorm to determine any anticipated barriers you might encounter during the evaluation process.
    
4. Decide what you will use as a measure of each outcome/goal.
    
5. Decide whether or not a comparison group will be used. If so, select the comparison group.
    
6. Draft your own version of your long-term outcome evaluation plan and then have others review your drafts of those sections of the plan.
    
7. Consider getting a grant to support development of your plan, e.g., maybe $3,000 to $5,000, particularly to have evaluation expertise to review your plans and your methods of data collection. If you can't get this grant, you still can proceed with your plan.
    
8. Develop a set of guidelines for administering the measures (e.g., where and when to administer, who should administer, order of administration).
    
9. Administer each measure before starting any program activities associated with that outcome.
    
10. Plan for administering each measure again after each participant completes the activity or activities associated with the outcome being assessed. Decide when to re-administer the measures to the comparison group.

Top

Tips for Best Results

Here are a few tips for getting the most out of your long-term outcome evaluation

• Make sure each measure is linked to an outcome from your Logic Model. For more about logic models, review Defining your Program Using Logic Models in Setting the Stage.
   
• Review the results of the Process Evaluation to ensure you understand exactly what the program is that you are planning to evaluate.
   
• Determine the amount of staffing needed to conduct the long-term outcome evaluation (e.g., to administer the measures).
   
• Identify resources available to assist (e.g., university students, volunteers).
   
• Be sure to gather baseline information before starting program activities. If you make your first measurement after the program activities have already begun, you will lose the ability to see the effect that has already occurred as a result of the program.

Top

--------------- 
1. Source: Outcome Measurement Resource Network. (1996). Measuring program outcomes: A practical approach: An introduction to outcome measurement. Retrieved June 10, 2004 from the United Way of America web site:
http://national.unitedway.org/outcomes/resources/mpo/intro.cfm
 
2. Source: Formative Evaluation Research Associates. (n.d.). Evaluation types. Retrieved June 10, 2004 from the Formative Evaluation Research Associates web site:
http://www.feraonline.com/typeeval.html
 
3. Source: Outcome Measurement Resource Network. (1996). Outcome measurement: What and why? Why measure outcomes? Retrieved June 10, 2004 from the United Way of America web site:
http://national.unitedway.org/outcomes/resources/mpo/why.cfm
 

4. Source: McNamera, C. (1999). Basic guide to program evaluation. Retrieved June 10, 2004 from The Management Assistance Program for Nonprofits web site:
http://www.mapnp.org/library/evaluatn/fnl_eval.htm
 
5. Source: Text adapted from:
McNamera, C. (1999). Basic guide to outcomes-based evaluation for nonprofit organizations with very limited resources. Retrieved June 10, 2004 from The Management Assistance Program for Nonprofits web site:
http://www.mapnp.org/library/evaluatn/outcomes.htm
 
University of Washington and University of Michigan. (n.d.). How libraries and librarians help: Outcome-based evaluation toolkit. Retrieved June 10, 2004 from the Information Behavior in Everyday Context web site.