“It does not take much strength to do things, but it requires
great strength to decide on what to do.”
-- Elbert Hubbard

Decide What to Evaluate

Programs come in all shapes and sizes, with vast differences in their complexity and components. Depending on your purpose and needs, evaluation can help identify the benefits of an entire program, specific components of the program, or even activities or services within a component. So, how do you decide which aspects of your program need to be evaluated to achieve your purpose? Sometimes, there is evidence of a problem in a specific program area and you are seeking information to understand what is wrong. Sometimes the decision is based on available resources. Other times the decision is made for you, driven by the larger organization or funders.

Tip: It is better to conduct an effective evaluation of a single program component than to attempt an evaluation of several components or an entire program without sufficient resources.1

In order to decide what to evaluate, remind yourself of the purpose, and then refer to your program’s logic model (see The Power of Proof: Setting the Stage). This table shows the questions you should ask yourself to help you decide what to evaluate for three different evaluation purposes and examples.2

Purpose Questions to Ask Example
To improve the delivery of the program • Are all program goals (or outcomes or outputs) being achieved? (Refer to your logic model. See Setting the Stage) If not, which program goals are not being accomplished?
 
• What inputs, activities, outputs and outcomes lead to this goal?
 
• Are the activities to meet this goal in place?
 
• Are the necessary inputs present?
 
• Have there been any changes in inputs or activities?
 
• Have any new activities been added that are having an unknown impact?
 
• Was formative information gathered for each activity?
 

Suppose your program is designed to increase the adoption of smoke-free workplace policies by employers in your county. While you have made some progress, only one employer has adopted a policy in the past two years. You wonder whether there are any problems that are contributing to this low adoption rate.
 
Some activities that occur as a part of your program are contacting Employee Assistance Program (EAP) directors by telephone to identify employers with an interest in the issue, providing draft smoke-free environment policies to these directors, and then meeting with the EAP directors of interested organizations to discuss the policies.
 
You can choose to evaluate whether any one or all of these activities is working. In addition, these activities require resources for identifying and contacting EAP directors, e.g., employer listings, telephone numbers, callers, copies of the policies, copying equipment, and so forth. Again, you can evaluate the availability of any or all of these resources.
To measure the program’s effectiveness. • Am I interested in the overall effectiveness of the program, or am I interested in a specific output, outcome, or impact?
 
• Has sufficient time passed for the long-term outcome to be achieved, or is it more appropriate to assess outputs and short-term outcomes?
 
• Do I need to know whether the activities that lead to the output(s), outcome(s), or impact(s) of interest are effective, or do I already have information about this?		 Let’s use the same example of a program that is designed to increase the adoption of smoke-free workplace policies by employers in a county. If the program has only been in existence for two years, it may be too early to assess the impact of the program upon lung cancer. Instead, evaluating an outcome like the number of employers who have adopted policies or an output such as the number of EAP directors who have agreed to a visit will provide early indications of program effectiveness.
To demonstrate the effective use of resources • Am I interested in specific resources, or resources overall?
 
• Do the resources of interest link to particular activities and outputs/ outcomes/impacts?		 In the program designed to increase the adoption of smoke-free workplace policies by employers in a county, recall that the activities included contacting EAP directors to identify employers with an interest in smoke-free environments, and to provide draft policies to these persons. One of the resources used was volunteers, who were used to make phone calls to EAP directors, copy materials, and mail the materials to the EAP directors. You want to demonstrate your effective use of these resources to the volunteer organization. Since the volunteers were specifically linked to the activities of making phone calls, copying policies, and mailing materials, your evaluation can focus on the outputs, outcomes, and impacts of these activities.
 
 

---------------
1. Source: Child Outcomes Research and Evaluation Team. (n.d.). How do you prepare for an evaluation? In The program manager's guide to evaluation. Retrieved January 25, 2004 from the Administration for Children and Families web site.

2. Source: Centers for Disease Control and Prevention. (2001). Focus the evaluation design. In Introduction to program evaluation for comprehensive tobacco control programs (pp. 37-48). Atlanta, GA: the Author.

 
 
Search TTAC