 |
| “It
is good to have an end to journey towards,
but it is the journey that matters in the end.”
- Ursula K. le Guinn |
Process Evaluation
The Basics of Process Evaluation
Description
Process evaluation is the mechanism for assessing
the delivery of your program. It’s
a kind of quality control that documents what makes up the
program each time it is delivered, and whether the program
is being delivered as planned. While outcome evaluation is
designed to find out what the program accomplished,
process evaluation is designed to find out what the program
actually was when it was delivered. It is then possible
to determine whether the actual program differed from the
planned program, and whether or not the deviations
from what was planned benefited or detracted from the program.
Purpose
Process evaluation can serve many purposes. Some of the
most important are to:
- Assess the degree to which the program is being conducted
as planned
- Compare program delivery across staff or sites
- Link specific methods of program delivery to program outcomes
- Provide information for improving program elements
- Provide feedback for program managers regarding the quality
of implementation
- Provide information on program accountability to stakeholders
such as funders, volunteers, or participants
When To Conduct
Process evaluation should begin as soon as the program is
put into action and continue throughout the life of the program.
Therefore, you need to design the forms and methods for process
evaluation while the program is under development.
Target Population
In process evaluation we study elements of the program,
rather than people or workplaces. We compare what the program
actually did with what the program planned
to do. In order to do this we might count the number of volunteers
trained, the number of legislators given information, or the
number of workplaces visited, because each of these is a program
activity. But the target population for process evaluation
is the number of people with whom the program actually interacted,
not the number of people with whom the program planned to
interact.
Top
Types of Information Produced
by Process Evaluation
Process evaluation is useful for fully understanding how
a program works, how it produces the results that it does.
Some typical questions that process evaluation might answer
include:
-
Was each program activity completed as
planned?
-
To how many people/workplaces/communities
was each activity delivered?
-
How many program materials were distributed?
-
What were participants’, workplaces’
or communities’ perceptions of each activity? Of
the program?
-
What were staff members’ perceptions
of each activity? Of the program?
-
What were the strengths of the way the
program was implemented?
-
What were the difficulties, barriers,
or challenges to implementation?
-
What strengths and weaknesses were present
for of each step of the program’s tobacco use prevention
intervention?
-
Did the people or organizations that participated
in the intervention understand it?
-
Were all the resources needed for project
activities available?
-
What was the nature of the interaction
between staff and participants?
As this list suggests, there are numerous questions that
might be addressed in a process evaluation. The evaluation
questions for your process evaluation can be selected by
carefully considering what is important to know about your
program.
Top
Process Evaluation Methods
and Tools
The method you select for process evaluation depends on
what you want to know and how you plan to keep track of
those data while the program is in operation. These three
important questions can help guide you in this exercise.1
-
What is the program intended to
do, and for whom?
Information to answer this question should be available
from documents such as program proposals, correspondence,
requests for funding, and recruitment materials. Additional
information can be obtained through interviews with those
who developed the program, and those who were recruited
to participate. For more about conducting interviews,
go to Individual
Interviews in the Data Collection section of The Power
of Proof series.
- What does the program actually deliver, and to
whom?
This question can be more difficult to answer than the first
one. The primary way of determining what your program is
delivering is to keep track of all contacts with the people,
households, workplaces, or communities served by the program.
It is also important to keep track of all program-related
services and items distributed to, or received from, these
people or entities. Let’s take a look at what each
of these entails.
Tracking materials sent and received
If you send out informational, educational, or other
materials, or items like press releases, you will want
to have a record of the number sent. If you collect
materials from other organizations, you will want to
keep track of this, as well.
Click
here to view a sample form for tracking materials
distributed and received.
Tracking people contacted: Directly and Indirectly
Counting direct contacts is straightforward.
One method of keeping track of direct contacts is to
use simple encounter forms which can be designed to
collect basic information 1) about each person, workplace,
etc. that has direct contact with the program and 2)
about the nature of the contact. These encounter forms
must be designed while the program is being developed
and ready for use as soon as the program begins. Click
here to view a sample encounter form.
Not all contact with a program is direct. For example,
many school-based tobacco prevention programs provide
information to schoolchildren (direct) who, in turn,
take the information home to parents (indirect). Other
programs train members of the target population as counselors
(direct) to work with their peers in the school community
(indirect). Often, a program’s stated purpose
is to reach community members through indirect methods.
Often, too, programs have an indirect effect that was
not planned.
To estimate the number of people the program reaches
indirectly, you could ask the people
with whom the program has direct contact to keep track
of their contacts (the people to whom they give the
program’s information or service). For this purpose,
they could use a system similar to the program’s
system of keeping track of its direct contacts. In many
cases, however, asking people with whom the program
has direct contact to keep track of their contacts is
impractical, unreliable, or both. When this is the case,
you can devise a reliable method for estimating the
number of indirect contacts. For example, you could
estimate that half the third graders who attended a
tobacco prevention program would speak to their parents
about the information given to them.
Click
here to view a sample form for tracking people contacted.
- How does the program that is delivered differ
from the program as planned?
Once you have described what the program was intended to
be (#1 above) and what the program actually is (#2 above),
you can determine the ways in which these two things differ
by comparing them. Sometimes, the actual program will be
an improvement over the program that was planned. At other
times, the deviations will be problematic, and need to be
brought in line with the program as planned.
Top
Using Process Evaluation
Results
The results of process evaluation have many uses. These
include:
-
Quality control for delivery of the program
-
Demonstrating which program elements produce
effective outcomes
-
Providing guidance for improving program
elements or activities
-
Giving feedback about implementation to
program managers
-
Demonstrating how well the target population
has been reached
-
Providing information to stakeholders
about the level and quality of program activity
In addition, much of the information gathered during process
evaluation can be used for short- or long-term outcome evaluation,
during which you will be calculating the effect your program
has had on the target population.
Top
Scenarios
Here are a few scenarios in which Process Evaluation would
be appropriate:
-
You are training program volunteers to
speak to minority youth about tobacco use.
-
Your program is mailing out materials
to local workplaces for the purpose of recruiting them
to participate in the smoke-free environment program.
-
Program volunteers are meeting with legislators
to discuss smoke-free environment legislation.
-
You are meeting with past program participants
to determine ways in which the program might be improved.
-
You are mailing a newsletter to workplaces
that have participated in your program to bring them up
to date on smoke-free legislative actions.
In each instance, you would be interested in the number
of members of the target population who were reached, the
number of materials or services delivered, the way in which
the services or materials were delivered, and what members
of the target population might have been missed.
Top
Getting Started
The most important steps for starting process evaluation
are:
-
Develop a clear description of the program
and its activities.
Unless you have a clear understanding of what the program
was trying to do, you will not know whether or not it
achieved its objectives. For a review of how to develop
and use logic models, go to Defining
Your Program Using Logic Models in the Setting
the Stage section of The Power of Proof.
- Develop a clear description of the target population.
Do you want your program to be evaluated on the basis of
the number of members of the target population
who are contacted or on the number of contacts
with members of the target population? The distinction is
important, especially when a member of the target population
may receive independent value or additional benefit from
each contact with the program.
Number of members contacted
For number of members contacted, count only once each
member of the target population who had contact with your
program, regardless of how many times that person/workplace/community
has contact.
Number of contacts with members
For number of contacts with members, count once each time
the program had contact with a person/workplace/community,
regardless of how many times some of the members had contact.
The number of contacts with members should be
the same as, or higher than, the number of members contacted.
- Develop necessary forms for use in documenting delivery
of program materials and services. For samples of different
forms for documenting services and materials, click
here.
- Start with the first delivery of program materials/services
to a member of the target population.
Top
Tips for Best Results
The following list includes important things to keep in
mind as you begin process evaluation.
-
Be sure stakeholders understand that you
are NOT evaluating the short- or long-term outcomes, or
the effect of the program.
-
Stress that you are evaluating “what
is happening,” and not how well any one staff member
or participant is performing.
-
Let the stakeholders know that the feedback
they provide may or may not result in changes in the program.
-
Monitor the collection and entry of process
data. Data should be reviewed on a regular basis to ensure
the quality of program delivery.
-
Make sure that the results get to those
who need to know them and are in a position to make use
of the information.
Top
---------------
1. Source: Bliss, M.J. & Emshoff, J.G.
Workbook for designing a process evaluation. Produced
for the Georgia Department of Human Resources, Division
of Public Health.
|
|
