Formative Evaluation

• The Basics of Formative Evaluation
• Types of Information Produced by Formative Evaluation
• Formative Evaluation Methods and Tools
• Using Formative Evaluation Results
• Scenarios
• Getting Started
• Tips for Best Results

The Basics of Formative Evaluation

Description

Formative evaluation is the process of testing program plans, messages, materials, strategies, or modifications for weaknesses and strengths before they are put into effect. Formative evaluation is also used when an unanticipated problem occurs after the program is in effect.

Purpose

Formative evaluation ensures that program materials, strategies, and activities are of the highest possible quality and effectiveness. During the developmental stage of a program, the purpose of formative evaluation is to ensure that the program aspect being evaluated (e.g., educational materials) are appropriate, meaningful, and acceptable for the tobacco control program and the target population. It can also ensure that the means of producing (e.g., printing, copying) and distributing these materials are feasible. When an unanticipated problem occurs after the program is in effect, the purpose of formative evaluation is to identify the reason for the problem and suggest possible solutions. Essentially, formative evaluation is a structured way to provide program staff with additional feedback about their work, products, and activities. This feedback is primarily designed to fine tune the elements of the program, and often includes information that is purely for internal use by program managers.¹

Formative evaluation:²

• Provides information to improve programs
• Helps identify issues in program implementation
• Helps clarify program strengths and limitations
• Helps refine data collection activities

Benefits

There are many benefits of formative evaluation, particularly when program components and/or modifications are fully evaluated before they are released. Formative evaluation can:³

• Identify problems very early in the design process, when only about 10% of the project resources have been expended
• Uncover problems that were not noticed earlier in program development
• Help program designers understand the perspective of persons who are trying to use the program/services in real situations
• Increase the interest of program users, and their eventual acceptance of the final program
• Give the first solid measurements of program performance
• Help program developers decide when different aspects of the program are ready for the next stage of development

When unexpected problems are encountered, formative evaluation can:

• Clarify the nature of the problems
• Identify the causes of the problems
• Allow program users to give voice to their concerns about difficulties encountered

When To Conduct

Formative evaluation typically involves gathering information during the early stages of your project or program. The focus is upon:¹

• Finding out whether your efforts are unfolding as planned
• Uncovering any obstacles, barriers or unexpected opportunities that may have emerged
• Identifying mid-course adjustments and corrections which can help insure the success of your work

But, formative evaluation can also be useful when an existing program is 1) being modified, 2) having problems with no obvious solutions, or 3) being adapted for a new setting, population, problem, or behavior.

Target Population

The types of people you recruit to participate in formative evaluation depend on the evaluation’s purpose and methods. For example, if you are pilot testing materials for a new program, select people or households from the population the program is intended to reach. If you are conducting a brief survey, select people at random. If you plan to use focus groups, select people who will be willing to talk. As another example, if you want to know the level of consumer satisfaction with your program, select evaluation participants from people or households who have already been served by your program. A third example is if you want to know why fewer people than expected are taking advantage of your program, select evaluation participants from among people or households in the target population who did not respond to your flyers or recruitment methods.

Top

Types of Information Produced by Formative Evaluation

As discussed in the last section, formative evaluation is used early on to make sure that program plans, procedures, activities, materials, and modifications will work as expected. Formative evaluation can also be conducted whenever an existing program is being adapted for use with a different target population or in a new location or setting. Here’s a look at the information that can be produced in either of these situations.

While a Program is Being Developed

Whether the program being developed is new or adapted, monitoring of tobacco use or intervention to change behavior, the formative evaluator's first concern is to answer questions similar to these:

• Introduction
  When is the best time to introduce the program or modification to the target population?
   
• Plans and Strategies
  Are the proposed plans and strategies likely to succeed?
   
• Methods for Implementing Program
  Are the proposed methods for implementing program plans, strategies, and evaluation feasible, appropriate, and likely to be effective; or are they unrealistic, poorly timed, or culturally insensitive?
   
• Program Activities
  Are the proposed activities suitable for the target population? That is, are they meaningful, barrier-free, culturally sensitive, and related to the desired outcome? For example, is the literacy level appropriate? Would a tobacco knowledge game appeal to minority teenagers or would they see it as childish? Is “lunch with a legislator” a viable means of educating them about smoke-free environments, or will some members of the population see it as a misuse of program funds?
   
• Logistics
  How much publicity and staff training are needed? Are sufficient resources (human and fiscal) available? Are the program hours and location acceptable? For example, would scheduling program hours during the normal school day make it difficult for some youth in the target population to attend the program?
   
• Acceptance by Program Personnel
  Is the program consistent with staff’s values? Are all staff members comfortable with the roles they have been assigned? For example, are they willing to distribute smoke-free environment literature door-to-door or to participate in weekend activities in order to reach working people?
   
• Barriers to Success
  Are there beliefs among the target population that work against the program? For example, do some people believe that individuals have a right to smoke and it is not up to others to interfere?

After a Program is in Operation

If a program is already in operation but having unanticipated problems, evaluators can conduct a formative evaluation to find the cause. They look at the same aspects of the program as they do during the developmental stage of the program to see: 1) what the source of the problem is, and 2) how to overcome the problem. Formative evaluation can also be used to collect ongoing feedback from participants in a program in order to identify areas that can be improved, and revise the program as needed.

Top

Formative Evaluation Methods and Tools

Because formative evaluators are looking for problems, obstacles, and areas in need of improvement, they need a format that allows evaluation participants the freedom to mention whatever they believe is important. Using a closed-ended quantitative method would only allow the evaluator to gather information about topics that were identified in advance by program staff or the evaluator. As a result, qualitative methods (personal interviews with open-ended questions, focus groups, and participant-observation) are almost always best.

Occasionally, however, quantitative surveys may be appropriate if the evaluation team already knows exactly what they need to ask about. For example, a quantitative survey would be useful when the purpose of evaluation was to find out the level of consumer or staff satisfaction with particular aspects of a program, like the training activities. In this case, the evaluator knows exactly what to ask about.

See the Data Collection section of The Power of Proof for more about personal interviews, focus groups, observation, and survey techniques.

Top

Using Formative Evaluation Results

Planning for Improvement

Formative evaluation gives those people your program intends to serve a voice in program development and modification. Undertaken early, formative evaluation allows the target population to express their preferences for plans, materials, activities, and so forth. As a result, appropriate use of formative evaluation can save the program from squandering resources on efforts that will not be accepted or effective. In other words, well-designed formative evaluation shows which elements of your program are likely to succeed and which need improvement.

Just as importantly, well-designed formative evaluation provides information about how problem areas can be improved. With sufficient attention to the formative evaluation’s design, as soon as the evaluation results are available, you can begin to modify the program’s plans, materials, strategies, and activities to reflect the information gathered.

Providing Continual Feedback

Formative evaluation is a dynamic process. Even after the tobacco control program has begun, formative evaluation should continue. The evaluator must create mechanisms (e.g., customer satisfaction forms to be completed by program participants) that continually provide feedback to program management from participants, staff, supervisors, and anyone else involved in the program.

Because formative evaluation data are often qualitative, usually in the words of program participants and staff, they are particularly valuable when you want to demonstrate the personal aspects of your program to funders, volunteers, or others in a position to provide resources.

Top

• You have been conducting a youth tobacco program and you decide to develop a new component specifically targeting minority youth.
   
• Your program promotes smoke-free workplaces and you have developed some new materials for mailing out to additional workplaces for the purpose of recruiting them.
   
• Recently, several of your program volunteers have stopped coming.
   
• The past two months’ participant satisfaction surveys suggest that participants are less satisfied than they have been in the past.
   
• You have a successful program for educating State legislators about smoke-free environments and a neighboring State has asked you to start a branch for them.

Top

1. Begin by talking with members of the target population to understand their perspectives on the tobacco problem to be addressed. Because you want their perspective, this should be done with in-depth interviews or focus groups. You will need to determine which to use. Be sure to identify persons who will be willing to talk to you.
    
2. Formulate some ideas for the program, based upon the initial input. These can range from recruitment materials, to location, to format, to activities, or all of these.
    
3. Present aspects of the program, as they are developed, to members of the target population and obtain more input. This will require a means of presenting the ideas. You might use mock materials for them to review, you might videotape an example of an activity. Whatever you select, it should give the formative participants a realistic sense of what is being proposed so they can discuss it meaningfully.
    
4. Revise the ideas or materials based upon the formative feedback.
    
5. Present the revised materials to members of the population to ensure that their suggestions have been captured.

   If unexpected difficulties arise in the program, it is time to go back to the target population for more input. You might do this:

6. Formulate a clear description of the problem, e.g., attendance at the program has been declining.
    
7. Present the problem to members of the target population. Again, because you want their perspective, this may best be done with qualitative methods.
    
8. Give participants some specific examples of when the problem has occurred, e.g., last Thursday evening when we held our community education program about smoke-free environments, only 4 people attended.
    
9. Ask participants for their ideas about why the problem may be occurring.

Top

• Begin formative work as early as possible in developing or revising a program.
   
• Get formative input on as many aspects of the program as possible, from planning, to recruiting, to activities, to revisions.
   
• Get input from all stakeholders including staff, participants, and referral sources.
   
• For interviews and focus groups, try to identify people who will be willing to talk.
   
• Get input on program elements as initially designed and again after they are revised, using the initial formative information.

Top

--------------- 
1. Source: Northwest Regional Educational Laboratory. (2001). Formative evaluation. Retrieved June 10, 2004 from the Northwest Regional Educational Laboratory web site: http://www.nwrel.org/evaluation/formative.shtml
 
2. Source: Bond, S., Boyd, S., & Rapp, K. (1997). Taking stock: A practical guide to evaluating your own programs. Chapel Hill, NC: Horizon Research Inc.
http://www.horizon-research.com/reports/1997/stock.pdf
 
3. Source: Maner, W. (1997). Formative usability evaluation. Retrieved June 10, 2004 from the Bowling Green State University, Department of Computer Science:
http://csweb.cs.bgsu.edu/maner/domains/Formeval.htm