Observation

This section includes these parts. Click on each for more.

• Fundamentals
• When to Use
• Observation Validity and Reliability
• Observation Instruments
• Sample Observations
• Format and Logistics

Fundamentals

Evaluation by observation is a very useful means of gathering data. It involves having members of the evaluation team present at the program event, watching the event as it takes place, and making notes about what they see taking place. Multiple observations should be made by more than one observer.

Observation is an effective way to see what actually occurs without relying on human recall. For that reason, it is useful in identifying the following:

• Physical barriers for participants (e.g., who never got there?)
• Smoothness of program operation
• Interactions among program staff and participants
• Areas of success
• Areas of weakness

Observation can be done in person, by videotaping an event, or with a combination of both. The advantage of videotaping is that several observers can review the tape without difficulty, and any conflict in notes can be easily resolved by looking at the tape again.

Participant observation is one particular type of in-person observation. In participant-observation, members of the evaluation team participate (to the degree possible) in the event being observed, look at events from the perspective of a participant while observing, and make notes about their experiences and observations. The presence of observers can alter the way the program is conducted. The added benefit of participant-observation is that, when the observers are also program participants, they can blend with the other participants and their presence is less apparent.

Top

When to Use

Observation is valuable when there is benefit to seeing the effect of your program on the behavior of the intended audience, rather than relying upon report. It is useful for several reasons:

• It allows you to witness, first hand, how a program activity actually works, rather than how it was intended to work.



• Unlike personal interviews or focus groups, participant-observation can produce information from people who have difficulty verbalizing their opinions and feelings.



• In the case of problems, the parties involved in the problem may not realize the effect of their actions or words on other people, or they may not be fully aware of their own reactions to particular situations. Thus, they would not be able to report the event accurately.

Problems that participants are not aware of can come to light. For example, the personnel director may believe that the company promotes clean indoor air, since employees are not smoking within 20 feet of the building entrance. In actuality, however, the location smokers are using may be directly beside the intake vent for the building's air system.

Top

Observation Validity and Reliability

The best method of ensuring that observation data are reliable and valid is to have more than one independent observer. The reports of these observers can then be compared and discrepant reports resolved, or not used. Observations can also be validated against other materials such as videotapes.

Top

Observation Instruments

Observation data can be recorded manually, or by voice recording. If there is particular information to be recorded, this should be noted on an observation form to remind the observer to cover each area. The following is a list of observations that might be made during a seminar on smoke-free environments:

• Number of people present
• People who left or were turned away
• Anyone who is acting as a leader
• People who were not participating

Observations should be made at different times of day, during different days of the week, with different program staff present, to ensure that the program is fully represented. This will require a schedule of observation, as well as the observation form. An observation schedule for observing legislative hearings about smoke-free environmental legislation may look like the grid below. The initials indicate who will be observing:

Click for another sample of an observation form.

Top

Sample Observations

The following is an example of how an evaluator might use observation to assess public reaction to a program's new advertising campaign about the dangers of secondhand smoke in the workplace using billboards, flyers, posters, radio, newspapers, and/or television.

Top

Format and Logistics

The key to gathering data by way of observation is well-trained observers and detailed note taking. For the observation to get the kinds of results that will allow you to judge the effectiveness of your program, observers should be:

• Alert
• Unobtrusive and ensure that their activities do not disrupt the program
• Trained in observational methods
• Aware of the type of observations of greatest importance to the program evaluation.

Observation requires that one or more observers watch participants interact with your program. What you select to observe may vary from participant behavior during a coalition meeting or seminar, to participants' use of a new product or new information provided by your program. Observation sessions must be carefully organized and timed. So, before beginning a session, make sure you have done the following:

1. Create a schedule for observing (or videotaping) participants interacting with your program.



2. Make arrangements for your presence as an observer and/or the presence of a videotape recorder well in advance.



3. Get permission from program management or others in charge.



4. Let participants know why you are observing and/or taping them, and how their privacy will be maintained.



5. If possible, have an expert among the observers who can tell you what to look for.¹
    
6. If there are certain items that must be observed, create a form on which these items are listed.



7. Be sure that observers review what they have recorded for clarity after each observation period, while the information is still fresh in their minds.



8. For analysis, compare notes across observers and look for commonalities and trends.

Top

--------------------
1. Source: Prime II Project. (n.d.). Data gathering methods. In Performance improvement toolkit. Retrieved April 12, 2004 from the Prime II web site.

2. Source: Institute of Education Technology. (1996). Data collection methods. Retrieved April 12, 2004 from the Open University web site.