"A place for everything, everything in its place."
— Benjamin Franklin

Before You Begin

While data collection can be rewarding, much work is required to be sure it is done well so that the data you collect truly reflect the impact of your program. To ensure that this process is organized and credible, do the following before you begin. Click on each for more.

1. Identify Existing Data Source(s)

Before you decide how to collect the data, you will need to decide where the data are going to come from and whether you must obtain new information, or whether you can use existing data (information that others have collected, usually for other purposes). Sometimes, state or local survey data or data that are part of another project will include the information you need. In other instances, program files or other records, public or private, may contain the information.

Some data, like participants' attitudes toward the program, will come directly from your program participants. Other data, like the proportion of minority youth in the community who smoke before and after the program, may be available from state surveys, or other existing data sources.

When appropriate, there are advantages to using existing data rather than going to the expense and energy of collecting new information. To determine whether it is even possible to use existing data, consider each of the following:

  • Can the existing data be obtained for my evaluation purposes?
  • How well can the existing data answer my evaluation questions?
  • How well do the existing data represent my target population?
  • How available are the existing data?

If the data can be obtained, can answer your evaluation questions, can represent your target population, and are available, consider yourself fortunate. Now you must decide whether it will be better to use these existing data or collect new data. To make this decision, ask yourself these three questions:1

  • What data sources, existing or new, are likely to provide the most accurate information for this target population (see Determine Credibility of Existing Data)?
  • What data sources will allow the least costly and most rapid data collection?
  • Will the collection of information pose an undue burden on the existing sources?

The answers to these three questions must be balanced. Clearly, accuracy is desirable, but at what cost in terms of money and time? And, how much burden are the existing sources willing to tolerate? For example, suppose the data you wanted could be obtained from files. Due to privacy, only designated people may be able to go through the files, and the task of data collection might add greatly to the burden upon these individuals. Conversely, if it were possible to use outside data collectors, the presence of these persons as they are going through the files may be disruptive to the setting in which the files are located.

Top

 

2. Determine Credibility of Existing Data

Tip: It's especially important to consider how credible each source will be to the persons to whom you intend to distribute the results of your evaluation (e.g., the funders).

Having the most accurate data sources possible is the most important factor for any successful evaluation. But, what makes one source more credible than another? Sometimes it's difficult to know. For example, carefully maintained records may be more accurate than the memory of an individual. On the other hand, overworked professionals may not attend to detail when recording information, especially if the record systems were not intended to provide evaluation data. Furthermore, different persons may record differently, creating inconsistencies in the record system.

There are two ways to assess the accuracy of data: Reliability and Validity.

Data Reliability is a measure of the degree to which the data can be reproduced, or replicated. For example, suppose two different members of the evaluation team reviewed the congressional record to determine a legislator's record of support for smoke-free environment legislation. If they both produced the same counts regarding instances of support, the data have reliability, specifically inter-rater reliability.

Data Validity is a measure of the degree to which the data actually measure what they are intended to measure. If the records of a vote indicate that a legislator voted against the smoke-free environment legislation, and, when asked, she confirms that she voted against it, the records have been validated. Conversely, if medical records indicate that 12 percent of pregnant Latina teens smoke, but 18 percent report smoking when interviewed, the medical records data have not been validated.

Other Credibility Factors

In addition to validity and reliability, several other factors contribute to the credibility of a data source. One such factor is whether or not those who collected the data have political, or other interests, that might be likely to influence the data collection methods or results.

Another factor is the reputation of the organization collecting the information among tobacco prevention and control advocates. For example, data from the Centers for Disease Control and Prevention (CDC) or the National Institutes of Health are viewed as highly credible. Efforts are made to see that their data are accurate, and both organizations are well known. In contrast, data from medical records can be suspect as they are rarely validated, and often collected for treatment purposes.

Top

 

3. Determine the Best Method for Collecting Data

Tip: Using mixed methods increases the cross-checks on different subsets of findings and generates increased stakeholder confidence in the overall findings. In addition, combining methods provides a way to triangulate findings, which maximizes the strengths and minimizes the limitations of each method. Using mixed methods enables you to validate your findings, enhance reliability, and build a more thorough evaluation for improving program effectiveness.2

Once you have determined whether your data will come from new or existing sources, it is important to select the data collection methods that will be the most appropriate for achieving your evaluation objectives. If you are using existing data sources, the main questions will be "What information do I select from the source?" and "How do I record it for your evaluation purposes?" When you are collecting new data, there will be many more decisions to make. The first of these is whether to use qualitative methods, quantitative methods, or both. In the Evaluation Planning section, we introduced these two types of data collection. Let's review.


Qualitative methods are open-ended and allow the evaluator unlimited scope for probing the feelings, beliefs, and impressions of the people participating in the evaluation, and to do so without prejudicing participants with the evaluator's own opinions. They also allow the evaluator to judge the intensity of people's preference for one item or another. Such methods include:

  • individual interviews
  • observation
  • focus groups

Quantitative methods are ways of gathering objective data that can be expressed in numbers (e.g., a count of the people with whom a program had contact or the percentage of change in a particular behavior by the target population). Unlike the results produced by qualitative methods, when correctly gathered, the results produced by quantitative methods can be used to draw conclusions about the target population. Such methods include surveys using:

  • respondent-completed instruments (e.g., questionnaires) administered by direct distribution or through the mail, web, or e-mail
  • interviewer-completed instruments administered face-to-face or through telephone surveys

Choosing the method best suited for your audience and objective(s) requires an understanding of what each method can reveal about your program. Here are some examples of when each method might be used in a program to promote smoke-free environments.

If... Then... Example
.you want to learn the full range of reasons for program participants' choices, and the topic is not sensitive .conduct focus groups How participants decided which public information session to attend.
.you want to learn the full range of reasons for program participants' choices and either (1) the topic may be considered sensitive, or (2) it is difficult to gather program participants in one location. .conduct qualitative individual interviews Two examples include:
  • Whether participants plan to vote for or against a smoke-free environment, and why.
  • Participants are spread out over a large rural area.
.you already know the range of potential responses, and you want to determine which response is the most common in the target population. .administer a questionnaire At which of the following locations they would be most likely to attend a public information session; or, how likely they would be to attend a public information session-from not at all likely to very likely
.you want to conduct the survey quickly and the people in your target population all have telephones and internet access. .administer a questionnaire by telephone or the web You want to determine whether legislators plan to support the Clean Indoor Air bill coming up for a vote in two days.
.you want to conduct a survey, but some of the people in your target population do not have telephones. .administer a questionnaire by mail, or by a face-to-face survey done by going door-to-door, or to a public place (e.g., shopping mall) where members of the population will be present You want to determine the number of persons in a predominantly Hispanic low income community who would be willing to attend a smoking cessation program conducted in Spanish.
.you want to conduct a survey, but some of the people in your target population have low literacy or have difficulty reading (e.g., uneducated, visually impaired, older) .administer a questionnaire by telephone or face-to-face by going door-to-door or to a public place (e.g., intersection) where members of the population will be present You want to determine the effectiveness of your program in reaching older adults with diabetes, many of whom are visually-impaired.
.you want to conduct a survey and the topic may be considered sensitive .administer a questionnaire by telephone, or a respondent-completed questionnaire to ensure anonymity Asking underage minority teens about smoking habits.


Tip: While the method you choose to collect data depends on the source, it also depends on the evaluation type and design (see Evaluation Planning: Evaluation Type and Design). For example, formative evaluations often call for qualitative information, collected in the form of conversations. On the other hand, outcome evaluations usually call for quantitative information, collected through surveys conducted before and after exposure to the program.

In choosing a data collection method, the CDC recommends that you consider the following:3

  • The purpose of the evaluation
    Which method seems most appropriate for your purpose and the questions that you want to answer?

  • The users of the evaluation
    Will the method allow you to gather information that can be analyzed and presented in a way that will be seen as credible by your intended audience? Will they want standardized quantitative information from a data source such as the Adult Tobacco Survey, or descriptive, narrative information from "real people", or both?

  • The respondents from whom you will collect the data
    Where and how can respondents best be reached? What is culturally appropriate? For example, is conducting a phone interview or a more personal, face-to-face interview more appropriate for certain population groups?

  • The resources available (time, money, volunteers, travel expenses, supplies)
    Which method(s) can you afford and manage well? What is feasible? Will your evaluation be completed in time for the next legislative session or prior to the end of the school year? Consider your own abilities and time. Do you have an evaluation background or will you have to hire an evaluator? Do program funds and relevant policies allow you to hire external evaluators?

  • The degree of intrusiveness-interruptions to the program or participants
    Will the method disrupt the program or be seen as intrusive by the respondents? Also consider issues of confidentiality if the information that you are seeking is sensitive.

  • Type of information
    Do you want representative information that applies to all participants (standardized information such as that from a survey, structured interview, or observation checklist that will be comparable nationally and across states)? Or, do you want to examine the range and diversity of experiences, or tell an in-depth story of particular people or programs (e.g., descriptive data as from a case study)?

  • The advantages and disadvantages of each method
    What are the key strengths and weaknesses in each? Consider issues such as time and respondent burden, cost, necessary infrastructure, access to sites and records, and overall level of complexity. What is the most appropriate for your evaluation needs?

Top

 

4. Select or Create Data Collection Instruments

An important part of data collection is using the same data collection instrument to get information from all of your information sources. If you are gathering information from files or other records, rather than from people, the instrument may be a form designed so that you can record all of the information needed from each file. This is called a data abstraction form. In the case of interviews and focus groups, the instrument may be a general outline of topics to be covered. This is known as an interview/focus group guide. In the case of surveys, the instrument is usually composed of a series of carefully-worded questions or statements for the respondent to answer. Often the answers are selected from answer choices that are provided on the instrument.

Creating a good data collection instrument can be difficult and time-consuming. For that reason, before you create a new instrument you should see if a suitable instrument already exists.

Whether you decide to use an existing instrument or to develop your own, the instrument you use should meet the following criteria.4

  • It should include questions that can be used to measure the concepts addressed or affected by your program (e.g., knowledge of tobacco prevention methods).

  • It should be appropriate for your participants in terms of age or developmental level, language, and ease of use. Questions should be written in simple and easy-to-understand language. These characteristics can be checked by conducting focus groups of participants or pilot testing the instruments.

  • It should respect and reflect the participants' cultural backgrounds. The definitions, concepts, and items in the instrument should be relevant to the participants' community and experience.

  • It should be possible to complete in a reasonable timeframe. Again, pilot testing can reveal these issues.

Creating Your Own Instrument

If you choose to create your own instrument, there are many evaluation, assessment, and testing web sites and sources that can help you get started. Here are just a few.

Ensuring the Effectiveness of Instruments and Procedures

Even after you have selected or created the instruments, you are not yet ready to begin collecting data. Before you can actually begin collecting evaluation information, you must pilot test your instruments and procedures. This means trying out the instruments on a small number of people from your target population, or from a group of participants from a similar project offered by your program or a program with a similar purpose.

The pilot test will determine whether the instruments:

  • Obtain all the information you need for the evaluation,
  • Are of the correct reading level for the target population,
  • Are understood by the target population,
  • Are not too lengthy or difficult for the respondents,
  • Are culturally appropriate for the target population, and
  • Are effective overall.

In most cases, you will need to make at least some changes the instruments as a result of the pilot test. If you do, it is better to have learned on a small number of people than to have wasted your evaluation resources collecting data with instruments that were not appropriate. Occasionally, you will not need to make any changes. If you use a sub-sample of your target population for the pretest and the instruments do not have to be changed, you can combine the data from this sub-sample into the actual evaluation data.

The kinds of information that can be obtained from a pilot test include:

  • How long it takes to complete interviews or fill out each instrument
  • Whether each of the questions was clear
  • Whether the instruments can be completed by participants without assistance from staff
  • Whether the necessary information can be collected in the established time frame
  • Whether instruments and procedures are culturally appropriate
  • Whether the notification procedures (letters, informed consent, and the like) are easily implemented and executed

For more about pilot testing your data collection instruments, see chapter 7 of The Program Manager's Guide to Evaluation.

Top

 

5. Determine Instrument Validity and Reliability

The instrument you choose to collect data must be able to collect the information you need to answer your questions about your program. Just as with data, the instrument must be valid and reliable.

Instrument Reliability is a measure of the degree to which an individual's responses are reproducible, or consistent, both over time and within the instrument. For example, suppose a question at the beginning of a data collection instrument asked, "Do you support increased tobacco taxes?" and 67% of respondents said, "Yes". Suppose that another question, toward the middle of the same instrument asked, "Are you in favor of increasing taxes on tobacco?" and 66% of respondents said, "Yes". This would suggest that the data were reliable. On the other hand, suppose data from a statewide web-based survey during the first week of April indicated that 65% of restaurant owners had a high level of knowledge about clean indoor air ordinances. If data from the same survey, conducted during the second week of April, indicated that only 35% of restaurant owners had a high level of knowledge about these ordinances, the data would not reliable (i.e., reproducible).

Instrument Validity is a measure of the degree to which the instrument actually measures what it is intended to measure. If the instrument is supposed to measure knowledge about tobacco prevention and control, then we would expect people who attended a tobacco use prevention course to get higher scores than people who did not attend such a course. If they do, this is evidence that the instrument is valid. If they do not, this suggests that the instrument is not valid.

There are three main types of validity: face validity, content validity, and construct validity.

  • Face validity is the degree to which the instrument appears to measure what it is intended to measure. Face validity is important for good rapport between interviewer (questioner) and respondent. If the interviewer informs the respondent that the survey is about smoking around children, but the respondent believes it is about something else, the respondent may distrust the evaluator's intent and may refuse to answer or may not answer truthfully. Assess face validity through pilot tests (e.g., focus groups or personal interviews with a subgroup of the target population) and by having subject-matter experts review the questionnaire.

  • Content validity is the degree to which all relevant aspects of the topic being addressed are covered by the survey instrument. Assess content validity by having subject-matter experts review the content of the instrument.

  • Construct validity is the degree to which the survey instrument accurately measures what it is supposed to measure. The easiest way to establish construct validity is to compare the results obtained using your instrument with results from a similar instrument that has already been shown to be valid. You can do this by administering both instruments to a group of people who are similar to, or selected from, the target population. If the results of your instrument correlate with those of the existing instrument, it demonstrates the validity of your instrument.

Top

 

6: Determine How Much Data to Collect

How much data you collect will depend upon the balance between your needs and the resources available. If you need to be able to draw conclusions about a number of sub-populations (e.g., various racial or ethnic groups) you will need more information than if you are only interested in the population as a whole. On the other hand, how many of the race/ethnic subpopulations you can draw conclusions about may be limited by the availability of funds, or staff who can speak the language.

There are two ways to think about the quantity of information you will need to collect:

  • How many questions do you need to ask?
  • How many people do you need information from (the sample size)?

How many questions do you need to ask?

The questions you ask should be restricted to those that are necessary to answer your evaluation questions with an acceptable level of detail. While it can be exciting to get lots of information, this places a burden upon the people who are providing the answers. In general, the more questions you ask, the less people will be willing to take the time to provide complete information. If you want a good response from your participants, keep it short and simple. In addition, long interviews and questionnaires also take longer for data processing and analysis. This will add to the cost of your evaluation. For more about how to write survey questions see Basics of Developing Questionnaires at http://www.mapnp.org/library/evaluatn/questnrs.htm

How many people do you need information from?

How many people you need to ask (i.e., the sample size) depends, again, on the level of detail you are interested in, as well as the types of comparisons you want to make. You will also need to determine the level at which you want to be able to draw conclusions (e.g., state, county, region, congressional district). Counties often appreciate and want county-level estimates; however, this usually means larger sample sizes and more expense.

Your study must have a certain minimum quantity of data to detect a specified change produced by your program. In general, detecting small amounts of change requires larger sample sizes. For example, detecting a 5% increase would require a larger sample size than detecting a 10% increase. If you use tobacco data sources such as the Youth Tobacco Survey, the sample size has already been determined.3

If you are designing your own evaluation tool, you will need the help of a statistician to determine an adequate sample size. The three pieces of information the statistician will want to know are:

  • What is your outcome measure? In particular, are you interested in averages (like average knowledge about the adverse effects of tobacco) or in the percentages of people in designated categories (like the percent who support Smoke-free Environments)?

  • How much does this outcome measure vary? For example, are most people pretty similar in their level of knowledge, or does it vary a lot from person to person? [NOTE: If your outcome measure has been used before, the statistician will LOVE you if you are able to find a value called the "standard deviation" of the measure.]

  • How much difference do you want to be able to detect? If your program increases average knowledge by 1 point, is that enough for you to care about or does it need to be at least 5 points?

Top

 

7. Establish Procedures for Collecting Data

Once you decide what type of instrument you will use to collect evaluation information, you must establish a set of procedures to ensure that this information will be collected in a consistent and systematic manner, especially if you are going to be making comparisons among groups. For this reason, standard data collection procedures should be established, and data collectors should be trained in using these procedures.

Everyone involved in collecting evaluation information must be trained in these procedures:

  • What instrument will be used to collect the information? Each data collector should be trained in the use of the instruments. If there is more than one instrument, then the order in which they are to be used should be made clear.

  • When will you collect the information? The timeframe during which the data are to be collected must be clearly specified. There is some information that may need to be collected before the program starts and other information that needs to be collected at the end of the program. Having the timeframe spelled out will ensure that the information is collected as scheduled.

  • Where will you collect the information? As discussed earlier, you will need to determine the sources from which the information will be collected. In some instances you may be using program records, while in other instances you may be relying on participants' coming to a specific location to complete the survey instrument or to participate in a group discussion about their experiences. When the data collected are sensitive, the location is especially important, and should allow privacy. You will need to determine where you will collect the information and convey this to program participants.

  • Who will collect the information? This responsibility must be clearly specified or you will risk having some information collection activities fall through the cracks. In some situations you will need to be sure that information collectors meet certain criteria. For example, they may need to be familiar with the culture or the language of the individuals they are interviewing or observing. If the questionnaire is being administered by interviewers (for example, residents hired and trained to conduct interviews), those persons must be properly trained to administer the questionnaire. Training will ensure that the interviewers are familiar with the survey instrument.

Top

---------------
1. Source: U.S. Department of Housing and Urban Development, Office of Policy Development and Research. (1997). How do you get the information you need for your evaluation? In A guide to evaluating crime control of programs in public housing. Washington, DC: Prepared for the U.S. Department of Housing and Urban Development by KRA Corporation (pp. 6.1-6.15).

2. Source: Bond, S.L., Boyd, S.E., & Rapp, K.A. (1997). Taking stock: A practical guide to evaluating your own programs. Chapel Hill, NC: Horizon Research.
[Referenced in: Centers for Disease Control and Prevention. (2001). Gather credible evidence. In Introduction to program evaluation for comprehensive tobacco control programs (pp. 49-56). Atlanta, GA: the Author.

3. Source: Centers for Disease Control and Prevention. (2001). Gather credible evidence. In Introduction to program evaluation for comprehensive tobacco control programs (pp. 49-56). Atlanta, GA: the Author.

4. Source: Child Outcomes Research and Evaluation Team. (n.d.). How do you get the information you need for your evaluation? In The program manager's guide to evaluation. Retrieved April 12, 2004 from the Administration for Children and Families web site.
 
Search TTAC