Questionnaires

This section includes these parts. Click on each for more.

• Fundamentals
• When to Use
• Questionnaire Validity and Reliability
• The Questionnaire Instrument
• Format and Logistics
• Sample Questions

Fundamentals

A questionnaire is a data collection instrument that is used for of collecting systematic, standardized information that can be expressed numerically or in short answers. Questionnaires can provide information on respondent behaviors, values, perceptions, desires, needs, preferences, priorities, opinions, and experiences.

Questionnaires offer the following benefits:¹

• They simplify the gathering of information from large numbers of people.


• Responses can be readily analyzed to determine underlying patterns and relationships, including trends over time.


• They can target specific respondent groups and/or specific issues or objectives of interest to the program staff and participants.


• They can gather information about people's perceptions, desires, and opinions unavailable from other sources.


• Their standard format provides consistency across respondents.


• The use of response choices (e.g., agree, disagree) can make data processing easy.

Questionnaires can be administered face-to-face, by telephone, through the mail, by e-mail, or on the web. The first two methods require an interviewer, and the last three are completed by the respondent. Questionnaires can be used in experimental and quasi-experimental designs, or in surveys that are conducted at one or many points in time. When used in a survey, a questionnaire may be referred to as the survey instrument.

Top

When to Use

Questionnaires are especially helpful when you know the questions you need to ask, and want information from a wide range of people. They are also valuable for use in measuring results in experimental and quasi-experimental designs, because of their standard and timely administration. For this reason, they are often used in short- and long-term outcome evaluation, but they can also be used during formative and process evaluation. For more about evaluation types, see the Evaluation Planning or Types of Evaluation sections of Power of Proof.

Telephone-administered and respondent-completed questionnaires are used when you want to gather data from a large number of people and it is not practical to meet them all face to face. Telephone, e-mail, and web-based instruments are especially useful when results are needed quickly.

While a program is under development, questionnaires are a valuable tool when you want to:

• Identify the aspects of the program that potential users like and dislike before those aspects are put into effect.

  Use of a questionnaire is possible because you know what the aspects of a program are and can ask about each. Such information allows you to modify the aspects that are unlikely to be successful. For example, you might ask youth to rate on a scale of 1 to 5 how well they understood the section on risks associated with smoking in the brochures distributed as a part of the Minority Youth Tobacco Prevention program. If many youth respond that they did not understand the risks, then it will be important to clarify the material covered in the brochures before distributing them on a large scale.

• Gather baseline data on the knowledge, attitudes, and beliefs of the target population.

  For example, suppose your program's goal is to increase the number of legislators who support smoke-free environment legislation. You can administer a questionnaire to legislators in the target population, before the program begins, to see:

  • how much they know about the risks associated with secondhand smoke,
  • what their attitudes are toward smoke-free environments, and
  • what they believe about whether or not people support such legislation.


• Gather baseline data on the rates at which members of the target population engage in behaviors of interest to the program.

For example, if your program goal is to increase the number of smoke-free workplaces, you can find out the number of workplaces that are already smoke-free. Having such information allows you to set a realistic goal for how much you want to increase that number.

After the program is in effect, you can use questionnaires to:

• Measure the level of participants' satisfaction with the program.

  You can determine whether people in the target population are receiving information about the program, what the most common sources for the information are, and whether the information they are receiving is correct. With such knowledge, you can eliminate the expenses (e.g., cost of newspaper advertisements) associated with aspects of the program that are not working.

• Determine why the program is having unexpected problems with no clear solution.

  Questionnaires administered through surveys can often locate the source of the problem, which may then lead to the solution. For example, they can show you how people who do not participate in your program differ from those who do. Perhaps you will find that the people who do not participate have an ongoing schedule conflict at the time of the program and are, therefore, unable to attend. Whatever the reason, once you know what it is, you can modify the program to fix the problem you discover.

• Measure the immediate, or short-term effect, your program is having on the target population's knowledge, attitudes, beliefs, and behaviors (i.e., how much they have changed since the program began).

  For example, if your minority youth tobacco prevention program is effective:

  • The target population's knowledge of and belief in the hazards or smoking will have increased, and
  • Their attitudes toward smoking will be less favorable.
     
• Show the increase in the number of people who report they are engaging in the behavior you are interested in (e.g., how many more CEOs report that their workplace is smoke-free than did so before the program, or how many more minority young adults report that they do not smoke) during short- or long-term outcome evaluation.

Top

Questionnaire Validity and Reliability

The following are a few tips for establishing the validity and reliability of questionnaires.

Here are some general facts about the reliability and validity of questionnaires:

• The fact that each person is answering identical questions on a questionnaire increases the reliability of responses.²
  


• When information is sensitive or private, anonymous questionnaires may reduce the likelihood of biased reporting and, thus, raise the validity of the information.³
  


• At first glance, it seems like it would be easy. But producing useful information through questionnaires is a complex and difficult task. Questionnaires that contain ill chosen questions and poor response options produce information that is likely to be invalid and not reliable, not to mention misleading and potentially damaging.²

In the Sample Questions section, we provide a list of samples questions, some examples of good questions that increase validity and reliability and some poorly written questions that would decrease questionnaire validity and reliability. To review that section now, click here.

Top

The Questionnaire Instrument

As we have discussed, the questionnaire is the most common data collection instrument used to gather the data in surveys as well as experimental and quasi-experimental designs. Other instruments include checklists, interview schedules, interview and focus group guides, and logs or records of phone calls or other program contacts.

Before designing the questionnaire, you must decide on the method you will use to administer it, because the method will dictate things like the questionnaire's length, complexity, and level of language. For example, questionnaires that will be completed by the respondent without an interviewer (i.e., self-administered) must be shorter and easier to follow than those to be administered by a trained interviewer.

This table shows the advantages and disadvantages of the three methods for administering questionnaires: face-to-face interview, telephone interview, or distribution (e.g., through the mail) to people who complete and return the questionnaire to the program.

The best method to use depends on the purpose of the evaluation and the target population. Let's say, for example, you want to evaluate a training program to assist sales staff in refusing to sell cigarettes to minors. If program participants have a moderate level of education, having them complete and return a questionnaire before they leave the classroom is clearly the least expensive and most efficient method of data gathering. On the other hand, if class participants have problems reading, a questionnaire to be completed in class would not be useful, and you may need to conduct face-to-face interviews. Another example might be if you were evaluating a smoking cessation program in a well-defined, low-income housing area. In this case, you may need to interview. If so, face-to-face would be better than telephone interviews, since income is an issue and some people with low income do not have telephones.

Questionnaire Cover Letter

When mailing or e-mailing the questionnaire to respondents, make sure to include a cover letter, personalized with the resident's name rather than using a generic "Dear Resident." The cover letter should spell out the purpose and objectives of the survey, who is being asked to respond to the survey and why, and how the resident was selected to receive the questionnaire. It should also give a deadline for completing the instrument. A two-week deadline is suggested. If an incentive will be provided to those who complete the questionnaire, information on how to obtain the incentive should be provided.

The letter should be signed by an official, such as the project manager or some other person, to show that the study has been given a high priority. The letter must include a contact person and telephone number in case the respondent has questions. In addition, the letter should include a pre-addressed, stamped envelope for returning the questionnaire.⁴

To view a sample cover letter, click here.

Designing Your Questionnaire

The process of designing your questionnaire requires organization and thoughtful consideration of how best to word the questions so they will elicit the most useful responses. Click on the link below for a list of steps to help you design your questionnaire.

Questionnaire Design Process

Questionnaire Creation Services

Here is a list of just a few of the services that will assist you in creating your survey instrument or questionnaire. Many will also help you administer them and analyze the results. (NOTE: TTAC does not endorse any of these products)

• Survey Console evaluation instruments


• Zoomerang - survey creation and analysis


• Survey Monkey - enable anyone to create professional online surveys quickly and easily


• EZSurvey - for the web, email, and mobile computing


• Key Survey - online business survey software for the internet.


• Pendulab survey creation service


• e-Questionnaire survey creation and marketing services


• Iventa survey, poll, and message board tools


• Insiteful Surveys: Do-it-yourself online survey solution

Top

Format and Logistics

Administering questionnaires requires a good bit of organization. Before you get started, do the following:⁵

• Decide how important the statistical validity and reliability of your data are. The more important they are, the more you will need to consult an expert to help determine your sample group, method of data collection, and how you will compute your results.
  
  
• It is best to use existing questionnaires, if they are available. They save development time, and are especially valuable if they have been tested to ensure they work.



• If you must design your own questionnaire, make sure you pilot test it.

For a review of pilot testing, go to Ensuring the Effectiveness of Instruments and Procedures. Click here to view that material now.

Administering Your Questionnaire

Here are some easy-to-follow steps for getting you started with questionnaire administration for face-to-face interviews, telephone interviews, and respondent-completed questionnaires.

• Face-to-face Interviews
  
  
  1. Develop your questionnaire
  2. Pilot test the questionnaire
  3. Assess validity and reliability
  4. Train your interviewers
  5. Determine your method of selecting those to be interviewed(e.g., go to every 3rd house starting with X; approach every 5th person entering the mall)
  6. Identify a private interviewing location
  7. Conduct the interviews
  8. Check for consistency across interviewers
     
     For more about face-to-face surveys, visit this site.



• Telephone Interview
  
  
  1. Develop your questionnaire
  2. Pilot test the questionnaire
  3. Assess validity and reliability
  4. Determine your method of selecting those to be interviewed (e.g., random-digit dialing; interview oldest female in household)
  5. Conduct the interviews
  6. Check for consistency across interviewers



• Respondent-Completed
  
  
  1. Develop your questionnaire
  2. Pilot test the questionnaire
  3. Assess validity and reliability
  4. If mailed or e-mailed, identify a sampling frame (i.e., a listing of members of the target population that has their addresses or e-mail addresses)
  5. Select those to receive the questionnaire (e.g., every 5th person on list)
  6. Prepare the cover letter and include a deadline
  7. Send (or distribute) the questionnaire
  8. Log in responses and note the date received
  9. Follow-up with non-respondents (if they can be identified, send targeted reminders, if not, send general reminders)
  10. Compare respondents and non-respondents on any information you have available (e.g., zip code of residence, gender)

Dealing with non-response

Non-response is a serious problem when it occurs. This is because people who actually complete the questionnaire and return it might not be representative of the entire target population. They may have a higher reading level, a greater interest in the subject of the questionnaire, more free time, or differ in any of a number of ways from those who do not respond. When non-response occurs, you should first try to increase the response rate. If you still have non-response, you should try to determine how the respondents and non-respondents are different, so you will know how your results are biased. Here are some specific things to do (in order):

1. Try again to get the people to respond. You can do this by resending the questionnaire, or phoning to encourage them to send the questionnaire back. After two repeat attempts you probably will not increase response any further.



2. Follow-up with non-respondents and ask them if they would please answer a few of the questions. Then, ask the questions you have selected as being the most important on the questionnaire. Compare the responses of these "non-respondents" to the responses of the people who completed and returned their questionnaire.



3. If you cannot follow-up with non-respondents because the questionnaire is anonymous, compare the respondents to the total target population with respect to any characteristics you have available (e.g., gender, zip code, etc.)

Top

Sample Questions

The most difficult part of creating a questionnaire is crafting the questions so that you elicit the best information possible for the results you seek. For best results, here are a few pointers for writing questionnaire items:

Top

--------------------
1. Source: U.S. Department of Housing and Urban Development. (1978). Task 6: Data collection. In Program evaluation and analysis: A technical guide for state and local governments. Washington, DC: Prepared for the U.S. Department of Housing and Urban Development by Public Technology, Inc. (pp. 20-25).

2. Source: Office of Juvenile Justice and Delinquency Prevention. (1989). Special topics in program evaluation. In Evaluating juvenile justice programs: A design monograph for state planners. Washington, DC: Prepared for the U.S. Department of Justice, Office of Juvenile Justice and Delinquency Prevention by Community Research Associates, Inc. (pp. 40-41).

3. Source: National Institute of Justice. (1989). Obtaining information for evaluations. In Evaluating drug control and system improvement projects: Guidelines for projects supported by the Bureau of Justice Assistance. Washington, DC: Prepared for the U.S. Department of Justice, National Institute of Justice by Abt Associates, Inc. (pp. 5-7).

4. Source: U.S. Department of Housing and Urban Development, Office of Policy Development and Research. (1997). How do you get the information you need for your evaluation? In A guide to evaluating crime control of programs in public housing. Washington, DC: Prepared for the U.S. Department of Housing and Urban Development by KRA Corporation (pp. 6.1-6.15).

5. Source: Prime II Project. (n.d.). Data gathering methods. In Performance improvement toolkit. Retrieved April 12, 2004 from the Prime II web site.