If questions are too vague or broad, or require data that is unavailable or unobservable, they are not answerable. When they feel uneasy, they will retreat into their shells and you will not get the input that you need. Fun Starter Question or Activity - Ask a fun question or incorporate a team building game to get the group’s creative juices flowing. The questions in an evaluation questionnaire should focus on the goals, objectives, strategies, and work plan. Questions should be carefully planned but appear spontaneous during the interview. References*Adapted from Rossi, P. H., Freeman, H. E., & Lipsey, M. W. (1999). Prepared for: U.S. Environmental Protection Agency Office of Human Resources and Organizational Services Office of Planning, Analysis, and Accountability Washington, DC 20460 Which of the following best describes your discretionary spending during the past six months? And what can be learned? It might also be useful to have some more specific questions under How is the project functioning from administrative, organizational, and/or personnel perspectives? No matter why a focus group is formed, you can help your members have a more productive session by asking questions that grab their attention and get them thinking beyond the obvious. Ratings scale - Select an appropriate ratings scale to measure different levels of approval. Image by Gerd Altmann from Pixabay, How to Develop a Change Communication Plan. What are the target population’s characteristics? Message and data rates may apply. A focus group lasting more than 90 minutes probably includes too many questions or topics for discussion. New York: Kluwer Academic Publishers. Focus Group Questions to Engage Your Group. Some sample questions are: 3. What are the outputs, outcomes, objectives, and goals of the project? If the problem is not identifiable by stakeholder communities, projects addressing it will not be successful. Are they satisfied? (1999). School of Professional and Extended Studies, Master of Public Administration and Policy, MS in Counter-Terrorism and Homeland Security, MA in Economics, Applied Economics Specialization, MS in Human Resource Analytics and Management, http://www.dfid.gov.uk/Documents/publications1/design-method-impact-eval.pdf, Sylvia Burwell Inaugurated as American University’s First Female President, School of Public Affairs Advances to #18 for US News & World Reports’ 2019 Top Public Affairs Schools. The benefits of doing business with company are clearly presented in this web page. What features of the product disappointed you? London: Sage Publications. Some sample questions are: 5. There is a trainer’s competency checklist and trainer attributes competency self-assessment. Contract 68-W-02-048 . How could those services be provided? 86–105). (Use a more or less tendency scale), Are you more or less likely to make a large purchase in the next six months? Before we decide what types of data we need (qualitative or quantitative) we need to know the bigger question of the project: What specific problem does the project or program address? These resources are sample evaluation forms and guides to adapt for your own use. Reasonable and appropriate, or realistic in the given project or program. Washington, DC: The World Bank.Stern, E., Stame, N., Mayne, J., Forss, K., Davies, R., & Befani, B. According to Owen and Rogers (1999), there are three levels of evaluation questions at this stage in project planning: The best questions must be developed with stakeholders in the evaluation, including program staff, sponsors and funders, local and regional decision-makers within and outside the program, and community representatives, when the community in which the evaluation or project will be carried out has already been identified. Focus groups are often used in concert with other product research methods to determine not only how the product could be improved but also how the product should be marketed. (Use a numeric or satisfaction scale), What is your impression of the product? FGD focus group discussions . your program is in its early stages, you may wish to focus on formative evaluation information, which will help you revise the program before undertaking a summative evaluation. Process evaluation questions focus on how a program is working, program performance, and involve extensive monitoring. If I wanted to purchase the product, I know where to click. This article will give you many effective focus group questions that should spark discussion. Focus groups can be a good way to collect qualitative data in monitoring and evaluation, especially when group interaction might produce insight into a topic. Project level (local, activity based, “little p”) – how does, or could, the evaluation effect project or local changes?**. (Use a numeric or favorable scale). About the AuthorAlly Krupar is an Adjunct Instructor at American University’s School of Professional and Extended Studies where she teaches Qualitative Methods in Monitoring and Evaluation. (Use a numeric or the degree of appeal scale). Many projects involve online marketing and gathering feedback on a website design or feature. Market Conditions: Do You Feel Like Splurging? For people to be engaged they first need to feel relaxed and comfortable in their surroundings. This tip sheet is intended for programs funded through the Centers for Disease Control and Prevention’s (CDC) Division for Heart Disease and Stroke Prevention. Community Food Security Coalition ii Overview The Community Food Project Evaluation Toolkit (Toolkit) is a hands-on workbook designed to help community food project (CFP) staff conduct evaluations of their programs to develop the kind of information that will be compelling not only to fellow staff but to funders, participants and Policy level – how does, or could, the evaluation impact relevant policy? Do they have negative effects? What is evaluation Focus groups are structured discussions among small groups of people. Generally, a focus group is comprised of six to twelve people (Morgan, 1998b; Stewart & Welcome and thank you for being here today. If I wanted to see more content, I know where to click. 2. Focus Group Questions to Engage Your Group. For example, ask them to think of three words to describe their favorite hobby. Are objectives met? But, without well-developed, relevant, and accurate evaluation questions, developed with stakeholders connected to the problem, projects can move around a problem without addressing the most appropriate issues. Strong focus group questions are critical to successful market research. Are the headlines/links easy to read and enticing for visitors to click on? (Use an agreement scale). Broadening the Sharing the of Designs and Benefits of Trade Methods for Impact Evaluations: Report of a study commissioned by the Department for International Development. What most contributes to your pessimistic outlook for the economy? Some sample questions are: 4. (Use a numeric or degree of likelihood scale). Here are a few pointers to help you in developing your own questions to motivate participants to open up and provide more honest and complete responses. Some sample questions are: (2012). The key to creating a positive experience for your participants is by asking questions that will awaken both their creative and critical sides. What is the structure or organization of the project/ program? To build off of qualitative interviews and focus groups, we need to think about what sort of questions we ask in program evaluation. Use this product-focused checklist for focus group discussions or interviews. She holds a BA in Anthropology from Case Western Reserve University and an MA from the School of International Service at American University. When thinking about navigating this page, what, if anything, is confusing to you? In addition to this guide, we have focus group proposals, moderators’ guides, presentations and reports, along with other manuals Introductions - Begin the session with a set of questions that will enable focus group members to get to know each other better. M&E monitoring and evaluation . Web Design: Does Everyone Know Where to Click? As stated above, program ... taker for focus group questions and/or activities which we develop with your program in advance. What three words would you use to describe the product? It illustrates how to order questions within a topic of inquiry. A focus group is an information-gathering tool where a selected set of participants, for example, a group of teachers, students, parents or members of the board, gather to discuss a particular area which has been identified as the focus for a school’s evaluation. Decide which questions should be included in an evaluation questionnaire. Questions for student focus groups . What, if anything, do you find frustrating or unappealing about the product? 5. As the project manager and facilitator of a focus group, your job is to fully engage the members to get their honest feedback. If the program has been in place for a while and is thought to be relatively stable, then it may be time for a summative evaluation. 1. Ask them about their favorite hobbies and interests and what they enjoy the most about the activity. The first order of business when considering conducting a process evaluation is to form a working group that includes key program stakeholders. Methodologically, evaluation questions focus on varied assessment types. Programs that are already running should have a theory of change, or how they think their intervention will lead to the stated outcome, objective, goal, or impact they hope to see. Please rate on a scale of 0 to 10, where 0 is extremely dissatisfied and 10 is extremely satisfied the following features: Sample Focus is the web’s premiere FREE community curated royalty-free sample library. Tracy (2013) explains that within a focus group, there is a cascading effect of conversation, as comments link into each other and participants share what can be rich, emic, qualitative, data with each other and the moderator. 3. If you would recommend this product, what would you say to convince your best friend to purchase this product? Are there alternative approaches that could have the same outcomes with less cost?*. Is the problem that the project/ program intends to address alleviated? NFI nonfood item . However, sometimes program assessment and teacher evaluation overlap. The purpose of this gathering is to get your feedback about how we can better serve students enrolled in math courses, such as yourselves, at the THINK TANK. Closing - Remember to thank the participants for their assistance and candor, and then ask them if there is anything that could be done to improve their experience as a member of the focus group. Writing focus group questions in the moderator's guide or focus group questionnaire is your first step. Are activities conducted with the target population? How best should the project/ program deliver those services (or activities)? Evaluation Questions Checklist for Program Evaluation Lori Wingate and Daniela Schroeter Evaluation questions identify what aspects of a program1 will be investigated. Step 1: Form a working group. Appendix B provides a sample timeline for a small focus group study. Learn more about how to file a complaint about a distance program or courses. RRA rapid rural appraisal Thinking of the answer to the previous question, what made you say that? Chapters 2 and 3. • The focus group needs to be appropriate for the type of participants in the group. Program theory assessment, or identifying how the program intends to address the problem. Copyright © 2020 Bright Hub PM. * Reliability Instead of asking “why” you should rephrase the question to ask respondents to provide some specific examples of what they liked or did not like about the item in question. Student Focus Groups for Program Assessment: ... away from teacher evaluation. Try not to have too many Key Evaluation Questions - a maximum of 5-7 main questions will be sufficient. The program evaluation was conducted by UCLA’s Center for Study of Evaluation … How would you rate your outlook for the economy over the next six months? The Various Types of Group Evaluation Forms. (Use a likelihood scale). To learn more about American University’s online Graduate Certificate in Project Monitoring and Evaluation, request more information or call us toll free at 855-725-7614. Answerable, similar to the reasonableness of a question, good evaluation questions must be able to be answered to some degree of certainty. Focus Group Interview Template. EVALUATION TOOL. How easy or difficult was it to navigate the content on the web page? 5. Impact/ outcome evaluation, or how does the program reach its outcomes or impact? I am providing my consent by leaving the opt-in checked. Evaluation questions, similar to research questions in academic research projects, guide the methods and tools used to collect data to understand the problem under investigation. * Speed The Road to Results: Designing and conducting effective development evaluations. KEQs usually need to be developed and agreed on at the beginning of evaluation planning - however sometimes KEQs are already prescribed by an evaluation system or a previously developed evaluation framework. 1. Quality Matters Rubric A6. Dr. Beverly Peters has written recently about qualitative interviewing and conducting and using focus groups. PRA participatory rural appraisal . 2 PRACTICAL GUIDE: FOCUS GROUP DISCUSSIONS CONTENTS The decision on when to conduct FGDs and—in case of a mixed-methods approach— their sequencing in relation to the other methods is influenced by the objective of the effort; the overarching assessment, monitoring or evaluation questions the FGDs seek What most contributes to your optimistic outlook for the economy? Gather the participants’ names. 1. * Ease of Use IR intermediate result . Using Focus Groups in Program Development and Evaluation by Roger A. Rennekamp, Ph.D. and Martha A. Nall, Ed.D. I personally like to have the question on impressions first because with online advertising first impressions matter the most in capturing and retaining a visitor’s attention. (extremely, somewhat, or not familiar at all), Overall how satisfied are you with the product? This question is for testing whether you are a human visitor and to prevent automated spam submissions. Once you learn basic questions… ! As the project manager and facilitator of a focus group, your job is to fully engage the members to get their honest feedback. IEC information, education and communication . How will the program work with the target population to sustain the project/ program? Surveys. What do they think of the services? For example, you could ask the group’s overall impressions of the web page last. How likely are you to purchase this product? Evaluation questions may seem intuitive, and thus be quickly developed to get to the more detailed program planning. OFDA Office of U.S. Foreign Disaster Assistance . According to Rossi, Freeman, and Lipsey (1999), evaluation questions must be: Once we have developed the larger question of the project, M&E practitioners begin to consider what data they need to answer the question, using a Theory of Change and a LogFrame. These consultations may be informal conversations, reviewing grant requirements and terms of reference (documentation review), or semi-structured individual and/or group interviewing. How does the target population interact with the program? Try to select scales that have more than five variations to give respondents more flexibility. Do you have any suggestions on how we could improve the product to make it more appealing? Sample focus group materials can be found in appendix B (PDF 101 KB). Post-Primary Focus Groups Focus Groups. Find the perfect sound in seconds. In Program Evaluation (pp. surveys and focus group questions for student participants, parents, teachers (survey only), program staff, and program partners (survey only). They focus on the merit, worth, or significance2 of a program or particular aspects of a program. Use questions that have relevant importance to a particular situation or issues. Needs assessment evaluation questions may include focus on the significance of the problem, drawing on literature, previous projects, and baseline data with potential stakeholders. What similar products have you used during the last 6 months? We give you the insight you need to write all types of focus group questions with the help of proven examples and expert advice. Yes, I do want to communicate with American University via SMS. Consumer sentiment and consumer confidence are the forerunners to the future success of many businesses. Similarly, formative evaluation questions look at whether program activities occur according to plan or the project is achieving its goals while it is underway. Specificity - Avoid vague open-ended questions that simply ask “why.” The word “why” has a negative connotation because it makes the respondent think their first answer was not good enough. Assessment Tool kit: Focus Group Template Focus Group Script . not These questions identify and support the problem that the intervention hopes to address. The following list is not exhaustive nor mutually exclusive but provides a framework for thinking about what kind of evaluation you are doing and what (generic) evaluation questions you may need to ask. Assessment of efficiency, or how cost-effective is the program. The main purpose of an evaluation plan is to focus on the evaluation to be undertaken while keeping a track of the desired outcomes and results. Retrieved from http://www.dfid.gov.uk/Documents/publications1/design-method-impact-eval.pdf Stufflebeam, D. L., Madaus, G. F., & Kellaghan, T. (2002). Are outcomes, objectives, and goals achieved? e. Is the target population affected by the project/ program equitably or according to the evaluation plan? Product Reviews: Is Everyone Completely Satisfied? It should always include less than ten questions. Sample questions are: Through asking questions, M&E practitioners can identify what their project specifically should address. Discovering these issues can help determine which of a number of options is the preferred way forward, or to determine what are the concerns that would prevent a proposal going ahead. Similarly, formative evaluation questions look at whether program activities occur according to plan or the project is achieving its goals while it is underway. Designing and Conducting Focus Group Interviews Richard A. Krueger Professor and Evaluation Leader University of Minnesota 1954 Buford Ave. … Please fill out both sides of this form: OBJECTIVES This program met the stated objectives of: 1. 4400 Massachusetts Avenue, NWWashington, DC 20016. Is the cost of the services or activities reasonable in relation to the benefits? Here are some sample questions to help a project manager evaluate a new website design or concept. Format - When choosing focus group questions, be sure to include a mix of multiple choice questions to provide structure and direction and open-ended questions to allow for further exploration and discovery. Describe the composition of the group and ensure that (1) the goal of the focus group was defined, (2) all information will remain confidential and (3) respondents introduce themselves to each other before proceeding with the discussion. For example, this scale has 7 levels to measure the level of satisfaction: completely satisfied, very satisfied, somewhat satisfied, neither satisfied nor unsatisfied, somewhat unsatisfied, very unsatisfied, and completely unsatisfied. Focus group question examples. There are many different types of Group Evaluation Forms that allow people to properly asses a group and determine if they’re doing a good job together or if they’re better off splitting up. (Use an agreement scale). London: SAGE Publications Ltd. Additional ResourcesMorra Imas, L. G., & Rist, R. C. (2009). Some sample questions are: 2. Topics to be covered are drawn from the following list. Evaluation: A Systematic Approach, 6th Edition. What resources does the project/ program need? Evaluation models: Viewpoints on educational and human services evaluation. If so, how? Process evaluation, or how the program addresses the problem, what it does, what the program services are and how the program operates. These questions can help assess the general business climate as part of the project’s risk analysis. If I wanted to contact the company, I know where to click. Can you describe your experiences in using these similar products? (Use a numeric or degree of likelihood scale), How likely would you be to recommend this product to a family member or friend? The order of these questions can be shifted around. What specific features of the product impressed you? * Specific Features (as named by you). Where it is necessary and/or helpful to hold a focus group to ascertain the views of a cross-section of students from across the subject, the following questions can be used: a. She is also a Doctoral Candidate in Adult Education and Comparative International Education at Pennsylvania State University and a Visiting Researcher with RET, an international organization providing secondary and post-secondary education to displaced peoples worldwide. Student Focus Group Questions A4. Is the target population adequately reached by and involved in activities? Author’s experience as a member of several focus groups involved with evaluating new food and investment products. Focus group questions should guide the group’s discussion. One of this group’s first tasks will be to determine whether to bring in an outside evaluator or have staff conduct the evaluation.17, 18! For people to be engaged they first need to feel relaxed and comfortable in their surroundings. Work Assignment B-16 . Needs assessment, or identifying the surrounding social conditions and need for a program. When respondents are relaxed, they will open up and talk more. The curriculum and intended learning outcomes . How much do you agree or disagree with the following statement. 6. ICT4D information, communication and technology for development . Evaluation involves a systematic collection and analysis of data to critically examine a program for better effectiveness. EVALUATION OF THE EPA INTERN PROGRAM . When using a focus group in marketing research, you must ask the right questions for accurate results. 1. Thinking of your previous response, what are some specific examples that helped you formed your opinion? Hints for Developing Your Own Sample of Focus Group Questions, consumer confidence are the forerunners to the future success of many businesses, Writing a Test Plan: Test Strategy, Schedule, and Deliverables, Writing a Test Plan: Define Test Criteria, Writing a Test Plan: Plan Test Resources, Writing a Test Plan: Product Analysis and Test Objectives, Innovate to Increase Personal Effectiveness, Project Management Certification & Careers, Project Management Software Reviews, Tips, & Tutorials, How familiar are you with the product?