The phrase “When the chef tastes the soup”

Part A
Part A includes fill-in-the-blank and short-answer items. There is no significance to the length of the lines used to denote blanks. However, for multi-part questions the number of lines denotes the number of separate responses needed. Also, there is no significance to the amount of space provided for short-answer questions. Strive to make your responses specific, succinct, etc.

Part B
Part B items require more discussion and elaboration. There is no minimum or maximum number of lines or paragraphs required. The length is determined by what and how much you want to discuss. While there is no specific requirement to cite sources for your answers, you are free to do so if you think it supports your case or argument. Readability and grammar are important. Use our standard format: double-spacing, 11 point font or larger, 1” margins, Chicago style–when appropriate.

Part A

  1. The phrase “When the chef tastes the soup” refers to Formative evaluation. The phrase “When the customer tastes the soup” refers to Summative evaluation. (2 points)
  2. In Kirkpatrick’s model, the two levels most associated with assessing whether a program is “Doing Things Right” as opposed to “Doing the Right Things” are: (2 points)

a. Level 1: Reaction
b. Level 2: Learning

  1. Internal Evaluation is to Process Evaluation as External Evaluation is to Impact Evaluation. (2 points)
  2. Four ways that program evaluation differs from research are: (4 points)

a. Program evaluation provides information for decision making while research develops laws and theories

b. Program evaluation is stakeholders and evaluators focused while research is researcher focused

c. Evaluation is based on accuracy, feasibility and accountability while research is based on internal and external validity

d. In terms of the preparation of those working in the area, evaluators use many methodological tools and interpersonal skills while researchers use fewer methodological tools and approaches

  1. “Assessing implementation and evaluating against plan” pertains primarily to what stage or step in the following two models? (2 points)

a. Stage or step 3(process evaluation) in Stufflebeam’s CIPP model.
b. Stage or step 4 Provus’ Discrepancy model.

  1. In Stufflebeam’s CIPP model, Product evaluation is most similar to level 4: Results and Level 3: Behavior in Kirkpatrick’s training evaluation model. (2 points)
  2. Needs assessment would most likely be conducted in the Context evaluation phase of Stufflebeam’s CIPP model. (2 points)
  3. Internal Evaluation would most likely refer to Reaction and Learning in Kirkpatrick’s model. (3 points)
  4. Some experts say that Merit (M) and Worth (W) pertain to different aspects of program quality or effectiveness. Place an “M” next to the evaluation model component that best measures Merit, and a “W” next to the evaluation model component that best measures Worth. (4 points)

Reaction___ Context___
Learning___ Input—–
Behavior___ Process—–
Results___ Product___

  1. An evaluability assessment was conducted prior to a program evaluation. Identify four conditions according to Fitzpatrick that might render it unwise for the evaluation to proceed:
    ______________; ________________;

______________; __________________. (4 points)

  1. Chapter 2 of the Program Manager’s Guide to evaluation describes several questions one could ask in evaluating Program Implementation objectives. List three of them. (3 points):

a. What training and services do you plan to implement?

b. What are the characteristics of the participant population

c. What is the number of people you are trying to reach?

  1. Two benefits of Program Theory are: (2 points)

a. Program theory aids in attributing outcomes to activities
b. It also facilitate both program planning, development and evaluation

  1. According to Carol Weiss, Implementation Theory pertains to Program activities, while Program Theory refers to mechanisms of change. (2 points)
  2. Three sources for determining the theory of a program are: Interview staff; Review the literature for evaluations of similar programs; and Review basic research. (3 points)
  3. Provide two examples of faulty program theory. (3 points)

a. __________________

b. __________________

  1. For 3 of the following 4 items, describe or define the item and provide a brief example (i.e., 2 or 3 sentences)

a. The principle that program evaluation criteria should be under staff control.
(3 points)

b. The principle that program evaluation criteria should match the purposes of the program. (3 points)

c. Triangulation: (3 points)

d. Non-Response bias: (3 points)

  1. Define and give an example (i.e., two or three sentences) of the following program evaluation terms:

a. Single Source Bias: (2 points)

b. Method Bias: (2 points)

c. Reactivity Bias: (2 points)

  1. Describe three ways that quantitative research differs from qualitative research in program evaluation. Elaborate with a brief example of each. (4 points)

a. ___________________________;

b. ___________________________;

c. ___________________________.

  1. Provide an example of the negative consequence(s) of failing to adequately examine the context of the need when doing a needs assessment. (3 points)
  2. Provide an example of how inaccurate assumptions about the underlying cause(s) of a need could lead to flawed program design. (3 poi
  3. Matching: Follow the instructions below in Column B to match the items in Column A with the most appropriate item from Column C. Items from Column C can only be used once, and only one item from Column C can be matched with a given item in Column A. Each correct match is worth 2 points.

Column A Column B Column C
Evaluation goal, purpose, key quote, key term or phrase, etc. Instructions: For each item in Column A, select an item from Column C that best matches the item in Column A. Indicate your selection by typing the item letter from Column C (i.e., a., b., c., etc.) in the appropriate cell in Column B. Evaluation Model, Theory, Principle, Method, Type of Information, etc.

  1. The goal is to “ensure key evaluative information reaches decision makers and program staff at the right time.” a. Utilization Focused Evaluation (Patton)
  2. “Intended use for intended users!” b. Qualitative evaluation
  3. “Goals serve as blinders for the evaluator!” c. Goal-Free Evaluation
  4. Allows evaluator to view the program from the inside d. Appreciative Inquiry
  5. Helps agency staff to conduct future evaluations e. Success Case Method
  6. Identifies mechanisms of change f. Program Impact/Logic Model
  7. Bottom line focus g. Mixed-methods
    h. Evaluation indicators
    i. Black Box evaluation
    j. Capacity-Building/Empowerment evaluation
    k. Discrepancy Evaluation

Part B

  1. It has been said that when conducting a needs analysis, it is better to ask people about the existence and nature of a need or problem, than to ask them about the adequacy of various possible solutions. Discuss the merits of this point and support your answer with an example. (5 points)
  2. Discuss the advantages of “mixed-methods” design in program evaluation. Use examples from the Riccio or other assigned case studies to support your answer. (5 pts.)
  3. Consider Two Scenarios:

a) In a program evaluation of a community bike sharing program an evaluator found that certain bike pick-up/drop-off stations were significantly under-utilized compared to other stations. Upon further review, it was determined that almost all of the under-utilized bike stations were located in neighborhoods where the average age of the adult population was over age 60.

b) In addition, a subsequent focus group with citizens from the under-utilized bike sharing neighborhoods revealed that Metro Access, a shuttle service for transporting disabled persons to nearby bus and metro stations was used frequently in the neighborhood. In addition, a free trolley that runs up and down the main east-west artery of the city was instituted six months ago.

For each scenario, discuss possible shortcomings or failures of the needs assessment process. (6 points—i.e. 3 points for each scenario)

  1. Using the method shown by Weiss for describing a planned causal sequence (See Carol Weiss Powerpoint titled “Carol Weiss on Program Theory”, slide 4), develop a planned causal sequence for how your anchor program would achieve its ultimate goal(s). Before describing the sequence, provide a 2 or 3 sentence reminder of what your anchor program is. (5 pts.)