Module 1:
My Thoughts on: Formative, Summative and Anecdotal Evaluations
Formal and summative program evaluations are, in my opinion, commonly used in many different service sectors. In the context of educational institutions, I propose that anecdotal evaluations should be used in conjunction with the traditional formal and summative means, in order to help create a more holistic outcome report. For this discussion post, I am using the student's educational journey through the secondary school "program" in order to illustrate the importance of these three forms of evaluation.
Boulmetis and Dutwin define formal evaluation as "an on-going process that allows for feedback to be implemented during a program's cycle" (2005) (The Pell Institute for the Study of Opportunity in Higher Education, the Institute for Higher Education Policy, and Pathways to College Network, 2019) . When looking at the evaluation of a student's learning, formative evaluation is both helpful and necessary to gauge their level of understanding, skill attainment and overall success. At the same time, formal evaluations occur with the goal of collecting feedback in order to make changes to the implementation of the program should this be necessary. These changes can be critical in moving towards summative evaluations, which then look at the overall success of the program. If these changes are implemented early enough, then the chances of the program's and student's success could increase. Hence, why several formative evaluations throughout a unit, subject or secondary school cycle, for example, are recommended. As a result, this method is similar to a needs assessment strategy whereby "determining and prioritizing the needs of a community" are at the forefront of the program evaluation (Chen, 2005).
Anecdotal evaluation is the thread that joins formal and summative evaluations in that it has the potential to yield higher success rates in students. A "detailed [record of] specific observations of individual student behavior, skills and attitudes" is kept with the goal of tweaking, accommodating, and improving methodological approaches to student learning (Assessment in Mathematics, 2008). When teachers pay particular attention to students' needs, and as a result actively choose to alter their approach towards them, the chances of a student's secondary school program success, increases. Anecdotal evaluation yields itself very well to ongoing observations with the potential for betterment in both students and educators.
Summative evaluations "occur at the end of a program cycle and provide an overall description of program effectiveness" (The Pell Institute for the Study of Opportunity in Higher Education, the Institute for Higher Education Policy, and Pathways to College Network, 2019). These evaluations occur at the end of an extensive thematic unit or subject, or at the end of an academic school year. Again, summative evaluations are important and necessary as they provide quantitative (exam marks) and qualitative (portfolios) evidence of a student's success. These assessments can then be used to determine whether program objectives have been met, where areas of weaknesses lie, and indicate the level of impact the program had on the stakeholders - the students. Therefore, summative assessment falls in line with outcome, impact, and goal-based evaluation strategies, in that a student's success at the secondary level is whether a high school diploma is attained (Chen, 2005). What better way to highlight whether the long-term goals of the program have been met than to reach graduation?
In summary, formative, anecdotal and summative evaluations can be used in an intertwined manner that could lead to better outcome and impact results in the educational sector. Furthermore, since education is largely a merit program, a holistic approach to its evaluation is required. It is "imperative to include contextual or transformation information when assessing" the overall impact of the program (Chen, 2005), and transformation information is at the forefront of anecdotal and formative evaluation. Through consistent and prompt anecdotal observations and concurrent changes to program implementation by teachers, success on formal assessments by the students may improve and in the long run, summative evaluations may be more profoundly impactful.
References:
Assessment in Mathematics. (2008, October 1). Retrieved from Mathematics: http://www.learnalberta.ca/content/mewa/html/assessment/anecdotalnotes.html
Huey-Tsyh, C. (2005). Practical program evaluation: Assessing and improving planning, implementation, and effectiveness (pp. ix-13; 45-68). Thousand Oaks, CA: Sage Publications.
The Pell Institute for the Study of Opportunity in Higher Education, the Institute for Higher Education Policy, and Pathways to College Network. (2019). Retrieved from The Pell Institute and Pathways to College Netowrk - Evaluation ToolKit: http://toolkit.pellinstitute.org/evaluation-101/evaluation-approaches-types/
Types of Evaluation Video: http://www.youtube.com/watch?v=IgwvRiGm424
AEA 365 Response to Jack Mills on Evaluation Theory
Mills’ stressed the three essential types of theory in evaluation: evaluation theory, program theory and social science theory, which Donalson & Lipsey put forth in their 2006 article (Mills, 2010). Separately, as outlined in detail in the 2011 article by Donaldson & Lipsey, they each bring benefits and limitations to an entire program’s evaluation.
In Mills’ example of an evaluation of a “training program designed to prepare ethnically diverse undergraduates for advanced careers in science” (Mills, 2010), the results were presented in a divided light on purpose to show the importance of the relationship between the three theories in evaluation. If the evaluator only emphasized the importance of mentoring and lab experience (from the results of a program theory perspective), as a strategy to encourage graduate level studies, then one would be concentrated on making changes to the program only. In doing so, one would be dismissing the fact that the ethnically diverse body of students in this program bring different background experiences to the science program. For example, some cultures do not encourage women to pursue science, whereas other cultures expect their youth to enter a science related career. Some cultures teach independence as a virtue and others focus on collaboration. Therefore, it would be more beneficial to look at the reasons “why” students are entering the program and what personal qualities act as the reason behind their further pursuit of the sciences; self-efficacy, for example. This brings the social science theory into the mix.
Therefore, in my opinion, applying the newly coined “Program Theory-Driven Evaluation Science” practice of evaluation by Steward I. Donaldson & Mark W. Lipsey (2011), would be the most holistic course of action in undertaking the program evaluation in Mills’ example. This being said, our readings repeat often enough that evaluations are never “one-size-fits-all”.
References:
Mills, J. “Jack Mills on Evaluation Theory” (retrieved on 2019 July 10) [blog] from https://aea365.org/blog/cap-tig-week-jack-mills-on-evaluation-theory/
Donaldson, S. I., & Lipsey, M. W. (2011). Roles for theory in contemporary evaluation practice: Developing practical knowledge. In I. F. Shaw, J. C. Greene, & M. M. Mark (Eds.), The SAGE Handbook of Evaluation (pp. 57-75). Thousand Oaks, CA: SAGE Publication.
Module 2:
Case Study: Logic Model
AEA365 Article and Response
Click Here
Erika Stanischewski’s Response Entitled: Lifting the Fog off of Program Evaluation’s Vocabulary
Sheila B. Robinson acknowledges the ongoing debate that evaluators are having over terminology in program evaluation in her blog post entitled Sheila B. Robinson on Theory of Change vs Logic Model – A LinkedIn Discussion (2013). “Terminology has always been a sticky point for evaluators as those from different sectors (i.e. health, education, non-profits, government, etc.) have developed their own preferences and in many cases, definitions of terms” (Robinson, 2013). As a novice evaluator, terminology is one of the most difficult aspects about program evaluation that I must overcome. The goal of this post, inspired by Sheila Robinson’s blog post, is to attempt to lift the fog off the relationship between program theory, theory of action, theory of change and the logic model.
Since reading Robinson’s post, I have requested to join AEA’s LinkedIn group so that I may locate the discussion that was had. However, in the meantime, I have pieced together what I believe is the relationship between these terminologies using what I have learned so far, and what Robinson summarized in regards to the LinkedIn discussion (the debate between Change Model and Logic Model).
At the very top sits Program Theory. A program theory “explains how and why a program is supposed to work” (Wilder Research, 2009) through its action model. It’s action model outlines the resources and activities believed to be required to set the program in motion and consequently prompt the emergence of change (Chen, 2011; Donaldson, S. I., & Lipsey, M.W. 2006; Robinson, 2013). The change model focuses deeply on the “why” that governs the effectiveness of the proposed activities, and associated outputs and outcomes, or lack thereof. It is an “explicit or implicit theory of how change occurs” (Robinson, 2013). Finally, a logic model is a concrete, visually organized tool that “helps identify inputs, activities, outcomes, [and outputs]” (Robinson, 2013) and, as a result, a logic model “illustrates a program theory” (Wilder Research, 2009). A logic model relies on a program theory’s change model. As such, frequent revisiting of the change model and consequently modifying the logic model seems sensible if program success is warranted.
Attached to this post is Figure 1.0, which graphically explains the relationship between the terminologies explored in this post, as per my understanding. You can find this attachment on the original Blog post.
I have also attached Figure 1.1, Evaluation Terminology by Kylie Hutchinson, which visually shows the relationship between the numerous players in the evaluation process through a concept map. This resource was recommended as a rad resource by Robinson (2013).
Let us begin our discussion using Figure 1.0 and 1.1 to discuss the interconnectedness, and arguably, the interchanging nature the terms in evaluation can be.
Resources:
Chen, H. (2005). Practical program evaluation: Assessing and improving planning, implementation, and effectiveness (pp. 15-43). Thousand Oaks, CA: Sage Publications.
Donaldson, S. I., & Lipsey, M.W. (2006). Roles for theory in contemporary evaluation practice: Developing practical knowledge. In I. Shaw, J. Greene, & M. Mark (Eds.), The Handbook of Evaluation: Policies, Programs, and Practices (pp. 56-75). London: Sage. https://goo.gl/vxULFs (https://goo.gl/vxULFs).
Robinson, Sheila (2013). “Sheila B. Robinson on Theory of Change vs Logic Model – A LinkedIn Discussion” (retrieved on 2019 July 18) [Blog] from https://aea365.org/blog/sheila-b-robinson-on-theory-of-change-vs-logic-model-a-linkedin-discussion/
Wilder Research. (2009). Program theory and logic models [PDF file]. Retrieved from http://www.evaluatod.org/assets/resources/evaluation-guides/logicmodel-8-09.pdf
Module 3:
Dilemmas in Evaluation Use
In today’s society, the role of the evaluator is becoming more charged with responsibility and expectations to work closely with the needs of the stakeholders. So much so, that the evaluators are sculpting the evaluation to ensure, without question, that the findings revealed are usable. “Evaluators [are] taking great pains to identify the type of use sought by program stakeholders and to custom design evaluations that would best promote such uses” (Shulha, L., & Cousins, B., 1997, p. 197-198). This intimate working relationship with the stakeholders may prove problematic in the sense that evaluators may struggle with their own personal ideals regarding the program under study, to the point where it has become a genuine concern that evaluators are unable to “guard [themselves] against undesirable influences and biases”(1997, p. 200). Although this may be true, my position is that the complex relationships that develop through the “stakeholder-service orientation”, as Shadish and Epstein (1987) coined (Shulha, L., & Cousins, B., 1997, p. 220), may be hiding the underlying truth that despite all efforts to justify, embellish or even mask the uses of evaluation, that in the end, the decision of use is simply politically motivated. To better comprehend this position, I must clarify some of the more predominant theories.
Dr. Michael Patton’s position on evaluations is that they should be designed with particular attention to the intended use of the evaluations’ findings to facilitate proactive decision making on the part of the program being evaluated (Patton, M., 2013; Saunders, M., 2012; Shulha, L., & Cousins, B., 1997); thus coining the term Utilization-Focused Evaluation (Patton, M., 2013; Weiss, C., 1998). He becomes more stringent in his position when he states that an evaluation should not even commence “unless there are users who will use the evaluation results produced” (Patton, M., 2013). He feels that the deeply evolving relationship between the two parties involved, the stakeholders and the evaluators, will ultimately “generate findings that would yield the type of information needed by the intended user” (Shulha, L., & Cousins, B., 1997, p. 197). He is confident that such a close working relationship will guarantee the efforts of all parties, and that they will not go unwarranted. He puts a considerable amount of responsibility on the part of the evaluator. He maintains that the evaluator is to blame should the findings not prompt instrumental and/or conceptual use on the part of the stakeholders advocating for the evaluation, therefore, constituting failure on the part of the evaluator.
Weiss’ position on Patton’s theory is “that evaluators should not be held accountable for failures to use their results” (1998, p. 22). Weiss goes on to argue that despite completely internalizing the findings of an evaluation, the very nature of the human, their respective circumstances within the program being evaluated, and/or the organizational obstacles themselves, “make the organization unable to respond to the need for change revealed by evaluation” (1998, p. 22). However, it is justifiably argued that given the close collaborative relationship between the stakeholders and the evaluator and his/her ability to pose the appropriate questions and make the necessary conclusions, it must be the fault of the evaluator should these results be dismissed or deflated. "Working with program staff enlists their attention, knowledge, and commitment to using the results" as stated by Weiss in summary of Patton's position (1998, p. 22). In another view, what if the reason behind the reluctance for action stems from symbolic use in that the purpose of the evaluation was purely symbolic in nature (Alkin, M. C. & Taut, S. M., 2003)? Or, as Alkin and Taut outline, what if the use of the evaluation was to legitimize a “previously made decision” (2003, p.7)? These possibilities are encroaching more and more onto the political side of evaluations.
Should there be a situation where a matrix of evaluations be conducted that yield similar results with no mandate for any change on the part of the stakeholders for the benefit of the program, then it begs the question of why? If evaluations are conducted and follow the standards outlined by Patton (utility, feasibility, propriety and accuracy) (2013), the evaluator remains unbiased despite being consciously aware of his/her position as being influential, knowledgeable and reliable, and the use of findings are still dismissed or arguably fall into the category of symbolic, then I maintain that the driving reason behind usability is that of political practicality, political motivation and bureaucratic red tape. Theories like Patton’s and Weiss’, although sound in their own regard, are attempts to put the silver lining on this possible reality.
**As a science teacher, I can’t help thinking and referring to the countless varieties of evaluations conducted about the status of our failing environment, the respective environmental programs that are designed to help with this reality and the stakeholders in the positions to use the findings revealed by the evaluations**
References:
Alkin, M. C., & Taut, S. (2003). Unbundling evaluation use. Studies in Educational Evaluation, 29, 1-12.
Patton, M.Q. (2013, June 13). Utilization-focused evaluation for equity-focused and gender-responsive evaluations [YouTube]. Retrieved from: www.youtube.com/watch?v=jQP1FGhxloY
Saunders, M. (2012). The use and usability of evaluation outputs: A social practical approach. Evaluation, 18(4), 421-436.
Shulha, L., & Cousins, B. (1997). Evaluation use: Theory, research and practice since 1986. Evaluation Practice, 18, 195-208.
Weiss, C. H. (1998). Have we learned anything new about the use of evaluation? American Journal of Evaluation, 19, 21-23.
AEA365 Connection with the Evaluation Community
Click here to access the AEA365 article that I chose to comment on , as well as the author's response.