PME 802 - Program Inquiry and Evaluation
Click Here
"The report, published by the Institut du Québec, found that between 2008 and 2015, the number of Quebecers graduating from public high schools in the required five-year time frame decreased — from 65 to 64 per cent. "(CBC News, 2018)
Module 1 B: Description of the Program's Context & Purpose for Evaluation
Name of Social Program
The Sir Wilfrid Laurier School Board's Academic Consolidation Support Plan, simply referred to as the AC program. The focus of this study will look at the AC program at Rosemere High School; one of the 4 high schools within the board that offer the program.
Program's Focus and Objectives
The Academic Consolidation program recognizes the struggles facing some students who have not yet mastered the basic skills required to enter the regular stream at the Secondary level. This program provides these students with the opportunity to receive extra, individualized, and customized support from a group of trained professionals. The focus of this support may change depending on the needs of the cohort, however, literacy and numeracy skills support are at the forefront of the AC program. Students will receive an extra year of structured support before entering the regular stream the following year. The AC program allows the student to complete Secondary Cycle 1 over a three year period as opposed to the regular two allotted years.
"Your child will receive individual attention in order to identify his or her needs, and to provide her or him with appropriate and effective instruction that addresses those needs. The Secondary Cycle One program is spread out over a three year period"(Sir Wilfrid Laurier School Board, 2018).
Community Demographics:
This program serves students who are behind academically on a global level including but not limited to having low DRA scores and/or dealing with a learning disability. Students are recommended to the AC program by their grade six teachers in collaboration with the advice from administration and the receiving teachers at the secondary level.
Goals of the Program:
-
To allow extra time for skill building and acquisition of an academic foundation.
-
To provide extra support in numeracy and literacy.
-
To push for secondary school graduation within a time span of 6 years.
Year of Implementation
2015-2016 welcomed the first cohort for the AC program. 2019-2020 will mark the fifth year of implementation. Students having successfully passed through the AC program at its debut, should find themselves in the regular secondary 4 stream for the upcoming school year.
Implementation
A program's defined...
can become it's...
First Year: The first year of the AC program focuses on strengthening literacy and numeracy skills as well as solidifying general knowledge from the elementary sector. Students will remain in low ratio classes for their main subjects: math, English, French, history/geography and science. They are fully integrated in their other options.
Second Year: Students will now enter the regular Secondary 1 stream except for one class, either math or English. This contained class will be chosen based on the needs of the particular cohort. Students will continue to receive support as her their IEP and the resources put forth by the school with the inclusion of an adult mentor.
Third Year: Students will continue into the regular Secondary 2 stream and will be fully integrated. Support based on their IEP will be legally maintained. At the end of this year, a decision will be made as to whether the student will continue along the path towards potential graduation or will enter the Work-Oriented Training Path which leads to an attestation of education with work experience.
Staff
The AC team is comprised of a hand-picked team. According to the Evaluator's Handbook by Herman et. al. (2011), it is imperative that "a description of, or an organizational chart depicting the administrative, management, and staff roles played be various people in the program" be presented.
Available Resources
-
Quebec Education Program for the Primary and Secondary Sector
-
AC Curriculum Guidelines for the main subjects (PDF documents)
-
Elementary School Curriculum Maps with suggested units (PDF documents)
-
Academic Consolidation Support Plan Flyer
-
In-house Group Chats using Office 365
-
Personally and collaboratively developed AC subject specific material
-
Laptops for students (although this was removed this year for the 2018-2019 cohort)
Click to access resources
Purpose for the Evaluation
The purpose of this evaluation is to closely examine what is working and/or not working within the context of the AC program's teaching and support staff, allocated resources/lack thereof and the realistic needs of the students with the hope of suggesting improvement strategies and resources for better implementation so that AC students have the potential to reach graduation within the 6 year allotted timeframe of the program.
Since 2017, I have been the AC Science teacher. I have put a significant amount of time and effort into the realization of the AC program. I have created resources both pertaining to my subject and to improving the literacy and numeracy needs of the AC students within my subject area. I have worked closely with my teaching colleagues, support staff as well as directly with the students and their parents. I am concerned about their graduation potential. I have experience first-hand all the collaborative activities, discussions, teaching efforts, student realities and a multitude of other factors that have led me to look at what is being done well and what is in need of improvement. Ultimately, I want to invest more into this program since the goal is to promote graduation. As Huey-Tsyh outlines in Fundamentals for Practicing Program Evaluation (2005), "stakeholders find evaluations useful if they both offer conclusions on how well programs have worked and provide information that assists the stakeholders in figuring out what must be done next to securely attain - or even surpass - program goals". As an invested stakeholder, the need to make AC a long lasting success is important. And, simply put, I care about the students.
Type of Program Evaluation
In light of my purpose for the evaluation, and the potential the AC program has, I am going to design a formative evaluation. "Formative evaluation is concerned with designing and using evaluation to improve a program"(Huey-Tsyh, 2011). Changes can be made to the program being evaluated based on the results of formative evaluations (Herman, et. al, 2011). Finally, I have the advantage of having "a close working relationship with the staff [of the AC program], which can help with data collection (Herman, et. al., 2011).
Evaluation Questions
"Evaluation is a tool that can both measure and contribute to the success of your program" (What is Evaluation?, 2012)
In the American Evaluation Association’s Better Evaluation Series, Patricia Rogers discusses what it is to be evaluated and stresses four categories of key evaluation questions: descriptive, Casual, synthesis and action. I will be using these categories to classify my questions for this program evaluation design. I will also use Herman, et. al.'s Evaluator's Handbook to help create formative evaluation questions within these categories (2011).
References
Module 2 B: Construction of a Program Theory
Program Theory Introduction and Link to Course Readings
In Practical program evaluation: Assessing an improving planning, implementation, and effectiveness (2005), Huey-Tsyh defines a program theory "as a configuration of the prescriptive and descriptive assumptions held by stakeholders and thus underlying the programs stakeholders create". In other words, a program theory is the relationship between the Action Model (Prescriptive) and the Change Model (Descriptive). How will the actions of a program cause changes for the participants of the program? Therefore, a program theory "explains how and why a program is supposed to work" (Wilder Research, 2009) and this can be visually displayed through a Program Theory's Logic Model since a logic model "illustrates a program theory" (2009) and this in turn, outlines the program rationale. A program's Rational serves the purpose of displaying the "determinants, interventions, and outcomes agreed upon by stakeholders". In a program rationale, "the goals to be pursued are named and reasons are given for the selection of these goals" in addition to the planned interventions to attain those goals with keeping the target group in mind (Huey-Tsyh, 2005).
I will outline the Program Theory for the Academic Consolidation Program using Wilder's Research (2009). This will be followed by a logic model to represent the theory of change incorporating the AC Program's 3 goals.
Program Theory of the Academic Consolidation Program
Overarching Program Theory Statement
Based on Wilder Research's Model of Program Theories (2009)
The Sir Wilfrid Laurier School Board's Academic Consolidation program aims to provide extra support in numeracy and literacy skills for students experiencing academic delays by providing them with additional time built into their secondary school journey and through the efforts of trained and specialized teaching and support staff who will concentrate on building an academic foundation for each student in the AC program based on their individual needs. Successful implementation of the AC program will result in secondary school graduation within a timeframe of 6 years.
-
If AC students receive structured and personalized support in numeracy and literacy within a time frame of 3 years (the beginning of the AC program) from trained and specialized teaching staff, then students will progressively acquire the necessary skills to build their academic foundation necessary to support their quest for a secondary school leaving diploma.
-
According to Kelsey Clark's article entitled, Helping students with learning disabilities succeed - these steps are key to helping educators help students succeed, "when a student with learning disabilities enters the classroom, this becomes even more important as a student’s disabilities will influence the types of support that they will need in order to be successful in their academic studies" (2019).
-
Evidence of successful intervention will result in the number students moving through the secondary levels with increased autonomy and skill. This will be accompanied by a general improvement in self-efficacy.
-
-
If this extra support is provided then it will prove essential in students'' acquisition of knowledge as the individual needs and possible learning obstacles experienced by each student will have been identified and strategies and methodologies to assist in dealing with/overcoming these obstacles will have been taught.
-
This support will work because "when a student is diagnosed with a learning disability, it is important for an individualized education plan (IEP) to be developed that will identify the areas in which a student struggles so that the proper supports can be put in place to encourage their success" (Clark, 2019).
-
Evidence of successful intervention will result in the number of students moving through the secondary levels with increased autonomy and skill. This will be accompanied by a general improvement in self-efficacy.
-
-
If this extra support and additional time is provided, then success of this intervention/program will be demonstrated by the number of Academic Consolidation students making it to graduation within 6 years, as well as an overall increase in personal self-efficacy and overall academic and personal growth.
-
The US Department of Education produced a report in 2018 indicating that "research suggests that academic support classes may have a positive effect on student outcomes such as average number of credits earned, high school graduation, and college enrollment ", and "Research has also shown that academic support and enrichment as a whole, which may include both academic support classes and academic tutoring, have a positive effect on student outcomes".
-
Evidence of success will result in a 75% graduation rate for Academic Consolidation students.
-
Logic Model:
The logic model I chose to use for each of the Academic Consolidation's three goals is based on Wilder Research Appendix C's example. This logic model includes short-term, intermediate and long-term outcomes. The intermediate outcomes are of utmost importance in the Academic Consolidation's Program Theory since the interventions put forth by the teaching staff will address "behavioral changes" in the students, and it is through behavioral changes as well as skill building (which are often connected to behaviors), that possible success can occur (2009). Below is some research to guide this process.
Goal ONE: To allow extra time for skill building and acquisition of an academic foundation
Goal TWO: To provide extra support in numeracy and literacy.
Goal THREE: To push for secondary school graduation within a time span of 6 years
Unintended Outcomes/Changes:
In addition to the measurable outcomes indicated in each of the 3 logic models above, there are possible unintended outcomes that may occur including:
-
AC students will begin to rely on each other for support as they enter the normal stream in the following years of this 6 year program. This support may look like:
-
Friendships
-
Working together on class assignments
-
Reliance on each other for reminding to use assistive tech
-
-
The AC teaching staff may become too weighed down by the intense responsibility of the AC program and this may result in change of teaching staff.
Module 2 B: Identify, Describe and Rationalization of Evaluation Approach
Program Planning Stage - Initial and Mature Implementation Stage = Multiple Entry Evaluation
It is important to stress that the Academic Consolidation Program will be entering its 5th year of implementation in 2019-2020. The students of the first cohort have not yet reached graduation since this program is looking at a minimum of six years before students are at graduation potential. Therefore, the AC program is classified in the Initial Implementation Stage, as outlined by Huey-Tsyh (2005). This phase deals with "providing timely information on implementation problems and the sources for assisting stakeholders to fix the problem and stabilize the program"(2005). However, since the AC program has now welcomed four cohorts, with the fifth beginning in Fall 2019, there is a lot of information that can be improved upon therefore further classifying the AC program in early Formative Phase (AKA the Mature Implementation Stage) (and arguably with certain aspects still in the Developmental Phase/Initial Implementation Stage) since, as mentioned above, the first cohort has not reached graduation.
Therefore, given the newness of the AC program and the evaluation questions pertaining to both the initial implementation AND mature implementation phases, a Multiple-entry Evaluation is what describes this program evaluation. "In multiple entry evaluation, evaluators are concerned with conducting an evaluation that focuses on two or more program stages" (Huey-Tsyh, 2005).
The importance of conducting an evaluation at this stage of the program proves beneficial since stakeholders are experimenting with methods and ideas of implementation. The proposed evaluation questions also aim to look at what is working, what needs improvement, and what can be done better, for example. Therefore, what is the propsed evaluation strategy?
Evaluation Strategy - Troubleshooting
Taken From: Huey-Tsyh, C. (2005). Practical program evaluation: A Practical Evaluation Taxonomy (pp. ix-46), Thousand Oaks, CA: SAGE Publications
Taken From: Template from Module 2B of PME 802 Program Inquiry and Evalaution. No Citation Provided.
As per Huey-Tsyh's Table 3.1 A practical Taxonomy for Program Evaluation Means and Ends, troubleshooting is the recommended evaluation strategy for both the initial and mature implementation stages (2005). Troubleshooting aims to:
-
"Identify trouble spots in programs and address them"
-
"Provide timely assessment of barriers and/or problems facing a program"
-
"Identify any implementation problem[s]" (Huey-Tsyh, 2005)
The Academic Consolidation Program is a perfect fit for a troubleshooting strategy based evaluation since the target group has a multitude of different needs making the testing of effective implementation strategies the heart of the program. What works and does not work can be unpacked through proper evaluator questioning. In fact, several of the outlined evaluation questions are targeting exactly that. For example:
-
Are the resources provided working?
-
What is not working?
-
How can teacher training be improved?
-
What resources do the AC staff need?
Evaluation Approach - Formative Evaluation
"Formative evaluation examines directly the program's implementation [...and] is associated with research methods that are flexible to use; are easy to adopt in the field; and have a short turnaround time, such as focus groups and participant observation to collect, in timely fashion, facts about barriers and problems in implementation that promise to strengthen it" (Huey-Tsyh, 2005).
The Academic Consolidation Program's goals support the reasoning behind a formative evaluation in that the focus of the prescribed activities aim to produce the outcomes that are centered on better teacher implementation of curriculum and a focused approach to teaching strategies and skills needed by the students of the program. A formative evaluation will look at, through observational data, the act of teaching and collect information from the students, teaching and support staff, through ongoing discourse, the methodologies that work or need improvement, as well as brainstorming the possible resources needed to work towards improving the program's overall implementation. The evaluation questions proposed in this program evaluation support this form of evaluation approach. The hope is to bring feedback back to the stakeholders so that improvement can be made and, more importantly, the viability of the AC program can be secured, in that the long-term goal of 75-85% graduation rate for AC students is both attainable and feasible given the implementation strategies proposed.
References
Module 3 B: Identify Data Collection Methods & Analysis Strategies
Data Collection Methods
Since this program evaluation is a Multiple-Entry Formative Evaluation, I will be basing my data collection on the recommendations outlined by Huey-Tsyh Chen and the Rainbow Network Better Evaluation. "[F]ormative evaluation uses research methods that are flexible and provide quick feedback. Participant observation, key informant interviews, focus groups, site visits, record reviews, and small-scale surveys are some popular tools" (Chen, 2015, p.3-4). The following tables will classify my choice of evaluation data collection methods into qualitative and quantitative methods as well as provide a description of each of the data collection methods I have chosen to use to better reflect a triangulation of data collection options "in order to ensure multiple data sources and perspectives" (Better Evaluation, 2019). This table should be referred to when reading the next section which will break down and classify each evaluation question with its respective data collection methods.
Data Analysis Strategies
"Data analysis involves reviewing the data to answer the research question and understanding its significance. It means studying connections, patterns and trends (as well as exceptions and unique cases) in the data" (Speakup, p.50).
Before I attribute data collection strategies with the evaluation questions for this program evaluation, I need to address analysis techniques that I intend to use in order to assume organized, un-biased, and fair interpretation of the program evaluation. It is important to recall that the stakeholders in the AC program are few. (Refer back to the table showing the hierarchical relationship of stakeholders). As such, the close proximal relationships between the teaching and support staff need to be taking into consideration. Limitations due to this small number of stakeholders and the possible relationships between stakeholders will occur. Therefore, I need to employ analysis strategies that will ensure, as much as possible, the un-biased reporting of data. The following table addresses the analysis strategies for the Quantitative and Qualitative Data being collected. There is significantly more qualitative data being collected in this program evaluation.
Table 5 is based on information from Speakup Chapter 5 How to Analyze Research Data, Approaches to Qualitative Data Analysis by Louis Cohen, Laurence Manion & Keith Morrison and Evaluation Design Checklist by Daniel Stufflebeam et. al., 2004.
Sampling Procedures:
In the case of the AC program's evaluation, I will be employing the deliberate sampling method which falls under the category of non-probability sampling methods. The sampling method is used when the researcher is able to select the participants in order to satisfy a research requirement. Since the AC program has a small number of stakeholders and that every one of these stakeholders is important in the successful functioning of the program, every member needs to undergo a series of evaluation questions so that the evaluator can better understand and then provide a well-rounded view of the findings. Therefore, every member will be participating in the questions and different data collection techniques.
Sample Questions SLIDESHOW (3 slides):
Final Points on Data Analysis:
-
When necessary, surveys and questionnaires will be administered at the beginning and then again after other data collection methods to compare data.
-
In qualitative data collection one strategy will include identifying the most frequent and unique responses.
-
When possible, hidden meanings will be identified.
-
Common themes will be identified.
-
Unique responses will prompt further investigation.
-
With the consideration of the small number of stakeholders, data collection methods will be administered both individually and then as a group when the situation calls for it. This way stakeholders will not be tempted to "follow" the others. Data will also be cross-checked for inconsistencies and in the attempt to prevent bias.
-
Teachers outside of the core courses of the AC program will also be subjected to data collection methods to harness their perspectives and compare them to the core teachers' input (These teachers are from gym, art and ERC)
Program Evaluation Questions with Data Collection Methods and Analysis Strategies SLIDESHOW
The slideshow below now associates each program evaluation question with its respective data collection methods and its associated data analysis strategies that have been explained above. There are 4 slides in total. It also takes into consideration the sample questions as listed in the previous table.
Description of the Approach to Enhance Evaluation Use
This program evaluation exemplifies Dr. Michael Patton's Utilization-focused Evaluation in that it is conceived on the premise that the "intended Use" of the evaluation will benefit the "intended Users" (2013). The Academic Consolidation team has been mandated with a difficult task of using their skill and exceptional implementation strategies to assist students in attaining benchmark skills that will allow them to reach graduation. This is a monumentous task and in order to achieve some sort of success, one needs to look closely at whether what currently exists is working or not.
Working closely with the stakeholders is of particular importance in Dr. Patton's view. As such, this program evaluation uses this idea and consistently aims at discovering data that can then be used to either pave the way for more questioning or allow for immediate implementation by the stakeholders for the benefit of the students in the AC program. This being said, the evaluation is for instrumental purposes in that the evaluation findings would help affect what can be done next and how these changes could improve the program (Weiss, 1998). Furthermore, as explained by Weiss on Patton's theory, "working with program staff enlists their attention, knowledge, and commitment to using the results" from the evaluation (Weiss, 1998, p.22).
The Academic Consolidation program focuses on catering to the specific needs of the students in the program. Since their needs are particular and often require a significant amount of instructional trial and error on the part of the teacher, an evaluation that looks deeply into what is working and what needs improvement through the involvement of the stakeholders and sifting through their constant feedback with an objective lens, may well be the road to useful evaluation practice for this situation (Providing data collection and analysis strategies are adhered to). Therefore, "the best way that we know to date of encouraging use of evaluation is through involving potential users in defining the study and helping to interpret results, and through reporting results to them regularly while the study is in progress" (Weiss, 1998, p.30).
Reporting Strategies and Methods for Enhancing Use
According to the Step-by-Step Guides for Conducting a Formative Evaluation In the Evaluator's Handbook, there are 3 main steps in reporting findings from a formative evaluation such as the one conducted here. The diagram shown here depicts these steps. I will interpret this further by combining the information from the Handbook and the Evaluation Design Checklist by Daniel Stufflebeam, et. al., 2004, by focusing on the following strategies I feel are most pertinent to this given evaluation:
-
Timely Reporting and methods of reporting
-
Focusing on the unique discoveries as potential for improvement
-
Accessibility to Evaluation Report (containing interpreted data)
Given the time constraint of the AC program in general (6 years until graduation), timely feedback from the evaluation findings are necessary in order to incorporate methods to prompt instrumental changes for the benefit of the students. As such, quick dispensing of information is required in the form of electronic communication for simple facts, and formal meetings to discuss findings will be primarily used before a final report is produced.
This evaluation would have to be ongoing and it would be preferable to follow at least one AC cohort from start to finish. This would then serve as an important resource to the AC team; having data that looks at the success of one cohort and the results/recommendations could be easily transferred. After this initial evaluation, spot checks at key stages in the program (like at the end of every academic cycle), would be a good idea to keep everyone involved focused on the program's goals.
Unique themes or findings need particular attention given the nature of this program. Through data analysis, particular attention will be put on ideas, issues and concepts that come up randomly. The AC team is striving to find where and how to improve. Often, there is only so much an experienced individual can offer to the team. Therefore, it is "important to pay attention to responses that are unique as they might provide you with some new insights" (Speakup, p.52). These issues need to be outlined and communicated in a timely manner.
Finally, a significant amount of data and interpretation will result from this program evaluation. As such, a record of the report along with helpful summaries of the findings, need to remain accessible to the AC team should another evaluation take effect or there is a change in staff or simply to refer to it on an ongoing basis. Therefore, copies of the report need to be provided to every member of the AC team with a digital copy held for reference by the administration and even at the board level with the consultant. The results of the evaluation can further be used as a reference when looking at the effectiveness of the other AC teams within the school board and can, in the future, be used to compare success between these schools to better build a picture and road map of what success looks like.
Commitment to Standards of Practice
The AC Program Evaluation will adhere to the Standards for Program Evaluation as described below.
Utility Standards
"The utility standards are intended to ensure that an evaluation will serve the information needs of intended users."
At the onset of this program evaluation, attention was taken to ensure that the stakeholders were carefully identified and that a full description was provided in regards to their positions and importance within the AC program. This was done to ensure that their particular needs were made clear, and that any and all outcomes will benefit their ability to effectively administer the AC program while keeping the needs of their students of utmost priority.
The questions that were designed for this program evaluation took into consideration the overarching realization that improvement is what is being sought out. As such, the questions pertain to extracting data that can answer the questions dealing with how to improve the program.
In order to safeguard and protect the integrity of the data discovered, a series of data collection methods and analysis techniques along with some guiding principals will be employed to ensure that unbiased judgements can be rendered.
Finally, the importance of delivering timely information to the stakeholders was recognized, and plans were put in place to meet this need as mentioned in the previous section.
Feasibility Standards
"The feasibility standards are intended to ensure that an evaluation will be realistic, prudent, diplomatic, and frugal"
Since the AC team is closed group, they are granted scheduled meeting times on a more frequent basis. As such, distraction and interruption will be minimal since meetings have already been pre-scheduled in order to administer and collect the data required for the evaluation. Furthermore, non-participatory data collection techniques can and should be administered during instructional time to generate genuine data.
Recognition pertaining to the responsible AC staff was given special attention in that evaluation questions also pertain to the different levels of staff in order to get a well-rounded and politically correct picture of the effectiveness/needs of the AC program. This is to ensure that all parties are cooperating and understand that the evaluator will be using the data collected to formulate a final report based on the findings yielded by all parties.
Propriety Standards
"The propriety standards are intended to ensure that an evaluation will be conducted legally, ethically, and with due regard for the welfare of those involved in the evaluation, as well as those affected by its results."
It is clearly stipulated that the evaluation is dealing with the welfare of the right to an education of minors experiencing developmental and academic delays. As such, all required protocol involving the protection of minors will be respected and adhered to.
It is recognized that every stakeholder has something of importance to offer the evaluation team. As such, all measures to protect the integrity of human interaction will be used. All participants will be made aware of this importance, and the consequences should this not be followed will be made known.
The evaluation will honestly disclose the data collected and outline the limitations of such data. All efforts will be made to ensure the final report has looked at the information from all possible angles and that the stakeholders will receive a complete report that reflects the meticulous detail put into such an evaluation. Findings will be fully disclosed and unaltered in any way.
Accuracy Standards
"The accuracy standards are intended to ensure that an evaluation will reveal and convey technically adequate information about the features that determine worth or merit of the program being evaluated."
A complete and thorough summary of the AC program will be rendered by the evaluator, and any pertinent areas of concern will be dually signaled and addressed in the evaluation report and/or as soon as needed.
Attention to the reliability of information gathered will be of paramount importance to ensure systematic review. A thorough analysis of both quantitative and qualitative information will be had as stipulated in the previous sections of this PED.
Any conclusions will be fully justified and thoroughly supported with data and research. These conclusions will be made known in the report as well as at any moment throughout the evaluation process when deemed necessary by the evaluator.
