Site MenuStudent Parallel Election Program (Student Vote) Evaluation
Section 2 Methodology
A mixed method approach was used for the evaluation. A series of surveys was conducted with students, teachers and parents from schools that had participated in Student Vote, and schools that had not participated in Student Vote, during the 2015 federal election. The survey data were supplemented by key informant interviews with participating and non-participating teachers, parents of students who had participated in Student Vote and program stakeholders. Additionally, focus groups were conducted with students who had participated in Student Vote. Finally, site visits were held at schools that had taken part in Student Vote.
2.1 Data Sources/Sample
The samples for the participating and non-participating surveys were provided by CIVIX. CIVIX provided two files containing the schools that had registered for Student Vote. The files contained a total of 5,598 schools. Before developing samples, non-traditional and adult educational institutions were excluded, leaving primary/elementary schools, middle schools, high/secondary schools and combined schools.
An initial sample of 250 schools, stratified by region and school type, was randomly drawn from the lists provided to recruit for the pre-program survey. The resulting sample was provided to CIVIX to recruit teacher participation at the school level. Teacher participation at the school level included administering student and parent surveys as well as completing a teacher survey. Teachers who agreed to participate at the school level were given a $50 gift card as a show of appreciation for their time and effort. Due to difficulties in recruiting teachers to participate at the school level, CIVIX requested additional samples for recruitment. An additional 500 schools were randomly selected from the list of registered schools and provided to CIVIX.
In addition to school-level recruitment, teacher-only recruitment was conducted. Teacher-only recruits were required only to complete the teacher survey. Given that few teacher surveys were expected to be completed at each school, a teacher-only recruitment sample was generated to supplement the school-level recruitment. A random sample of 400 schools was drawn from the registered schools list and provided to CIVIX for recruitment of teachers to complete the teacher survey.
The recruitment of school-level participation for the post-program surveys was mainly conducted with schools that had participated at the school level in the pre-program surveys. Teachers who had administered pre-program student and parent surveys were asked to administer post-program student and parent surveys. However, due to concerns about adequate participation of Quebec schools, a list of 82 additional schools in Quebec was provided.
Post-program, teacher-only recruitment included all teachers who had participated in Student Vote. CIVIX sent out survey invitations to all schools that had registered.
At the close of Student Vote, after the federal election, CIVIX provided a list of all the schools that had not registered for the program. The list contained a total of 8,735 schools. After non-valid educational institutions were excluded, 7,699 schools remained. An initial sample of 1,500 schools was randomly selected from the provided list. After contact information for the schools was generated, schools were contacted to participate at the school level in the non-participating (control) surveys. The principals of the schools were asked whether their schools would be interested in participating in the survey by administering student, parent and teacher surveys. Interested schools were sent packages outlining survey-administration instructions. Follow-up calls were made to schools that had initially expressed interest to encourage them to complete the surveys. Non-registered schools that had agreed to participate in the surveys were given a $200 honorarium to help defray the cost and time required to participate. Due to challenges with recruiting schools for the non-participating surveys, an additional sample of 1,500 schools was generated and contact information found.
Similar to the participating sample, a supplemental sample was generated to recruit teacher-only participants. A sample of 750 schools was generated for teacher-only recruitment. After contact information for the sample was generated, principals of the schools were recruited. Interested schools were asked to distribute the survey among appropriate teachers. To help recruit teachers who had not participated in the Student Vote program, teachers were given a $10 gift card for completing the survey.
Key informant interviews were conducted with teachers, parents and program stakeholders. Interviews with teachers included both teachers who had and teachers who had not registered to participate in Student Vote for the 2015 federal election. Parent interviews were conducted only with those parents whose child(ren) had participated in Student Vote. Participants for these interviews were primarily located among survey respondents. Surveys of teachers, both participating and non-participating in Student Vote, asked respondents whether they would be interested in participating in an interview about Student Vote. Parents of students who had participated in Student Vote were also invited at the end of the survey to participate in an interview. Interested respondents left their contact information and were contacted for an interview. Stakeholder interviews were conducted with individuals from CIVIX and Elections Canada who were familiar with Student Vote as well as with partner organizations that worked with CIVIX. The names and contact information for stakeholder interviews were provided by CIVIX and Elections Canada. The individuals invited for stakeholder interviews were selected based on their perceived ability to speak about Student Vote.
The recruitment of participating schools for site visits and focus groups was primarily conducted by CIVIX. After the geographical locations of the site visits and focus groups were determined, CIVIX inquired with teacher leads at schools in those areas about their interest in participating. Schools that agreed to participate were put in contact with the contractor to make arrangements.
2.2 Data Instruments
2.2.1 Surveys
A series of 11 surveys was conducted with different participant groups (see Table 2-1). Elementary students, secondary students and teachers at schools that had participated in Student Vote were asked to complete pre-program and post-program surveys. Non-participating schools were asked to complete a post-program survey for teachers and their students. Parents of students in both participating and non-participating schools were asked to complete a post-program survey only.
Teachers participating in Student Vote were first invited to complete the pre-program surveys on September 17, 2015, and the survey remained open for completion until October 16, 2015. Teachers participating in Student Vote were first invited to complete the post-program surveys on November 2, 2015, and the survey remained open until December 11, 2015. Non-participating school administrators were first contacted to participate in the survey on November 10, 2015. Due to a low response rate, the survey and recruitment efforts for non-participating schools continued until February 19, 2016.
| Participating Schools Pre-program |
Participating Schools Post-program |
Non-participating Schools Post-program (Control) |
|
|---|---|---|---|
| Students – Elementary |  |
|
|
| Students – Secondary | |
|
|
| Teachers | |
|
|
| Parents | – | |
|
Surveys for the current evaluation were largely based on the versions used in the 2011 evaluation. The surveys were modified and updated with extensive input from Elections Canada and CIVIX to reflect the research objectives and expected outcomes. The surveys asked respondents about their interest in politics, their knowledge of politics, their level of engagement with the 2015 federal election and their satisfaction with Student Vote. Additionally, students and teachers were asked about their confidence in talking about/teaching politics as well as the learning activities they had engaged in during the election period.
The surveys for the elementary and secondary students were the same within each treatment condition (i.e. pre-program, post-program, non-participating), with the exception of the knowledge questions, which had different difficulty levels for elementary and secondary students. Additionally, student surveys administered after the federal election (post-program, control) included follow-up questions associated with election-related activities and satisfaction with these activities. Parent surveys (participating, non-participating) were largely identical for most questions; however, participating parents were asked about their child(ren)'s experience with Student Vote. Teacher surveys also contained a core set of questions across treatment conditions. However, post-program participating teachers were asked about their experience and satisfaction with Student Vote. Additionally, non-participating teachers were asked about their reasons for not participating in Student Vote.
2.2.2 Key Informant Interview Guides
A series of four interview guides was prepared for the different participant groups. The participant groups for the interviews included:
- Teachers who had participated in the 2015 federal Student Vote.
- Teachers who had not participated in the 2015 federal Student Vote.
- Parents of students who had participated in the 2015 federal Student Vote.
- Student Vote stakeholders (see Section 2.1).
Participating teachers and parents were asked about their personal interest in and knowledge of politics and government, experiences with Student Vote and perceptions of the impact of Student Vote on their students/child(ren). Non-participating teacher guides included questions about why teachers did not participate in Student Vote and future intentions to participate. Stakeholder guides addressed questions concerning the need for, and effectiveness of, Student Vote. The discussion of the effectiveness of Student Vote included the recruitment of school participants for the program, engagement with current and future stakeholders, the use of appropriate communication processes and ongoing performance monitoring and its impact on student outcomes. All the guides were reviewed and edited by Elections Canada and CIVIX before the interviews.
2.2.3 Focus Group Guides
Two focus group guides were developed for the participating student groups: a pre-program guide and a post-program guide. Both guides were similar, asking students about their knowledge of, interest in and comfort with talking about politics as well as their beliefs about voting. Students in the pre-program focus groups were asked about their satisfaction with the Student Vote activities they were currently involved in. Students in the post-program focus groups were asked about their satisfaction with Student Vote and any perceived impact it may have had on them. The guides were reviewed by both Elections Canada and CIVIX before the focus groups took place.
2.3 Data Collection
2.3.1 Surveys
All surveys were available for completion online or in hard copy in either official language, based on the preference of the respondent. CIVIX administered the distribution of the surveys for participating schools, both pre- and post-program. Recruited schools were sent survey-administration instructions, including links to the online survey, or physical copies of the survey, as preferred by the teacher lead. Completed surveys were returned to the contractor at no expense to the teachers.
The contractor administered the survey for non-participating schools. As recruitment of non-participating schools was conducted through school administrators (e.g. the principal), the school administrator liaised between the contractor and the teacher(s). Interested schools were sent survey-administration instructions. When requested, hard copies were provided to schools, to be returned to the contractor in pre-paid express-mail envelopes.
Completed hard copies were data-entered by the contractor. Data-entry staff received training before entering the surveys. Data-entered surveys were verified by a supervisor to ensure their accuracy and completion.
2.3.2 Key Informant Interviews
Teachers and parents were primarily recruited through the survey, by means of a question that asked them to indicate their interest. Individuals who had expressed interest in being interviewed were contacted by the contractor to confirm interest and arrange a date and time for the interview. Interviewees were sent a copy of the interview guide in advance of the interview to allow them to prepare. All the interviews were conducted by telephone and took from 30 to 90 minutes, depending on the participant group.
2.3.3 Focus Groups
Recruitment for the student focus groups was conducted by CIVIX. CIVIX approached teachers, in certain geographical locations, to participate in the focus groups. Interested teachers were put in contact with the contractor to arrange a date and time. The focus groups were hosted at the schools during regular school hours. The nature of the groups differed by teacher; some teachers selected students to attend a focus group, while other teachers requested that the group be conducted with the entire class. The focus groups were 45 to 60 minutes in length.
2.3.4 Site Visits
A series of four site visits was conducted at three schools. The first two visits occurred before the federal election, when students were participating in the Student Vote program. The last two visits occurred after the election, when students had completed Student Vote. The site visits were originally designed to allow the evaluation to observe schools pre- and post-program. Unfortunately, due to logistical issues, it was not possible to revisit one of the schools that had participated in the pre-program site visit. As a result, an additional school in the same region hosted the post-program site visit. Furthermore, due to time constraints and classroom demands, the observation of students was limited during the majority of site visits. As a result, some visits were limited to conducting a student focus group and discussing Student Vote with the teacher. All four of the site visits occurred at elementary schools.
2.4 Challenges and Limitations
Obtaining school-level participation from both participating and non-participating schools was challenging. School-level participation required teachers to administer the student survey to their class, recruit parents to complete the survey and complete a teacher survey themselves. As such, school-level participation required a great deal of time and energy from teachers. The level of commitment required was especially great for participating teachers, who were asked to complete the survey twice, once before the election and again after the election. The time requirements for teachers to participate at the school level impeded recruitment efforts.
The amount of time required to participate at the school level posed unique problems for recruitment in Quebec and Ontario. Collective agreement negotiations and job actions were occurring in these provinces during the 2015 federal election. As a result, many teachers were hesitant to agree to the amount of time required to participate in the surveys. Some teachers were uncertain about possible negotiation breakdowns, while other teachers were working to rule until negotiations were completed. As a result, many teachers were unwilling to take on any new responsibilities, such as agreeing to school-level surveying.
Since there are only a few teachers in a school who are eligible to complete the survey, recruiting a school to conduct the teacher-only survey generated very few survey completions. As such, a large number of schools/teachers were invited to participate to ensure adequate numbers of teacher completions. This was especially challenging with non-participating teachers, where communication about the survey was mediated by school administrators, who needed to determine who was eligible before distributing the survey.
The success of Student Vote in registering schools for the 2015 federal election impacted the ability to collect a control sample. Almost three-quarters (72%) of the unregistered schools in the non-registered sample were primary/elementary schools. While the final sample of non-participating students reflects these proportions, it resulted in a more unbalanced sample than the participating student sample; in both the pre- and the post-program samples, the final samples were roughly 50% from each group.
2.5 Profile of Participants
2.5.1 Survey
A total of 7,542 surveys were collected. Table 2-2 shows the number of individually completed surveys by survey type. Roughly half the pre-program (49%) and post-program (50%) student surveys were completed by elementary students. Over two-thirds of non-participating (70%) student surveys were completed by elementary students.
| Participating Schools Pre-program |
Participating Schools Post-program |
Non-participating Schools Control |
|
|---|---|---|---|
| Students – Elementary | 1,104 | 928 | 674 |
| Students – Secondary | 1,128 | 940 | 288 |
| Teachers | 171 | 1,448 | 134 |
| Parents | – | 604 | 123 |
Students: Table 2-3 shows the demographic breakdown of the student surveys, by survey type.
| Demographic Characteristic Participant Response |
Pre-program | Post-program | Non-participating (Control) |
|---|---|---|---|
| GenderMale | 49% | 50% | 51% |
| Gender Female | 51% | 51% | 50% |
| Born in CanadaYes | 91% | 91% | 91% |
| Born in Canada No | 9% | 9% | 9% |
| ProvinceUnassigneda | 1% | 1% | 2% |
| ProvinceAlberta | 11% | 12% | 21% |
| ProvinceBritish Columbia | 19% | 19% | 33% |
| ProvinceManitoba | 6% | 2% | 8% |
| ProvinceNew Brunswick | 7% | 8% | 6% |
| ProvinceNewfoundland and Labrador | 5% | 3% | 9% |
| ProvinceNova Scotia | 13% | 9% | 0% |
| ProvinceOntario | 20% | 16% | 17% |
| ProvincePrince Edward Island | 2% | 3% | 1% |
| ProvinceQuebec | 16% | 24% | 0% |
| ProvinceSaskatchewan | 3% | 4% | 3% |
| Age7 or younger | 0% | 0% | 0% |
| Age8 | 2% | 1% | 0% |
| Age9 | 7% | 6% | 10% |
| Age10 | 16% | 17% | 26% |
| Age11 (elementary) 11 or younger (secondary)b |
14% | 18% | 27% |
| Age12 | 11% | 11% | 17% |
| Age13 | 6% | 7% | 10% |
| Age14 | 16% | 14% | 4% |
| Age15 | 14% | 18% | 1% |
| Age16 | 12% | 5% | 2% |
| Age17 | 2% | 2% | 2% |
| Age18 | 0% | 0% | 0% |
| Age19 | 0% | 0% | 0% |
| Previously Participated in Student VoteYes | 18% | 37% | 29% |
| Previously Participated in Student VoteNo | 66% | 52% | 48% |
| Previously Participated in Student VoteDon't know/not sure | 17% | 11% | 23% |
aSome students did not know, or provide, their school code. As a result, it was not possible to assign these students to a school or province.
bThis was the lowest age category that student completing the secondary survey could select.
Teachers: Table 2-4 shows the demographic breakdown of the teacher surveys, by survey type.
| Demographic Characteristic Participant Response |
Pre-program | Post-program | Non-participating (Control) |
|---|---|---|---|
| GenderMale | 31% | 30% | 37% |
| Gender Female | 69% | 70% | 63% |
| Born in CanadaYes | 89% | 90% | 89% |
| Born in Canada No | 11% | 10% | 11% |
| ProvinceUnassigneda | 2% | 20% | 4% |
| Province Alberta | 16% | 13% | 13% |
| Province British Columbia | 19% | 15% | 33% |
| ProvinceManitoba | 6% | 3% | 9% |
| ProvinceNew Brunswick | 6% | 2% | 9% |
| ProvinceNewfoundland and Labrador | 2% | 1% | 10% |
| ProvinceNova Scotia | 14% | 4% | 1% |
| ProvinceOntario | 22% | 33% | 15% |
| ProvincePrince Edward Island | 2% | 1% | 0% |
| Province Quebec | 9% | 3% | 0% |
| ProvinceSaskatchewan | 2% | 5% | 6% |
| Length of Time Teaching Civics or a Subject Related to Canadian Politics and the Workings of GovernmentLess than 1 year | 7% | 9% | 11% |
| Length of Time Teaching Civics or a Subject Related to Canadian Politics and the Workings of Government 1–2 years | 15% | 10% | 11% |
| Length of Time Teaching Civics or a Subject Related to Canadian Politics and the Workings of Government 3–5 years | 19% | 20% | 25% |
| Length of Time Teaching Civics or a Subject Related to Canadian Politics and the Workings of Government 6–7 years | 8% | 11% | 8% |
| Length of Time Teaching Civics or a Subject Related to Canadian Politics and the Workings of Government8 or more years | 52% | 50% | 44% |
| Prior Participation in Student VotePreviously participated in a Student Vote federal election | – | 41% | 16% |
| Prior Participation in Student Vote Previously participated in a Student Vote provincial election | – | 45% | 14% |
| Prior Participation in Student Vote Previously participated in a Student Vote municipal election | – | 23% | 5% |
| Prior Participation in Student Vote Never participated in the Student Vote Program | – | 40% | 70% |
| Prior Participation in Student VoteThis is my first time. I have never participated in the Student Vote program. | 34% | – | – |
| Prior Participation in Student Vote I previously participated in one Student Vote program. | 23% | – | – |
| Prior Participation in Student Vote I previously participated in two Student Vote programs. | 15% | – | – |
| Prior Participation in Student Vote I previously participated in three or more Student Vote programs. | 29% | – | – |
aSome teachers did not know, or provide, their school code. As a result, it was not possible to assign these teachers to a school or province.
Table 2-5 shows the breakdown of the grades and subjects the teachers taught, by survey type.
| Demographic Characteristic Participant Response |
Pre-program | Post-program | Non-participating (Control) |
|---|---|---|---|
| Grade Level(s) TaughtaGrade 1 | 1% | 3% | 10% |
| Grade Level(s) Taughta Grade 2 | 2% | 3% | 10% |
| Grade Level(s) Taughta Grade 3 | 6% | 7% | 14% |
| Grade Level(s) Taughta Grade 4 | 15% | 20% | 28% |
| Grade Level(s) TaughtaGrade 5 | 33% | 45% | 46% |
| Grade Level(s) TaughtaGrade 6 | 35% | 42% | 37% |
| Grade Level(s) Taughta Grade 7 | 22% | 29% | 30% |
| Grade Level(s) TaughtaGrade 8 | 23% | 26% | 25% |
| Grade Level(s) TaughtaGrade 9 | 25% | 20% | 16% |
| Grade Level(s) TaughtaGrade 10 | 20% | 21% | 16% |
| Grade Level(s) TaughtaGrade 11 | 20% | 16% | 17% |
| Grade Level(s) TaughtaGrade 12 | 15% | 15% | 16% |
| Subject(s) TaughtaThe arts | 32% | 31% | 34% |
| Subject(s) TaughtaGeography | 23% | 19% | 26% |
| Subject(s) TaughtaLanguage arts | 57% | 53% | 64% |
| Subject(s) TaughtaMathematics | 39% | 42% | 57% |
| Subject(s) TaughtaCivics | 20% | 17% | 21% |
| Subject(s) TaughtaHistory | 35% | 26% | 31% |
| Subject(s) TaughtaInformation studies | 11% | 4% | 9% |
| Subject(s) TaughtaPhysical/health education | 19% | 28% | 35% |
| Subject(s) TaughtaScience | 37% | 38% | 49% |
| Subject(s) Taughta Social studies | 59% | 65% | 74% |
| Subject(s) Taughta All of the above | 29% | 14% | 6% |
| Subject(s) TaughtaOther | 18% | 15% | 17% |
aThe question allowed for multiple responses.
Parents: Tables 2-6 and 2-7 show the demographic breakdown of the parent surveys, by survey type.
| Demographic Characteristic Participant Response |
Post-program | Non-participating (Control) |
|---|---|---|
| Grades of Child(ren)Grade 1 | 1% | 5% |
| Grades of Child(ren)Grade 2 | 0% | 1% |
| Grades of Child(ren)Grade 3 | 1% | 1% |
| Grades of Child(ren)Grade 4 | 6% | 7% |
| Grades of Child(ren)Grade 5 | 34% | 19% |
| Grades of Child(ren)Grade 6 | 23% | 27% |
| Grades of Child(ren)Grade 7 | 8% | 12% |
| Grades of Child(ren)Grade 8 | 6% | 16% |
| Grades of Child(ren)Grade 9 | 8% | 7% |
| Grades of Child(ren)Grade 10 | 9% | 1% |
| Grades of Child(ren)Grade 11 | 3% | 3% |
| Grades of Child(ren)Grade 12 | 0% | 2% |
| ProvinceUnassigneda | 0% | 2% |
| ProvinceAlberta | 6% | 19% |
| ProvinceBritish Columbia | 15% | 26% |
| ProvinceManitoba | 0% | 1% |
| ProvinceNew Brunswick | 9% | 1% |
| ProvinceNewfoundland and Labrador | 4% | 24% |
| ProvinceNova Scotia | 10% | 0% |
| ProvinceOntario | 18% | 24% |
| ProvincePrince Edward Island | 0% | 3% |
| ProvinceQuebec | 28% | 0% |
| ProvinceSaskatchewan | 9% | 0% |
aSome parents did not know, or provide, their child(ren)'s school code. As a result, it was not possible to assign these parents to a school or province.
| Demographic Characteristic Participant Response |
Post-program | Non-participating (Control) |
|---|---|---|
| Age20 to 24 | 1% | 0% |
| Age25 to 29 | 1% | 2% |
| Age30 to 34 | 7% | 11% |
| Age35 to 39 | 20% | 20% |
| Age40 to 44 | 34% | 34% |
| Age45 to 49 | 24% | 20% |
| Age50 to 54 | 10% | 8% |
| Age55 to 59 | 1% | 2% |
| Age60 or older | 1% | 2% |
| GenderMale | 20% | 24% |
| GenderFemale | 78% | 73% |
| GenderPrefer not to say | 2% | 3% |
| Born in CanadaYes | 84% | 74% |
| Born in CanadaNo | 16% | 26% |
| If Not Born in Canada, How Long They Have Lived in Canada1 to 5 years | 9% | 6% |
| If Not Born in Canada, How Long They Have Lived in Canada6 to 10 years | 15% | 13% |
| If Not Born in Canada, How Long They Have Lived in Canada11 to 15 years | 24% | 34% |
| If Not Born in Canada, How Long They Have Lived in Canada16 to 20 years | 12% | 28% |
| If Not Born in Canada, How Long They Have Lived in Canada21 or more years | 39% | 19% |
2.5.2 Key Informant Interviews
A total of 37 interviews were conducted. Table 2-8 shows the number of interviews, by participant group.
| Participant Group | Number of Interviews |
|---|---|
| Participating Teachers | 18 |
| Non-participating Teachers | 5 |
| Participating Parents | 6 |
| Stakeholders | 8 |
2.5.3 Focus Groups
A total of six focus groups were conducted in four cities across Canada. Two of the groups were hosted at elementary schools before the federal election. The remaining four groups were hosted after the election, with two held at elementary schools and two held at secondary schools. One of the elementary schools hosted pre- and post-program focus groups. A total of 119 students attended the six groups. Table 2-9 shows the breakdown of students attending the different types of focus groups.
| Focus Group Timing | Elementary School | Secondary School |
|---|---|---|
| Pre-program | 38 | – |
| Post-program | 60 | 21 |