Public Opinion Survey Following the November 25, 2013 By-elections
Appendix B: Study Methodology
This section describes the methodology of the project in detail.
Preparing the Survey Instrument
Elections Canada prepared a fifteen minute questionnaire suitable for telephone administration, which aimed to evaluate electors' opinions, experiences, attitudes and knowledge of Elections Canada's services and to measure various aspects of electors' experiences with the electoral process. This questionnaire has been used in previous public opinion surveys following by-elections allowing for comparisons of results across elections and by-elections.
Defining the Target Population and Preparing the Sample
A minimum of 400 individuals from the general population of eligible electors were to be surveyed in each of the four ridings. The initial sample frame for the survey was drawn from three sources, with the following proportions of each, approximately:
- Listed landline sample: 80%
- Cell phone sample: 10%; and,
- RDD sample: 10%.
Each riding was divided into a number of sub-regional quotas to ensure survey responses were geographically representative. They were based on the first three digits of postal codes (forward sortation areas or FSAs). See "Administering the Survey" (next page) for a full listing of the sub-regional quotas. This approach was formalized in a sampling strategy which was approved by Elections Canada.
Later it was determined that additional sample was required. It was drawn entirely from the direct landline sample but maintained the sub-regional proportions set out in the Sampling Strategy. A summary of the sample frame for each subject riding appears in the table below (Figure B.1).
Riding | Listed Sample | Cell Phone Sample | RDD Sample | Initial Sample Frame | Additional Listed Sample | Final Sample Frame |
---|---|---|---|---|---|---|
Bourassa, QC | 4,800 | 600 | 600 | 6,000 | 1,200 | 7,200 |
Brandon–Souris, MB | 2,400 | 250 | 300 | 2,950 | 2,501 | 5,451 |
Provencher, MB | 2,400 | 250 | 300 | 2,950 | 2,501 | 5,451 |
Toronto Centre, ON | 4,800 | 600 | 600 | 6,000 | 1,200 | 7,200 |
Total | 14,400 | 1,700 | 1,800 | 17,900 | 7,402 | 25,302 |
Conducting a Pre-test
Before the full administration of the survey commenced, the questionnaire was pre-tested. The pre-test was performed to ensure that the survey was working as expected and to gauge the ease of contacting the sample and its responsiveness.
The survey was field tested in both languages on December 19, 2013 by the R.A. Malatest & Associates' survey house in Ottawa. During the field test, 1352 calls were made and 42 interviews (3%) were completed through outbound calling. A disproportionate number of calls were made to respondents in Bourassa, QC, in order to obtain a significant number of completes in French. The pre-test confirmed that the survey performed well and was approved for a full launch on December 23, 2013.
Administering the Survey
The survey was administered by telephone from the Consultant's Ottawa, Edmonton and Victoria survey houses. Potential survey participants were initially contacted by telephone to confirm contact details, establish the individual who would be completing the survey and introduce the survey topic. Respondents could then choose to complete the survey immediately over the telephone, or arrange for another time to complete the survey. The questionnaire duration ranged from 3.6 to 40.8 minutes, with an average of 14.6 minutes. The survey was completed on January 23, 2014.
Postal codes and other eligibility questions were reviewed to ensure all completions existed in the riding. In total, 65 cases were removed.
A total of 1,631 valid survey completions were obtained. The overall response rate was 13.5%. All riding and most sub-regional targets were met or exceeded. Figure B.2 below shows the breakdown of completes by sector and region, as well as the original target completes.
With these completions in mind, as well as the population numbers included in Appendix A, the following margins of error have been calculated for the sample subgroups, with a confidence level of 95% (19 times out of 20):
- For each riding, +/- 4.9 percentage points; and,
- Overall, +/- 2.4 percentage points.
Riding | Sub-region | Target | Completes | Progress (%) |
---|---|---|---|---|
Bourassa, QC | Montréal-Nord (North) | 228 | 211 | 93% |
Montréal-Nord (South) | 118 | 143 | 121% | |
Rivière-des-Prairies (Southwest) | 32 | 39 | 122% | |
Bourassa - Other | 22 | 10 | 45% | |
Total | 400 | 403 | 101% | |
Brandon–Souris, MB | Brandon (East and Vicinity) | 113 | 121 | 107% |
Brandon Region | 108 | 115 | 106% | |
Brandon (South; Northwest) | 100 | 92 | 92% | |
Southwestern Manitoba | 79 | 76 | 96% | |
Total | 400 | 404 | 101% | |
Provencher, MB | Southeastern / South Central Manitoba | 165 | 200 | 121% |
Eastern Manitoba | 122 | 105 | 86% | |
St. Adolphe / Steinbach / Ste. Anne | 113 | 116 | 103% | |
Total | 400 | 421 | 105% | |
Toronto Centre, ON | Regent Park / Harbourfront / Ryerson / Other Downtown | 154 | 157 | 102% |
Church and Wellesley | 93 | 80 | 86% | |
Rosedale / Moore Park / Summershill East / Willowdale (South) | 66 | 81 | 123% | |
Cabbagetown / St. James Town | 50 | 54 | 108% | |
Yorkville / The Annex / North Midtown | 37 | 31 | 84% | |
Total | 400 | 403 | 101% | |
Total | 1,600 | 1,631 | 101% |
Weighting the Data
The sample of survey respondents was compared to the actual population of each riding by age and gender, through the use of riding profiles maintained by Statistics Canada and based on the 2011 census. The survey sample was skewed disproportionately older in all four ridings. In all ridings, there was a surplus of respondents aged 55 years or older and an under representation of youth (34 or younger). The survey sample was also skewed somewhat by gender, with women being somewhat overrepresented compared to men.
Weights were created in order to adjust the data to better reflect the age and gender distribution of electors in the four ridings. Technically four distinct weighting schemes were created – one for each riding. This maintains that each riding is roughly one quarter of the overall sample (1,631 cases). Fewer than 5% of respondents (n=77) did not provide their birth year, and thus they are assigned a neutral weight of one. The impact of the weights on the distribution of electors by age and gender is illustrated in the table below (Figure B.3).
Variables | Without Weights | Weighted by both Age & Gender where possible |
---|---|---|
Riding | 403 in Bourassa 404 in Brandon–Souris 421 in Provencher 403 in Toronto Centre |
403 in Bourassa 404 in Brandon–Souris 421 in Provencher 403 in Toronto Centre |
Gender | 733 males (44.9%) 898 females (55.1%) |
794 males (48.7%) 837 females (51.3%) |
Age | 129 18-34 (7.9%) 471 35-54 (28.9%) 954 55+ (58.5%) 77 missing (4.7%) |
475 18-34 (29.2%) 546 35-54 (33.5%) 533 55+ (32.7%) 77 missing (4.7%) |