open Secondary menu

Public Opinion Survey Following the November 25, 2013 By-elections

Appendix B: Study Methodology

This section describes the methodology of the project in detail.

Preparing the Survey Instrument

Elections Canada prepared a fifteen minute questionnaire suitable for telephone administration, which aimed to evaluate electors' opinions, experiences, attitudes and knowledge of Elections Canada's services and to measure various aspects of electors' experiences with the electoral process. This questionnaire has been used in previous public opinion surveys following by-elections allowing for comparisons of results across elections and by-elections.

Defining the Target Population and Preparing the Sample

A minimum of 400 individuals from the general population of eligible electors were to be surveyed in each of the four ridings. The initial sample frame for the survey was drawn from three sources, with the following proportions of each, approximately:

  • Listed landline sample: 80%
  • Cell phone sample: 10%; and,
  • RDD sample: 10%.

Each riding was divided into a number of sub-regional quotas to ensure survey responses were geographically representative. They were based on the first three digits of postal codes (forward sortation areas or FSAs). See "Administering the Survey" (next page) for a full listing of the sub-regional quotas. This approach was formalized in a sampling strategy which was approved by Elections Canada.

Later it was determined that additional sample was required. It was drawn entirely from the direct landline sample but maintained the sub-regional proportions set out in the Sampling Strategy. A summary of the sample frame for each subject riding appears in the table below (Figure B.1).

Figure B.1: Sample frame for each subject riding, by source
Riding Listed Sample Cell Phone Sample RDD Sample Initial Sample Frame Additional Listed Sample Final Sample Frame
Bourassa, QC 4,800 600 600 6,000 1,200 7,200
Brandon–Souris, MB 2,400 250 300 2,950 2,501 5,451
Provencher, MB 2,400 250 300 2,950 2,501 5,451
Toronto Centre, ON 4,800 600 600 6,000 1,200 7,200
Total 14,400 1,700 1,800 17,900 7,402 25,302

Conducting a Pre-test

Before the full administration of the survey commenced, the questionnaire was pre-tested. The pre-test was performed to ensure that the survey was working as expected and to gauge the ease of contacting the sample and its responsiveness.

The survey was field tested in both languages on December 19, 2013 by the R.A. Malatest & Associates' survey house in Ottawa. During the field test, 1352 calls were made and 42 interviews (3%) were completed through outbound calling. A disproportionate number of calls were made to respondents in Bourassa, QC, in order to obtain a significant number of completes in French. The pre-test confirmed that the survey performed well and was approved for a full launch on December 23, 2013.

Administering the Survey

The survey was administered by telephone from the Consultant's Ottawa, Edmonton and Victoria survey houses. Potential survey participants were initially contacted by telephone to confirm contact details, establish the individual who would be completing the survey and introduce the survey topic. Respondents could then choose to complete the survey immediately over the telephone, or arrange for another time to complete the survey. The questionnaire duration ranged from 3.6 to 40.8 minutes, with an average of 14.6 minutes. The survey was completed on January 23, 2014.

Postal codes and other eligibility questions were reviewed to ensure all completions existed in the riding. In total, 65 cases were removed.

A total of 1,631 valid survey completions were obtained. The overall response rate was 13.5%. All riding and most sub-regional targets were met or exceeded. Figure B.2 below shows the breakdown of completes by sector and region, as well as the original target completes.

With these completions in mind, as well as the population numbers included in Appendix A, the following margins of error have been calculated for the sample subgroups, with a confidence level of 95% (19 times out of 20):

  • For each riding, +/- 4.9 percentage points; and,
  • Overall, +/- 2.4 percentage points.
Figure B.2: Targets and Final Completes by Region and Sector (Unweighted)
Riding Sub-region Target Completes Progress (%)
Bourassa, QC Montréal-Nord (North) 228 211 93%
Montréal-Nord (South) 118 143 121%
Rivière-des-Prairies (Southwest) 32 39 122%
Bourassa - Other 22 10 45%
Total 400 403 101%
Brandon–Souris, MB Brandon (East and Vicinity) 113 121 107%
Brandon Region 108 115 106%
Brandon (South; Northwest) 100 92 92%
Southwestern Manitoba 79 76 96%
Total 400 404 101%
Provencher, MB Southeastern / South Central Manitoba 165 200 121%
Eastern Manitoba 122 105 86%
St. Adolphe / Steinbach / Ste. Anne 113 116 103%
Total 400 421 105%
Toronto Centre, ON Regent Park / Harbourfront / Ryerson / Other Downtown 154 157 102%
Church and Wellesley 93 80 86%
Rosedale / Moore Park / Summershill East / Willowdale (South) 66 81 123%
Cabbagetown / St. James Town 50 54 108%
Yorkville / The Annex / North Midtown 37 31 84%
Total 400 403 101%
Total 1,600 1,631 101%

Weighting the Data

The sample of survey respondents was compared to the actual population of each riding by age and gender, through the use of riding profiles maintained by Statistics Canada and based on the 2011 census. The survey sample was skewed disproportionately older in all four ridings. In all ridings, there was a surplus of respondents aged 55 years or older and an under representation of youth (34 or younger). The survey sample was also skewed somewhat by gender, with women being somewhat overrepresented compared to men.

Weights were created in order to adjust the data to better reflect the age and gender distribution of electors in the four ridings. Technically four distinct weighting schemes were created – one for each riding. This maintains that each riding is roughly one quarter of the overall sample (1,631 cases). Fewer than 5% of respondents (n=77) did not provide their birth year, and thus they are assigned a neutral weight of one. The impact of the weights on the distribution of electors by age and gender is illustrated in the table below (Figure B.3).

Figure B.3: Impact of Revised Weights on Demographic Variables
Variables Without Weights Weighted by both Age & Gender where possible
Riding 403 in Bourassa
404 in Brandon–Souris
421 in Provencher
403 in Toronto Centre
403 in Bourassa
404 in Brandon–Souris
421 in Provencher
403 in Toronto Centre
Gender 733 males (44.9%)
898 females (55.1%)
794 males (48.7%)
837 females (51.3%)
Age 129 18-34 (7.9%)
471 35-54 (28.9%)
954 55+ (58.5%)
77 missing (4.7%)
475 18-34 (29.2%)
546 35-54 (33.5%)
533 55+ (32.7%)
77 missing (4.7%)