Electoral Insight – 2004 General Election
Opinion Polls and the Canada Elections Act
Professor, Department of Sociology, Université de Montréal
The public right to good, reliable information is unquestionably one of the cornerstones of our democratic societies." Footnote 1 Recognition of this principle led Parliament, when amending the Canada Elections Act in 2000, to include a section on the publication of and access to methodological information about opinion polls published during an election period. In the Act adopted in February 2000, subsection 326(1) creates the obligation to publish, together with the poll results, the names of the polling organization and sponsor, the timing of the survey, the sample population, the number of people contacted and, if applicable (author's emphasis Footnote 2), the margin of error for the results presented. Subsection 326(2) requires publication of the wording of the survey questions and instructions on how the reader can obtain a report with additional methodological information. Subsection 326(3) describes the information that must be presented in that report: the sampling method and a variety of information about the sample, the response rate, the dates and time of day the interviews took place and the method used to adjust the data, particularly to account for non-responses (respondents not answering a specific question).
A little background
We should remember that there was no such section in the 1993 Act, Footnote 3 despite the representations on this matter made before the Royal Commission on Electoral Reform and Party Financing (Lortie Commission). At that time, an analysis of the way polling methodology was described in the media Footnote 4 had demonstrated serious shortcomings. The author noted in particular the lack of information on the response rate, the weighting or adjustment, and the size of regional samples. Moreover, the wording of the questions was not always included. He proposed that the Act require the publication of a methodological insert with certain specific information. After the Act came into effect, Lachapelle analyzed the presentation of survey methodology during the 1993 election, and his conclusion is just as bleak as in 1991: "The methodological information required to judge the quality of a survey is often missing or obscured behind the mass of data provided." Footnote 5 During the 1997 federal election, Andersen "took up the torch" and analyzed the basic methodological information in all the articles presenting survey data. He found that 64% did not mention the polling period, 33%, the size of the sample, 25%, the margin of error, 21%, the wording of the question, and 14%, the proportion of undecided responses. Footnote 6
After the enactment of the new legislation in 2000, Durand showed that the basic information required by subsection 326(1) is generally presented and subsection 326(2) concerning the publication of the wording of the question on voting intentions and the mention of the availability of a detailed report is generally respected. Footnote 7 However, with a few exceptions, most of the information that should appear in the report provided by the polling organization (subsection 326(3)) is missing.
What about 2004?
During the campaign of May 23 to June 27, 2004, 17 national surveys were published in the national anglophone and francophone media (The Globe and Mail, The Toronto Star, The National Post, La Presse, Le Journal de Montréal, The Gazette and Canadian Press). Four firms did the polling: Ipsos-Reid (7 polls), EKOS (4), Leger Marketing (4) and COMPAS (2). In addition, Environics published a poll on its Web site and SES Research published a series of polls for CPAC on its Web site – we chose four, to avoid redundancy.
Media Compliance with Subsections 326(1) and 326(2) of the Canada Elections Act
|Media||Number of articles||Polling firm (number of surveys)||Polling period||Sampled population||Number of respondents (National)||Margin of error (National)||Number of respondents (Regional)||Margin of error (Regional)|
|The Globe and Mail||7||Ipsos-Reid (7)||information present||Canadians (1/7)
Not published (3/7)
|information present1||information present2||0/7||0/7|
|The National Post/The Gazette||3||COMPAS (2)||information present||Canadians (2/3)
Not published (1/3)
|information present||information present||0/2||0/2|
|La Presse/The Toronto Star3||5||EKOS (4)||information present||Canadians, 18+ (3/5)
|information present||information present||2/34||1/3 (Que.)|
|Le Journal de Montréal||3||Leger
|information present||Canadians (2/3)||information present5||information present||2/3 (Que.)||2/3 (Que.)|
|information present||Adults (1/1)||information present||information present||0/1||0/1|
|SES Research||47||SES Research (4)||information present||Not published (4/4)||information present||information present||1/1||1/1|
|Environics||1||Environics (1)||information present||Adult Canadians (1/1)||information present8||information present||0/1||0/1|
|Media||Number of articles||Wording of question||Information on access to reports||Insert on methodology|
|The Globe and Mail||7||7/7||0/7||0/7|
|The National Post/The Gazette||3||1/3||0/3||0/3|
|La Presse/The Toronto Star3||5||5/5||0/5||0/5|
|Le Journal de Montréal||3||2/3||0/3||0/3|
1 The Globe and Mail also systematically mentions the number of people who answered the question on voting intentions.
2 The Globe and Mail also mentions the specific margin of error for the question on voting intentions in three of its articles.
3 Four articles in La Presse and only one in The Toronto Star were analyzed, since the articles in the two papers were equivalent.
4 One of the articles mentions the size of samples from Quebec and Ontario; the other mentions Quebec only.
5 Le Journal de Montréal also mentions in one of its articles the specific number of people who answered the question on voting intentions.
6 Name of survey sponsor not specified.
7 Four SES Research polls were chosen, one per week, to avoid overlapping samples.
8 Enviroincs also mentions the number of respondents eligible to vote, that is, Canadian citizens aged 18 and over.
Table 1 shows the level of media compliance with subsections 326(1) and 326(2). The 24 articles selected are those that presented the surveys for the first time. Some of the information required by subsection 326(1), such as the names of the polling organization and sponsor, the survey period, the total number of respondents and the margin of error for the national sample, is now automatically presented by most media. The information provided on the margin of error – except in The Globe and Mail – is the maximum margin of error (at 50%) for the entire sample and not for the sample of people who indicated their voting intentions. However, non-response to the question on voting intention varies between 7% and 26%, substantially modifying the margin of error. The information on the sampled population is presented, but in very succinct form (adults, Canadians).
Regional voting intentions are presented in 18 of the 24 articles. The spirit of the Act calls for information on the sample size and margin of error to be reported at the same time. With a few exceptions, neither the size of the regional samples nor the associated margin of error was reported. When they are included, they are reported only for Quebec (Le Journal de Montréal and an article in La Presse) or for Quebec and Ontario (an article in La Presse). A single article, on the SES Research site, gives the size of all the regional samples and the associated margins of error. Voting intentions in various regions were thus the subject of much media comment about the regional nature of party support, but with no possibility of learning the margin of error for the data.
|Pollster||Sampling method||Initial sample size||Response rate||Refusals and eligibility||Time of day of interviews||% of non-responses/ % undecided||Treatment of non-responses||Weighting and adjustments|
|Random representative sample (6/7)
Random sample (1/7)
|–||Region, age, sex (4/4)|
|Representative national sample (1/2)
Representative sample (1/2)
|1/2||Region, age, sex (1/2)1|
|Random sample (2/4)
Random stratified sample (1/4)
Undecided and refused 3/4
|–||Region, age, sex (4/4)3|
|3/4||Age, sex, mother tongue (3/4)4
Sex, mother tongue and region (1/4)
|Random sample||–||–||–||Evening||Non-responses 3/4
1 Report mentions that the data come from Statistics Canada, without specifying the exact source.
2 Panel consists of follow-up interviews with all the poll respondents, June 7 to 9.
3 Report does not mention the year of the census data used.
4 Report mentions that the data come from Statistics Canada, without specifying the source.
As for compliance with subsection 326(2), four out of 24 articles did not include the wording of the question and, unlike what happened in 2000, a single article, from the Canadian Press, informed readers of the availability of a detailed report and how to get it. During the 2000 election, most of the media referred readers to the polling firms' Web sites. Accordingly, we checked to see if those Web sites did post the information required by subsection 326(3). Table 2 shows the available information. The information normally presented bears on the sampling method, the proportion of non-disclosers or undecided respondents and the adjustment used. Leger Marketing and Environics do not specify the sampling method they use. The information provided by the other firms is scanty: it does not indicate whether unlisted numbers are included in the sampling base or the selection method used within households. As for adjustment, two firms, SES Research and Environics, provide no information. EKOS does not explicitly mention the year of the census data used for reference, while Leger Marketing and COMPAS mention only that the data comes from Statistics Canada. The information on the response rate is normally missing, Footnote 8 except for Leger Marketing's most recent survey. Only Leger Marketing (three out of four surveys) and COMPAS (for a single survey) explicitly mention how cases of non-response are dealt with, and only SES Research mentions the time of day they conducted interviews.
Finally, with respect to regional voting intentions, the proportion of non-responses within the samples is presented for the most recent surveys done by Ipsos-Reid, COMPAS, SES Research (undecided responses only) and Leger Marketing, but not for the earlier polls.
The conclusion to be drawn from this analysis is that, overall, the media respect the formal provisions of section 326 in their publications, except for mentioning the availability of a detailed report. However, the information related to the response rate required in the reports normally provided by the polling organizations Footnote 9 is notably absent, with one exception. Moreover, the information available is sometimes scanty and the basic information needed to judge regional voting intentions is very often absent.
Looking at the general quality of the published information, two points deserve mention. One, it is sometimes very difficult to find the methodological information in articles. It can even be presented as a comment by the pollster. Only Le Journal de Montréal, SES Research and Environics present the methodology separately in an insert. Two, Lachapelle was already pointing out, in 1991, the poor quality of the graphics illustrating changes in voting intentions. That situation has not improved. None of the graphics illustrating changes in voting intentions, over whatever time span – the campaign, a few months, since the last election – respect the timeline. Thus, the points representing voting intentions in June 2000, in January 2004, in April 2004 and in three surveys done in May 2004 may all be equidistant. This type of graphic gives a false picture of the trends in voting intentions and can mislead the reader.
Although the situation is not catastrophic, Elections Canada could help make life easier for everyone – the media, pollsters, readers and experts – by reminding the media of the provisions of the Act during the next election and offering a model presentation, such as an insert on methodology for publication purposes and a model report for the polling firms. It would also be a good idea to specify that the basic information must be presented for all the voting intentions indicated, including voting intentions by region. The Act could, moreover, be amended to require that information on undecided respondents, which is essential, be presented in the articles rather than in the detailed reports. Finally, the media should work on improving the graphic presentation of trends in voting intentions, which is clearly deficient.
Return to source of Footnote 1 Guy Lachapelle, "La réglementation des sondages au Canada et l'élection de 1993," Revue québécoise de science politique No. 27 (Spring 1993), pp. 175–201.
Return to source of Footnote 2 This probably refers to the fact that, theoretically, the margin of error applies only to strictly random samples, as opposed to quota-based samples.
Return to source of Footnote 3 The 1993 Act provided for a ban on the publication of opinion survey results during the final three days of the election period.
Return to source of Footnote 4 Guy Lachapelle, Polls and the Media in Canadian Elections: Taking the Pulse, Vol. 16 of The Collected Research Studies, Royal Commission on Electoral Reform and Party Financing (Toronto: Dundurn Press, 1991), 182 p.
Return to source of Footnote 5 Lachapelle, "La réglementation des sondages au Canada et l'élection de 1993," p. 182.
Return to source of Footnote 6 Robert M. Andersen, "Reporting Public Opinion Polls: The Media and the 1997 Canadian Election," International Journal of Public Opinion Research Vol. 12, No. 3 (Fall 2000), pp. 285–298.
Return to source of Footnote 7 Claire Durand, "The 2000 Canadian Election and Poll Reporting under the New Elections Act," Canadian Public Policy Vol. 28, No. 4 (2002), pp. 539–545.
Return to source of Footnote 8 The paper La Presse tried to understand why there were noticeable differences in the results of two polls it had commissioned from EKOS and CROP. Reporter Denis Lessard ("Les écarts entre les sondages," La Presse, June 12, 2004, p. A34) thus obtained and published the response rates for these polls, suggesting a difference between the rates as a possible explanation for the differing results. After that, the response rate for the CROP poll done in Quebec only was also published.
Return to source of Footnote 9 Strictly speaking, however, it is the media's responsibility to see that the information is available.
The opinions expressed are those of the authors; they do not necessarily reflect those of the Chief Electoral Officer of Canada.