Report on the Evaluations of the 41st General Election of May 2, 2011
Appendix 2: Limitations of Surveys in General
This report draws on data from several surveys. While surveys are highly valuable research tools that enable Elections Canada to evaluate its performance and services, they are also subject to several limitations. Some of the principal limitations associated with survey research are as follows:Footnote 19
- Low response rates – It is increasingly difficult to encourage people to respond to telephone surveys, while those who do take the time to respond are in many ways different from people who do not. This "self-selection bias" makes generalizing the results of a survey risky.
- Cellphone-only households – It is very difficult or impossible to contact respondents who do not have a landline telephone. This problem is becoming more prevalent with the increase of cellphone-only households (from 8 percent in 2008 to 13 percent in 2010, according to Statistics Canada). These respondents tend to be younger, more politically informed and more likely to use the Internet, social networking and blogs.
- Small-case exceptions – Surveys are very useful for identifying large trends. For example, having 2,500 respondents for a particular question yields a 1.96 percent margin of error but cannot be used to understand exceptional cases, while having 25 respondents for a particular question yields a 19.6 percent margin of error.
- Small populations – For purposes of generalization, surveys must be adequately representative. This makes it difficult and more expensive to obtain accurate information on small populations or groups.
- Self-reporting nature – Surveys rely on the self-reported evaluation of respondents. What they remember and report cannot be controlled for.
- Social-desirability bias – Respondents can consciously or unconsciously give the answer that they think the surveyor wants to hear or that they think will put them in the best light.
- Oversimplified interpretation – As a result of using pre-constructed categories of responses (which speeds up the surveys), there is a natural tendency to oversimplify what respondents really think. This can be a particular problem when trying to explain complex issues.
Return to source of Footnote 19 For example, see W. Lawrence Neuman, Social Research Methods: Qualitative and Quantitative Approaches, 3rd ed. (Boston: Allyn & Bacon, 1997).