The Distinctions Pollsters Don't Make
Rappeport, Michael
The Distinction the Pollsters Don't Make by Michael Rappeport Public opinion polls on policy issues have become an unquestioned part of our political life. They are taken seriously as an...
...The second question was, “To the best of your knowledge, does any governmental agency regulate the profits and rates of your local electric light and power company...
...Our second example is from a poll done in early January under the direction of Warren Mitofsky of CBS News...
...We divided the respondents into three groups according to whether they answered correctly both, one, or neither question...
...From these data a politician might theorize that there was considerable hostility to the local utilities, and strong potential political support for public ownership of the power companies...
...Where the survey was taken, the entire geographic area studied was served by privately owned utilities...
...if people don’t know about the issue, they’ll say so to the questioner...
...The four alternatives offered were individual investors, or city, state, or federal government...
...The first poll was a regional study, containing a section on public attitudes towards electric utilities...
...On many issues-the Middle East among them-the very best informed people are those who are passionately committed to a side and who have a constant appetite for information on the subject...
...Two examples from recent surveys conducted by Opinion Research Corporation illustrate this problem...
...The key question in this section was, “Which of these groups do you think should own and operate the electric light and power company...
...This is certainly not to suggest that the better-informed opinions should automatically be taken more seriously...
...Perhaps it should be de rigeur that the reporting of polls on public policy issues should include some breakdown by defined levels of knowledge...
...They are taken seriously as an index of national sentiment in domestic affairs and, admittedly to a somewhat lesser extent, in foreign policy...
...The Distinction the Pollsters Don't Make by Michael Rappeport Public opinion polls on policy issues have become an unquestioned part of our political life...
...It is my experience that, startlingly often and over a wide spectrum of issues, substantial minorities or even majorities of people questioned do not have even the barest minimum of factual information on which to base an opinion...
...Two of these factual questions were used to classify respondents into three groups: To the best of your knowledge, approximately how much of the total oil used in the United States usually comes from the Arab countries...
...This attitude question is particularly interesting because it presupposes knowledge of Israeli...
...The chart shows a clear difference of opinion between those who are factually knowledgeable and those who aren’t...
...The distinctions are useful guides for interpreting polls, and a knowledge guide would be, too...
...Both of these questions are clearly relevant to forming an opinion as to who should own the local power company...
...A major element in the study was the attitudes toward public ownership of utilities...
...Even with these data stated, the public is more skeptical now than before about the objectivity of the survey questions themselves...
...Perhaps the bestinformed people of all said “don’t know” because the questions contained such vague terms as “the Arab god...
...Which do you think is true-are the Arabs out to destroy Israel, or are they only interested in regaining their land...
...Two of these illustrate both the low level of public knowledge and the impact which this general ignorance can have on attitudes...
...But breakdowns by sex and region don’t imply that one group’s opinion is worth more than another’s...
...Table 1 shows the proportions of each group which favored various forms of ownership and, for reference, the overall figures...
...In publishing polls, the press has grown more scrupulous about stating essential facts about the survey: when it was taken, and whether the interviews were done in person or over the telephone...
...The difference is greater than any of the differences which normally appear in such standard breakdowns as age, sex, income, or region...
...Their Michael Rappeport ir a vice president of Opinion Research Corporation and a contributing editor of The Washington Monthly objective is to word the question in such a way that people on both sides of a policy issue will agree that the survey is not “loaded” against them...
...Rather, it is the sharp difference of opinion between those who know and those who don’t know a few key facts...
...Others say the Arabs are not out to destroy Israel, that they are only interested in regaining the land lost to the Israelis in the 1967 war...
...This assumption is wrong, because the proportion of these responses is misleadingly low...
...The polls are not what some people once feared they would become : unofficial referenda that automatically determine a politician’s actions...
...Interestingly, not knowing that Israel had gained ground did not prevent people from answering “regaining the ground” as the Arabs’ goal...
...The first question was, “Which group on this card do you think owns the electric light and power company here...
...TABLE II ARAB INTENTIONS Number of Destroy Regain Bothor No respondents I s t - 4 Land neither opinion All respondents 1231 * 28% 52% 5% 15% Both knowledge One knowledge questions correct 351 41 % 45% 8% 6% question correct 328 34% 50% 5% 11% Neither knowledge question correct 320 15% 59% 4% 22...
...The questions themselves are not what is significant...
...The poll dealt with attitudes toward problems in the Middle East...
...The study contained several other questions dealing with factual knowledge rather than attitudes...
...It is human nature not to want to admit ignorance...
...Is it less than 20 per cent of all the oil we consume, is it somewhere between 20 per cent and 50 per cent of all the oil we consume, or is it more than 50 per cent of our total oil consumption...
...Since a public opinion survey is one test in which the subject is not penalized for guessing, respondents who have not thought the issue out-or perhaps even heard of it-may simply blurt out an answer...
...This failure to consider the “ignorance factor” leads to two wrong conclusions: it overstates the percentage of the public holding strong opinions on an issue, and it obscures how quickly public opinion could change if a certain issue dominated the headlines and the facts became widely known...
...It contained a fairly long series of attitude questions, interspersed with several factual questions...
...Thus the support for a fairly hard-line policy would appear to be much greater among the knowledgeable, but the overall results obscure this finding...
...Moreover, this analysis suggests that the facile assumption of widespread dissatisfaction with the TABLE I Number of %favoring % favoring 96 don’t Respondents investors government know All respondents 1546 52 36 12 Both knowledge questions correct 523 82 16 2 One knowledge Neither knowledge question correct 526 53 37 10 question correct 497 12 57 31 local utilities is probably wrong...
...and Based on what you know, has Israel mostly gained or mostly lost territory since it was set up in 1948, or has there been no change in Israel’s territory...
...Another question dealt with American views of Arab intentions: Some people believe the Arab countries are out to destroy Israel as a nation...
...Again, the differences based on factual knowledge are dramatic...
...As polling has become a more serious ingredient in our political process, the public and the press have become increasingly aware of the importance of interpreting survey data properly...
...Again the possible answers were private investors and the three levels of government...
...They are, more modestly, conceived as indicators which can help inform political decisions...
...Almost half of those who got both questions right believed that the Arab goal is to destroy Israel, while the figure drops to one in five among those who knew the correct answer for neither...
...The results: Individual investors 52% City, state or federal government 3 6% No opinion 12% Thus only a bare majority actually favored private ownership which prevailed at the time...
...Just about one third of the respondents fell into each group...
...There is an entirely different type of problem which we must also face if the polls are to enlighten us rather than mislead...
...These corrections are welcome, but they are far from sufficient...
...Before assuming that a poll reflects public opinion, we should find out just what proportion of the adult public has any opinion to reflect...
...Table 2 shows the results...
...The sentiment for public ownership is more a reflection of ignorance than a desire for a change...
...Those who take polls and those who rely on their results are also aware of the risk of biased questions, and they have tried to design and implement polls that present accurate reflections of public opinion...
...One’s natural assumption is likely to be that this problem is solved by the “don’t know” or “no opinion” categories on most published polls...
...This eagerness to answer often leads to survey results that distort public opinion...
...territorial gains, one of the two factual questions used for analysis...
Vol. 6 • March 1974 • No. 1
