Political Polling: The German Shepherd Factor
Wheeler, Michael
Political Polling: The German Shepherd Factor by Micahael Wheeler Pollsters have been able to insulate themselves from criticism by constructing a barricade of statistical and scientific...
...If you would not, then you must discount the results of the poll, no matter how conclusive the statistics seem...
...The house-tehouse surveys taken by the major pollsters are getting longer and longer...
...The liars are lumped together with the truth-tellers...
...Irwin Harrison, of Decision Research in Massachusetts, thinks that this is an illusory advantage...
...In one case the Times reporter learned the interviewer had talked to four people playing cards and incorporated all their answers into one interview...
...You just don’t need surveys that long...
...They play the expert, but that’s wrong...
...George Gallup states, “Almost all our interviewers are women...
...Women have always been better interviewers...
...Albert Sindlinger is a self-styled maverick within the polling profession...
...If Harris’ firm had not erred by leaving several questions out of the original poll, the fabricated interviews would have slipped through...
...Ask the other pollsters if they ever listen to their interviewers,” he says...
...The late Oliver Quayle candidly stated that the best ones are “not too intelligent so they will not get too curious and involve themselves in survey analysis...
...Diane introduced herself, though with the over-sized button she wore on her blouse, that was hardly necessary...
...There are other criteria for interviewers...
...Goldwater ultimately did somewhat better than the pollsters had reported, even those using secret ballots, which indicates that even more people were refusing to disclose their actual feelings to the pollsters...
...And the same is true for other firms all over the world...
...The Digest’s sample was based in part on telephone books, which at the time made the sample heavily biased toward Republicans...
...Another successful pollster, who asked not to be identified, says he knows several former Gallup and Harris interviewers who have admitted to him that they faked interviews regularly and were never caught...
...The man with whom she lived (the poll had categories only for “married” or “single”) was a truck driver...
...Judged by Diane’s standards, none of the neighborhood looked very promising, but she managed to find a street where there were a few cars...
...Other pollsters who do personal interviewing follow similar practices...
...The woman interviewed by Diane Bentley was unemployed...
...People are really less reluctant to talk with a woman, and women are much more conscientious...
...Someone who says she is a pollster may actually be with the FBI or the Internal Revenue Service...
...We are reluctant to do it, and so are our interviewers...
...Even if a poll is based on perfectly constructed samples and even if the results are scrutinized by a sophisticated analyst, the results will be meaningless if the questions were simplistic...
...An older man answered Diane’s knock on the other, and she was pleased, as it is harder to find men at home during the day...
...In a number of recent surveys for various Democrats, Pat Caddell has regularly asked people for whom they voted in 1972, Nixon or McGovern...
...Another group that may be underrepresented is owners of German shepherd dogs...
...Only sheer luck prevented them from printing it...
...The client insisted on getting the information it had paid for, so Harris’ firm sent out a special questionnaire by mail to those people who had supposedly been interviewed...
...Newspapers almost never doublecheck polls they print, but the Gallup incident suggests that perhaps they should...
...Phonv Interviews The pollsters claim they are very careful about whom they hire and that they summarily fire anyone who fabricates data, yet the very nature of personal interviewing means the pollsters have little control over their employees in the field...
...Harris notes that complete information about his published surveys is on file with the University of North Carolina and that his firm has always made it a practice to disclose wording or questions and other information to anybody who requests it...
...I just don’t think that way...
...Diane Bentley approached a woman returning from the supermarket with two large bags of groceries, who said she would be willing to talk for a little while...
...This practice of piggy-backing allows the pollsters to spread the costs of interviewing among several clients, but it also means that the interviews are long and ponderous...
...In practice, however, certain types of people are much easier to find than others...
...More than 90 per cent of the households in the country have listed phone numbers...
...That is no longer a real problem today...
...But that answer, as she could plainly see, would be taken as an endorsement of Kissinger...
...If polling is a science, it is at best a crude one...
...Consider some of the questions Diane Bentley was asking...
...I have an asset, I can listen to every damn interview we make...
...Diane did not mention that the survey would take more than an hour...
...Both before and after the incident, Gallup has repeatedly made the claim that his firm validates every interview made by new interviewers and every fourth or fifth interview done by one of their experienced employees...
...Some firms tell their interviewers to start at a certain house and work in a particular direction, but this day she was simply given a map of the neighborhood and told to begin wherever she wanted...
...Diane persisted, saying it was important to express one’s opinion and the whole thing was strictly confidential, but the woman still said no...
...The occupants, one guessed, were conservative in their politics, but no one answered the door, so whatever their views, they went unrecor de d. Next door was a rather run-down, two-family house...
...Ordinarily it is difficult to measure the extent of lying that goes on in issue polling...
...Originally, she did it principally to get out of the house, since her children were both well along in school...
...No analyst, no matter how sophisticated, could divine the woman’s real views from the “not sure” recorded by Diane...
...On the other hand, she could not bring herself to say “yes,” as that implied she had until recently supported him...
...Access is the big thing...
...As helpful as that may be to serious scholars, it does little good for the average reader, who is left in the position of having to trust Harris’ judgment in interpreting the meaning of his surveys...
...Obviously that was not done in this case...
...The door had two locks on it and a small decal which said the premises were protected by Lectronic Alarm Systems...
...Therefore, we feel respondents are exaggerating the amount of schooling they’ve had...
...Sindlinger believes that there is a paradox which undermines personal interviewing...
...We publish the questions, tell how they were collected, and let the reader draw his own conclusions from the facts...
...To the extent that other pollsters have felt this sort of criticism, their usual response has been simply to ask more questions, not better ones...
...The major pollsters agree the most important criterion is sex...
...Her pleasure was brief, however, for the man spoke so little English that an interview was impossible...
...To the same end, many pollsters affect the same pretensions assumed by faith healers, astrologers, and other quacks: it does not escape our notice that we are dealing with Dr...
...Pollsters behave as though opinion can only be located through asking questions, often artificially narrow ones, but in fact we express ourselves in many ways...
...Gallup’s American Institute of Public Opinion in B-inceton, New Jersey...
...According to Pat Caddell, the post-election studies which Yankelovich did in 1972 for The New York Times were unsuccessful in locating the full extent of Wallace’s support...
...As the crime situation actually gets worse in some cities, it becomes more and more a problem to send interviewers into areas where the crime rate is high...
...I look for parked cars, toys in the yard...
...Unfortunately, these clues to our attitudes and opinions are almost always lost on the pollsters...
...Open-ended questions which ask people to state their views in their own words allow each individual to express himself accurately, but the very diversity of expression which is produced necessarily means that it cannot be tabulated into neat “for” and “against” columns...
...Most pollsters imply they have their own special force of field workers, but in fact many interviewers like Diane work for two or more polling firms...
...Even as people walked out of the voting booth, they still were reluctant to admit they had voted for Wallace...
...Nothing seems to be wrong with our samples, and there is no indication of error in recording or processing...
...An editor was so pleased with the poll he decided to play it up by sending a reporter and a photographer to get a story about some of those who had supposedly been interviewed...
...To do objective research, the interviewer has to be the dummy, not the other way around...
...Burns Roper says that perhaps 98 per cent of his firm’s interviewers are female...
...The next time you read what purports to be a scientific survey of public opinion, recall what really goes on as interviewers like Diane Bentley move from door to door...
...Diane was pleasant but persistent...
...Most of the polling establishment, including Gallup and Harris, regards the use of the telephone as the sign of a minor-league operation, claiming it has many built-in drawbacks which make it unreliable...
...The way a person lives may tell much about his attitudes...
...she said she had too much house work to do before her daughter came home from kindergarten...
...People get tired talking...
...It is much easier to hang up a phone than to get someone out of your living room, particularly when they have been trained how to stay long after they have worn out their welcome...
...Now, however, with family money tight, the pay is an important incentive, so she takes on any polling assignment she can get...
...To follow her as she makes her rounds is to peer behind the wizard’s curtain and see polling as it really is...
...Moreover, five other people who had allegedly been polled could not be traced-the addresses existed but apparently the people did not...
...The question, typical of many on that survey, illustrates a serious problem common to much of public opinion polling: complex attitudes are artificially forced into neat little boxes-agree/disagree or favor/oppose...
...It is true, for example, that in the past the rate of refusal was higher for telephone interviewing than for that done in person...
...Both those considerations would seem to be essential to any intelligent response to the question, yet the pollsters demand a simple yes or no...
...Pat Caddell, for example, prides himself on the depth of his interpretation of his surveys, but his analysis can be no better than the questions he asks...
...She said she had never liked Kissinger...
...That people feel the need to be deceptive is of interest in and of itself, but unfortunately from the bare statistics of a public opinion poll, there is no way to tell who really believes what...
...In zeroing in on simple agree/disagee questions, the pollsters often ignore other indicators of opinion which may be much more revealing...
...Ask Me No Questions Sindlinger says that he has occasionally hired Gallup and Harris interviewers to do his telephone canvassing, and from this experience he has a very low opinion of them...
...Harrison has found that in a 15-minute telephone interview, he can ask 35 or 40 questions, which will produce more than enough information for most clients...
...Diane began working as a polltaker several years ago...
...Many pollsters are reluctant to reveal the trouble they have getting to talk to people...
...If you have...
...and on many issues homebodies tend to have different opinions than those who are on the move...
...Though the woman’s interest flagged after 45 minutes, Diane kept firing off questions, carefully recording a response for each one...
...The surveys which Harris, Caddell, and others conduct often take well over an hour to administer...
...Choose whichever is the lesser of two evils...
...Diane’s sample included one...
...Harris conducted a private survey in which several questions which had been contracted for were inadvertently left out...
...Recently, however, personal interviewers have encoun tered an increasing number of locked doors, so now the telephone produces an equivalent level of cooperation...
...That means that it’s morb,@cely people are around and I don’t have to waste time at empty houses...
...He simply does not trust interviews done without supervision...
...Not all interviewers are so diligent...
...Given Gallup’s experience with the Columbia Law School students in Harlem, Sindlinger may be right...
...Diane faithfully carried out her assignment...
...A year later Gallup’s rival Lou Harris had exactly the same problem, but Harris was more fortunate in being able to keep his client from knowing it...
...Whether it’s a commercial client or a political client, they use only a fraction of the data you give them...
...Checking their responses is simply too difficult when 100 or more questions have been asked, and after all, the pollster may not really care if the responses are legitimate as long as they are plausible...
...In 1968 The New York Times commissioned Gallup to do an intensive survey of attitudes of Harlem residents...
...Gallup himself says, “We regard ourselves simply as fact-finders...
...In 1964 Elmo Roper discovered that a significant portion of people would not say outright they intended to vote for Barry Goldwater...
...If people lie to pollsters about how much money they make and are not always truthful about whom they will-or did-vote for, they also are likely to be coy about giving their true feelings on controversial issues...
...Gallup is concerned that some urban people may become inaccessible...
...A few identifying details have been changed to protect her anonymity...
...Other pollsters are more guarded on this point, for they understandably do not want to slander their own employees, but it is clear that you can be considered “over-qualified” for public opinion research...
...Most pollsters try to hire people who will feel comfortable in various kinds of neighborhoods, but problems still persist...
...There was no explanation, however, why the Gallup organization had not discovered the fakes themselves...
...Caddell is hardly alone, however, in forcing people to choose between two positions, neither one of which they support...
...This is the big argument I have with Burns Roper and others in our field...
...Diane’s real name is not used here...
...Diane moved crisply through the questionnaire, politely but firmly insisting on specific answers to the questions she put...
...The Gallup surveys commonly start with questions which will be used for his newspaper column, and then go to matters which are being probed for one or more private businesses...
...McGovern won by an eight-point margin...
...She felt trapped by a question which asked her to agree or disagree with the statement, “Since - Henry Kissinger failed to make peace between Egypt and Israel, it looks as though he is losing his touch as a peacemaker...
...Nor can you qualify your answer by saying your support of energy independence depends on how much more energy will cost when it is produced solely from domestic sources...
...Chivalry isn’t dead-men will be more responsive to a woman than they will be to a man...
...Diane tried to explain what she wanted, but her attempt left him confused and Diane embarrassed...
...Pollsters try to control the problem by hiring interviewers who will have the greatest chance of winning people’s trust...
...He is also justifiably suspicious of the information which personal interviewers elicit toward the end of interviews...
...The practice is a far cry from what the pollsters would have us believe...
...Diane followed the woman into the kitchen and sat down with her clipboard to conduct the interview while the woman put away her groceries...
...Just answer the question as best you can...
...Diane Bentley” does interviewing for both Louis Harris and Pat Caddell...
...Polling theory in fact is not complicated, and to understand it is to know just what polls can and cannot do...
...Michael Wheeler teaches at the New England School of Law...
...Housewives, retired people, and the unemployed may be found at any hour, but young working people are hard to track down...
...Caddell says, “California was the first state in which we found this...
...People either say that they are undecided or that they are going to vote for someone else...
...When the theoretical precepts are not scrupulously applied-and they seldom are-the results become all the more shaky...
...No matter how polling is conducted, in person, by telephone or by mail, the pollsters have to take on faith that people are being honest with them...
...He pointed out the interviews in question had been submitted by two Columbia law students who had been specially hired for the project...
...Everything you’re asking me is yes or no, black or white...
...The first house she tried was surrounded by a chain link fence...
...Would you like to see the U. S. become totally independent of all foreign sources of energy, or not...
...It showed a half-dozen greedy and bloated figures, each one representing a major oil company...
...Unfortunately, not all pollsters follow this procedure...
...Only the person who is answering the poll knows whether the opinion he expresses is truly his own...
...Tell Me No Lies There is other evidence that people do not always tell the truth to polltakers...
...Political Polling: The German Shepherd Factor by Micahael Wheeler Pollsters have been able to insulate themselves from criticism by constructing a barricade of statistical and scientific jargon...
...Diane’s manner is naturally friendly, and three years of work as a poll-taker has polished her ability to put people at ease...
...Diane crossed the street to talk with a woman who was hanging up her wash...
...W. Norton in June...
...As the interview went on, however, the woman got increasingly impatient...
...Not even all the remaining interviews were legitimate...
...There were four posters taped on her kitchen walls...
...The pigeon-hole effect is the result of the polling mechanism...
...The woman still could not subscribe to either of the two offered alternatives...
...In large part the hostility to the telephone seems to be a legacy of the Literary Digest prediction that Landon would defeat Roosevelt in 1936...
...The Ontology of the Fact The Gallup Poll is careful to include the wording of all questions in the releases...
...If the pollster sets out to report public opinion in some unified and comprehensible fashion, he must get answers which fit into categories which can be easily tallied...
...The Times tracked down one of the special interviewers whom Gallup had tried to make a scapegoat...
...Harris usually does not supply the wording of the questions which are asked for his syndicated column, and he is sensitive about criticism that he should...
...In some states which Nixon actually won, a majority now say they voted for McGovern...
...When questioned on this they generally say that “people are most cooperative...
...And women aren’t afraid of another woman, as they might be of a man coming to the door...
...People who are willing to be interviewed get nothing but the satisfaction of knowing their responses will fractionally affect the totals spewed out by the pollster’s computer . Diane did not know who was the principal client for the survey she was conducting, but most of the questions dealt with energy problems, which made her guess that the client was either an oil company or a public utility...
...Lou Harris similarly reports that over 90 per cent of his house-tehouse interviewers are women...
...He has all his interviewing done by telephone so that he can monitor his employees’ work...
...A revealing statement recently appeared buried deep within an instruction circular the Harris firm prepared for its interviewers: “It’s been brought to our attention that almost all of our surveys are showing the p,opulation to be more educated than what the census says it actually is...
...Look at the newspaper polls...
...Were it not for their desire to do long interviews, more pollsters would probably use the telephone...
...All the questions are ‘agree/disagree.’ Well, life isn’t that simple...
...The problem pollsters are least inclined to acknowledge is cheating by their interviewers, but it occurs fairly often...
...She said she just wanted “your reaction to these questions-it doesn’t have to be something you would stand by...
...People are apparently somewhat more honest in assigning themselves to a category than they are in stating their education outright...
...Were she to take the question literally, she must answer no, for she believed he never had a touch as a peacemaker...
...Well, all our columns are based on eight or ten questions and there is no way to get all that information in with a word limit of 800 or 850...
...The first questions asked whether there was an energy shortage, either locally or nationally, and whether there would be one in the future...
...If anything, Caddell’s questions tend to be even more restrictive than those of his major rivals...
...We sent him a telegram saying, ‘Demand a recount!’ ” If people do not always tell pollsters the truth about how they voted in the past, then we must suspect what they say they are going to do in the future...
...Well, I think the gas crisis was manufactured by the companies...
...On a Tuesday morning in June, Diane went to a working-class section of New Haven, Connecticut, where she had to complete ten interviews...
...Even a public opinion poll with a perfect sample and unbiased questions has inherent limitations...
...A typical Harris interview takes more than an hour...
...Other studies have confirmed that, when asked, most people will say they are somewhat better paid and more highly educated than they really are...
...You cannot say that you are more concerned with dependence on Middle Eastern oil than with Canadian sources...
...It was accidental that the Times uncovered the specious poll...
...If pressed, however, Louis Harris admits that upwards of 20 per cent of the people his interviewers contact refuse to talk, and this figure does not include those who are not at home...
...That faith may often be misplaced...
...Harris questionnaires can run up to 30 pages with perhaps 200 questions...
...Caddell’s polls are usually commissioned by a single political client, but most of the other pollsters use one interview to ask questions for a number of subscribers...
...The pollsters have responded by making minor changes in their method of asking those demographic questions...
...When the Times confronted him with this fakery, Gallup ducked the responsibility for the phony interviews...
...If she had, the woman would probably have declined, as would most people...
...Nevertheless the woman refused to be interviewed...
...Whatever the reason, the kind of people who do not feel comfortable talking to strangers probably have different attitudes about themselves and the world they live in, yet their opinions are under-represented in public opinion polls...
...This article is adapted from his book, Lies, Damn Lies and Statistics, to be published by LiverightlW...
...Diane eventually put her down as “not sure,” when in fact the woman had a clear and strong opinion about Kissinger...
...Big Dogs and Blacks One after-the-fact explanation for the pollsters’ blunder in predicting that Dewey would win the election in 1948 was that many of their interviewers stayed out of the black neighborhoods in the Northern cities, where the vote went heavily to Truman...
...If I get to choose, I like to drive around the area a bit...
...Like Sindlinger, he, too, now relies principally on telephone interviewing...
...The fourth poster said more about her attitude toward the energy crisis than her answers to any of the questions which Diane asked her...
...Agree/Disagree This pigeon-holing is a fundamental weakness of public opinion polls...
...Also, there are door-to-door salesmen who introduce themselves as poll-takers...
...This kind of background never finds its way into polls, and as a result, the full texture and shape of public opinion goes unreported...
...In theory every person in the United States must have an equal chance of being selected in a sample if the survey is to be reasonably representative...
...Cross-tabulations,” “area random sampling,” “flowcoding”the language sounds so technical, so formidable that most of us are intimidated into believing we are not qualified to judge whether public opinion polling is a legitimate science...
...They can’t...
...It isn’t just ‘excellent, good, fair or poor.’ It makes cleaner newspaper copy to make everything ‘agree/disagree’ but that’s an awfully simple way of looking at public attitudes...
...For example, Harris interviewers now hand people a card on which different levels of education are placed in numbered categories...
...In sum, though the theory of polling is scientifically sound, the actual practice is not...
...The language is blandly bureaucratic, but the fundamental meaning is nothing less than startling: The Harris organization admits that “in almost all” of their surveys, people have been lying to them...
...A simple test for determining the meaningfulness of poll results is to read the questions to see if you would be comfortable giving agree/disagree answers...
...She took one look at the dog and without hesitating walked to the next house on her list...
...people who have been taken in by this pitch may be wary of others who make the same claim...
...The information was collected, tabulated, and submitted to the Times for publication...
...The rate of refusal has gone up in recent years, a fact that leads some to speculate that Watergate and all its fall-out have caused people to be suspicious of anyone asking questions...
...Peter Hart, a young Washington, D. C. political pollster, is one of the few in his profession to even acknowledge the problem...
...Underneath, there was a caption: “Don’t blame the truckers, it’s these motherfuckers...
...Three were of pop music and television stars...
...all they’re thinking about is how they can get the interviewer out of the house without being rude...
...They usually only call back people to see if they were interviewed...
...The woman hesitated...
...Those done by Cambridge Survey Research, Pat Caddell’s firm, often last an hour and a half...
...Each individual agreeldisagree question is flawed, but most pollsters think that the total value of a survey can somehow be more than the sum of its parts...
...A more limited study, he believes, can help the client understand what the real issues are, instead of being overwhelmed with page after page of computer printout...
...You must answer yes or no...
...Diane checked the boxes that said there was no energy shortage...
...If you hire somebody with any intelligence, after the tenth interview they’re going to sit down and make up the other 25...
...According to a person who worked for Harris at the time, 25 per cent of the questionnaires were returned as undeliverable: there were no such people or addresses...
...Even the most conscientious pollsters are limited in how much they can do to check the validity of interviews...
...In primary polls, George Wallace’s strength has almost always been underestimated...
...Most pollsters say they routinely double-check a portion .of all their interviews to be sure they were really conducted, but incidents involving both the Gallup Poll and the Harris Survey indicate this validation process is not nearly so rigorous or effective as the pollsters would like us to believe...
...I know that it was really done and that the questions were asked the way they were supposed to be...
...Women are superior at getting cooperation from either sex...
...He says that we shouldn’t report a question unless we also report the wording and the tabular results...
...No one was home on one side...
...somebody who is too dumb to do that, they’re too dumb to record people’s answers...
...New polling firms spring up right and left, and each one seems to be called Survey Research Associates or some such name which conjures up images of white-jacketed computer technicians poring over print-outs...
...It would be very difficult to get people to stay on the telephone that length of time...
...When he told people to mark an ostensibly secret ballot, Goldwater invariably did four percentage points better than when they were asked directly for their preference...
...At seven of the 23 addresses Gallup had given them, the newsmen could not even find a dwelling...
...He claimed he had actually talked to people “in the streets,” but admitted he had made up the names and addresses because he was under pressure from the Gallup organization to complete his assignment-“I was uptight to get it in...
Vol. 8 • April 1976 • No. 2
