The Guilty Secrets Of An Ets Grader
Vopat, Jim
The Guilty Secrets of An ETS Grader by Jim Vopat The “table aide” set down 45 or so pink examination booklets to my right and collected the finished product from my left. Across the table...
...There is as much cafeteria food as one can eat, and for many teachers the $350 paid for their six days of gradingis nota negligible sum...
...To hear ETS tell it, the Advanced Placement Program is the pinnacle of American secondaryschooleducation...
...ARECURRINGTHENLE In order to achieve “quality control” and the uniformity that results, the ETS employs a subtle yet not so subtle system of rewards and punishments...
...While the table leaders checked on the grading performance of their readers, the question leaders checked the table leaders...
...Besides, why worry...
...An estimated 20 percent of the 22,000 American secondary schools participate in the program, which culminates in the yearly exam...
...I asked for a ruling from the question leaders as to how the examination should be treated...
...The successful high school participant gains college credit and/or advanced placement, not to mention saving thecost ofup toafullyear’scollege tuition...
...ter of the novel...
...to be baron...
...Moreover, on the final day of each year’s scoring session, each reader fills out aform requesting that a “letter of commendation” be sent to his or her academic dean, and it is a letter predictably full of praise for a job well done...
...was scrubbed...
...Robert Jameson, who is considered the architect of Advanced Placement, wrote in a book about AP tests that “one English examination included a question which asked candidates to write a unified essay commenting on three of five prose passages...
...Within the six days of the I980 scoring session, for example, 3 18 readers graded 180,328 essays written by 45,082 candidates for Advanced Placement in English...
...At the head ofeach table presided the table leader, whose job was to ensure that the grading reflected “qualitycontrol...
...others keep personal records on their scoring patterns from day to day...
...they have read over previous Rubrics...
...In more than one sense of the term, we must all admit that the AP students are “gifted...
...Everything is under the professional care of the Educational Testing Service...
...Only 3.3 percent of all high school students taking the Enghsh Advanced Placement exam failed to be judged “qualified” or better...
...He supplied them with a bowl of paperclips tomakeit easier...
...After all, the candidates have been in training an entire year for this particular examination...
...The students comply, and it is profoundly disheartening to read an identical essay written all over the United States,especially when these are supposed to be the brightest high school students in the nation...
...A 10-or 15-pageshortstoryby Ray Bradbury has the same testingvalueas Crime and Punishment...
...I once objected to approving an essay on The Hobbit, only to be immediately overruled...
...Eliot’s Silus Murner, Nathaniel Hawthorne’s Antigone, and Sophocles’s Orhello...
...Most of the high school candidates taking the AP exam have been trained in the correct response...
...The air conditioning is on high because cold readers have been found to read faster...
...At the front of the room were the “question leaders...
...58695 7) “A recurring theme in literature is the classic war between passion and responsibility...
...it’s an entire curriculum designed to promote college-level achievement while the student is still in high school...
...Of the 41,975 candidates in 1979 only 1,385, or 3.3 percent,failed to bejudged a degree of “qualified...
...AP is more than a test...
...Consider these opening sentences from the 1980 public checkings: “A recurring theme in literature is the classic war between a passion and responsibility...
...Then there are the public humiliations in the form of periodic “checkings...
...Although the examination questions clearly stipulate that candidates write ona “novel or play of recognized literary merit,” in practice no distinctionsare made...
...A“sp1it”is when one readergivesan essaya 5whileanothergivesitan8...
...for safety...
...Instead ofdrawing all that attention and making the question leader go to all that trouble, it is easier to resolve “splits” oneself by glancing quickly under the white label...
...Not that it makes much difference once the test begins...
...In a world of diminishing enrollments, there must be some incentives...
...The best technique is a paperclip or top of a Bic pen...
...This grading fest is unparalleled in American education...
...As soon as the idea of not giving academic credits for classes students don’t take is mentioned, someone important in the admissions office throws a fit...
...Within the narrow confines of standardized education, that may even be true...
...The original rubric stated that a non-unified essay would be penalized...
...The grading standard is an official “Rubric Scoring Guide,” which itemizes what “qua1ifies”a student for advanced placement...
...Not only did the Educational Testing Service expressly forbid it, but the edges of the labels could cut your fingers...
...The gradershigh school and college teachers-are screened carefully by the ETS, and many (especially the high school AP instructors) have a vested interest in the continued growth of the program...
...The trouble is that formulaic essay-writing and assembly-line gradingdon’t often have much to do with theclear thinking and writing one should expect in collegelevel work...
...The test is a combination of essay and multiple-choice questions, and its purpose is to measure the student’s command of introductory college work...
...The ETS operates the only grading standard I know of in which words like “weak”and “distort” are used to define a positive intellectual achievement...
...M 26122) “A recurring theme in literature is the internal struggle between a passion and responsibility...
...But the ETS, which needs a good success rate in order to encourage students to plop down $38 per test the next year, justifies this percentage by pointing out that such impressive scores should be expected from such bright students...
...I have seen many grades crossed out and changed aftersuchaclandestineconsultation...
...These are the best high school students in America, the ETS directors intone at the pep rally before the mass gradings begin, and we are the finest, most committed teachers the nation has to offer...
...Sometimes, when the white labels pop off...
...they have been repeatedly exposed to the “Rubric,” previous questions, and sample essays...
...The Advanced Placement Examination is the Educational Testing Service’s flagship account...
...they have attended special lectures on the nature of the exam...
...But at least The Hobbit is a book...
...Two rows of reading tables filled the auditoriumit was another record-breaking year of more examinations, more readers, and more college credits...
...Candidates are more than willing to fill in the 40-minute time period with almost anything, and what I witnessed was a three-year succession of sentences so innocent about the nature of writing and literature that theybecamecomiconeliners: “The theme of Hamlet is innocent until proven “Captain Ahab decided to take his crew ashore “Obviously Anna Kerenina is the main charac“Henry VIII’s wife was childless and appeared guilty...
...But if you think the teachers have been turned into Sperry Univacs, try reading some of the essays by these bright young students...
...By the device of the scoring guide, even admittedly “very poorly written” essays are encouraged into the middle range of qualified scores...
...Something called the PREDIC system, one hears, can tell what grade you are going to give before you even give it...
...The reason readers label-peek may be that the AP scoring session makes them feel insecure about their ability to discriminate good writing from bad...
...Readers also label-peek because they are afraid their score will cause a “split...
...So why don’t colleges just stop granting advanced credits for course work not done and thereby make more money, employ more professors, and provide a full four years to students who could surely use it...
...My “table leader” said that the ETS did not engage in such subterfuge...
...K 18505) “A common theme in literature is the internal struggle between a passion and a responsibility...
...Two tables away there was a round of peek-aboo scoring going on-readers prying up the white labels befoie recording their scores to see what the previous scorer had given...
...There are also essays on T.S...
...Grading of the examinations is accomplished during a sixday “scoring session...
...There was a sudden expectant pause in the shuffle of paper, followed by the sound of chairs scraping against the linoleum floor...
...At these sessions the chief reader, the ETS statistician, the program director, and various consultants determine the number of “extremely well qualified,” “well qualified,” “qualified,” “possibly qualified,” and “no recommendation” scores...
...In 15 minutes the bell sounded again, and we returned to our respective places at our respective reading tables and started grading the Advanced Placement Examinations all over again...
...This is certainly a very novel approach to evaluation, in many ways more novel and inventive than peek-a-boo scoring...
...It’s on Portrait of the Artist as a Found Man,” the reader told the rest of us, “and there can’t be two of those...
...Based on these results, errant graders are admonished to lower or raise their scores, no matter how valid are the reasons for disagreement...
...They have rehearsed questions...
...The rewards are basically food, money, status, and the illusion that, as a reader for the ETS, one is promoting a “superior educational experience...
...But by passing three tests, a candidate can earn a full year of college credit, saving as much as $10,000 to $12,000 if he or she attends one of the more expensive colleges...
...Someone had changed the backing and returned the booklet to her, she said...
...After a few eye-glazing days, a number of readers lose entirely their ability to discriminate...
...Splits” have to be read a third time by the question leader, and there is a fear common among readers that if you come out on the wrong end of too many “sp1its”you won’t beinvited back to Rider College the next year...
...Jim Vopat teaches English ai Carroll College...
...during one of these private consultations, readers play around by pasting the label across their forehead-like white bandage to show where the operation took place...
...the bell-ringing, physical discomfort of the reading situation-the eyestrain, the headaches, and the shared fatigue...
...they are familiar with previous essay questions...
...This modern definition of literacy is determined at what is termed a “cutoff session...
...But after 10,000 or so essays many readers begin feeling the need to confirm their judgments through a quick peek under the label...
...When it became clear that almost nocandidate had written anything that could be called unified, the word...
...What’s worse is that the official ETS Rubric Scoring Guide itself is practically designed to accommodate the deficiencies of the student writing...
...ETS, PHONE HOME At the 1980 reading, a high school teacher remarked that she could not understand why college professors returned year after year to the scoring session...
...What’s good for ETS and the pocketbooks of parents trying to cut college costs might not be so good for students...
...After all, she said, what thecollege professors were doing was literally reading themselves right out of their jobs as the “service”of the ETS manufactured academic credits for courses that will never have to be taught...
...Finally, the “reading bell” sounded, and the rush for the bathroom was on...
...Not only are students being told that they are “qualified” after having just demonstrated a desperate need for freshman English, but thevery teachers whocould haw taught them become unemployed because they don’t need the academic credits...
...In a checking, an essay that has already been graded by the table leaders is circulated and then compared with the reader’s grades...
...Some roll the booklet back on itself to pop up the label, others use clips and tops of Bic pens...
...Every June these readers are transported to Rider College outside Princeton and housed in dormitories, and every day for seven-and-a-half-hours they devote their energies to readingandevaluating the essay sections of the exam...
...If you’re the president of a small liberal arts college who’s spending half your time recruiting next year’s freshmen, this argument can sound especially convincing...
...That’s about 100 essays per day per reader...
...The question is asked, and when the students are found unable to answer it, the scoring guide is rewritten to either ignore the level of error or translate that error into a positive argument for recommendation...
...Such redefinition goes a long way to explaining the extremely high success rate of AP candidates in English...
...In fact, the Rubric’s middle range of encouragement includes those essays that set out and prove the opposite of what the examination question asks...
...Eventually the essay was returned to me with the instruction to “reward the essay for what it did well in response to the question...
...The punishments involve not having sufficient time to recover from one day’s scoring before beginning the next...
...Her name is in the title...
...J 65 160) “The personal psychological conflict between passion and responsibility tends to be classical in literature...
...At the 1980 scoring session, the table leader two rows over took pity on his readers because they were slicing their fingers prying up the labels...
...in 1980, the percent was 3.7 The 96-97 percent success rate may surprise you4specially during a period of growing concern about the ability of high school and college students to write and think clearly...
...Some start giving the same score at every public checking...
...The purpose of the checkings is to ensure that readers hold to the detailed prescriptions of the Rubric...
...The ETS places a white label across each score recorded on the back of the pink examination booklets to protect the reader from being influenced by already recorded scores...
...Rumors periodically sweep through the reading room about studies the ETS is simultaneously conducting against its own readers...
...Label-peeking was, of course, risky business...
...The answer is demographics...
...Across the table a fellow reader flipped a pink bopkletclosed withan exasperated gesture, observing that it wasanother essay on “that stupid thing by Thomas Mann...
...K49545) When the reader’s ability to distinguish between such essays diminishes, the need for a quick, reinforcing look under the label enforces itself...
...After all, enrollment is a competitive business and the college that can grant the most credits ahead of time has the edge...
...In fact, the stress is such that many readers become so confused and insecure that they regress to the most absurd of scoring stratagems...
...The reader at the end of my own reading table threw down an examination booklet and announced that she wouldn’t grade it because she already had graded it the day before...
...Readers often cheat by peeling back the white label that covers the grade given by an earlier reader...
...At the 1979 reading, I received an essay in which the student chose a movie for his “novel or play of recognized literary merit...
...The ETSestimates thatearning therequired gradeona single AP exam is worth between $300 and $700...
Vol. 14 • November 1982 • No. 9
