Last fall my son, who is 11 and in the fifth grade at Park Forest Elementary in State College, made a short film about always doing your best for an all school assembly. It was a great success.
A couple months ago he came me with an idea for another film project. This time he wanted to make a movie at his school about the high stakes NCLB testing, in Pennsylvania called the PSSA. Specifically, he wanted to document how test was hurting his education and that of his school mates.
Then Ted read something about the Bartleby Project. I loved the short story about Bartleby when I was in school. Ted and Bartleby the Scrivener got me doing some research about the upcoming PSSA testing and the No Child Left Behind law. And what I discovered shocked me. I'd like to share what I've learned so far with the world here.
The problems are so much worse than a debate over whether high-stakes standardized testing is hurtingor helping our kids and our school. So much worse. And the solution is so simple. Option one works in PA. Option two can work anywhere.
The PSSA tests are prepared and scored by a private firm in Minnesota called Data Recognition Corporation.
The actual test booklets arrived in boxes last week all over Pennsylvania. Right now they are being sorted and stacked in every principal's office in every school. Parents have the right to review the tests before their children take them. At least that is the law in Pennsylvania. We just have to sign a confidentiality agreement. I reviewed the ones at my children's school on Monday.
The taxpayers of Pennsylvania pay $30 million every year to Data Recognition Corp (DRC) to develop, print and score these tests every year.
In May of 2009, the Pennslyvania Department of Education (PDE) signed a $200 million contract with DRC to create and administer the Keystone High School Graduation Exams.
The tests are scored by temporary workers with no training in education in what some have described as “sweat shop conditions.”
DRC annually hires 4,000 temporary workers at $11-$13 an hour to score these tests. Other testing firms do the same. April through July is test scoring time. A test scorer can spend all day scoring tests for DRC and then go home and score tests all night for Pearson. Dan DiMaggio, veteran test scorer, picks up the tale:
“Wait, someone scores standardized tests? I thought those were all done by machines.” This is usually the first response I get when I tell people I’ve been eking out a living as a test-scoring temp. The companies responsible for scoring standardized tests have not yet figured out a way to electronically process the varied handwriting and creative flourishes of millions of third to twelfth graders. Nor, to my knowledge, have they begun to outsource this work to India. Instead, every year, the written-response portions of innumerable standardized tests given across the country are scored by human beings—tens of thousands of us, a veritable army of temporary workers.
I often wonder who students (or teachers and parents, for that matter) picture scoring their papers. When I was a student, I envisioned my tests being graded by qualified teachers in another part of the country, who taught the grade level and subject corresponding to the tests. This idea, it turns out, is as much a fantasy as imagining all the tests are being scored by machines.
I called the human resource department of DRC (1-800-835-4697). To be hired as a test scorer one only needs to claim a four year degree, provide a short writing sample, solve two math problems, and attend an orientation/training session. The person from personnel told me that the "training session" was actually more of a recruitment event. DRC is now actively hiring scorers for the PSSAs. The pay is $11 in Minneapolis. In their Pittsburgh scoring center, though, you can earn $13 an hour! I have now spoken personally with four people who used to work as test scorers in Minnesota, Iowa, and Texas. This is the industry standard.
There is a shocking article in City Pages, a Minneapolis weekly. The reporter interviewed people who had worked as scorers. The stories they recount clearly demonstrate how arbitrary and capricious the scoring is. Every parent, teacher, and student needs to read this article at a bare minimum.
I have since spoken with two people in the article, Dan DiMaggio and Todd Farley. They are both incredulous that this country is apparently moving toward even more standardized high stakes testing. Todd has even written a book about his experiences as a supervisor in the industry. Last night he told me feels like the worst whistleblower in the world. He wrote a book exposing how corrupt the testing industry is and essentially meaningless the results are, and today we are moving toward even more of this testing. The book is Making the Grades because that is literally what they do. (My copy just arrived. It's actually very funny, proving the futility of trying to quantify the imagination of children.)
The article described haphazard grading by people who have no idea what constitutes good writing. They are given a brief training and off they go. The rubrics that they use frequently become useless once the reality of children's imaginations hit. The graders who do care are cancelled out by those who don't. For supervisors, pressure is extreme to create a bell curve so that no matter how well schools or students are actually doing, a certain percentage will ALWAYS be on the wrong end of that curve. Psychometricians predetermine what these curve will look like. Supervisors just changes scores to get the desired results so that they will get paid.
[Todd] Farley now understood the reasons why, when he'd been a scorer, his team leaders would tell the room he wanted to start seeing more 3s or 4s or whatever. Supervisors were expected to turn the test scores into a nice bell curve. If his room did not agree at least 80 percent of the time, the tests would be taken back and re-graded, wasting time and money. The supervisor would be put on probation or demoted.
When Farley complained to a fellow supervisor about his problem, she smiled wryly and held up a pencil.
"I've got this eraser, see," she told him. "I help them out."
So Farley simply began changing Harry's scores to agree with his peers'. The practice soon spread well beyond Harry.
"I'd just change a bunch of answers to make it look like my group was doing a great job," Farley says. "I wanted the stupid item to be done, and so did my bosses."
I have spoken now with serveral people who have done this kind of work. They all confirm the details. This is in no way the fault of the army of temporary workers. They are just trying to make a living in tough economic times and doing the best they can. But a bunch of temps in a basement somewhere with no training in education or child development trying to figure out if a child who says that his favorite food is grass and that it tastes bitter gets credit for answering a question about the human tongue. (Are gummy worms a food? Real worms? Depends who you ask.)
To see an actual example of how repetitive, confusing essays are given perfect scores, while clear, concise writing is penalized look at this example of actual tests and how they were scored. And those are just the samples that the company uses to train the scores. The actual tests are another thing altogether.
For more information also see Dan Dimaggio's article of his experiences as a test scorer. Or read Making the Grades by Todd Farley
There are hundreds on articles everywhere from Huffington Post and Daily Kos to academic journals all documenting how NCLB and high stakes testing is failure, that it has actually made the problems in education worse. Even if this testing were a measure of anything, it's not working. But the actually testing from these private-for-profit companies is crap. These people are the Halliburtons and Blackwaters of education, making billions off our public education system, while the billionaires blame teachers for all the problems. I really want to just go throw up now.
Okay, I'm back.
The solution for those of us in Pennsylvania who know that NCLB testing is hurting our school and our children is simple. Parents in PA can opt their kids out of testing. Here's how to actually do it.