Okay, so we're up at my sister's house this past Christmas, and somebody says we just have to play this Wii game they've got called Just Dance. It's sorta like dance karaoke: there's a list of songs to pick from, and each song has a video of a choreographed dancer to accompany the music. The object is to mimic the video dancer's moves as closely as possible; the more precisely you duplicate the moves, the higher your score.
Let me tell you, this game is a freaking riot. Even sober. We were ROTFLOAO as siblings, nieces, nephews and cousins embarrassed themselves to "Rock the Casbah" and "Rasputin" and "Tik Tok."
There's no place to hide in my sister's living room, it's just not that big, so I knew that sooner or later somebody after being ROTFLOAO'ed at would turn and extend the Wii controller to me, insisting that I, too, debase myself just as they had, all in the name of good wholesome family holiday fun. There would be no escaping it.
Two things here:
First, I try to make it a point to include myself out of any Public Displays of Rhythm. Anyone unfortunate enough to have witnessed my participation in one will understand why this is so: the sight is sufficiently alarming that "Someone call 911!" seems a response both reasonable and compassionate.
Second, I am a wily competitor, always looking for any advantage to be gleaned by dint of sheer cunning. What this translated to in the case of Just Dance was that my brain was preparing to write a check that I knew my hips couldn't cash -
- but I also knew my hips wouldn't have to.
See, here's how I figured it: the Wii system (the basic Wii system, mind you) is a fairly primitive one, involving just one controller (in the case of Just Dance, to be held in the right hand), the motion of which is tracked by the Wii sensor mounted on the television set. Regardless of whatever gyrations the karaoke/video dancer was going through on the screen that the competitors were supposed to mirror, it was no great feat to realize that the way to get the highest score was to focus on the motion of your right hand; nothing else mattered. The system could only track the motion of the competitors' right hands; everything else was irrelevant.
So I got up to "dance." It would not be much of a stretch to say that if god forbid whatever it was that I was doing in that living room that evening had made it onto YouTube, no one even vaguely conversant with all of the countless iterations of that millennia-old human art form known as dancing would have recognized my movements as such.
Now, my wife - she knows how to dance. Seriously, girlfriend got it goin' on. She has rhythm, she's utterly uninhibited, she has complete command over her body as a vehicle for expressing whatever's happening in the moment with her mind and spirit - she's an open book. I can watch her on the dance floor for hours, getting variously prouder and hotter by the minute. So you'd think that Just Dance would be the perfect opportunity for her to clean up at the expense of the, ah, more choreographically challenged among us.
But I kicked her butt. Which - rightly - pissed her off.
Let me be more precise:
By the standards that the game was able to measure,
I Am An Awesome Dancer. A regular freaking Fred Astaire.
But, see, the thing is, I wasn't dancing. I was trying to score the greatest number of points on Just Dance.
HUGE difference.
And I understood the difference going in.
Yeah, yeah, I know: I'm a total buzzkill, a Debbie Downer - how could I take something as joyful, as creative, as fulfilling as dancing, and make it so cut-and-dried, so boring, so depressingly quantifiable?
Because I recognized Just Dance for what it was: an overly simple way of purporting to measure a very complex set of skills, skills whose asssessment are subject to as many interpretations as there are humans on the planet. I mean, really:
How could something so intangible, so individual, so complex and multi-faceted be reduced to a number?
Well, of course, it can't.
Which brings us, unavoidably, to the question of "standardized" testing and teacher evaluation.
The Measure of a Good Teacher
You wanna know what makes a good teacher? I'll tell you what makes a good teacher. In fact, no - I'll let the Los Angeles Unified School District tell you what makes a good teacher, given that they have just spent a buttload of my money figuring that out.
Here, according to the Los Angeles Unified School District, is the measure of a good teacher:
y = Xß + Zv + e where ß is a p-by-1 vector of fixed effects; X is an n-by-p matrix; v is a q-by-1 vector of random effects; Z is an n-by-q matrix; E(v) = 0, Var(v) = G; E(e) = 0, Var(e) = R; Cov(v,e) = 0. V = Var(y) = Var(y - Xß) = Var(Zv + e) = ZGZT + R
And in response to your next question, No, I am not making this up.
I must admit, I do feel a bit cheated, knowing that my daughters are nearing the end of their student careers in the LAUSD and as such will likely never reap the benefit of this simple common-sense assessment method. I guess we will have to satisfy ourselves that the rusticated practices of talking with and listening to and observing over time the various instructors our daughters have had, and asking our daughters and school officials what they thought of those teachers, provided sufficient insight into those teachers' respective levels of effectiveness.
But you know, now that I think about it, an equation like
y = Xß + Zv + e where ß is a p-by-1 vector of fixed effects; X is an n-by-p matrix; v is a q-by-1 vector of random effects; Z is an n-by-q matrix; E(v) = 0, Var(v) = G; E(e) = 0, Var(e) = R; Cov(v,e) = 0. V = Var(y) = Var(y - Xß) = Var(Zv + e) = ZGZT + R
really does just make so much more sense. I mean, who doesn't hold a special place in their heart for those teachers who made it a point to help them raise their percentiles on the STAR test, who took the time to show them that the truly important thing in life was achieving a higher AYP in their reading scores? Right? Especially the ones who helped us understand that cheating was totally justified in order to achieve all of that - those lessons will always stay with us. On the other hand, forget all those teachers who wasted our time with stupid things like "character building," or "life skills," or "critical thinking" - who remembers any of that shit, anyway?
Because, as everyone knows - especially Bill Gates, Michelle Rhee and Arne Duncan - the truest measure of a teacher is how well his students do on a "standardized" test.
No matter what it takes.
I mean, what's a little cheating between friends?
Right, Michelle? Riiiight?
From yesterday's USA Today:
When standardized test scores soared in D.C., were the gains real?
USA TODAY examined testing irregularities in the District of Columbia's public schools because, under [former Chancellor Michelle] Rhee, the system became a national symbol of what high expectations and effective teaching could accomplish. Federal money also was at play: Last year, D.C. won an extra $75 million for public and charter schools in the U.S. government's Race to the Top competition. Test scores were a factor.
(Read the whole article; it's damning.)
Yeah, that Michelle Rhee - the one the Republican state legislature of Florida creamed its pants over, the one "Waiting for Superman" deified, the one even Barack Obama was duped by - the one who, as long as four years ago, astute observers - people who evidently had developed critical thinking skills before public schools dumped them in favor of multiple-choice erasable tests - expressed serious, ahh, reservations about. See, even the sainted Michelle Rhee can't escape the pressure of resorting to cheating on "standardized" tests. Besides the more recent examples discovered by USA Today, in an effort to improve her position as educational reform's Deliverer, down from the mountain with a scannable test form in one hand and a No. 2 pencil - with an eraser - in the other, it looks like Ms. Rhee might've found it more expeditious - 15 years ago - to, ahh, enhance her former students' test scores in a way that strains the bounds of standard deviation.
And Bill Gates - lemme tell ya, that guy really knows a thing or two about education, yessirree, bob. Like this:
"Once somebody has taught for three years their teaching quality does not change thereafter."
-- Bill Gates
(Yes, he actually says that, about 12:33 in; the entire “education” portion of the lecture is painful to watch, it’s so rife with stupidity.)
Whew, boy, that's good to know - I wonder if software CEOs have the same three-year shelf life.
My wife and I had a dance teacher once, for a few months before we got married. We hired him to teach us to waltz, so that we wouldn't embarrass ourselves at our wedding reception when we stepped out to do our first dance.
Had he been so motivated, I am certain that that dance instructor could have made it his mission to see to it that we scored the greatest number of points while playing Just Dance. I will be forever grateful that instead he made it his mission to see to it that my wife and I could pull off a passable first dance.
For the record, he succeeded. Our waltz was reasonably well-executed and politely well-received.
If I were asked to provide a standard by which to measure to what extent our dance teacher was effective, I would cite the preceding paragraph.
Amazingly, though, there are a lot of people out there in the world right now - including Michelle Rhee, Bill Gates, most Republicans and, sadly, our current president and his education secretary - who think it would make much more sense to evaluate our dance teacher based not on how well we actually danced, but rather on our scores while playing Just Dance.
Not to put too fine a point on it, these people don't know what the fuck they are talking about.
Right now in this country a number of jurisdictions – including, indirectly, the United States government - are seriously weighing proposals that would tie teachers' compensation - and, in some cases, even their employment status - to their students' "standardized" test scores.
High-Stakes Testing: If You Build It, They Will . . . Cheat
When compensation, tenure and indeed entire careers are riding on the outcome of high-stakes testing, guess what happens?
CHEATING.
Duhhhh! To wit:
When standardized test scores soared in D.C., were the gains real?
-- USA Today,
March 28, 2011
Under Pressure, Teachers Tamper With Tests
-- New York Times,
June 10, 2010
L.A. Unified set to renew charter contract despite evidence of cheating
-- L.A. Times,
February 28, 2011
When test scores seem too good to believe
-- USA Today,
March 7, 2011
Questions arise over big gains on AIMS
-- Arizona Republic,
March 7, 2011
But no one should be surprised by the cheating; it's hardly a new phenomenon:
In 1969, what was then called the U.S. Department of Health, Education and Welfare wanted to increase reading and math scores for some 300 junior high and high school students in Texarkana, Ark. The district was under intense pressure to desegregate its schools and narrow the achievement gap between black and white students. (Does this goal sound familiar today?)
Under a program called performance contracting, federal funds would be returned for students who failed to pass the "standardized" tests at a stipulated level. The plan provided incentives for teachers, administrators and students. The initial evaluation seemed too good to be true. After only 48 hours of instruction, students averaged gains of more than two grade levels in reading and one in math. But the Texarkana miracle turned out to be a mirage when it was discovered that cheating was rampant. In the hope that what transpired in Texarkana was an aberration, performance contracting moved on to 18 other cities in the state. The lack of results there eventually put an end to the experiment.
There are other examples from other countries, but they all make a similar point: When too much is on the line, educators will engage in conduct that undermines taxpayer trust in public schools. Unless Campbell's Law can be repealed, expect to see further evidence of cheating.
Wait – So You Think “Value-Added” Teacher Assessment is Terrific, but Global Warming Is a Hoax?
The National Academy of Sciences would beg to differ with you. Regarding “Race to the Top,” here’s what they had to say:
“Too little research has been done on these methods' validity to base high-stakes decisions about teachers on them.”
Research is pretty clear that the correlation between “standardized” test scores and teacher effectiveness is, generously, tenuous.
- Last year researchers at Stanford and UC Berkeley found this (emphasis added):
[T]eachers who were teaching greater proportions of more advantaged students may have been advantaged in their effectiveness rankings . . . Each teacher appeared to be significantly more effective when teaching upper-track courses than the same teacher appeared when teaching lower-track courses.
The default assumption in the value-added literature is that teacher effects are a fixed construct that is independent of the context of teaching (e.g., types of courses, student demographic compositions in a class, and so on) and stable across time. Our empirical exploration of teacher effectiveness rankings across different courses and years suggested that this assumption is not consistent with reality. In particular, the fact that an individual student's learning gain is heavily dependent upon who else is in his or her class, apart from the teacher, raises questions about our ability to isolate a teacher's effect on an individual student's learning, no matter how sophisticated the statistical model might be.
Even if it's this "sophisticated?:"
y = Xß + Zv + e where ß is a p-by-1 vector of fixed effects; X is an n-by-p matrix; v is a q-by-1 vector of random effects; Z is an n-by-q matrix; E(v) = 0, Var(v) = G; E(e) = 0, Var(e) = R; Cov(v,e) = 0. V = Var(y) = Var(y - Xß) = Var(Zv + e) = ZGZT + R
- Here’s a nice compilation of other debunkings:
* The Board on Testing Assessment wrote an open letter to Department of Education Secretary Arne Duncan in October, 2009, stating that not enough research had been done on VAM’s validity to use it as a basis for determining teacher effectiveness. It also concluded that a student’s scores can be affected by various factors other than their teacher.
* The Department of Education’s Institute of Education Sciences published a 36-page analysis of VAM in July, 2010, in which it stated that “more than 90 percent of the variation in student gain scores is due to the variation in student-level factors that are not under the control of the teacher.”
* The Economic Policy Institute (EPI) assembled a panel of experts in August, 2010, who warned [PDF file] against giving substantial weight to VAM scores as a tool for measuring teacher effectiveness.
* Researchers for the RAND Corporation, a nonprofit institution that helps improve policy and decision making through research and analysis, concluded, “The research base is currently insufficient to support the use of VAM for high-stakes decisions about individual teachers.”
* And in the wake of the L.A. Times debacle, Rutgers Professor Bruce Baker concluded, after an analysis of the study, that its ratings of teachers are racially biased. He cited that the lowest VAM scores were earned by black teachers while the highest were earned by Asian teachers.
But It MUST Be Meaningful – We Spent A Lot of Money on It!
Let’s just drop in a few other inconvenient facts that supporters of “No Child Left Behind” and “Race to the Top” – both programs that tie school funding with “standardized” test results - would rather you not know:
- Here's how well No Child Left Behind and all of its associated "standardized" testing and "accountability" has worked (from 2007):
Since 2002, fourth grade reading scores have gone up 2 points, and eighth grade reading scores have dropped one point. (The national average for grade 4 in 2007 was 221, with the lowest 10 percent scoring 174, the highest scoring 264. A two point gain is very small.)
The gap between students from high and low income families is also nearly unchanged, reduced by one point in grade 4, and two points in grade 8.
Students in Reading First, the reading component of NCLB, get much more instruction, an extra 100 minutes per week, or an extra semester every two years. What the NAEP scores tell us is that this huge investment is not paying off. Even if Reading First were only mildly effective, we would see noticeable improvement, not just two points after several years at one grade level and a drop of one point in another.
- Here's how successful “No Child Left Behind” has been:
SAT scores FELL under NCLB.
Holy shit!! You mean to tell me, all that time and money we spent on getting our kids to raise their "standardized" test scores actually HURT their scores on another "standardized" test?!? WTF!?!!?
- Teaching to the test – selecting a very narrow band of “correct” information to transmit to students, and then rewarding them for successfully anticipating what the expected answer is, can have very negative long-term consequences:
[D]irect instruction made the children less curious and less likely to discover new information . . .
[The study’s authors] provide scientific support for the intuitions many teachers have had all along: Direct instruction really can limit young children's learning. Teaching is a very effective way to get children to learn something specific - this tube squeaks, say, or a squish then a press then a pull causes the music to play. But it also makes children less likely to discover unexpected information and to draw unexpected conclusions.
(Of course, if you’re a rightwing corporatist feudalist, who wants future serfs “discovering unexpected information” or “drawing unexpected conclusions” anyway, right?)
- Standardized test scores are higher in unionized states than in non-union states:
[T]he states in which there are no teachers covered under binding agreements score lower than the states that have them.
- Guess what? News flash: Poverty directly affects test scores:
American students in schools with less than 10% of students on free and reduced lunch averaged 551, higher than the overall average of any OECD [Organization for Economic Cooperation and Development country. Those in schools with 10% to 25% of students qualifying for free and reduced lunch averaged 527, which was behind only Korea and Finland.
In contrast, American students in schools with 75% or more of children in poverty averaged 446, second to last among the 34 OECD countries.
But – but – but! But Bill Gates! Finland!! That’s right: Bill Gates, that educational titan, has decided that the performance of Finland's students is a good benchmark for American students. So has that Genius-of-all-trades Tom Friedman.
Finland? Seriously? You want to point to Finland as an example of how to raise teaching standards? Brilliant idea, assholes:
Finnish children never take a standardized test. Nor are there standardized tests used to compare teachers or schools to each other.
Oh, and -
Ninety-five percent of Finland's teachers belong to labor unions, and those teachers tend to enjoy many of the same rights and privileges (including "tenure") that unionized teachers in the U.S. do.
But don't try to convince the "free-market" "reformers" of the truth or wisdom in any of these mere facts - they couldn't care less. That's because "free-market" "reformers" in education don't give a shit about children, or education, or the future of this country. No, "free-market" "reformers" in education want exactly what every "free-market" "reformer" in every industry wants: a completely unregulated business environment where they can make as much money as possible in the shortest time possible with absolutely no accountability or consequences for their action, irrespective of the welfare of American citizens, ideally while acting with the full support and assistance of government in their systematic extraction of money from the American taxpayer. And the reason they're going after K-12 education is a reason that Willie Sutton would understand very well: It's Where the Money Is - $500 billion a year, to be precise.
It's that simple. All the rest of the bullshit - "accountability," "excellence," "Race to the Top" - all of it - you can shove up your ass.
Millionaire Union Thug Teachers are Limiting My Freedom of Choice
No offense, but these people are morons.
Well, actually, they're not morons. They're actually old-school greedy fucking motherfuckers who don't give a shit about children or this country. The people who buy into the bullshit these greedy motherfuckers are peddling about "standardized" testing, however, are morons.
"Standardized" testing does a few things very well: (1) it enriches companies that sell "standardized" tests, as well as the shareholders and executives of those companies, and the hucksters and politicians who back those companies' products; (2) it incentivizes those whose livelihoods depend on their students' scores on those tests to find ways to increase their students' scores on those tests; and (3) it shifts resources away from other educational goals and programs. That's about it.
"Standardized" testing is fraught with pitfalls and caveats and limitations - but it is also in service to several pillars of the Republican corporatist vision for this country. To wit:
It leads to the limitation of curriculum, away from hard-to-quantify skills like critical thinking and creativity. This ultimately makes for much more pliable worker drones, well conditioned for plugging in to the low-pay service-oriented work force much desired by so many corporate titans (cf. "feudal society").
When used in the context of a narrowly defined, failure-guaranteed program like “No Child Left Behind” and “Race to the Top,” it makes inevitable the portrayal of public schools - and the teachers who staff them - as failures.
As such, it opens the door to the "logical" replacement of public schools by private schools.
Thus, it is the Shock Doctrine brought home to your child's classroom.
In the process, it allows private test-making and administering companies to make a buttload of money selling their crap to school districts.
Republicans and their greedy corporatist cronies hate the idea of an informed and critical-thinking electorate and work force. Much better for their purposes to have a compliant, servile horde of serfs who are content with the thin gruel of their miserable lives, and unaware of anything better that might once have been in this country.
In case you've forgotten the "education crisis narrative," let me try to summarize. According to the reformers, American public schools are failing. Reformers know public schools are failing because according to a bevy of "standardized" tests, students in American public schools are not proficient, below basic, and failing. Not only are students failing, they also drop out of school in mass numbers. However, if American public school students do graduate, most of them are not even capable of signing their own name. To address this mass failure, a ton of taxpayers' money has been dumped into public schools over the past 15 years and nothing has changed. The public schools are still failing, the economy is in distress, and it's all because of the "crappy" teachers and a misguided public school system. Done!
But why go to such lengths? Why the effort? The reality is that if the reformers were honest they would just admit that they abhor a "public" anything. It's hard to find any honest reformers, but I actually know one. This person at least has the decency to admit that according to her, "government" schools (most of us call them public schools) are federally unconstitutional and at the state level public schools should be eliminated and replaced by a "thriving" private school system. Also, this person believes the government should not "steal" her money and redistribute it to bureaucrats and create a government institution (public schools) designed to indoctrinate children with socialist ideologies. It is an individual's responsibility to pay for and obtain an education. For some readers, this may sound a little scary but at least it's honest. What it also reveals is that many reformers are not really interested in making public schools better. Their main goal is to thrust their market based ideology on society and dismantle the American public education system -- scary, but honest.
This makes me wish that the reformers would just admit their position and honestly try to convince the American public that their ideas for the privatization of the American public school system would better serve our nation's children. Let's stop all the No Child Left Behind, Race to the Top, Common Core Standards dishonest reforms that are damaging children and educators and truly debase the wisdom of a totally private American school system.
This latest push for ""standardized"" testing - including the demonization of teachers and unions – demonstrates, if nothing else, three things: (1) People want easy answers; (2) There's a sucker born every minute; and (3) Right-wing corporatist Republican types who want to suck the profits out of every public sector endeavor are relentless and persistent.
Bartleby and Me
I always have been the beneficiary of "standardized" testing. I kicked ass on the Stanford-Binet when I was 11; as such, I was "tracked" into gifted classes throughout the rest of my school career. I had SAT scores high enough to get into virtually any school in the country. But it was only once in high school that I had my eyes opened to the soul-crushing destructiveness of so much of the educational system. While I was a very successful student, I also saw that the system was not nearly so well constructed as it could have been to address the needs of as many students as possible.
The first book I read on education was John Holt's classic How Children Fail. From there in short order I devoured Crisis in the Classroom and various other tomes, culminating in what I still consider some of the finest books on education (and on the much larger world as well), The Student as Nigger by Jerry Farber, and Neil Postman and Charles Weingartner's Teaching as a Subversive Activity and The Soft Revolution.
I haven't done an exhaustive study of the subject of education and “standardized” testing. But I lived in it, I swam in it, for 12 years, and so did my kids. Does that make me an expert? Mmm, I don't know. But I damn well know it doesn't make me someone with a financial axe to grind. Let me repeat: I have been a lifelong beneficiary of "standardized" testing.
Somehow, that path led me, when I was a senior in high school, to engage in what to date has been my greatest act of civil disobedience. I led an uprising among my senior classmates to utterly sabotage the statewide "standardized" test that was administered every year to California students.
The test was scheduled for some time just after the middle of the school year, as I recall. The school (there was only one high school in our district at that time) bused us over to a local hall to administer the test. I had started the buzz a few days earlier: we were going to deliberately perform as poorly as possible on the test.
Before hatching the plot, I had taken care to ask several school personnel what the purpose of the test was. I never got a satisfactory answer. I asked whether any school funding rode on the outcome of the test; I was told no. No teacher’s job or salary was tied to the outcome of the test. I knew that the test was certainly not diagnostic - it was anonymous, as I recall; in any event, we were seniors, and our respective individual results meant nothing. No one's graduation hung in the balance; in fact, the state never released results until after the end of the school year.
It was therefore clear that the test was a huge waste of time and resources. By the time the buses dropped us off at the testing venue, the electric tingle of rebellion was in the air.
I remember that testing session being one of the most satisfying experiences of my high-school career. Virtually every student in the room treated the process with the contempt it deserved. I know for a fact (since the questions on the test all were fairly easy) that I got every single answer on the multiple-choice test wrong. A monkey with a No. 2 pencil literally would have done better.
The school administration had caught wind of some of this, and were nervous about it, but powerless to do anything. When the results finally came out - after all of us were safely graduated - they expressed their righteous indignation, telling of "a small group of misguided students," or something like that. The scores were the lowest the district had ever received by orders of magnitude.
This was, incidentally, long before I had ever heard of the Bartleby project (PDF file), which urges students to respectfully decline to take any “standardized” test:
Mass abstract testing, anonymously scored, is the torture centrifuge whirling away precious resources of time and money from productive use and routing it into the hands of testing magicians. It happens only because the tormented allow it. Here is the divide-and-conquer mechanism par excellence, the wizard-wand which establishes a bogus rank order among the schooled, inflicts prodigies of stress upon the unwary, causes suicides, family breakups, and grossly perverts the learning process - while producing no information of any genuine worth. Testing can't predict who will become the best surgeon, college professor, or taxicab driver; it predicts nothing which would impel any sane human being to enquire after these scores.
Value-added screen testing
No assessment system is perfect. The story goes that when Fred Astaire had his first Hollywood screen test, the MGM underling who was tasked with reviewing it jotted the perfunctory comment,
"Can't sing. Can't act. Balding. Can dance a little."
(For the record, Astaire allegedly remembered it slightly differently; in any event, David O. Selznick agreed that the screen test was "wretched.")
Now, the fact that Fred Astaire dancing in front of an MGM scout and me "dancing" in front of a Wii console produced nearly identical assessments, one generated by a human, the other by a machine, should be enough to give one pause.
(I'm guessing the scouting report on Michelle Rhee would've read something like this:
Can't teach. Can't administer. Lying. Can sell a little.
Thanks for reading.