We often hear that spots in school choice programs like KIPP and the New York City small schools are allocated "by lottery" from those who apply to participate. The idea of a lottery is critical to our attempts to evaluate how a program, a drug, or a school is working - because we can be sure that the people receiving the "treatment" - whether it is a place in a school or a chance to take a smoking cessation drug - weren't simply the most likely to benefit from the treatment to begin with. As it turns out, neither KIPP nor the small schools in New York City are using a true lottery, which raises some serious issues about how to evaluate the effectiveness of these programs. This is not to say that some kids are not benefiting - certainly they may be - but that we need to look closely at how these schools' students are different before calling these programs a "success."
We often hear that spots in school choice programs like KIPP and the New York City small schools are allocated "by lottery" from those who apply to participate. The idea of a lottery is critical to our attempts to evaluate how a program, a drug, or a school is working - because we can be sure that the people receiving the "treatment" - whether it is a place in a school or a chance to take a smoking cessation drug - weren't simply the most likely to benefit from the treatment to begin with. As it turns out, neither KIPP nor the small schools in New York City are actually using a true lottery, which raises some serious issues about how to evaluate the effectiveness of these programs. This is not to say that some kids are not benefiting - certainly they may be - but that we need to look closely at how these schools' students are different before calling these programs a "success."
- KIPP
Even KIPP’s critics acknowledge that KIPP students are chosen by lottery. What it means to be chosen by lottery is that there is no intermediate step between picking your name out of a hat and deciding whether you get to show up on the first day of school. From media depictions of KIPP, I was under the impression that a certain number of names are chosen and, give or take a handful of kids that represent random attrition, these are the kids who matriculate at KIPP.
It turns out that’s not exactly true. For example, consider the four KIPP schools that are located in New York City. From their websites, I found that students are chosen, and then are asked to come to the school for a meeting or have a phone conversation to discuss the expectations for the school; after these interactions, parents – or potentially the schools - decide if they will enroll their students at KIPP.
Contrary to many of KIPP’s critics, I don’t see this as strategic action (i.e. an attempt to cream the best students). KIPP wants to run schools where kids are committed to the culture, and this makes a lot of sense. Meeting kids individually before school starts is a necessary part of this process. KIPP schools ask kids to do things that are not required by regular public schools, and it’s impossible to run such a school without student and parent buy-in.
That said, here is the text from the KIPP A.M.P. application form, which is available at their website:
KIPP: A.M.P. Academy will extend offers of admission to the first 77 children whose names are picked in the lottery. If my child’s name is selected in the lottery, KIPP: A.M.P. Academy will contact me at the above-listed address and phone number to schedule a meeting. After my meeting, I will have to decide whether to accept the offer of admission and register my child at KIPP: A.M.P Academy for the 2007-2008 school year. If KIPP: A.M.P. makes a good faith effort to reach me but is unable to schedule a meeting, my child’s place at KIPP: A.M.P. Academy will automatically be surrendered to the first child on the waiting list.
I thought this might be specific to New York, so I checked out KIPP Indianapolis, where there is not just a phone call or a meeting, but a home visit before formal enrollment:
KIPP Indianapolis College Preparatory will extend offers of admission to the first 85 children whose names are picked in the lottery...If my child’s name is selected in the lottery, KIPP Indianapolis College Preparatory will contact me at the above-listed address and phone number to schedule a home visit.
It’s certainly possible that there is not significant attrition as a result of this process. From KIPP’s end, it would be helpful to know how much lottery to first day of school attrition we’re talking about. In other words, if a KIPP school draws 100 names, I would like to know what proportion of the original 100 attend KIPP, and also know the reasons why lottery winners chose not to attend. (This is a different kind of attrition than the post-matriculation attrition covered in Ed Week’s recent article; I’ll discuss this on Thursday.)
What’s the big deal? From the researcher's and public policymakers' perspective, the primary benefit of lottery selection is that it results in balancing the treatment and control groups on observable characteristics – i.e. students’ test scores, free lunch status, etc – and their "unobservable" characteristics – i.e. their motivation, aspirations, and commitment to KIPP’s principles. On any dimension we can think of, the treatment and control groups should be indistinguishable. We can then compare these groups to figure out how effective KIPP is, as I wrote in yesterday's post.
But if we draw a lottery of 100 people and after learning more about KIPP’s approach, 30 kids who were less motivated select out or are counseled out because they are a bad fit, we end up comparing very different treatment and control groups. The treatment group is now much more motivated/committed than the control group – many of whom might have opted out as well if they had these meetings before entering the lottery.
Again, I'm not sure that school leaders are aware that these post-lottery screening processes effectively result in non-lottery admission. For example, an old friend of mine recently started a KIPP-style charter school. This is a remarkable and brilliant guy who is deeply committed to improving educational options for disadvantaged kids; he’s not trying to play the system. His school drew 150 kids in the lottery for 100 seats at his school. His rationale for over-drawing was that many kids would realize they wouldn’t like the extra time, their parents wouldn’t accept the culture of the school, or the school would feel the family wasn’t sufficiently committed, and about 50 kids would withdraw even before school began. He still believed his students represented random draws from the lottery pool – after all, he argued, these are all poor and minority kids. I am not sure I was successful in convincing him otherwise.
The take-home lesson from this post is that we need to beware of the term "lottery" – it doesn’t always mean what we think it does.
- New York City Small Schools
I received a number of emails last week in response to my post comparing Evander Childs High School and the new small schools in the same building, which found that the small schools enrolled a very different group of kids. The most common question was how these differences were possible if these students are assigned "by lottery."
I spent a few days looking into this question, and here’s what I learned: it’s a common misconception that the NYC small schools are admitting students by lottery. The majority of the new small schools fall under the city’s "limited unscreened" selection mechanism. Here’s how it works: as a student, I can apply to up to 12 schools; students rank these choices from 1-12. Every 8th grader applies to high school, and these applications are entered into a central database. Each school then receives a list of the students who have applied to them, but the schools do not know if the students have ranked the school 1st or 12th.
Limited unscreened schools then dichotomously rank their students (yes or no); students are chosen if the school can verify that the student is making an "informed choice" to attend the school. Students that choose the school and that are also chosen by the school are admitted in order of the students' preferences (i.e. students who ranked the school first are admitted before those who ranked the school 10th).
Schools vary in the criteria they use for verifying informed choice. Some schools require students to attend information sessions; in the past, schools have required that students attend a session with a parent or guardian, but this has now been forbidden by the Department of Education. Other schools have used "applications" that students needed to fill out to verify informed choice. For example, a New Yorker passed along the application used in the 2004-2005 school years by the network of schools affiliated with Replications, which include schools like the Frederick Douglass Academy and Mott Hall replications, which are "limited unscreened" schools. Here are the essay questions:
- What are three things your teachers would say about you?
- What makes you want to attend a school that will demand your very best academically and will expect you to work harder than you probably ever have before?
- What are five future goals you have for yourself?
- Mention the title and authors of some books you would like to discuss during your interview.
- What are some activities to which you belong either in school or outside of school?
In addition, until this year, all limited unscreened schools had access to individual students’ prior attendance, grades, their test scores, their date of birth, their address, their sending junior high schools, and their special education and English language learner status. Interestingly, the Department of Education did not provide this information to limited unscreened schools beginning with admission for the 9th grade class of 2007 - was this choice made because unscreened schools were using data they weren't supposed to?
What this means, however, is that the limited unscreened schools about which we’ve heard a lot of crowing had access to a great deal of students’ achievement data. Given the data I presented last week, which show large differences between large and small school incoming populations, I have a hard time believing that schools ignored these data entirely.
Certainly the formal rules of the system prohibit these schools from doing so – but how tightly was this regulated? The Department of Education could easily demonstrate that "informed choice" doesn’t limit lower-achieving students’ access to these schools by testing for differences in the mean test scores, attendance, etc of the applicants, the applicants chosen by the schools themselves, and the students who ultimately matriculate. This is a trivial (i.e. easy) analysis, and certainly something that the Department of Education needs to do if they want to maximize educational opportunity for New York City kids.
To sum up, it appears that some of these disparities are created by who applies to the small schools themselves, by who schools verify as making an "informed choice," and potentially by these schools use of students’ achievement characteristics available through the database.