Another Pre-emptive Strike: Accountability

Sorry this is so long. The topic is complex.

Yesterday I talked a little bit about the changes in Milwaukee Public Schools funding provided in the Sinicki bill addressing the issue of Milwaukee Parental Choice Program caps. Today it's accountability, as the Sinicki plan--which mirrors the plan Governor Jim Doyle proposed in November--radically changes the accountability structure for voucher schools.

Now, I need to be honest again up front: I do not like standardized testing. As I have written here many times before, I do not think the regimen of testing imposed by No Child Left Behind provides an accurate assessment measure for either students or schools, and to use standardized test scores as the sole basis for judging the public schools is bad public policy and bad educational practice. As I have noted, I would be a hypocrite to advocate the kind of testing that I despise in my own school for voucher schools. However, my own plan for accountability seems not to have drawn any sponsors. That leaves the two choices on the table--the Sinicki bill or the Darling/ Vukmir bill. Again, I point you to Seth's place for a comprehensive side-by side (also note Seth's follow-up).

While I would not wish testing like this on my worst enemy, on the merits, the Doyle/ Sinicki version is the one I side with, under protest. There are many reasons why I would rather have that one, but first I want to look at present levels of accountability and the changes under the Darling/ Vukmir proposal; I will end this post with a discussion of the Sinicki bill.

As it is now
Up until 1995, the state received an annual report from John Witte, a respected and unbiased professor of Political Science and Public Affairs at the UW-Madison's LaFollette School. But for the last decade, there has not been a single independent analysis of the program. Anything we know about voucher schools comes from information that the schools themselves choose to release to the public. MPCP schools are not required to release any data except the number of students attending in the program (and schools have been caught inflating that number). So, with the exception of selectively released information, the schools are or can be utter black holes when it comes to graduation rates, truancy rates, achievement and proficiency levels, or anything else that you can think of. (I am not suggesting all or any specific voucher schools would show poorly in those areas, just that any information we get from schools in those areas is likely calculated to make the schools look good. If you don't have to make public bad test results, you probably won't.)

A number of outside studies of quality have been done since 1995 by arguably biased researchers; those who like vouchers generally have found that the program does well while those who dislike vouchers find that it is not any better for kids than MPS. Several studies, though, have been done on process, and accountability that accrues through that process. The Public Policy Forum, in particular, has done well with this. For example, despite the news that 92% of voucher schools give some flavor of standardized test at one or more levels, the PPF discovered that many of the data collected are never made public, or even provided to the parents of students in those schools.

In fact, the pro-voucher Milwaukee Journal Sentinel, in its excellent series on the MPCP last spring, found that concerns over academic quality just don't factor into parents' decisions to send their children to particular schools as often as we might like. Howard Fuller, president of one of the groups now advertising hard against Gov. Doyle, admitted that the marketplace idea--the idea that accountability inherent in the free market's process for rewarding success and punishing failure--has failed when it comes to MPCP. The Public Policy Forum identified one solitary school that had been shut down by parents' decisions in the first 12 years of the program.

That is not to say that schools don't shut down--they do, sometimes because the sponsors pull the plug or sometimes because the state's Department of Public Instruction steps in. DPI can shut down a school for basically three reasons: the school does not offer a safe physical environment for students (including checks on employees); the school does not have a sound business model or attempts to defraud the state; or the school does not meet the minimum statutory definition of a "school." This does not give me much of a sense of certainty regarding the quality of the schools that remain.

George Mitchell, when he was on WPR this morning where I could call in and question him, said that there were "many requirements that schools must follow" to continue in the program--but aside from the three I mentioned above, the requirements are quite thin. Basically, schools have to do one of these: advance 70% of voucher students one grade a year; claim that voucher students attend at least 90% of the time; assert that 80% of the voucher students make "significant academic progress"; or involve 70% of the voucher parents. The standard basically is, create your your own standard and then say you met it.

What's of greatest concern is that for the schools shut down by DPI--just a handful overall, and most of them in the past 18 months--parents didn't get involved in closing them, and on occasion have been dismayed that the schools closed. I will never forget the words of a parent interviewed following the shuttering of Academic Solutions about a year ago. She was upset at the closure, as her daughter--previously a failing student in MPS--was getting A's at Academic Solutions. But as DPI and even the Milwaukee Police Department made clear, there was no learning going on at the school--the A's earned by the girl were meaningless by any objective standard. Yet the inflated grade gave the mother, and possibly her daughter, an inflated sense of the quality of Academic Solutions.

The plural of anecdote (like the one above) is not, of course, data. But therein lies the current conundrum: There are no data. There is nothing to show that any of the present levels of accountability do anything to guarantee that the state is getting good returns on its investment in the MPCP.

Their proposal
In late 2003, the Republican legislature sent Gov. Doyle a bill that included a long-term study of the voucher program using standardized testing data. There were a number of problems with that bill. Despite what George Mitchell said to me on WPR this morning, that study would not have provided the level of comparable data that the Doyle/ Sinicki bill would. I discussed that at the bottom of this long post. But the same ideas are being recycled in the present Darling/ Vukmir proposal. There is an article in paper this morning about the study gearing up:

The study would be conducted by the School Choice Demonstration Project at Georgetown University, with Patrick Wolf, a well-known researcher in education policy, as the lead figure. Jay Greene, a professor at the University of Arkansas who has written numerous research works favorable to voucher programs, would be a partner with Wolf, as would University of Wisconsin-Madison professor John Witte, who was the main researcher in the studies in the early 1990s and who is regarded as more neutral on the merits of vouchers.

Wolf said the study was expected to cost $9 million and last five years, with funding to come from private foundations. He said about 40% of the money had been lined up from foundations such as the Walton Family Foundation, the Annie E. Casey Foundation and the Joyce Foundation. Other major national foundations are considering funding requests. Researchers are seeking funding from a variety of sources, hoping the results will be regarded as even-handed and scientifically sound.

A major part of the research would be using a randomly selected sample of students in MPS, in charter schools in Milwaukee and in private schools that are in the voucher program to analyze and compare the educational success and progress of each group of students. Scores on Wisconsin's battery of standardized tests called the Wisconsin Knowledge and Concepts Examination would be used in the analysis. [. . .]

The questions that the research would aim to answer include how well voucher students are doing academically, how their performance compares with other students and what are the explanations for the findings, Wolf said. The researchers would also aim to identify and describe the traits of schools that are the most successful.

Here we see some of the potential problems: The funding for this study, as was planned in the 2003 bill, will come entirely from groups which are not exactly disinterested. I mean, the Waltons and the Joyces are to vouchers what Orville Reddenbacher is to popcorn. Jay Greene never met a voucher program he couldn't skew data to support. Most importantly, the program will only test a representative sample of students from MPS and voucher schools.

Now, I am not a statistician, but I know enough social science to recognize that representative samples can produce valid data. But I have serious concerns about that part: For one, I am concerned that schools will be allowed to opt out of the study. The sampling is supposed to cover that, I know, I know; but if the only schools participating are ones with above-embarrassing test scores, the researchers will have to work hard to make the sample representative.

For two, and perhaps most importantly, while the study this time (as opposed to the 2003 proposal) involves the state WKCE exams, the study will not in any way at any time provide any kind of comprehensive picture of how individual schools perform on those tests. While the samples will provide an idea of whether the voucher program overall is better or worse or comparable to MPS, the study does not one damned thing to paint a picture of how well the individual schools that parents have to choose from are doing at present or over time.

Right now on the Milwaukee Public Schools home page, you can find the district's report cards. And this isn't just the big document that tells you how the district is doing overall; you can download the reports for any school and for any racial or other subgroup. You can find data, and then find it again flipped upside-down around and backwards, the district is so thorough with its reports. If this study of the voucher program goes through with sampling of data, there will be no comparable set of data for the schools in the program. When parents hear whatever the answer is from this study, whether it be that voucher schools are great or that they are weak, they will have no way to know whether or not the particular school they are choosing is reflective of the program as a whole.

MPS has some dynamite programs, many of them. Even within bad schools, there are pockets of success in distinct programs that draw from that population. It would simply be a fallacy to suggest that because MPS's fourth-grade reading level, as a whole, is low, that every MPS school fails its young readers. The same will hold true if this voucher study is the only information we have--it would be a fallacy to suggest that any individual school a parent wants for a child is as good or as bad as the overall study results show the program to be. Yet the pro-voucher folks insist that there is no need to collect or make public data on voucher schools the way we must, by state and federal law, do in MPS.

The Doyle/ Sinicki plan
As I said above, I reluctantly back the Sinicki bill's accountability measures. I do not do so because it involves a No Child Left Behind-style testing regimen, but rather in spite of that. As the plan indicates,

MPCP schools must administer the Wisconsin Knowledge and Concepts Examination (WKCE) Criterion-Referenced Tests in grades 3-8 and grade 10. DPI must report the results for MPCP schools so parents and taxpayers can compare the academic performance of choice schools with public schools. In addition, the scores of MPCP pupils will determine whether schools meet adequate yearly progress objectives or are labeled “in need of improvement” under the provisions of the No Child Left Behind Act--just like under-performing public and independent charter schools.

Note that this plan requires the testing of all students in all schools, not just a "sample" from schools that did not opt out. This is key, because then there is no other way for the taxpayers, the public, parents, or the state to know the quality (as much as a standardized test can judge that) of individual schools in the program. Even if you feel, as I do, that standardized tests are not valid and authentic assessments, this plan provides two important bits of data. One, we will be able to track schools' performance over time, and gauge the "value-added" effect of these schools. If they really are able to take students who are failing miserably in the public schools, two or three or four grade levels behind, then we should be able to track whether or not the school is effective in identifying and remediating the problems. Two, if the voucher schools are giving the same tests at the same time as public schools, there can be some real, legitimate comparative data. Even if we know that "proficiency" is an arbitrary and useless benchmark, we still will be able to compare the achievement on "proficiency" of voucher schools to those of public schools and, perhaps more importantly, to each other.

But this extensive and far-reaching NCLB-style testing plan is not without its detractors on the other side. If the funding piece I talked about yesterday will generate mind-blowing spin from the right, the accountability piece is equally dangerous to the state of our craniums. For example, when I pressed George Mitchell about accountability this morning, he was shocked--shocked, I tell you!--that anyone would dare suggest that every school administer the same test. "The idea that the state's test should to be mandated on more than 100 different schools in Milwaukee, I don't think is appropriate. Schools ought to have the option of choosing which of many acceptable standardized tests they desire to use." I wanted to shout into the phone about how nutty it is that MPS's 200 different programs have to all take the same test! Where is our freedom to choose how and when we test? Hm?

This is something I'm toying with calling the "Accountability Paradox": Conservatives who cannot mandate standards and testing fast enough for the public schools absolutely abhor the idea of applying those same standards and tests to private schools that take state money.

Campus Tavern's Ryan Alexander points to a couple of Marquetters' thoughts on the matter displaying this paradox. Law student Steve at Eminent Domain explains,

A lot of this testing talk is just a means to control the private schools. Let's say that Doyle gets his way and his chosen test is used to measure the quality of the private schools. The private schools don't want to look bad based on these test results. They will abandon their chosen curriculum in order to "teach for the test" as it has been called. That kills the whole idea behind school choice: parents should chose the kind of school that they want for their kids. The private schools that once offered a diverse range of educational programs will now be forced to teach for Doyle's test or look like they are "failing". And believe me, if they look like they are "failing" based on the test, school choice opponents will be happy to claim that as proof that school choice doesn't work.

This is different from the way people use the test to label public schools as "failing" . . . how? Steve himself complains that he doesn't want his "tax money being wasted in the public school system as it currently is." How does he know it's a waste? Could it perhaps be because the test data (or other data collected by MPS but not voucher schools) show that the schools are "failing"?

Even more to the point: Should the creative and varied programs in MPS be forced to "teach to the test"? Is it right to take away the choice of parents to send their children to unique programs within MPS? Is it possible that NCLB and its requirements are stifling the kind of innovation that would make the voucher program unnecessary? Okay, that's a little much with the rhetorical questions, but you have to ask them when Marquette Prof. John McAdams can write,

Even if the testing doesn’t have a tendentious political agenda, it’s likely to be burdensome. The bureaucrats imposing the testing will likely have a “public school” mentality. They will care more about “standards” and “outcomes” and “assessment” and “procedures” than about actually teaching kids.

They will tend, in other words, to undermine the distinctive ethos of private schools.

Don’t parents looking for a school in which to enroll their kids have a right to know how effective the school is? And don’t parents with kids enrolled in the school have a right to know how their kids are doing?

The answer is simple: in a competitive market, all schools will be under pressure to show that they are effective, but there are different ways of doing that.

Schools may choose to administer some form of standardized test, but will be free to pick a test that reflects their educational goals and philosophy.

Parents will want to know how their kids are doing, but this might involve sitting down with a guidance counselor to discuss scores on standardized tests, or it might involve having parents review all their children’s graded assignments.

The key thing to remember is that public school bureaucrats will, either by design or because they don’t know how to think any other way, subvert the benign organizational culture of private schools.

But it's all right to entrench a monolithic culture, through testing, upon the diverse--some would say "distinctive"--ethos of more than 200 different public school options in the city of Milwaukee? And, professor, the straw men in that post . . . unacceptable from someone who should know better. "Do the private schools fail to teach kids in junior high how to use condoms? They may be found to violate educational 'standards.' Do religious schools fail to teach that gays are the victims of 'homophobic' prejudice? Same thing." You of all people, professor, ought to know how to Google up the state Model Academic Standards, none of which cover condoms or homophobia. I hope they cover the logical fallacies, though. As much as people like Steve will say that "Hasn't it been the political Left that has done nothing be berate standardized testing for years? Now it's the lynchpin of school choice regulation?" in some kind of mock disbelief, I can just as strongly question how the pro-testing side is running away from their standards here. At the very least, I hope I can expect to see Steve and Professor McAdams at the next anti-NCLB protest.

More seriously, there is more to this than just a simple what's-good-for-the-goose proposition on my part and the part of Chris Sinicki and Gov. Doyle. If voucher schools are teaching to the Wisconsin Model Academic Standards, then at least we know that there is something more to the school than just a drawerful of paper work. And if a school chooses not to teach to exactly those standards, they can freely explain why their results don't matter, or rest secure in the knowledge that their quality, innovative program's success will shine through even if they don't "teach to the test." After all, good students with a good education ought to be able to perform up to the level of "proficiency" in reading, math, or other core subject areas. There should be nothing in the test results that schools will want to run away from--and if there are results they want to run away from, I think we, as taxpayers funding these schools, are owed an explanation.

In the end, that's the crux of the accountability matter: We are the taxpayers. We are paying for these schools' existence, and we deserve some assurance that the trains are running, even if not on time. If these schools were completely private, and parents' private funds were all that got invested in them, then I could (and happily do) stomach the idea of schools that don't follow state standards or open meetings laws or special education requirements. But, as I've explained before, the vouchers signed over to these schools are not just parents' money. The money is their money, my money, your money, all of our money. Even Charlie Sykes throws a few pennies into the hopper for every voucher student. And even Charlie Sykes deserves better than the non-specific, long-term study proposed (for the second time around) by Republicans in the legislature.