Generalizing about the performance of charter schools and comparing the performance of charters with noncharters are complex challenges for a number of reasons. First, the charter segment in California is ever-changing. Every year a substantial number of new charters open. At the same time, a few schools close of their own accord or have their charters revoked.

In addition, the great diversity within the charter universe makes it difficult to talk about "charter schools" as a group. Charters are found in urban, suburban, and rural areas; and though most are small, a few are quite large. This type of diversity can be found among noncharters, but there are other ways in which the charter community is more diverse than the traditional public school segment. Among charter schools, the exception is the rule. Along with some back-to-basics programs, one finds independent study schools, project-based learning programs, "virtual schools" using distance learning, networks of home-schoolers, and programs helping young working adults earn their high school diploma or a GED. Most are single "mom and pop" schools, but a growing number belong to networks or are run by charter management organizations. A minority of charters are converted traditional public schools that are in many ways conventional.

There are also important differences between charters and noncharters. For example, a substantial portion of charters have atypical grade configurations, with many serving grades K–8 or K–12.

In addition, charter schools as a whole enroll somewhat different students than noncharters. Charter schools are less likely than traditional public schools to serve Hispanic students, English learners, or students from low-income families. These demographic factors and several others are summarized in the state's school characteristics index (SCI). Among elementary schools, charters tend to serve students who are somewhat less disadvantaged according to SCI values. In contrast, charter middle schools serve somewhat more disadvantaged students than traditional schools on average, and charter high schools tend to have students who are markedly more disadvantaged than traditional high schools.

Charter high schools also tend to be much smaller than their traditional counterparts. The difference in size between charters and noncharters is less marked but still substantial among middle and elementary schools.

Meaningful comparisons of the academic performance of charters with that of traditional public schools must try to control for these differences in students' background factors because of the association between background characteristics and performance.

Evaluators face additional difficulties: Should recently opened charter schools be included or only those that have had a few years under their belts? Should—or can—evaluators adjust for the fact that charter school students (or their parents) have chosen that school, as opposed to most noncharter students who are assigned to schools based on the neighborhood where they live? Should nonclassroom-based charter schools be compared with traditional classroom-based schools?

Researchers address these issues—to the extent that they can—in different ways. During the past five years, EdSource has conducted its own analysis of charter performance in California. In the course of that work, we have addressed some of these complexities. For example, the EdSource analyses have dealt with the diversity of the charter segment by placing charter schools into categories such as elementary, middle, high; classroom-based versus nonclassroom-based; and conversion versus start-up.

Certain patterns in the performance have emerged based on these categories. For example, EdSource reports in 2007 through 2009 showed that, after controlling for differences in student background and school size, charter elementary schools do not perform as well as traditional elementary schools, while charter middle and high schools have outscored their traditional counterparts. Similarly, conversions tend to outscore start-ups, but not always by statistically significant margins. Further, classroom-based charters have consistently outperformed nonclassroom-based charters.