Saturday, October 29, 2016

Data analysis is hard

As I said in the previous post, I find the student standardized test score data valuable mostly for what it shows us about the significant gaps between subgroups of students. What I find less informative, though, are year-to-year comparisons and comparisons between our students and students elsewhere. It is too easy to impose preconceived conclusions on those comparisons, when there may be other plausible hypotheses that are also consistent with the data.

For example, at our meeting last week, the board were presented with bar graphs showing that our students’ scores compare favorably to those of students nationwide. Then we saw more bar graphs showing that our students compared favorably to statewide students on a measure of annual score growth.

There is always an air of self-congratulation to that kind of discussion. When our students compare favorably with others, that means our district is doing a good job, right? But actually, that isn’t what it means. Even if our students compare favorably with students elsewhere, it doesn’t follow that our district’s policies and practices are the reason why. We live in a university town where a lot of people are highly educated; that alone could potentially explain why our students sometimes score better than the average student. For all we know, they would get even higher scores if our district made different educational choices; our district might actually be holding the students back! The data neither prove nor disprove that hypothesis.

Moreover, the data also show that our students suffer by comparison to students elsewhere in many ways. For example, the Iowa Assessments data show that on all three tested subjects (reading, math, and science), at every grade level tested (third through eleventh), the percentage of our students who are non-proficient is higher than it is for students statewide (with the one exception of eleventh-grade science). Those differences appear to be largely a function of the scores of our subgroups of students on free and reduced-price lunch, black and African-American students, students receiving special ed services, and students who are English-language learners. Here’s an example of the comparisons (no helpful bar graphs on this data set!):

And our students are often doing even worse, relative to statewide, than they did last year. Here’s what that cohort’s chart looked like last year:

If we let the district take credit for every student success story, we also have to blame it for every unsatisfactory result.

But in fact, the data does not dictate either conclusion, because it is very difficult to evaluate all the potential causal factors that affect our student test scores. There are sophisticated statistical methods, like multivariate regression analysis, that could at least try to do so, but they have their own limitations. In any event, I’m fairly sure that our district is not even attempting such sophisticated analyses on its test score data. It’s probably much closer to the truth to say that our district, when given a new set of test score data, has a tendency to simply eyeball the changes from the previous year and then make assumptions (not dictated by the data) about what those data tell us about the effects of the district’s practices.

For example, this year we’re instituting a weighted resource allocation model, shifting resources toward schools that have more kids in the subgroups that are showing proficiency gaps. Next year, we’ll look at how the proficiency scores have changed. If the gaps are at all smaller, it will be tempting to attribute the improvement to the weighted resource model. But the data would be consistent with other plausible explanations as well, since we can’t possibly hold all other variables constant like in a laboratory experiment. At the same time that we’re shifting resources, we might also be changing disciplinary practices, and trying to diversify the teaching staff, and providing bias training, etc. (And of course we’re usually measuring different actual kids from one year to the next.) If the gaps subsequently improve, it’s even possible that the weighted resource model actually made them worse, but that the other variables more than offset that effect.

If we shift resources and then the test scores go up, we’ll want to assume a causal link. But if we shift resources and the scores go down, we won’t assume that the shift caused the decrease—and if anything, people might even say, “See, the scores went down, so we need to shift resources even more.” My point is just that our explanations often come from sources other than the data itself, since the data is consistent with multiple hypotheses. I don’t think that’s crazy; I think it makes sense to bring experience, judgment, and even instincts to bear in interpreting any data. But it can certainly lead people into temptation, as it becomes fairly easy to use data to justify preconceived conclusions.

In any event, if someone tells you that our district is doing great because the kids outperform their counterparts elsewhere, remember that what you’re hearing is probably closer to sales than to science.


Amy said...

Interesting. We seem to be unwavering in our commitment to the Average White Kid, and not all that terribly interested in anyone else.

You can send that along to Steve, btw. Since he's in charge, I'd call that a criticism.

Anonymous said...

Wow. Just about every subgroup in that cohort saw a decrease in proficiency. FRL, Black, Hispanic, etc.

This is what you get with the "busing for balance" district plan supported by Kirscling, Lynch, DeLoach, Roesler, and Murley. It has overshadowed other effective strategies and will only get worse with time.

When are we going to put this social engineering experiment behind us? No one wants destabilized and always changing boundaries. Lincoln families don't want to be pulled east to west to east (I really don't give a crap that some guy pounded his fist and yelled, "Give me Lincoln"). Alexander students don't want to be ostracized from the east side. Kirkwood parents don't want to be pulled away from West. Horn parents don't want to be at their third school in three years. And parents of elementary school students don't want to be part of split feeder systems.

People want a solid, stable, dependable school district that is predictable and hard working. And right now we are anything but.

Anonymous said...

12:09. Think please. The bussing plan hasn't started yet

Anonymous said...

12:48 I beg to differ, busing for balance has been an educational focus of this district leadership for eight years...

...The FMP was based on proposed busing boundaries.

...Horn, Weber, and Borlaug have been moved around for the sake of "balance" for years.

....A Superintendent's directive was recently put in place making "busing for balance" a district priority.

....Buses have been moving neighborhoods away from one school and towards another since Lincoln was moved to CHS in '09.

As long as "fluid boundaries" and "busing for balance" are educational priorities in this district we are going to fall behind.

Just because the bullet isn't flying from the barrel doesn't mean you aren't hunting.

Anonymous said...

Even assuming thats true, and as Chris correctly points out in another post, correlation does not equal causation. The achievement gap discussed here has existed long before any "busing for balance" was implemented. It's also ironic you pointed out the Borlaug, Weber, Horn boundaries. The last board asked for and examined data specific to those schools and found a "statistically significant narrowing" of the acheivement gap measured by gains in the lowest testing cohort. Again, correlation does not equal causation but you can make what you want of that data.

Anonymous said...

12:09 and 12:48
The FMP looks for new buildings more like its based on where cities want development or had promised cities a school would be built. Otherwise, no one would have built Alexander in a location where it is has such a high FRL rate.

Prior to "balance" as you put it, Pheasant Ridge kids were bussed by Horn to Roosevelt. This wasn't right and should have been changed.

There may have been a decision to move Lincoln in 09 but kids Murley grandfathered in so many kids and siblings that not many Lincoln kids went to City High until 13-14 and even then, not many went.

2:29 is spot on, "correlation does not equal causation."

Anonymous said...

Anon 2:29 and 3:35.

1, Borlaug, Weber, and Horn were outpaced by a similar three school cohort that was not redistricted for balance. The fact that the last board was so short sighted to miss that basic comparison is a huge problem and is yet another reason why we need new leadership.

2. Correlation is not causation but with massive amounts of descriptive data correlation can be assessed in terms of probability. And there is a massive amount of data that shows a high probability of a school district weakening under a redistricting/busing for balance approach to education. So seeing the data in this very small set of data points it is no surprise.

3. And the old argument, "we just need more time for redistricting and busing to work" has never proven true. In fact there are compelling examples that show the sooner that approach is abandoned the better off you are.

This district has a widening achievement gap.... DINA 9 anyone?

Anonymous said...

What is going on at Weber? Their "closing the gap scores" are the worst in the district, and maybe the state.

There are 19 elementary schools in the district and each one has been assessed by The Iowa Department of Education for Closing the Achievement Gap (v school and v state). And looking at that data one jumps out like a sore thumb:

Weber's Proficiency Gap scores are 11.2 (v school) and 14 (v state). The next lowest elementary school in the district is 33 and 39. And the majority score above 50.

You put this small piece of data together with other larger pieces of data and the overwhelming results show again and again that "redistricting and busing for balance plans" are failures. But we didn't have to run these pointless experiments in the district, you could have done a reasonable amount of homework and seen that these plans are worse than a lemon on a used car lot.

And BTW Lynch, Kirschling, DeLoach, Roesler, Murley all want to continue the social engineering experiment that led to Weber's outcomes. Come on ICCSD, you're smarter than this!

Anonymous said...

To me, what is significant in the reported test scores is the discrepancy between how Iowa City is performing re the level of nonproficient kids to to the rest of the state. In this regard, Iowa City is simply doing much worse than the rest of the state, especially in terms of educating black, hispanic, FRL, and IEP kids.

Being Iowa City, we like to come up with new buzzwords like weighted resource allocation, etc. At the end of the day, its educational leadership. The lack of this got us into the situation we are now in - it didn't happen overnight.

Keep the question simple: why are other districts in Iowa better than we are?
Keep the solution simple: hire an Iowa administrator with proven competence in improving outcomes. We don't need more consultants or fancy buzzwords.

Anonymous said...

Anonymous at 12:48. Perhaps without busing/boundary changing things would be even worse? Regardless of what building low performing kids go to, they would still be reported in our district wide numbers. Other Iowa districts have the same issue but tend to have better outcomes. What are they doing which makes them better? Why can't our district do the same?

Anonymous said...

Thinking of redistricting/busing/boundary as a way of slowing down decline is a mistake. In itself the approach is corrosive and the nationwide results are there to show it. And if you prefer to look locally, can Weber get much worse?

No, things aren't going to get worse without busing/boundary in the ICCSD. Things are going to get better when that policy is gone and an emphasis on school culture and accountability returns to the district.

Chris said...

Thanks, everyone, for the comments. I guess I'm struck by how people are debating the role of socioeconomic balance and district culture, etc. -- all of which are certainly worth discussing -- while we hear very little discussion about actual curricular choices the district is making as potentially affecting how much/well kids learn. I would think that our choices about what to teach (and the underlying approaches they represent about how kids learn) would be at least one of the things at the center of any discussion about kids' proficiency rates. Yet the debate never seems to get much into that topic.

Is that because of a feeling that we should defer to our administrators to use their expertise to make those decisions? Curriculum decisions do ultimately come before the board for approval, though. How should the board evaluate them? Moreover, the board at the very least would have to hold our administrators accountable for making good choices about curriculum. How could we do that if we defer completely to their judgments of what the best choices are?

Is it plausible that we could improve our proficiency scores (and reduce the gaps) if we were doing something different with our curriculum? If so, by what process would the board pursue change in that area?

Anonymous said...

Chris, the board should not be expected to second guess the administration or to do the administration's job - it is not composed of educational experts. But that assumes that you have the benefit of administrators who have this expertise. I am not confident that you do.

The simplest thing to do is to look at the current Iowa districts that are performing well and try and poach an administrator or two from these districts. We obviously need their knowledge. Although whether these administrators would be willing to come to our district with its current administration is another question.

The whole busing/boundary thing seems like a red herring to me. The district's poor performance is based on district-wide numbers. What building these kids are in won't improve the district wide numbers.

EDJ said...

"Anonymous Anonymous said...
12:09 and 12:48
The FMP looks for new buildings more like its based on where cities want development or had promised cities a school would be built. Otherwise, no one would have built Alexander in a location where it is has such a high FRL rate."

The FMP doesn't choose the locations for schools. The (then) board chose and bought the land to build the schools that the FMP specified as needing to be built. Note that the board just bought more land in North Liberty, anticipating future needs. That's not on the FMP now, but it could be incorporated into it now.

I think the Alexander location wasn't a great choice, but I'm not at all sure that there were many better choices available either. The district was to some degree constrained by the need to find a piece of land that's suitable in terms of size and layout, that's for sale, and that serves an attendance need. Opening Alexander relieved overcrowding at Twain (which was about 120 students over capacity) and at Wood (which was more than 200 students over capacity and which had numerous temporaries set up outside of it for years).

At the board meeting where the boundaries for Alexander passed, I spoke against the specific boundaries they chose, and specifically asked that they include more low poverty schools in the mix. But, in retrospect, I'm not sure there were many other options. The closest low poverty school is Longfellow, which is directly north of Twain. There's not much you can do for Wood or Alexander or Hills by involving Longfellow in the mix. You could maybe involve Horn in the mix? But the schools aren't close, and there is a whole lot of non-residential space between them.

The gambit with Alexander's location was to hope that the combination of the new school, and the soccer park, and Terry Trueblood park, will help bring more affluent development to the area. This isn't a case of putting the school "where cities want development," it's looking at where people are, and what the city is doing in a particular area, and reacting to it. At the same time, the board began having regular meetings with the District municipalities to encourage them to adopt inclusionary zoning policies which would spread affordable housing out and not bunch it up together where it causes school zones to become economically isolated. And, they began drafting what would eventually become the WRAM.

I have some skepticism about whether this is going to work or not. On the one hand, there are some nice houses in the newer neighborhoods close to the school, and you can get a lot more house down there than you can in many other locations in the district. On the other hand, (mostly unconscious) racism will keep even self-proclaimed Iowa City liberals from flocking down there, and opening a school as a high-FRL school isn't going to encourage more affluent development. There's also the problem that, if you do successfully encourage gentrification in that area, then people are going to need affordable housing elsewhere. That's where the lobbying for inclusionary zoning is supposed to come in. We'll see how well this works. I'd sure like it to.

Musings about the history of this aside, there needed to be a school somewhere in that area to relieve already-existing overcrowding, and the District had limited options in terms of available land to put a school on.

EDJ said...

Sorry Chris- your last post there is a great point, but just one more on the balance/boundary thing:

"The whole busing/boundary thing seems like a red herring to me. The district's poor performance is based on district-wide numbers. What building these kids are in won't improve the district wide numbers."

There's actually a good amount of scholarship that suggests that this isn't the case. And, per the state report cards (which should just be called data collections and not typified as "report cards") the building does make a difference. Minority and low SES kids consistently have done better in schools with an overall FRL% that's under the 50-60% line, and consistently do worse over that line. As the subgroups do better in each school, then the district-wide numbers will be better as well.

EDJ said...

"Is it plausible that we could improve our proficiency scores (and reduce the gaps) if we were doing something different with our curriculum? If so, by what process would the board pursue change in that area?"

Maybe so. It might be helpful to see how well our curriculum speaks to the actual demographics of the district, both economic and racial. If we're looking at a gap that's being driven by differences in race and socioeconomic status, and the structural factors associated, then maybe there is curricula that does a better job of addressing all groups. I don't specifically know what that looks like though. In terms of how to pursue it, I'd say research is in order, but I don't know exactly how that would work on a board level.

I think it would also be useful to disaggregate the race and SES data to a greater degree. We know that race and class act in consort, but to what degree are our minority kids being held back because many of them are also poor, and to what degree are they being held back because our school system, in fact all three municipalities in the district, are institutionally white and unfriendly to or unfamiliar with the needs and wants of black and latino kids?

I'm curious to what degree differentiation is central to our teaching practice across the district? I know that in many of our high-poverty schools its been a necessity in order to keep advanced learners moving forward while bringing kids who are coming in needing more up to speed.

Other than those thoughts, its hard for me to see how different curricula helps us overcome a gap mainly caused by structural inequalities in American society, and compounded by building demographics that keep most of the poor and minority kids in the district firmly inside just a small percentage of our elementary schools.

Anonymous said...


Iowa City has long wanted the Alexander site to be a school and IC staff wants to gentrify that part of town. The school district went along with what Iowa City wanted and there is now a high FRL school. After Iowa City builds soccer fields in the east, does anyone really believe Kickers will stay where it is? IC also promised Windsor Ridge years ago it would get a school there and at one time there was land for it. Now Windsor Ridge is getting a school. One of the north elementary schools was to be built in a cornfield with a lot of initial bussing costs. Does this benefit kids or developers?

Is there going to be enough inclusionary zoning across the district to make a difference?

Anonymous said...

EDJ, once again short of facts and long on overly generalized conclusions. 50-60% tipping points can not be taken at face value. There are other factors that need to be considered and many of those other factors are within the ICCSD's range of influence. But keep in mind that as long as the district's current fluid boundary policy (redistricting and busing for balance) is in place that range of influence is greatly reduced.

And here is a little mind expanding exercise for you EDJ and all of the others (DeLoach, Lynch, Roesler, Kirschling, Murley) ...

1. Explain CR Washington, why are they making great progress in closing the achievement gap while being 7% higher FRL than City High, which is seeing a widening gap?

2. Explain Weber, at 42% FRL they are the worst school in the district for closing the achievement gap? Are they the worst in the state? What happened to your splendid "redistricting for balance" results there Murely? And don't give try and pawn off the non comparative data here. We need accountability not deception.

We need leaders in this district looking at The National Title 1 Distinguished School Program and The US Department of Education National Blue Ribbon Schools Program. We can make those lists of recognition but it is going to take new leadership.

How many visits have EDJ, Lynch, Murley, Kirschling, DeLoach made to districts that have tried redistricting for balance? Safe to say, ZERO? So you embarked on and support a massive plan that has occupied countless hours of adult focus but you couldn't find a day or two to visit La Crosse, WI?

EDJ said...

Anonymous November 2, 2016 at 11:08 AM- Is that you Mike Colleran? That sounds just like the stuff you post on your ICCSD Monitor Page.

EDJ said...

Anonymous November 2, 2016 at 11:08 AM:

There's nothing about a school that has 35% kids on FRL and 5% kids on Reduced Lunch scoring in the 70s and 80s for proficiency that contradicts what I am and have been saying on the effects of building demographics.

I can't answer the question on Weber right now because I haven't looked at those numbers, and given the blog you run and the obvious thrust of your comments here, I'm not going to rely on your characterization. I'll get back to you though.

Anonymous said...

I think people post anonymously for a reason. Personally, I like the freedom it brings to the discussion. I see a lot of substantive posts being made by people choosing to be anonymous.

And I'd like to hear from some of our publicly elected community members. Roesler follows this page but hasn't said much. How about Kirschling, DeLoach, Lynch, or Murley?

Mitchel Starc said...

Wow, great post.