1. Statistical analyses of the sort that the Dallas Morning News published and the Amarillo Globe-News reprinted to imply cheating is always circumstantial evidence. Because we are berated by statistics through our media and by our politicians, and because those statistics are so used, misused and abused to manipulate us, I have long believed that basic statistics should be taught as part of the regular curriculum in high school. The understanding of statistics is a citizenship issue. Statistical analysis has also demonstrated that the results of the 2004 national election were extremely suspect. Did we see these statistics in the Morning News?
2. Accordingly, although statistics can tell us the chances of something happening, they cannot prove why or how it happened. That the scores improved is known. The newspapers provided numbers and context that demonstrated the unlikelihood of it happening. But without additional specific evidence, we jump to a conclusion if we assume cheating.
3. Amarillo Independent School District did a good job of educating the media and the public about the statistical issues involved. They credibly demonstrated how small results could look bigger under the Dallas Morning News methodology. Superintendent Rod Schroder also—like a good defense attorney—established reasonable doubt by providing credible alternative explanations for the improved results. These alternative explanations included increased tutoring and outstanding teaching.
4. However, appeals to sentiment remain logical fallacies. The coverage in the Amarillo Globe-News was surprisingly thorough and accurate, with some exceptions noted below. Through the magic of my S.O.’s roommate's TiVo, I was able to also view the coverage on KCIT Fox 14 and KVII Channel 7 (the ABC affiliate). (Kim Fischer of Fox 14 amusingly questioned the “validididity” of the results.) A viewer of the television coverage only might be forgiven for concluding that the essence of AISD’s defense involved a vulnerable-looking blonde crying crocodile tears. This blonde- whose picture also graced the front page of the AGN- was Lee Elementary School principal Karen Atkinson. Apparently using the same tactic seventeen-year-old girls sometimes use to get out of speeding tickets, Atkinson cried for the cameras that her students had been upset by the coverage and fretted that people thought they were “stupid” because they were “poor.” This testimony was unseemly, unprofessional and unnecessary, since the facts (so far) appear to be on AISD’s side.
5. Furthermore, although the TAKS results were interpreted by the Morning News in such a way to maximize unusual results, and although the AISD provided credible evidence which should settle the matter unless other evidence of cheating appears, the results do remain unusual. Therefore, the victory-crowing of the headline “Parents, principals knew it was quality education, not cheating” seems premature and misleading. “Believed” might be a better word. Comments in the article such as
"You can't compare third-graders with fourth-graders. It's like comparing an apple and a lemon, and it just don't work. They had a bad set of figures to start with; you can't compare across grade levels."
prove my main point: that the general public is undereducated about statistics to the point that it is a danger to the republic.
6. AISD’s “internal investigation” remains just that: an internal investigation, with all that implies. Schroder is perhaps best viewed as a defense attorney. His job is to provide a credible narrative for why the scores appear as they do that does not involved cheating on the part of AISD employees. At this point in time he appears to have done so, and absent further evidence from the prosecutors (Dallas Morning News) the matter should be dropped.
SPACEDARK
I KNOW there are PTS readers/contributors who are better statisticians than I. Your thoughts?
ReplyDelete(Full disclosure- I work part-time in the sports department at the Globe-News, and my sister is a third-grade teacher at Lee that was really hurt by this whole story, so I think my bias is a little more towards the person that i've known my whole life than the paycheck)
ReplyDeleteBasically, the Morning News treated the data like SAT scores, instead of a set of binary data, because the TAKS is essentially a long pass-fail test, unlike the SAT. You may get a low or high score on the SAT, but you don't pass or fail it.
I understand why the AGN went with the story, but I don't see what made the Morning News run their numbers in such a way in the first place. Comparing pass-fail rates between 2003 and 2004 third-grade classes would give you the data you wanted to see if looking for cheating, but with a pass-fail test, many scores are clustered around the pass-fail line, and a jump or drop of a few points in any score changes the pass-fail status. In addition, reading the report, it turns out that the Morning News threw out all scores from schools with fewer than 30 third-graders (around 500). While such a method would be okay with a study based on pass-fail, throwing out these schools had the effect of making percentile changes appear even greater by decreasing the population. The same thing would happen with any set of data treated like this. Hank Blalock's percentile rank in hitting improvement would look a lot greater if all lefties or switch-hitters were removed, for example.
The comment in the AGN story today about comparing apples and lemons sounds odd on its face, but it does reveal a truth that no one likes to admit about our Bush-imposed system - Since students have to pass the third-grade reading TAKS to go to fourth grade, resources are funneled into third grade to keep the kids from having to retake the third grade. Students who get literally days of extra instruction in third grade don't always get the same attention in fourth grade.
Thank you for your comments. I should add that, although we frequently critique the AGN here, I see nothing wrong with the local newspaper's reporting the story. Once the Dallas Morning News reported it, it would have been irresponsible for the Globe-News to ignore it. I'm also not sure the Dallas Morning News should have reported it the way they did with no evidence other than suspect statistical analysis.
ReplyDeleteI totally understand how the Globe-News gets a fax, or call, or email from the biggest paper (by reputation) in the state, and runs the story without really digging through the data themselves, but what's going on with the Morning News?
ReplyDeleteThere are stories of students from elementary schools around Dallas that had 90-100% passing rates on the TAKS as third, fourth and fifth graders who got to middle school and were incredibly far behind everyone else. Obviously, something was going on, but the DMN used the wrong statistical method to try to find out, and wound up accusing a lot of innocent schools who really did work hard.
That's what I really don't get, it seems like the DMN would have an education beat writer, or at least someone on staff who would have some grasp of what these numbers mean and how skewed they are before publishing them.
Welcome to PTS Travis!
ReplyDelete