Tuesday, August 2, 2011

Washington Post's "Challenge Index" - What Does It Really Mean?

The Washington Post has published an update to their annual “Challenge Index”, which they describe as a measure of a high school’s willingness to challenge its students.

In the preamble to their results, the Post described their methodology as follows: “The formula is simple: Divide the number of Advanced Placement, International Baccalaureate or other college-level tests a school gave in 2010 by the number of graduating seniors. While not a measure of the overall quality of the school, the rating can reveal the level of a high school’s commitment to preparing average students for college.”

They went on to describe two additional measures which the Post reveals in their summary of statistics for the high schools covered by their survey: “E & E is the percent of all seniors who had at least one passing grade on an AP or IB exam. Subs lunch is the percent of all students designated low income.” It appears that neither of these two categories of information are used in their Challenge Index calculations, so you may simply think of them as informational data points.

This is a survey which was originally produced annually by Newsweek magazine, which was sold by the Washington Post in 2010, with the Post obviously retaining this survey which seems to generate interest among some parents. To my mind, an index which is determined by a number of tests taken, without any regard for the actual grades obtained by the students taking such tests, is of very little value indeed.

Nevertheless, it may be a good indicator that a particular high school might be taking a much-too-exclusive attitude about who may take such tests, or even an attitude that it fails to sufficiently challenge the students in that high school.

However, if someone were to focus solely upon the Washington Post’s “Challenge Index”, without simultaneously paying careful attention to the NY State assessments (in the case of schools within this state) of college readiness, as determined by actual scores on NY State Regents exams in certain categories, they could obtain a very misguided sense of the value of one high school versus another.

As an example of this, I would invite readers to refer to a community commentary which provided analysis of the State assessments, as revealed in Newsday this past June, and which may be viewed on the following web page:
http://elwoodilluminations.blogspot.com/2011/06/sed-survey-shows-elwood-did-very-well.html

There was a subsequent article in the NY Times, which provided greater insight of the State process and of the meaningfulness of these assessments for John Glenn HS, and you may refer to the related community commentary on this web page:
http://elwoodilluminations.blogspot.com/2011/06/ny-times-article-offers-greater-insight.html

Now, to illustrate the somewhat mythological “value” of the Washington Post “Challenge Index”, contrast the Challenge Index results for Cold Spring Harbor HS, followed by the results of most of the other high schools in the Town of Huntington, with that of John Glenn HS, as seen in the chart below.

Then, contrast the results of the State assessments based upon actual grades achieved, rather than the number of tests taken without regard to scores achieved (i.e., the Washington Post “Challenge Index”).

You will discover that John Glenn’s Class of 2010 had the second best NY State key Regents scores among the nine high schools in the eight school districts in the Town of Huntington.

As a further analytical note, while I initially wondered why the E & E rate of Glenn was so low, in contrast with how well we did on the State’s assessments (again, based on actual scores on NY State Regents), it finally dawned on my slow mind that any school with a low percentage of its students taking these AP/IB/other college level courses (such as Glenn, in comparison to its peers), is predestined to have a low E & E score which is based, as the Post stated and as I cited in the third paragraph of this commentary, on “...the percent of all seniors who had at least one passing grade on an AP or IB exam.”

That was my “Ah-Ha” moment of the evening.

Now, for the dubious honors, and dubious value, of the Washington Post’s “Challenge Index” for the Class of 2010:

HS_____ Index ____Rank-NE ___Rank-National __E & E ___ Sub. Lunch.

Cold S H _4.085 ____ 17 ________117 __________74.50 ___ 0.00

Harborf. _ 3.258 ____36 ________244 __________57.70 ___6.00

HHH-W _2.542 _____75 ________478 __________46.00 ___11.00

HHH-E _2.457 _____88 ________533 __________44.40 ___ 10.00

Commack _2.174 ____ 115 _______694 __________40.00 ___ 3.00

Hunting. _2.104 ____125 ________750 __________34.40 ____32.00

Northp. _ 2.088 ____131 ________ 765 __________52.10 _____6.00

Glenn ___1.632 _____199 _______1,181 _________34.00 _____14.00

Whitman _1.583 ____212 ________1,240 _________37.40 _____36.00

Having dismissed the “Challenge Index” as a meaningful assessment of quality, there is nevertheless one possible application which can be used as a Kaizen Moment, or Continuous Improvement Opportunity, for the administrative management of John Glenn HS: How about challenging the students to try more demanding courses, as another way of improving their overall performance, and, particularly, their overall opportunities to excel in their post-secondary education endeavors?