SAT Coaching Found to be Giant Waste of Money and Time

Is probably what the headline ought to have read. The Wall Street Journal has an article casting doubt upon the extravagant SAT score improvement claims offered by outfits such as Kaplans. I've always been a little dubious about the whole test prep enterprise, but using a harder test at the beginning to make it look like the student has made massive improvements is just cheating.

This is pretty good money we are talking about here too: $100-$200 an hour! I'd do it for a discount, only $75/hour. Probably pretty good work for unemployed ivory tower types. If you think about it in standard deviation terms, these companies claim they can move you up 1 SD (the SAT is normalized such that the standard deviation is 100 for each sub-test, and there are three sub-tests now).  If we were talking percentiles, this would be equivalent to a move from the 50th percentile  to the 84th percentile, or from the 84th to the 97.7th. For those of us who like pictures, see the images below:

From well-designed studies I have read before this, the figure of 30 points average increase sounds much more likely. That is a third of standard deviation, a move from the 50th percentile to the 62th percentile. That is about the limit of any kind of focused intervention on intellectual skills applied to a large number of people. I've seen similar numbers come out of studies on efforts to improve the academic performance of inner-city kids (think Head Start). This is what that would look like:

This is an interesting conjunction of the basic SWPL attitude towards anything related to intelligence. On the one hand, the SWPL crowd would be aghast that the SAT is highly g-loaded, and as such is a pretty accurate estimate of intellecutal ability (IQ test). Ranking people by intelligence is so gauche! Hard work and determination must be more important! On the other hand, getting a high score is important to getting into a good college, and is probably secretly considered proof that you are smart, and therefore better than dumber people. This is exactly backwards, you didn't do much to deserve your smarts, and it has nothing to do with whether you are better than anyone. It does have a lot to do with SAT scores however.

h/t Steve SailerUpdate on 2009-05-23 16:03 by Ben Espen

Oops. The third figure shows +3 SD, instead of +2 SD. This is what happens without an editor.