The Uses and Misuses of the ‘Gatekeeper’ SAT

Standardized test scores can provide some useful information; however, they are just one point in what should be a much fuller, more comprehensive and accurate picture of the applicant (“The Gatekeeper Tests,” Review, March 10). The danger is in lionizing these scores and being
blind to their limitations and the great harm that can come to individuals and institutions by fetishizing these scores without appreciating their serious shortcomings.

The problem is that the relationship between performance on standardized tests such as the SAT and ACT and later school and career outcomes, is at the level of a large group, in contrast to what can be expected for any one individual. This is very misleading since what we are interested in is how to predict outcome for a particular, specific individual.

Perhaps the most unfair and pernicious misuse of relying on standardized test scores as gatekeepers to professions such as medicine and law is to the 20% of the population who are dyslexic. Without accommodations such as extra time on standardized tests, dyslexics typically will do poorly—not because they do not know the correct answers, but rather because they are unable to finish the test. Dyslexia is a paradox. No matter their high level of intelligence, dyslexics are slow readers yet at the same time fast, creative thinkers.

Sadly, colleges and professional schools which, rather than focus on the whole candidate rely primarily on standardized test scores, will miss out on having extraordinary graduates who are both dyslexic and brilliant and who score poorly on standardized tests.

Sally E. Shaywitz, M.D.
Bennett A. Shaywitz, M.D.
Yale University
New Haven, Conn.

Read the article here. (Must have subscription to the Wall Street Journal)

Read the full response by YCD&C Co-Directors Sally and Bennett Shaywitz to the Wall Street Journal piece on standardized tests.

No, Virginia, Standardized Tests Do Not Predict Future Achievement

Wouldn’t it be wonderful if we could confidently predict not only achievement throughout school but also success in careers by the score on a single test administered at the end of high school? In a recent Wall Street Journal article Professors Kuncel and Sackett, two very sophisticated industrial psychologists, argue that standardized tests such as the SAT and ACT do in fact accomplish this legerdemain. Many in the field have acknowledged that this relationship is not at all as usefully predictive as these authors would lead us to believe. For example, while there are studies that some researchers selectively cite to support the argument that these standardized test scores predict future performance, other studies can be selectively cited to show the reverse, that is, that standardized test scores are not very good predictors of these same outcomes.

To paraphrase Richard Pryor, whom should we believe, Kuncel and Sackett’s claims or our lying eyes? We all know of, and read almost daily about those men and women who in real life excel in a range of fields, yet who would have been overlooked, that is, not have been predicted to be outstanding if judged solely by their scores on standardized tests.

The problem is that the relationship between performance on standardized tests such as the SAT and ACT and later school and career outcome is a relationship at the level of a large group in contrast to what can be expected for any one individual. This is very misleading since what we are interested in is how to predict outcome for a particular, specific individual. But even considering what happens at the group level, in the authors’ own data , the strongest relationships (correlation about .5) are found between standardized tests such as the SAT and outcome when outcome is measured by other standardized tests such as licensing exams and qualifying exams. It is important to know that even here, only 25% of the score on the licensing exam is explained by the SAT. This means that 75% of the score on the licensing exam is not explained by the SAT scores but is rather due to other influences. Much weaker relationships (correlations .1 and .2) are found between standardized tests such as SAT and meaningful career outcomes such as research productivity and publication citations. Here, less than 5% of the outcome is explained by the SAT. This means that over 95% of achievement in research is not explained by the SAT scores but is rather due to other factors.

Kuncel and Sackett cite a study of students who scored at the very top of the scale on the SAT math test at age 13 and seek to relate these scores to what they consider exceptional achievement two decades later. As an example, they report that students who scored at the very top at age 13 were more likely than their lower scoring peers to secure patents working for Fortune 500 companies. Yes, but even here this remained an extremely rare event so that even in this highest scoring group, just over 4% secured patents–96% of these extremely high scorers never secured a patent, not much help in predicting to this outcome. It is not difficult to see why scores on standardized tests such as the SAT will not be very helpful or even misleading to either college admission committees or graduate school admission committees trying to select the best students, or those in industry tasked with selecting the most productive employees.

Perhaps the most unfair and pernicious misuse of relying on standardized test scores as gatekeepers to professions such as medicine and law is to the 20% of the population who are dyslexic. Without accommodations such as extra time on standardized tests, dyslexics typically will do poorly – not because they do not know the correct answers, but rather because they are unable to finish the test. Dyslexia is a paradox — no matter their high level of intelligence, dyslexics are slow readers yet, at the same time, fast, creative thinkers.

Sadly, colleges and professional schools who rather than focus on the whole candidate rely primarily on standardized test scores, will miss out on having extraordinary graduates who are both dyslexic and brilliant and who score poorly on standardized tests. Wise were the schools that chose to admit dyslexic applicants with less than the top scores such as renowned attorney David Boies; cardiac surgeon and Cleveland Clinic CEO and President Dr. Delos Cosgrove who has been awarded over 30 patents; innovative financier Charles Schwab and highly respected economist Diane Swonk. Thinking of all the incredible good these individuals have contributed to society, imagine what would have been lost had their respective schools looked only for high scores on standardized tests and not at the real person.

Which brings us to the question relating to college admissions: What would happen if schools chose not to use or to have standardized test scores optional in their admissions process? Back in 1984, at the suggestion of then Admissions Director, William Hiss, Bates College chose to make submission of standardized test scores optional and have, one must say, gleefully reported on the results. According to current Dean of Admissions Leigh Weisenburger, “Students who submit scores at the point of application and those who do not, perform almost precisely the same at Bates.” She states that “while standardized tests are useful, we have found that three and a half years on a transcript will tell us much more about a student’s potential than three and a half hours on a Saturday morning.” Reflecting Bates’ positive experience, there are a growing number, now more than 1000 accredited colleges and universities with optional submission of standardized test scores.

Yes, standardized test scores can provide some useful information. However, it is just one point in what should be a much fuller, more comprehensive and accurate picture of the applicant. The danger is in lionizing these scores and being blind to their limitations and the great harm that can come to individuals and institutions by fetishizing these scores without appreciating their serious shortcomings.