Like Phoebe, I was vaguely troubled by the annual NYT announcement that selective college admissions grows ever more selective. Unlike Phoebe, I'm at the U of C right now (in the Reg even!), and I can attest to the devastating effects of this stringency first-hand: the undergrads, especially the women, have become a lot more attractive, or at least, cleaner since we were there. They were always more hygienic than the men, but now, whoa. Also, they all wear the same casual-but-actually-calculated side-flip hairstyle. It is a travesty.
However, the question this raises for me is: can you design an experiment that tests the quality of decision-making that goes on under conditions analogous to current high-prestige college admissions? These admissions counselors are always saying that they're turning away thousands of "qualified" applicants. But since they're human beings, their choices among this overflowing pool of the qualified cannot be or even approximate random selection. They're also consciously trying to select the best of the qualified. And, indeed, the casual inference to be drawn from an 8% acceptance rate is that the accepted students are in some kind of 92nd percentile and up of awesomeness out of the available options. Is there some way to test whether people making choices that they assume will have an enormous impact on people's life outcomes under such high-selectivity constraints results in their making, on average, better or worse choices?
I realize that the vagueness of what constitutes a correct choice in this case might present problems for such an experiment. But, I suppose you could use a sample of the applications of recent graduates who ended up doing very well academically at a school for your baseline of correctness and applications of students who fared worse for incorrectness. This is subject to some difficulty of course, since some students do poorly at a school for reasons that could not possibly be predicted by their applications, but maybe there is a way to correct for that? If so, then maybe mix these two sets of applications together, impose a drastically low selection rate on one set of deciders and a much higher one on another, and set them to work to see who picks more correct applications? Or, something else?
My suspicion is that, on the whole, the dull but solid applicants are more likely to get passed over under conditions of extreme selectivity where the stakes are high (that is, those deciding believe that admitting someone will substantially improve his life outcome), but that they are also more likely to be successful in college than colorful iconoclasts (and also might become more colorful later as the precociously colorful get duller), so that highly selective admissions processes produce slightly worse academic outcomes than moderately selective ones. But, having been a dull but solid admit in my day, I of course would suspect that. So I want this to be tested.