Assigning Scores

For several years I have served on a committee that selects scholarship recipients.  A local community foundation administers philanthropic endowments of all types. Some donor largesse seeks worthy high school seniors and later students pursuing professional degrees to defray their university expenses.  Most awards barely nudge the astronomical tuition of private colleges or medical schools but would probably cover books or allow a graduate student to maintain a car.  Each season, the Foundation assigns me about twenty-five applications to review.  This brings me to the intersection of efficiency and effectiveness.

Applications have a standardized format.  Biographical information.  Classroom grades, Test Scores.  An open-ended activity statement.  One recommendation.  An essay whose topic changes each year.  The reviewer assigns a grade to each section, usually a 4 or 5.  The computer scores the composite, which the committee largely skims the top scores for the awards.  No unworthy person has ever gotten an award.  Unfortunately, some very worthy people, in fact many who would be at the top of my list to receive a grant, get swept away by a B or two on the transcript or the inexperience of the mentor who submitted the recommendation.  Having done hiring, skimming the top data does not segregate the people you would be most eager to bring aboard.  There is something not quite just about penalizing somebody who challenged himself with a course or two outside their strengths but fell short in the process.  The most successful people all failed at something.  That should reflect growth, not disqualification.

The applications in some ways expose larger problems with America's educational processes.  Several prestige universities suspended their SAT/ACT application mandates.  Most restored them when they found that kids who did not submit them underperformed in the classroom once on campus.  The applications I review are replete with disparities between grades and tests.  Some high schoolers take six AP Courses but don't score above a mediocre 3 on any of them, which suggests that either their enrollment in that class was premature or they were really not ready for college's academic rigor.  Yet in all six, their class grade was A.  People with stellar classroom scores have marginal SATs.  Even medical students already accepted with no B appearing on their transcript had mediocre MCATs.  There are certainly reasons why some students underperform relative to their abilities on timed high stakes testing, but the extent of this disparity really makes me wonder less whether the tests are right but the classroom assessment of performance is faulty.

I think final decisions on who gets money should basically abandon the algorithms, no matter how expedient.  Accept some realities like grades are inflated, though not universally inflated, so accepting at face value does not always isolate the best talent.  A better way to do this would be to have a threshold that keeps people in a pool to go on to the next step.  There is something amiss about penalizing somebody for challenging themselves or rewarding somebody for somebody else's leniency.  Dealing with mismatched grades and testing is much harder but university admissions committees have created their own methods for two generations.

Recommendations have gotten too automated.  I think they would be better evaluated as part of the application package by scoring them as subset scores instead of the current gestalt composite that really does not separate the candidates very well.  Perhaps separate scores for relation of person to applicant, or at least degree of familiarity, detail of specific activities to support the endorsement, difficulties or challenges that the applicant managed irrespective of success, evidence of tenacity, evidence of relations the applicant was able to form.  Replace a composite score with a scoresheet.

The one part I would leave relatively unchanged is the essay, but again scored by subsets.  Caliber of writing, creativity dealing with the assigned topic, specific illustrations.

And finally an interview with two or three committee members, the same number that currently score each individual's application, though not the individuals doing the written scoring.  A chance to assess understanding, explain mismatches of grade and test performance, find out what motivates the person.  Certainly the people in professional school programs have already acquired experience with this, though the high school seniors may not.

Reviewing the applications from medical students for several years, differentiation was difficult.  They all went to but a few schools for college and medicine, all had the same grades, all had a recommendation by somebody experienced at writing them.  Those are screenings.  We would do better having an assessment of the person, as each person is different, even if their paper credentials are not.