Friday, December 30, 2011

Beyond Scores

I have written before about how hard it is to recruit good students (e.g. here and here). My department  is ranked in the top 15 or so, therefore it's quite a good place but we don't routinely bring in superstar student candidates. However, I believe that if we are a bit clever in how we recruit, we can find some true gems among students who get passed up by more highly ranked schools because there is something a bit off with their record.

In the physical sciences, American students have multiple fellowship opportunities available to them and they are therefore (on average) easier to fund than international students. This is part of the reason why good American students are very attractive to hire and they routinely get snatched by places more highly ranked than mine. So, in order to find some good talent willing to come here I typically look hard at international student applications. I generally give a lot of weight to the quality of the undergrad institution, grades in math and physics, letters of recommendation, research experience and papers (if the student has any), as well as my previous experiences with students from the same school. As for tests, I look at TOEFL scores for international students and GRE Quantitative but not at Verbal or Analytic. Unfortunately, there are several countries that produce students who have very high Verbal scores, but when they come to the US it turns out they don't speak English fluently at all and have trouble following classes. The Analytic part, since they changed it to "Analytical Writing" is -- in my humble opinion -- completely useless for selection in my field. These are essays scored by humans; being able to write persuasively in essay format in English is not a skill most international students have when they take the GRE test, even if they speak the language well and have decent listening comprehension. The average quality of English instruction and the availability of resources  for test preparation vary greatly from nation to nation and should be taken it into account when looking at test scores. Also, in many countries the GRE and TOEFL tests aren't offered very often, may still be paper-based, and taking the tests more than once may be prohibitively costly for the student.

My best two students came in with really mediocre test scores. However, they were from countries with selective admissions processes at the undergraduate levels and had very good grades in math and physics and some research experience. So this year I was going to bring in another student from the same university from which I recruited good students before; even spectacular students from this country tend to do so-so in the GRE.  The new candidate has excellent math and physics grades, stellar letters of recommendation, even published a paper in a reputable journal. GRE Quantitative maximal, he even took a GRE Subject test (not necessary for admission to this program, which is in engineering) and received a very good score. We have been in contact and I was sure he would get formally admitted without a problem.

The student contacted me a few days ago, very upset, saying he was rejected. I was stunned. I inquired around a bit with our admissions committee, and apparently the person who looked at the student's file thought his GRE Analytical Writing Score of 3 was too low and that was what got the student nixed. For comparison, my two excellent American-born and -educated undergrad researchers, who both went on to top-5 places for their PhDs,  received 4's on their GRE Analytical Writing; I think holding international students to the same standard is  pretty ridiculous. The colleague who examined the student's file appeared defensive about the decision, which leads me to believe he hadn't even examined the rest of the student's file carefully.

Anyway, I had to make a stink about it (thank God for tenure that enables me to do this) and the student's file will be reconsidered. But this leaves a bad taste in the mouth -- that a good student can be dismissed based on a pretty irrelevant test score. I know people on admissions committees work hard, there are a number of applications to go through, and it's very time-consuming as far as  service roles go. Picking cut-offs to streamline the process seems like a must. But, I have been less than happy with the quality of the average graduate student admitted in my department ever since I started my job here; I feel we admit too many poorly prepared or insufficiently motivated candidates, but most importantly -- too many candidates who don't have the background for the type of research that the faculty  in this department do. I generally hand-pick students on my own before the selection is made and then make comments to the committee, but this time I didn't get to the student's file before the formal decision was made.  It looks like it's time for me to get more involved in the admissions process at the department and university levels and push for giving admissions criteria at my place a big-picture do-over.

Having formally high and inflexible selection criteria may seem to convey that we are a top-notch, hard-core place. We are, in the sense that there is very good science being done here, but let's not kid ourselves. We are not a top-5 place, we are not a magnet school to which everybody applies and everybody wants to go,  so we do not have our pick among the cream of the crop. We need to be smarter than inflexible admissions and test scores, accounting for the fact that the obviously superb students won't come here. We need to be more efficient at identifying and attracting the not-so-obviously superb students who will be interested in coming here. All this seems obvious, but is surprisingly hard to implement, especially at a level higher than a research group (department, college, university) where the different stages of the admission process are decoupled from one another.

How is the graduate admissions process handled at your school? If you are a faculty, are you happy with the process? Do you pick from admitted applicants or pre-select on your own? Please tell us also what your area (broadly) is and how highly ranked the department is, and anything else that you deem pertinent for the successes or failures of graduate admissions at your school.

And Happy New Year everyone!


Anonymous said...

At my institution & department (Biomedical Engineering at a well respected university in Canada) , the official standards for admission are fairly vague and relatively easily obtainable such that once an advisor has verbally agreed to hire a student, it would be qiute surprising for the student to not be granted admission. This is the way I think it should be in departments where you are admitted under the funding of a specific advisor.

I don't think that it makes sense for grad school to have a lot of formal cut-offs at the institution or department level, however, the school is concerned about graduation rates, etc and it is in their best interest to have some degree of regulation.

I do think that any formal cut-offs that are deemed important need to apply equally to both domestic and international students. If it is thought that international students shouldn't be held to a certain level on a GRE score, etc, because that metric is not particularly useful for the particular field, than domestic students shouldn't be held to the level either.

Anonymous said...

I think I was in a very similar position as your student was. My issue was a GPA that was near the cutoff. I'm sure that my low GPA put me out of the running at a lot of schools. The place that eventually admitted me was a top 10ish department (public R1). They basically had a "pick who you want and they will be admitted" system. Which makes sense in the non-rotation graduate school style. Even so, my advisor did have to "go to bat" for me to make sure I was admitted.

Of course I'm biased, but I think it was a good move to be flexible. I did better than most, gradated, wrote some papers and landed a pretty good TT position.

gasstationwithoutpumps said...

We only have a formal cutoff for TOEFL, and it is so low that we have never accepted a student anywhere near the cutoff.

Our department is mainly bioinformatics, possibly the best in the world at that, though at a school that is otherwise only top 100 in US. Because of the expense, we generally only admit 1 foreign student a year. Our domestic pool is generally strong enough that we can do that and still have decent students.

Our department is tiny, so every faculty member is expected to participate in admissions. Every domestic file gets read by at least 3 faculty, and every foreign file by at least 1. The grad director usually reads all the domestic files. We rank all the students by the ratings faculty give, and have more faculty rate students who are near threshold (which is determined by how much support money we can get faculty to squeeze out of their grants). We admit near-threshold students as MS students, some of whom turn out better than they looked on paper (and are encouraged to move into the PhD program) and some of whom turn out to be worse than they looked on paper (and are encouraged to leave with an MS).

gc said...

I was in one of the last few batches that had to take the old school GRE and remember being glad that I didn't have to deal with the new version, with the new analytical section. I thought TOEFL was sufficient to test english proficiency. But the problem with the quant section in GRE was that it was a bit too easy and so not a useful tool for distinguishing students.
I'm surprised that you take such an active role in admitting students you think are good for your work - that's a very nice way of tapping into a good pool...all through grad school (and in other schools I know), I've heard PIs basically tell students that the admission itself is not in their hands and that they could talk if the student was admitted in the program first.
I once noticed an unusual enrichment of students from a particular school near my hometown abroad (the school was totally unheard of in the US in general) in a top 5 school in the west coast. I learnt that they had identified the school based on the quality of students they observed initially and then started taking more consistently. I don't know if other top schools do this routinely, but its good to know that good students anywhere can get a shot sometimes despite rigid test cutoffs and perceived school reputations.
Happy 2012! (The last year of the world according to....the movie 2012)

Anonymous said...

This situation so resonates. I too handpick my potential students and go to bat for them as the only way to build a program. Domestic students are 'preferred' in the same way for funding but are almost non-existent in my field (they tend to go overseas), so its a constant battle against english proficiency test scores and telling the story of the student. Any department that tells you it is 'out of their hands' is either receiving so many good applicants that they can choose or is using it as an excuse. If you want to build a program you have to hand select to be able to take risks as described in the post. This year I have recruited 3 exceptionally good students who for various reasons dont 'fit' a standard box. My gain, well worth the effort. My advice to potential students like this - think broadly also, good people in non top-5 universities are keen for good students. Make sensible contact with potential advisors and pursue it. If you have got what it takes to be a good PhD student there will be a match there, you simply have to think more broadly. Sometimes its 'famous professor X' who is actually at non-famous university Y for some reason (perhaps family, perhaps something else), or 'not yet famous but really energetic rising star researcher' potentially at any university. Look at people's profiles and work at it. However it may appear most professors are working just as hard to attract good students as potential students are in attracting a placement.

Anonymous said...

To both GMP and Anon 4:48: it seems that both of you have been very successful recruiting good students who do not "fit the standard box". Could you tell us a little about what kind of things you look for in a student? What is it that indicates that a non-standard student is good?

--Another Faculty in a top-20 but non-top-10 school

Anonymous said...

Unfortunately, there are several countries that produce students who have very high Verbal scores, but when they come to the US it turns out they don't speak English fluently at all and have trouble following classes.

That's interesting, I tend to see the opposite: countries that produce students with very low Verbal scores, but actually speak English quite well. Turkey and Iran are the ones that spring immediately to mind.

GMP said...

Anon at 2:29 -- yes, I agree with you completely. Luckily, I find that these students usually do quite decently on the TOEFL, which I feel is a reliable predictor of one's English proficiency. Also, one can also infer a lot from prolonged email correspondence and talking to students on the phone (which I generally do before formalizing the offer).

Thanks to all other commenters for their thoughts! Anon at 2:08 -- I will respond in more detail a bit later. A lot of what I do was described in the main post, but it's by no means an exact science...

HFM said...

I don't understand why anyone would use GRE-style verbal to judge a non-native English speaker, much less use it as a cutoff. It's not designed to test English proficiency - it's a mass-market IQ test.

Knowing GRE vocabulary means you read a lot, especially high-brow literature and older stuff, which means you're smart (or bookish, or at least rich). Okay, I can see the logic, and there's probably a correlation between smart/bookish/rich and good grad student. But what about the kids who grew up devouring the Chinese classics - they would have the same desirable traits as the ones who read Victorian literature, but the GRE would tell you otherwise. They can fake it by memorizing flashcards of useless words, but that's a waste for all concerned, except as some twisted measure of "how bad do you want this". (I think that's why scores correlate to country - in some countries, the students believe they need to do this, and most of them will. Other countries don't have that culture.)

Not sour grapes, either - I'm a US native, and smart/bookish/semi-rich, so I'm awesome at standardized tests. But if I were looking to poach good students from the Top 5, an international with a weak GRE-V (and decent TOEFL) would be fine with me.