Sunday, October 10, 2010

Experiment or Theory?

In my experience, in many physical sciences research groups are either primarily experimental or primarily theoretical/computational. Of course, all experimental groups have to do at least a little bit of theory, because obviously one has to design and interpret experiments, but, in my field at least, only very few very large experimental groups do have a resident theorist or two, usually as research scientists.

My group is a theoretical/computational one. We do theory and numerical simulations in an applied physics field that is at the interface of physics, materials science, and electronics. I have a number of experimental collaborators and a number of collaborative projects, and then of course I have my own theory-only projects. So I am pretty busy, which is awesome. There is never a shortage of projects, only of time and money.

What I want to write about here is a conversation, or rather a piece of a conversation, that I had with a couple of my close experimental collaborators over dinner a little while back. We sort of started chatting about this but got distracted and never finished.

Is it better to be a theorist/computational scientist or an experimentalist?

This question obviously makes sense in the fields where such a distinction exists, i.e. where people specialize in one or the other. I am not going to pretend I can list all the fields in which this is true. I know it is true in many physical sciences. I have no idea about biological sciences, and am equally clueless about social sciences. (Also, I know there are theory purists in physics who would not be caught dead numerically solving anything and don't consider analytical theory and computational science to be in the same purity category. To them I only give an evil eye and wonder how they managed to stay under a rock for this long.)

Anyhoo, what's better: being a theorist or an experimentalist?
I vote experimentalist.

Don't get me wrong: being a theorist, I think theory has a lot of advantages:

1) I need less funding than an experimentalist in the same field -- once I have my computational resources in check, all I need is money for students and summer salary and an occasional conference, and we're in business. As my postdoc would say: We're nice and compact.

2) I don't need hands in the lab. Even if I were to hypothetically totally run out of money, I could still work on my own, or with a couple of TA-supported students, generate preliminary results, and soon be able to apply for more money. It is my impression that if an experimental lab is broke and empty, it is much much harder to ramp back up.

3) There is a much greater flexibility in switching fields, as there is less of a funding barrier. If I want to switch fields, I don't need shiny new equipment to get preliminary data. All I need time and a little money to fund a student and we're in business.

But then there are cons:

1) At least in my field, the potential to get GlamourMag papers as a lone theorist is virtually zero. A theory-only paper can only go so high in terms of prestige.

2) When I write proposals on my own, everyone wants to see letters of support from experimentalists, as insurance that what I do is real or relevant. No one asks my experimental collaborators to show a letter of support from a theorist. I have the impression that theorists are viewed as naughty children who are about to go off into mischief unless anchored by the wise overseeing experimetalists.

3) We are always second fiddle. Whenever there is an experiment/theory collaboration, the experimentalist is always the lead. My peeps work as hard as experimental students on joint papers, but my student is always second author and I am always next-to-last, never last (i.e. lead senior) author. If my student is first author, you can bet your ass it's just us theory folks on the paper.

4) With some of my collaborators, my group is only contacted when the experimental data is ho-hum. When the experimental data is fantastic, the experimentalists go and publish alone (as one collaborator said "If we get [effect], we don't need you." Being untenured at the time, I couldn't spit into his face.) When the data is ho-hum, theory collaborators are asked to help make sense of the data so the complete paper becomes much more alluring (read: published in higher IF journal).

5) There are fewer employment oportunities in academia as well as industry for theorists. That's why I always say, to any students interested in working with me, "If you can envision at all being happy in an experimental lab, you should by all means do experiment." One professor says that you only need 1 theorist for every 10 experimentalists in academia... Hm.

When I had this brief chat with my collaborators, one of them said that we theorists have it better because "You don't have to sit around waiting for a broken piece of equipment to get fixed."

As though all theory/computation is effortless and there are no roadblocks. I can assure you that sitting for months over a piece of code that you know should work but gives you results that are ever so slightly off is completely mind-boggling. Every branch of science has plenty of frustrating moments.

When I was an undergrad, I was pretty good in the lab, but none of it ever truly appealed to me. Nowhere near to how evaluating contour integrals made my heart flutter (very few men had the same effect on me). How irreducible representations of Lie groups made me positively giddy. And how now that I'm all grown up I get to actually do the best thing of all: come up with mathematical models to describe experimental situations and then solve those models numerically by any and all means necessary. How good models can predict results of measurements to come many years down the road. That's some kick-ass high for a theorist.

Bottom line: I loooooove theory. That's all I ever wanted to do and I get to do it. But, I think experimentalists overall have a better deal. More recognition, more employability. They do have to bring in more money, but they also have an easier time getting money as no one judges their sanity as they do that of "crazy theorists". Sure, equipment breaks, but codes have bugs too. I think at the end of the day most of the big shots in my field are experimentalists, so recognition is strongly on the experiment side. And within collaborations, I think the work of theorists is often undervalued (or maybe it's just within my collaborations).

But, of course, it can be the grass is always greener on the other side.

However, when my experimental collaborator says theorists have it better he says it because he thinks our equipment doesn't break so we supposedly somehow have it easier. When I say experiment is better, I don't say it's becasue they have it easy. It's hard, but you get the recognition. It is ultimately about how well your hard work correlates with the accolades from the scientific community. I think experimentalists win this one, no contest.

Anyhoo, that's my bit of experience. I have made my peace with occasional slights from collaborators, as I feel most of them do respect me, at least to a point to which people doing something as trivial as theory can be respected. ;) I am not even going to go into the whole female thing. (Too many variables make for an ill-posed model!) Let's just say that I am taller than many of my collaborators and am fairly direct, so I don't exactly project as a demsel in distress. While I am sure some of my experiences have to do with gender, I am fairly certain a lot also have to do with the theory/experiment divide as I have witnessed it play out similarly for male theorists.

I am a good theorist, if I do say so myself; I can tell you for sure that without my work some of the experiments would ultimately not have been interpreted as the nice effects that they were and all of us in the collaboration would not have ended up with certain high-impact papers. So I know the worth of what I do, and I certainly know the enjoyment that it brings me. I just wish my group and I got a bit more respect for it from the experimental colleagues. But there is only so much you can do about it, I guess, and I don't lose sleep over it (any more).

So dear readers, who do you think has an overall sweeter deal in today's supercompetitive science (and why): theorists or experimentalists? Comment and/or take the little poll below.

Who's got an overall sweeter deal in science: experimentalists or theorists (including computational scientists)?
I am so awesome that I do both extremely well free polls


Alex said...

I did experiment for most of grad school. I wasn't bad enough at it for my (very, very patient) advisor to kick me out, but it wasn't working very well. I came up with a theory project on the side, did it after hours, and showed it to my (very, very patient) advisor once I had results. He surprised me by finding funding for another theory project. My thesis wound up being 2/3 theory and 1/3 experiment. I did a postdoc in a group that had people from both specialties, pretty much did theory there (not always to the delight of my supervisor), and now I just do theory as an assistant professor.

Interestingly, though, a sort of side project of mine involved some statistical questions related to a data analysis procedure. That side project yielded important insights, I shifted a lot of focus onto that, and now if you were to see a description of what I work on you might assume that I must have a lab somewhere generating data. Nope, I just do the theory and mathematics, occasionally apply it to real data from friends, and mostly apply it to simulated data. (We almost have to use simulated data, because if we knew what was in the experimental data we wouldn't need the analysis procedure. Since we don't know what's in the real data until the analysis tools are applied, it's necessary to do a realistic simulation of what the data would look like under given conditions, and see how well the tools work.)

So, I do "The theory of experiments" these days.

Alyssa said...

I'm in Astronomy, which I guess could be considered all theory since our "labs" are in space! We tend to split ourselves into observational and theoretical astronomers (where the former gather data from telescopes, and the latter do things like create models).

I know there are theory purists in physics who would not be caught dead numerically solving anything and don't consider analytical theory and computational science to be in the same purity category.

I was a judge at a research symposium for graduate students at my university, and one of the students I was judging was in pure math. He totally had this attitude about him - first asking me if I wanted him to explain the project as if to "the public" (with an eye roll) or to someone who "knows about this stuff". I said since it's a conference for the general grad student population, it would be great if he could explain it in layman's terms. He then went on to say how annoying that was, but okay. Then, when I asked him if there were any physical/experimental applications of his work, he responded with a very snarky "Uhhh...I don't care about that stuff. Us mathematicians do this stuff because we love it, not to make money."

Awesome. I'm sure you can guess what kind of score I gave him on his presentation.

Venkat said...

Interesting post! I like experiments because you can see the outcome of complex scenarios that are beyond the predictive reach of theory. I like theory because it helps us understand what is going on by capturing the essence.
No experimentalist can even think of an experiment without a theory in mind, however vague or incorrect it may be. No theorist can even state any assumptions without some basis on prior experiments.
So, I actually like them as a couple :)
(experiment and theory that is, not the experimentalist and theorist - not that there is anything wrong with that :)

Female Computer Scientist said...

Interesting question. In my department I definitely see social tension between the theoretical computer scientists and everyone else. Most of the theoreticians I find friendly, but a few of them look down upon anything that is not Real Computer Science (i.e., theory). This has created some tension in the past. For example, one of my students couldn't get his project approved because some theoreticians decided it didn't have a strong enough algorithmic component to it. Which made no sense to me, because the project was completely applied and experimental.

And on the other side, I know of a few people in my department who talk about the theory people somewhat disdainfully as well. As in, hey, why don't you just move over to the math department and leave us alone.

I can sort of see both arguments, but at least looking at the CVs of the pure theory faculty in my department, I really do kind of wonder why they're with us an not in Math. I suspect it's politics and history that underlies a lot of the strife.

This all being said, I have sees theoreticians be positively revered intellectually in other contexts. It's as though the simple act of being involved in the development of new ideas in mathematics makes one a god. (Compared to us lowly experimentalists who just take other people's ideas and apply them to real-world problems).

pika said...

I am originally a mathematician (but moved to an interdisciplinary field) and in those past days in a Maths department, I remember exactly the kind of double-sided snobbery there as FCS describes, except it was between Pure Mathematicians and Applied Mathematicians.

Pure Mathematicians were those who dealt with only "pure theory". You know, the things where you only need a piece of paper and a pencil to work and you go and prove things all day long. Everything else was below their attention and not worthy of respect. Typically this group would inclue faculty working on topology, algerba, theory of measure, analysis, etc. They would not want to touch a computer, nor write code and nor having to do with anything numerical. They would also frequently told the applied guys to move to the CS dept.

Applied guys on the other hand would scorn the pure ones as just playing around and having no idea what the "real life problems" are like. They'd work more towards CS side, develop new methods, algorithms, etc. These would primarily be discrete maths people and numerical algebraists/analysts.

I've since left Maths, but in my current university they have Maths/Stats people together in one department and I sometimes go to their seminars. From questions and social interactions during post-seminar coffee, my impression is that there is the same kind of tension between Maths and Stats people here.
So, I guess these tensions are present everywhere in some way or another?

And, one more thing, when it comes to "real life problems" of the Applied Mathematicians of my youth, now that I am in a interdisciplinary field really having to use applied maths/CS for something else, in retrospect I noticed that even these applied guys didn't have much idea what "real life problems" are like. They worked at a level several steps towards ideal abstraction than what we actually have to do when dealing with real data in my interdisciplinary field. I find this very interesting, that there are these two communities working on similar problems, who are so unconnected and mostly unaware of each other's existence.

Anonymous said...

I'm also a sort of "theory of experiments" person. In my experimental field, some people see the value of what I do, but many do not. It seems to correlate with career stage---most grad students don't think much about what they're measuring/testing and instead just follow the standard procedures.

We actually can experiment on our theories of experimentation, so I might even describe my work as "experimental theory of experiments".

Requin said...

In the social sciences - at least in political science but I think you'd find this in economics even more strongly - there are those scholars who use observational (or experimental) data and those who do purely theoretical work. So at the theoretical extreme you have the game theorists who write models and don't particularly care about empirical implications and sometimes say that they don't "do" real-world applications. But that's a caricature - most of the theorists do try to tie things into the empirical side (often by co-authoring).

Anonymous said...

I was once told that there are two kinds of senior PIs: theorists and managers. With a fragmented time schedule and demands from all corners of academic life, very few senior PIs can afford to work at the bench.

Under these constraints, for the paper one publishes on his/her own, one could probably only choose to do theoretical projects. In that sense, all PI eventually becomes theorists. (or managers)

Jean Grey said...

GMP, sounds like your research area is similar to mine.

I have always worked in primarily experimental groups. In my first group, some students had both experimental and computational projects and eventually full time computationalists were incorporated. In my second lab, I did some computational work myself (with the aid of local theorist friends) in addition to my experiments. Now in my current group, I want to do some computations, but am seriously avoiding it.

I thought it would be 'easy' to do both at the same time...don't ask me why. Prior to doing my own computations, I thought that theorists had 'the life.' The ones in my building seemed to only work sporadically, so I envied their massive amounts of free time. Set up a calculation, run it, then go hang out at a coffee shop or go surfing while you wait for it to finish. Make fancy figures, publish papers, etc etc.

Man was I wrong. After running my first set of calculations, I had a whole new understanding of and respect for what you guys do all day long. Hence the reason why I am currently avoiding running any calculations: theory is best left to the theorists.

If I was a PI/when I am a PI, I envision a perfect world where there is a theorist in my department with whom I work extremely well. We have an agreement to collaborate whenever possible. Both credit for the work and first authorship go to the group that conceives the original idea. Everyone is happy in the end. (Is your department hiring experimentalists? ;) )

Regarding theorist con #2, I think there is some legitimate disconnect about what is interesting from a theoretical standpoint and what can actually be verified experimentally. Is this possibly what causes the sentiment that you express here, even to a small extent? Case in point: I recently came across a purely theoretical paper with some really interesting results. I thought it would be perfect to do the experiment, and it would have been a pretty easy experiment to boot, until I tried to buy the material in question. Turns out it is nearly impossible to isolate, but this fact is mentioned no where in the original paper. It would have been nice to know that before I had mentally written the first draft of the paper!

GMP said...

Thanks everyone for the comments!

@ Alex and Anon 10:10: I like how your term "theory of experiment." I do that too, I just call it "service theory." It's not as nice as "theory of experiment."

@Venksat: So, I actually like them as a couple :)
... not the experimentalist and theorist - not that there is anything wrong with that :)

:)) My husband is an experimentalist, me a theorist. I have been trying to use this argument to get him to cook -- he's the hands-on guy, how come he lets a theorist cook?!-- but with zero success. I'm telling you, experimentalists have it made from where I'm standing. ;))

Alyssa, FCS, Pika -- I totally know what you are saying about the "purity fractions" within departments and between disciplines. Generally, when people dismiss the work of another group it's because they don't really know and thus can't appreciate what the problems are or how hard the field is.

Requin -- thanks for the insight from the social sciences aspect!

Anon at 12:12: In that sense, all PI eventually becomes theorists.

My PhD advisor did this. He said that, once he had become a PI, he dropped doing experiments in order not to be in the students' way, which resulted in him being bored and restless. So he started doing theory and is now largely known for his theory work.

@Jean Grey: After running my first set of calculations, I had a whole new understanding of and respect for what you guys do all day long. Hence the reason why I am currently avoiding running any calculations: theory is best left to the theorists.

:) Thank you! My husband started as a theorist and was pretty good but his heart was in experiment so he switched. He always tells me that the people who don't appreciate that theory is difficult have not really tried doing it.

I envision a perfect world where there is a theorist in my department with whom I work extremely well.

I think this is EXTREMELY importnant. When I was interviewing, one of the aspects I was looking for is whether there were good experimentalists to work with. The place I am now is ideal, as there are several experimental groups I work with and I am the only theorist with my specialty, so I get to play with all the cool problems! Seriously, getting a job in a place where you can have a couple of really good collaborators is the most important thing for your success. See what Prodigal Academic says here . Re hiring -- we have had a hiring freeze for 2 years now, and it looks to continue in the foreseeable future. I don't envy you for looking for a TT in this climate... Best of luck! Let me know off line if I can be of help.

Anonymous said...

Catch the machines if you can! :)

Alex said...

Hey! Cooking ability and experimental ability have little to do with each other! I'm a good theorist, I'm not terribly good at experiments, but I cook really well.

My wife (not a scientist at all) does most of the desserts, I do most of the dinners, and some of her desserts wind up in the faculty lounge, making her the most popular spouse on campus.

Alex said...

Oh, and "service theory" is not an adequate description of the awesomeness of what can be done in "theory of experiments." Most of the time I'm posing questions that the experimentalists haven't even thought of yet, rather than servicing them by doing theory that they request. The questions I'm asking really boil down to "How much information is actually contained here? What does it take to extract it?" I consider these to be very fundamental questions about nature and our knowledge of the world.

GMP said...

@ Alex: Hey! Cooking ability and experimental ability have little to do with each other! I'm a good theorist, I'm not terribly good at experiments, but I cook really well.

:) Of course not, I was just joking above. I am a theorist and I do 100% of the cooking (and I'm not half-bad if I do say so myself). But you can't blame me for trying any and all angles to get my husband (experimentalist, not an academic) to cook a bit, but so far no luck... ;)

Oh, and "service theory" is not an adequate description of the awesomeness of what can be done in "theory of experiments." Most of the time I'm posing questions that the experimentalists haven't even thought of yet

True. I think what you are describing above is what I would say is simply good theory, that is, theory with a high predictive value, obviously the only meaningful kind -- where you ask questions with a clear understanding that they must be experimentally verifiable in the future or that they have actually already been indirectly asked in existing experiments.

I am henceforth officially adopting your name "theory of experiments" to describe the full awesomeness of doing theory tightly connected to experiment!

Alex said...

To see the full awesomeness of what I have in mind when I talk about "theory of experiments", check this out:

Not my work, but same subfield. By using statistics and simulations, they can write down the limit for what can and can't be done in an experiment. I find that pretty awesome. And those same ideas of maximum likelihood estimation can then be used in other contexts, to get more "physicsy" theories, like determining the smallest concentration that a cell can possibly measure:

Again, not my work, but stuff I like to read about.

FrauTech said...

Well my whole department could probably be described as experimentalist at heart. But I have great respect for those more on the theoretical side of things. It takes a lot of digging and running numbers and simulations to arrive at numbers you can only guess as to whether they are close or correct. Things so rarely work out as perfectly as in calculations from our experts, and we often have to use their analysis to justify doing something based on some numbers we can't measure but are only inferring. Me I can go run the experiment or get the data directly, so it seems a lot easier. Maybe the drawbacks are I need to build a rig, or have equipment I can blow up, versus a theorist can work from just their computer. But still I think us experimentalists have it pretty easy, and am in awe of someone who's got a knack for analysis.

Anonymous said...

I was originally trained as a mathematician, then moved into computer science and eventually into a biomedical field. I like experiments better. I am good at it and I enjoy collecting and analysing data, putting together pieces of the puzzle and design new experiments. I was giddy about math too for sure. I don't know what happened. Something changed one day and I was drawn to experimental stuff...

GMP said...

Alex, thanks for the awesome links! This dude Ned Wingreen did some pretty important stuff in my field back in the day.

FrauTech, thanks for the shout-out to theory folks!

Anon at 1:30, it's great to have a perspective of someone with great math background as well as strong experimental skills. It's interesting how your passion shifted!

prodigal academic said...

I am an experimentalist who has collaborated with good theorists for my whole career. We really need the help of theory to understand some of our results.

I think theorists get a raw deal. Everyone assumes life for them is so much better because they need so much less money to run their research, and can often work from home, but they always play second fiddle to experimentalists in collaborative papers. I also think that it can sometimes be difficult to attract students, especially for a lone theoretician in a department, since it is risky for a theoretically-minded student to put all their eggs in the lone theorist's basket. I don't know if it is easier or harder to get proposals funded, but I suspect that is a wash.

GMP said...

Thanks, Prodigal! It's wonderful to hear how appreciative you are of your theory collaborators. And you are very right about the hardship in attracting students.

Massimo said...

Hey GMP, ask any of your non-scientist friends to name the most famous (or the one and only) physicist(s) they have heard of, or know. I bet no one will mention a single experimentalist. They will say Hawking, Einstein, Feynman... Someone may say Newton, if you want to regard him as an experimentalist (I would disagree with that).
Nature/Science/PRL ? Yes, experimentalists have an edge there. But when it comes to becoming a household name, it's theory all the way.

GMP said...

Thanks, Massimo. A friend of mine who's a molecular biologist said the same thing a couple of years back. I don't think this commonality has anything to do with theory/experiement. People know of people of whom they have learned in school (Newton), or those who write popular books (Hawking, Feynman, Carl Sagan), or are somehow cool and/or notorious for excentricity (Einstein, Feynman). I bet if you ask random people whom they know in the biomedical sciences, they will offer Darwin and Pasteure (learned in school), but not much beyond that.

I'm thinking that, if being a household name if one's objective, getting a PhD is probably not the way to go. Having octuplets or getting on a reality TV show would be more efficient ways to achieve such dubious fame..! ;)

Anonymous said...

This is a very nice post. The comments too, are quite interesting.
As an about-to-graduate PhD student, I stand at a point where I feel I must must must shift from experiment to theory. I love to do theory, but somehow ended up doing experiments in my PhD (several reasons, field is very cool, I liked the advisor etc.). But now, when I look back, I see I am not so much good as an experimentalist, and I hate working long hours in the lab. I would rather sit infront of a computer and play with cool math for hours.

But I feel rather stuck when I am trying to get a postdoc in theory. I am getting very positive responses from exp. people, because they know my advisor, but not from the theorists with whom I am so desperate to work with. This has brought me to a dead end, which is again scary, because I have never considered doing anything except academia.

Well, I felt like writing this, since you and others who read the blog might have some general suggestion for me (and might have some experience in making a shift after their PhD).

GMP said...

Anon on March 26, I am sorry to hear you are having no luck finding a theory postdoc, but I must say I don't find it surprising. It's always very hard to move from a "less pure" to a "more pure" field or subfield (a s per this xkcd comic than it is the other way around (e.g. a physicist can get a job in an engineering department, but I have never heard of converse being true). So that's happening with your transition, you are not being considered a serious/deep enough/trained enough theorist. Theorists also tend to have less money overall, so may be reluctant to spend the hard-won and scarce grant money on someone whose training they don't trust... Plus, theorists can be very territorial, I've seen it on grant panels, when they see an experimentalist try to endanger their turf.

I think your best bet is to get an experimental postdoc in a good, well-known group that collaborates with good, well-known theorists or is large enough that they have resident theorists, and to let your postdoc advisor know that you would like to do to divide your time between experiment and theory for the project you will be working on (maybe 70-30 or 60-40 or so), and then engage seriously with the theorists available. I don't know if you will find a friendly advisor ear, but I collaborate with a couple of big-name experimentalists whose groups are large enough that they can and are willing to accommodate these types of shifts (more often theory-->experiment, but I am sure they'd be open to converse as well), so I think you may have some luck.

Once you have a faculty position, you will have more freedom to do what you want. I know several people who went from theory to experiment, most when they became faculty. I know only one who did it the other way around -- my PhD advisor -- who became a theorist, pretty much trained himself -- once he was faculty, because he was no longer able to be in the lab, as he'd be in the way of his students, and out of sheer boredom started looking into theory. He now has a mixed theory/experiment program, whereas he himself has been doing only theory for the past 30+ years.

I would consider your training as an experimentalist a blessing, on which you can always rely and don't abandon it. When you are faculty, you'll have other people spend lots of time in the lab, whereas your duties do shift to more theory-like anyway (big picture, writing grants, doing calculations if you want to). Take the next few years and perhaps a couple of postdocs to slowly build up your theory expertise and "street cred"; I think you have the potential to be a successful combo if you keep advancing your skills in both areas, but it may take a bit of time. Good luck!