Nobel Predictions from Thomson Reuters: Yawn
Every year, Thomson Reuters predicts who’s going to win the Nobel Prize, and every year, the media eats it up. Here’s the 2010 press release.
Each year, Thomson Reuters uses data from its research solution, Web of Knowledge (SM), to quantitatively determine the most influential researchers in the Nobel categories of Physiology or Medicine, Physics, Chemistry, and Economics. Based on citations to their works, the company names these high-impact researchers as Thomson Reuters Citation Laureates and predicts them to be Nobel Prize winners, either this year or in the near future.
Thomson Reuters is the only organization to use quantitative data to make annual predictions of Nobel Prize winners. Since 2002, 19 Citation Laureates have gone on to win Nobel Prizes.
“We choose our Citation Laureates by assessing citation counts and the number of high-impact papers while identifying discoveries or themes that may be considered worthy of recognition by the Nobel Committee,” said David Pendlebury, Citation Analyst, Research Services, Thomson Reuters. “A strong correlation exists between citations in literature and peer esteem. Professional awards, like the Nobel Prize, are a reflection of this peer esteem.”
I get that Reuters is an organization great at funneling news to lazy subscribers, but scientific news organizations should treat this stuff with a bit more scutiny. For instance, a Nature blog implies that the Thomson method is especially ”analytical”. Bulldoody. Thomson’s picks are just as subjective as everyone else’s. Note that Thomson does not release any details about its method… how, exactly, do they “identify discoveries or themes that may be considered worthy of recognition”? What quantitative method are they using to do this? My guess: none. After the citation data are passed through the qualitative filter of what some guy at the company thinks is important, the quantitative aspect of the “method” is practically meaningless. C&EN, Nature, or someone, should get David Pendlebury on the horn to explain himself.
And with that said, I’d like to take this opportunity to direct members of the press to what is, unquestionably, the definitive set of predictions for who’s going to win the Nobel Prize in chemistry. I am available for interviews, and I can provide a detailed description of my rigorous, highly objective, quantitative method. It entails reading stuff, talking to people, guessing, and incessantly congratulating myself when I stumble upon winners.
September 23rd, 2010 at 3:13 PM
I think it is a little early to be giving the Nobel for MOFs. It would be sad if MOFs won at the expense of zeolites since much of the ideas and mathematical heft come from the older field.
Also interesting is the observation that whilst T. Ebbesen’s discovery of “extraordinary optical transmission” through nanoholes spurred the recent interest in the field of plasmonics, his initial measurement was two orders of magnitude off.
September 23rd, 2010 at 7:42 PM
Dear Paul:
You asked someone to get me on the horn and here I am. I can not claim the predictive ability of Paul the Octopus or even Paul the blogger, but I can explain what we do. Our initial analysis is quantitative: we look at numbers of citations to individuals over the last three decades and look at individual highly cited papers over the same period (the typical lag time between a discovery and a Nobel is about two decades, but some have come faster — PCR in 1993, RNAi in 2006 — both about 8 years). So, the citation analysis directs us to the people in the top 1/10th to 1/100th of one percent by total cites; then we filter by most cited papers; then we look for reports of fundamental discoveries (rather than methods papers or reviews); then we consider the recent history of Nobel prizes (subfields or topics recognized) assuming a subfield will not be recognized back to back; then we look at other prizes awarded to those remaining from our filters — other indicators of peer esteem. It is, of course, not strictly analytical — but neither are the decisions of the Nobel committees. Is this more analytical than guesses? I think so. The other point is that I have no expert knowledge but rather rely on the expert decisions in aggregate of chemists as represented in their citations. We do this chiefly to demonstrate that citations in the literature, when analyzed in quantity, are robust indicators of peer esteem, and peer esteem seems closely correlated to the awarding of prestigious prizes. I see you have actually given odds on particular discoveries and discoverers. How, precisely, do you determine such odds other than guesses? I would not presume to go that far. But insiders may attempt this and be more accurate than the predictions of Thomson Reuters. Good luck with your picks and your odds.
September 23rd, 2010 at 10:54 PM
@ David. Thanks for your response.
Let me begin by reaffirming the admission that my Nobel predictions are guesses. They’re educated guesses based on criteria that I’ve enumerated here, but they’re guesses nonetheless. The same goes for assigning odds to the candidates. Obviously, this isn’t like a game of craps where you can determine true probabilities for each outcome; it’s more like betting on the eventual champion at the beginning of a football season. I set the odds based on my educated guess of each candidate’s chances. If this were the real deal and I were actually trying to make money instead of trying to be accurate, the odds would shift based on how people were betting. I think the result would be a less accurate rundown, because there are a lot of idiots out there. There are so few data (only one prize per year), that any assessment of my predictions is going to be tenuous. That, coupled with the fact that all of this business is silly in the first place, is why this subject is more about entertainment than substance. (And why it makes for excellent blog/water cooler discussion.)
Thank you for providing more details about your method, although as scientists, I bet more than a few of us would love to take a look at the algorithm. I think it is noble that you try to take the subjectivity out of the analysis by relying on a numerical data (citations), but I still maintain that your method is going to be rooted in a great deal of subjectivity. I question whether citations are a good metric for what subjects are important or who has “conferred the greatest benefit on mankind”. Also—and you allude to this—you are going to have to make some subjective calls on things like deciding what data to ignore (e.g., I imagine highly-cited things like review articles are thrown out of your analysis) and judging what scientists to lump together (based on discovery). That can’t fall directly out of the citation analysis without some human intervention. I’d love to see how you weight your qualitative factors.
Even if we saw your actual algorithm, we’d probably have a hard time deciding how much was quantitative and how much was qualitative. I’ve got no problem introducing subjectivity into the analysis. I just find it funny that Thomson trumpets how quantitative and objective its analysis is, while probably engaging in much of the same qualitative/guesswork as Internet blowhards (like me).
I think your model is interesting. My only beefs are that I feel that it is probably less accurate than, say, just picking Lasker winners, and that you oversell it to a gullible press.
Everyone’s predictions should be judged based on accuracy. We could even run a challenge. I could take the same number of people off of the top of my list of predictions, and we’ll see who has a better run over the next few years. Perhaps it is less fair, but we could take a look back at previous predictions, too.
First, something seems fishy to me about the laureates you count as winners. Apparently, ISI Thomson awarded a bunch of citation laureate awards in 1989 and 1990, and every one of them won a Nobel Prize (?!). No citation laureate awards were again given until 1997, and all of them won Nobel Prizes as well. Your continuous (year-by-year) predictions did not begin until 2002. Since the previous data seem weird and the range of years 2002-2010 is what you discuss in your press release, let’s analyze only those laureates.
From 2002-2009, you named 22 citation laureates in chemistry. Two won the Nobel Prize. On my 2007 list of predictions, of the first 22 scientists listed (i.e., those to whom I gave the best odds), four won the Nobel Prize. The next person (23rd) also won a Nobel Prize in chemistry, while numbers 24, 25, and 26 won a Nobel in medicine. An alternate way of looking at it is that my top 22 hit all three prizes in chemistry (2007, 2008, 2009). Your 22 guys covered 2008 (Tsien), but missed 2007 and 2009 completely. Despite this disparity, ABC News, Nature, SciAm, and the LA Times run your predictions instead of mine. Where is the justice? When do I get to go to the ball? Why don’t you care about my feelings?
I guess that’s all for now. Thanks for leaving a comment. All of this Nobel business aside, I think you guys produce a great product (the Web of Knowledge). I am a big fan of it.
September 24th, 2010 at 2:48 AM
Also, I don’t look forward to the inevitable day where I will be reading a chemistry blog only to find one of my own papers being discussed under a poorly drawn picture of a bull taking a dump.
September 24th, 2010 at 3:34 AM
Huzzah for blogging! An uncertainty is vented, and 4.5 h later an answer is served. Just show how rapidly the drums are heard…
September 28th, 2010 at 8:39 PM
The Coleman and Friedman (leptin) pick is ridiculous, even if it got the Lasker (which clearly appears to be the Thomson metric). It’s cool science but really not groundbreaking, and not important to human disease. How that is bigger than, say, how vesicles traffic through cells, how proteins fold in cells, gene regulation (Ptashne), oncogene discovery/p53, the cytoskeleton…yeeesh.
October 4th, 2010 at 9:35 AM
[...] time to focus on it. Paul over at ChemBark has some nice posts on Chemistry Nobel predictions- and poo-pooing those who maybe take it a wee bit too [...]
October 4th, 2010 at 10:02 AM
“How, precisely, do you determine such odds other than guesses?”
the frequentist is screwed. But since we’re all going with the keynesian statistical model here, usually the way it’s done is by information markets. Like intrade.
September 30th, 2011 at 3:25 AM
The Coleman and Friedman (leptin) pick is ridiculous, even if it got the Lasker (which clearly appears to be the Thomson metric). It’s cool science but really not groundbreaking, and not important to human disease. How that is bigger than, say, how vesicles traffic through cells, how proteins fold in cells, gene regulation (Ptashne), oncogene discovery/p53, the cytoskeleton…yeeesh.