The perils of self-evaluation (created 2009-06-30)

This page is moving to a new website.

A survey by New Scientist magazine examined a phenomenon called "citation amnesia." This is the tendency of researchers to overlook previously published work in the bibliography of their articles. Most of the respondents felt that citation amnesia was a problem.

"Indeed, the vast majority of the survey's roughly 550 respondents -- 85% -- said that citation amnesia in the life sciences literature is an already-serious or potentially serious problem. A full 72% of respondents said their own work had been regularly or frequently ignored in the citations list of subsequent publications. Respondents' explanations of the causes range from maliciousness to laziness."

The summary of the results and the survey itself can be found at the New Scientist website. You may need to set up a free account in order to view these files. There are several problems with this survey, though.

First, there is almost certainly volunteer bias at work here. An article on the website asked people to volunteer a few minutes of their time to fill out this survey. Someone whose work was recently snubbed by a competitor's bibliography would have a greater interest in spending the time on a survey like this. Perhaps they thought this would help right a grave injustice. Perhaps they wanted to opportunity to vent their feelings. Not everyone who filled out this would necessarily have these motivations, but such individuals would probably be overrepresented in this sample.

Second, the survey requires a self-evaluation. The question reads: "How often has highly relevant work by you been ignored in subsequent publications by others?" Who is the best judge of whether a work is highly relevant? Scientists are human and are prone to overstate the value and importance of their own work. Perhaps their work isn't being cited because it wasn't good enough to merit citation.

In fairness, some of the people who complain have a legitimate complaint. It's just impossible to gauge how many of the 72% have a valid objection and how many of the 72% are just whining.

Self-evaluation does have its place in research. Pain measurements, for example, rely on self-evaluation. You just have to be careful about how you collect self-evaluation data.