Category: Critical appraisal (created 2007-07-21). Critical appraisal is the ability to judge the persuasiveness of the evidence in a research study.  You have to strike the proper balance between being too harsh and being too accepting of research findings.

Articles are arranged by date with the most recent entries at the top. You can find outside resources at the bottom of this page. Other entries about critical appraisal can be found in the critical appraisal page at the StATS website.

2008

  1. P.Mean: Reading abstracts instead of the full article (created 2008-11-05). An interesting inquiry on the Evidence-Based Health email discussion group generated a lot of responses. A busy clinician has a limited amount of time to answer a clinical question. They carry out a quick search and find 5 decent abstracts. They have two options: 1) Look at one full-text article. 2) Look at 5 abstracts. Which do people think is preferable? It's a tricky question because both approaches have problems. Here are my thoughts on this issue.
  2. P.Mean: Errors in statistical methodology (created 2008-10-19). From those of you who review/appraise articles regularly, I would like to hear what kinds or errors you find most often in the statistical methodology. I will be training nurses to critically appraise the statistical methodology sections of articles, and since their time and knowledge of statistics are limited, I hope to focus mainly on errors they are likely to find in real articles.
  3. P.Mean: Godwin's Law (created 2008-10-05). There is a tendency in some discussions (mostly with a political focus, but also in some with a medical focus), to invoke the name of Adolph Hitler in criticizing one's opponents or to compare one's opponents to Nazis. This is sometimes called the Reductio ad Hitlerum fallacy or the Argumentum ad Nazium fallacy.
  4. P.Mean: The depths of anti-intellectualism (created 2008-09-05). My brother-in-law is an avid conservative and often sends me political commentary that would make Attila the Hun blush. That's actually a good thing, as it makes me think things through more carefully. He recently made a sarcastic comment about the lack of experience of Barack Obama ("the guy with the resume consisting of two good speeches"). It made me think a bit more about a topic of general interest to me and one that goes well beyond politics: the rise of anti-intellectualism in the United States. Here's what I wrote back to him in response.
  5. P.Mean: Where to look for information in a controversial area (created 2008-08-20). I am currently researching vaccinations and the negative effects they may cause. My ex-boyfriend is against vaccines and I am for them. I was wondering if you could give me some references to research supporting the autism link not being caused by vaccines or actually any other theory regarding vaccines being harmful to the body. I went to a seminar and they told us there is a problem with molecular mimicry where the body may attack itself looking for a sequence similar to the disease. Any help you can offer would be greatly appreciated. Desperately seeking the truth.
  6. P.Mean: Is there a scientific basis for EBM? (created 2008-08-20). A pair of articles in Chest, along with two rebuttals examines two sides to the debate over the validity of Evidence-Based Medicine (EBM). Point: evidence-based medicine has a sound scientific base: P. J. Karanicolas, R. Kunz, G. H. Guyatt. Chest 2008: 133(5); 1067-71, and Counterpoint: evidence-based medicine lacks a sound scientific base. M. J. Tobin. Chest 2008: 133(5); 1071-4; discussion 1074-7.
  7. P.Mean: Quote on anti-intellectualism (created 2008-08-08). I want to write an article about the growing mistrust of experts in our society. A working title is "The Rising Tide of Anti-Intellectualism." Here's a quote that might be worth starting out with.

Outside resources:

Creative Commons License All of the material above this paragraph is licensed under a Creative Commons Attribution 3.0 United States License. This page was written by Steve Simon and was last modified on 2008-11-26. The material below this paragraph links to my old website, StATS. Although I wrote all of the material listed below, my ex-employer, Children's Mercy Hospital, has claimed copyright ownership of this material. The brief excerpts shown here are included under the fair use provisions of U.S. Copyright laws.

2008

  1. Stats: The post-modern assault on evidence-based medicine, part 3 (January 7, 2008). I have volunteered to give a talk for a group of statisticians which discusses an article I am preparing: "The post-modern assault on evidence-based medicine." I might give this talk on the first Monday in February or the first Monday in March. Here is a tentative abstract.

    2007
     
  2.  Stats: The post-modern assault on evidence-based medicine, part 2 (December 18, 2007). As mentioned in a December 7, 2007 weblog entry, I sent an email to Michael Shermer, editor of Skeptic Magazine. In it, I included a brief pitch for an article, "The post-modern assault on evidence-based medicine" and he gave me some encouragement to work on this. Guidelines for contributions are at http://www.skeptic.com/the_magazine/contribute.html and they accept articles of varying lengths (500 to 5000 words) though they prefer shorter articles. the current draft is already 2700 words with a lot left to be written. So I'll have to find a way to condense many of these thoughts. Here are some of the talking points in the article. I'm including heading in this draft, but they are just to help me organize my thoughts. This draft still needs a lot of work, but I wanted to have something to show to people.
  3. Stats: The post-modern assault on evidence-based medicine (December 7, 2007). I sent a brief email to Michael Shermer, editor of Skeptic Magazine. In it, I included a brief pitch for an article. The title would be "The post-modern assault on evidence-based medicine."
  4. Stats: Which expert should you believe? (August 3, 2007). There's a common saying in research circles that goes something like this: "For every PhD, there is an equal and opposite PhD." That saying is certainly true in my experience. For just about any scientific controversy, you can find an expert on either side of the issue. Quite often the experts on both sides know far more about the controversy than you or I will ever get a chance to know. So the question becomes: "Which expert should you believe?"
  5. Stats: The research world is not black and white (July 7, 2007). I was asked to review a commentary about studies involving remote healing (which includes prayer studies as a subgroup). It is not available yet on the web, but I'll try to link to it when it becomes available. One of the documents that this commentary did refer to, however, is available on the web. Mistakes in Experimental Design and Interpretation. Peter Norvig, norvig.com/experiment-design.html. This is a good commentary, but it falls into a philosophical trap that plagues most of the discussion on how to critically evaluate research, including (I have to admit) some of my own commentaries.
  6. Stats: The Seven Deadly Sins of Researchers (April 4, 2007). I was reading an article in written by Steven Goodwin, The Seven Sins of Programmers, published in issue 17 of the Free Software Magazine and thought it would be fun to use a similar theme in research. So here are the seven deadly sins of researchers. Lest I be accused of the sin of pride, let me admit that everyone, including myself has been tempted by and has indulged in some of these sins at one point in their research career.
  7. Stats: Ignore weak evidence at your own peril (March 13, 2007). I ran across an interesting article recently: Incorporating quality of evidence into decision analytic modeling. R. S. Braithwaite, M. S. Roberts, A. C. Justice. Ann Intern Med 2007: 146(2); 133-41. I have not yet read the full article, but the message seems to be that using evidence from weak data sources is better than ignoring it.

    2006
     
  8. Stats: Ambiguous nature of the word "risk" (September 13, 2006). A recent discussion on the Evidence-Based Health email discussion group focused on the word "risk". It means harmful event (risks versus benefits) as well as a probability (risk of heart attack).
  9. Stats: Why do we need research? (May 31, 2006). I help a lot of people to perform research and it's important to understand why research is important. We do research because good research drives out bad medical practices.
  10. Stats: No tolerance for ambiguity (May 10, 2006). I was at a meeting tonight and put in a plug for my book, Statistical Evidence in Medical Trials, by mentioning that it was intended to help people understand the controversies and the seemingly contradictory research that appears in the medical journals. A woman talked to me afterwards and wanted to know what I thought about a particular author who had written about hormone replacement therapy. I had to defer any comments because I was unfamiliar with this particular author. She then informed me that she had taken hormone replacement therapy and it gave her breast cancer. Thankfully, the cancer has responded well to treatment, but I was struck by the certainty of her comment about how the estrogen supplements caused her cancer.
  11. Stats: Contradictory research (March 30, 2006). The Washington Post published an article that gets to the heart of the difficulty with Evidence Based Medicine, Fat or Fiction? Is There a Link Between Dietary Fat and Cancer Risk? Why Two Big Studies Reached Different Conclusions. Lisa M. Schwartz, Steven Woloshin, H. Gilbert Welch, Published March 14, 2006 in The Washington Post, Page HE01. It starts out with a rather provocative statement, but one that is very hard to argue with, "The public is bombarded with messages about diet and cancer prevention. Unfortunately, the advice is pretty inconsistent. One day a diet prevents cancer, the next day it doesn't. In the early '90s, beta carotene (a vitamin A precursor present in fruits and vegetables) was said to prevent lung cancer. But several years later, headlines read, "Beta carotene pills yield no benefit" (The Post, 1996). And while people have been told for years to eat a high-fiber diet to reduce the risk of colon cancer, recently we were told "High-fiber diets are not anti-cancer miracle" (Montreal Gazette, 2005)."
  12. Stats: Integrating internal and external evidence (March 17, 2006). Evidence based medicine involves the combining of the best available evidence with your clinical knowledge and the patient's individual values. A good article that emphasizes how this should be done is on the web at: Evidence-based decision making--the six step approach. Franz Porzsolt, Andrea Ohletz, Anke Thim, David Gardner, Helmuth Ruatti, Horand Meier, Nicole Schlotz-Gorton, Laura Schrott. Evid Based Med 2003: 8(6); 165-166.
  13. Stats: More on the Emily Rosa experiment (March 10, 2006). One of the more interesting research studies from an Evidence-Based Medicine perspective started out as a simple science fair project by a fourth grade student. Emily Rosa wanted to see if practitioners of Therapeutic Touch could detect the energy fields in a carefully controlled condition. The topic of this project was not too surprising, since her parents both worked for the QuackWatch website, but Emily came up with the idea entirely on her own. The science project received a lot of publicity and Emily was encouraged to publish here results in a medical journal. With the assistance of several adults, the publication, A close look at therapeutic touch. L. Rosa, E. Rosa, L. Sarner, S. Barrett. Jama 1998: 279(13); 1005-10, appeared, giving Emily Rosa something nice to put on her resume when she applies to college. I'm still waiting for my first publication in an "A journal" like JAMA, so I am quite jealous.

    2005
     
  14. Stats: A nice definition of anecdotal evidence (October 24, 2005). Robert Todd Carroll, author of The Skeptics Dictionary and creator of the website Skepdic.com, mentioned in a recent newsletter that he has updated his definition for anecdotal evidence. He points out that anecdotes are unreliable because they  are "prone to contamination by beliefs, later experiences, feedback, selective attention to details, and so on." and although these are "scientifically worthless", their vivid details make them popular. But Dr. Carroll wisely does not totally rule out their use.
  15. Stats: What is critical thinking? (May 17, 2005). A lot of people use the term "critical thinking" in an offhand way. Usually the writer who uses this term is trying to imply that anyone who disagrees with the writer's theory or belief is naive. Critical thinking is also produced as an excuse to attack a particular theory or to promote "equal time" for a competing belief. Critical thinking, however, is much more complex than this. It involves more than just fault finding.
  16. Stats: Effective communication about randomized clinical trials (February 22, 2005). The most recent issue of BMJ has a nice article evaluating a training program for health care professionals. This is an unusual thing to do; most of the time, training programs are just put together with the assumption that because intelligent people are designing the program, it must be effective. I'm just as guilty of this as anyone else, of course. The particular training class is of great interest to me, because it works on how health care professionals communicate to patients about randomized clinical trials.
  17. Stats: How good is your intuition? (January 21, 2005). One definition of Evidence Based Medicine is "the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients. The practice of evidence based medicine means integrating individual clinical expertise with the best available external clinical evidence from systematic research. Sackett et al 1996." In the same article, the authors point out that "Good doctors use both individual clinical expertise and the best available external evidence, and neither alone is enough. Without clinical expertise, practice risks becoming tyrannised by evidence, for even excellent external evidence may be inapplicable to or inappropriate for an individual patient. Without current best evidence, practice risks becoming rapidly out of date, to the detriment of patients." A recently published book by Malcolm Gladwell highlights the value of individual expertise, which you might also call clinical judgment or simply intuition.

    2004
     
  18. Stats: Entrenched beliefs (December 1, 2004). A member of the Evidence Based Health listserv wrote about a workshop he taught on Evidence Based Medicine (EBM). It went pretty well, he thought until he overheard a conversation during one of the breaks. A doctor was recommending an herbal remedy to a friend with the comment: "It cleared my problem up really quickly." This demonstrated to him how readily we still accept anecdotal evidence instead of EBM. I think it is just a fundamental characteristic of human nature to apply critical thinking skills to everything except a core set of beliefs that you hold near and dear to your heart.
  19. Stats: Anecdotal information (November 2, 2004). Anecdotal information is problematic for several reasons, and I have highlighted some of these reasons in a speech I gave at the 2004 meeting of the  Midwest Society for Pediatric Research. Catherine Fiorello highlights the weakness of the anecdotal argument "I was spanked as a child and I turned out okay." Such an observation ignores the need for a comparison group. Such a group would help answer the question "What would I have been like if I hadn't been spanked as a child?"
  20. Stats: Craniosacral therapy (September 24, 2004). An educational email circulated at our hospital during Pain Awareness Week has two interesting questions that draw a sharp contrast between traditional medicine and alternative medicine.
  21. Stats: Statistical nihilism (July 6, 2004). There's an enormous mistrust of statistics in the real world. To the extent that it makes people skeptical, that's good. To the extent it turns them cynical, that's bad. There's a viewpoint, championed by too many people, that statistics are worthless. I call this viewpoint statistical nihilism.
  22. Stats: Overlooking one's own flaws (May 3, 2004). Can we ever be truly objective about ourselves? Maybe not. A recent article in Scientific American  by Michael Shermer, The Enchanted Glass, talks about the tendency to see ourselves more positively than our peers. For example, when asked the probability that certain people will go to heaven, the surveyors listed Bill Clinton at 52%, Mother Theresa at 79% and so forth. But these same people rated their own probability at 87%.

    2003

    2002

    2001

    2000
     
  23. Stats: Flaws in a research paper (January 27, 2000). Dear Professor Mean: A small group I've been teaching has gotten extremely interested in how to decide when there are too many flaws in a paper which would completely invalidate (and circular file) it.
  24. Stats: Jargon in Statistics (January 27, 2000). Dear Professor Mean: I have to review a paper for journal club and I don't understand all the obscure statistical jargon that the authors use.

Closely related categories:

What now?

Browse other categories at this site

Browse through the most recent entries

Get help

Creative Commons License This work is licensed under a Creative Commons Attribution 3.0 United States License. This page was written by Steve Simon and was last modified on 2008-11-26.