Leading Horses to Water


Ancient Greeks began the way of thinking originally known as natural philosophy but which we now call science.  Science emerged as we know it during the Renaissance, in an age dominated by fear, superstition, injustice, and brutality.  In other words, pretty much like the present.  These musings are aimed at explaining how science works, and how science can serve even nonscientists in their efforts to make sense of the world.  I can try to explain things but it’s up to you to decide whether or not you wish to drink from these waters.

#12 - The industry standard for science:  Peer review

American Heathen:  aired: 14 April 2012

Ever been in a bar having a half-tipsy discussion with a drinking partner (call him Roger, for no particular reason), when Roger announces, with regard to the topic under discussion, “Well, I’ve got a theory about that!” In science, the word “theory” describes something very different than the meaning of the word in Roger’s announcement.  Scientific ideas are called “Theories” only after having passed many, many tests of their validity – for example, the Theory of Relativity.  Roger’s statement simply signals the arrival of an idea into the conversation and, if it pertained to science, this vague, half-baked idea might be described as a hypothesis.  As the barroom discussion suggests, anyone can have an idea.  From the standpoint of many discussion topics in a bar, pretty much anyone’s idea can be considered to be equally deserving of respect.  In such a discussion, if some idea comes to the person with whom you’re talking and you happen to have known this person for quite some time, it’s likely that this person’s ideas carry with them a credibility based on your experience with him/her.  You probably have an idea of what topics they’re qualified to comment on and receive your due attention.

Imagine that Roger is a professional bass fisherman in his real life.  If the idea has something to do with bass fishing, his ideas about bass fishing must be given considerable credibility.  If it’s about bear hunting, and Roger has no experience hunting bears, then your initial reaction to his idea may be that it may contain some value but it’s nowhere near as credible as his ideas about bass fishing.  If it’s about politics or religion, it’s likely he’s just as clueless as the rest of us.

Regardless of your initial reaction to Roger’s announcement that he has an idea, the validity of that idea should be tested if it’s possible to do so.  It has to make logical sense (for example, it can’t be based on an obviously false premise – such as that gravity causes objects to repel each other), and it has to be evaluated on the basis of predicted outcomes to experiments designed to test it.  If some sort of experiment can test it, the validity of Roger’s idea can be determined.  That experiment has to be designed properly, so that its outcome isn’t mostly the result of something unrelated to Roger’s idea.  And the results have to analyzed by methods suitable for that sort of experiment.  Obviously, the test results shouldn’t be falsified or forced by some deception on the part of the experimenter.

If an idea can’t be validated by some sort of test, then it just lives in metaphysical limbo.  It might be correct, or it might not – no one can know if it’s correct until a way can be found to test it.

In science, ideas from scientists working within the field are tested by those scientists (and/or other scientists) and the results of their tests are evaluated by their peers – other scientists working in that same area of specialization.  Ideas about the climate are evaluated by climate specialists, those about hurricanes are evaluated by hurricane specialists, and so on.  As a meteorologist, I might also have some idea about the climate.  If I carry out a study aimed at establishing the validity of my ideas about the climate, its logical reviewers aren’t other tornado scientists.  Rather, my climate research work should be evaluated by climate scientists. 

The whole principle of peer review seems rather elitist.  It presumes that the logical person to evaluate work on some specific topic is someone who is doing similar work.  Hence, many other scientists are simply unqualified to review the research.  There’s no implied disrespect for my contributions as a tornado scientist if I’m not asked to participate in evaluating a paper submitted to a scientific journal on a very different topic, like atmospheric chemistry.  I might be able to evaluate how clearly the authors expressed their ideas, but I could offer virtually no meaningful comment on the scientific work that is the topic of the paper.  No offense is intended, and none is taken.

Peer review is the industry standard for science.  If your ideas consistently pass peer review and are accepted for publication in scientific journals, then you’re recognized as someone contributing to the science – by virtue of your own research, and also by your critical reviews of the work of your colleagues.  This is all well and good, but just how well does peer review actually work, in practice?

I have a more extended discussion of this on my website, but the short version is that peer review isn’t perfect – this should surprise no one, of course, since science is done by humans, not machines.  Some good ideas are rejected by peer review that shouldn’t have been rejected.  Some bad ideas are published after peer review when those ideas, in fact, are seriously flawed.  Scientific frauds can pass peer review, from time to time.  Hence, the journals are not an ultimate test of the validity of some idea.  In fact, there is no ultimate test of a scientific idea’s validity.  All ideas forever remain open to question, and are subject to rejection or revision on the basis of new work producing new data.  Journals are not sacred documents, containing only truth, the whole truth, and nothing but the truth.  There are no sacred documents in science.

But peer review is an important and mostly successful way to weed out bad ideas.  Peer review works precisely because it’s elitist!  Those most qualified to criticize a piece of work are those experienced in just that sort of work.  They’re familiar with the data, and are familiar with the methods best suited for that sort of analysis.  Science is not an egalitarian world, where everyone’s ideas are given equal respect.  Even great scientists have bad ideas from time to time.  Their work is not automatically correct, just because they’ve done great work in the past.

If you haven’t done work in any scientific field that’s good enough to pass peer review, then your opinions about science have little credibility.  Imagine how little credibility your ideas about bass fishing would be afforded if you’ve never actually fished for and caught a bass!  There’s a sort of “Horatio Alger” myth that outsiders can come in and revolutionize some scientific field despite not having worked in that field before.  Such examples can be found – for one, the meteorologist Alfred Wegener’s ideas about continental drift, which eventually revolutionized geology.  But for every such isolated case, there are hundreds, if not thousands, of worthless ideas proposed by outsiders.  Our love for the underdog shouldn’t cloud our judgment.  People outside of a field are usually too ignorant to make much of a contribution.  It’s just not the case that everyone’s opinion carries the same weight.

If your barroom discussion with Roger turns to science, Roger’s abilities as a bass fisherman might be marginally relevant in some areas of science, but mostly don’t merit serious consideration when the conversation turns to scientific topics.  In order to provide meaningful input, you must first become a peer.

Science is not a religion but rather a tool for those who wish to think for themselves about the natural world.  Its primary characteristic is its willingness to entertain questions from those who wish to obtain believable answers.