Leading Horses to Water
Ancient Greeks began the way of
thinking originally known as natural philosophy but which we now call
science. Science emerged as we know it during the Renaissance, in
an age dominated by fear, superstition, injustice, and brutality.
In other words, pretty much like the present. These musings are
aimed at explaining how science works, and how science can serve even
nonscientists in their efforts to make sense of the world. I can
try to explain things but it’s up to you to decide whether or not you
wish to drink from these waters.
#12 - The industry standard for science: Peer review
American Heathen: aired: 14 April 2012
Ever been in a bar having a half-tipsy discussion with a drinking
partner (call him Roger, for no particular reason), when Roger
announces, with regard to the topic under discussion, “Well, I’ve got a
theory about that!” In science, the word “theory” describes something
very different than the meaning of the word in Roger’s
announcement. Scientific ideas are called “Theories” only after
having passed many, many tests of their validity – for example, the
Theory of Relativity. Roger’s statement simply signals the
arrival of an idea into the conversation and, if it pertained to
science, this vague, half-baked idea might be described as a hypothesis.
As the barroom discussion suggests, anyone can have an idea. From
the standpoint of many discussion topics in a bar, pretty much anyone’s
idea can be considered to be equally deserving of respect. In
such a discussion, if some idea comes to the person with whom you’re
talking and you happen to have known this person for quite some time,
it’s likely that this person’s ideas carry with them a credibility
based on your experience with him/her. You probably have an idea
of what topics they’re qualified to comment on and receive your due
Imagine that Roger is a professional bass fisherman in his real
life. If the idea has something to do with bass fishing, his
ideas about bass fishing must be given considerable credibility.
If it’s about bear hunting, and Roger has no experience hunting bears,
then your initial reaction to his idea may be that it may contain some
value but it’s nowhere near as credible as his ideas about bass
fishing. If it’s about politics or religion, it’s likely he’s
just as clueless as the rest of us.
Regardless of your initial reaction to Roger’s announcement that he has
an idea, the validity of that idea should be tested if it’s possible to
do so. It has to make logical sense (for example, it can’t be
based on an obviously false premise – such as that gravity causes
objects to repel each other), and it has to be evaluated on the basis
of predicted outcomes to experiments designed to test it. If some
sort of experiment can test it, the validity of Roger’s idea can be
determined. That experiment has to be designed properly, so that
its outcome isn’t mostly the result of something unrelated to Roger’s
idea. And the results have to analyzed by methods suitable for
that sort of experiment. Obviously, the test results shouldn’t be
falsified or forced by some deception on the part of the experimenter.
If an idea can’t be validated by some sort of test, then it just lives
in metaphysical limbo. It might be correct, or it might not – no
one can know if it’s correct until a way can be found to test it.
In science, ideas from scientists working within the field are tested
by those scientists (and/or other scientists) and the results of their
tests are evaluated by their peers – other scientists working in that
same area of specialization. Ideas about the climate are
evaluated by climate specialists, those about hurricanes are evaluated
by hurricane specialists, and so on. As a meteorologist, I might
also have some idea about the climate. If I carry out a study
aimed at establishing the validity of my ideas about the climate, its
logical reviewers aren’t other tornado scientists. Rather, my
climate research work should be evaluated by climate scientists.
The whole principle of peer review seems rather elitist. It
presumes that the logical person to evaluate work on some specific
topic is someone who is doing similar work. Hence, many other
scientists are simply unqualified to review the research. There’s
no implied disrespect for my contributions as a tornado scientist if
I’m not asked to participate in evaluating a paper submitted to a
scientific journal on a very different topic, like atmospheric
chemistry. I might be able to evaluate how clearly the authors
expressed their ideas, but I could offer virtually no meaningful
comment on the scientific work that is the topic of the paper. No
offense is intended, and none is taken.
Peer review is the industry standard for science. If your ideas
consistently pass peer review and are accepted for publication in
scientific journals, then you’re recognized as someone contributing to
the science – by virtue of your own research, and also by your critical
reviews of the work of your colleagues. This is all well and
good, but just how well does peer review actually work, in practice?
I have a more extended discussion of this on my website,
but the short version is that peer review isn’t perfect – this should
surprise no one, of course, since science is done by humans, not
machines. Some good ideas are rejected by peer review that
shouldn’t have been rejected. Some bad ideas are published after
peer review when those ideas, in fact, are seriously flawed.
Scientific frauds can pass peer review, from time to time. Hence,
the journals are not an ultimate test of the validity of some idea. In fact, there is no ultimate test of a scientific idea’s validity.
All ideas forever remain open to question, and are subject to rejection
or revision on the basis of new work producing new data. Journals
are not sacred documents, containing only truth, the whole truth, and
nothing but the truth. There are no sacred documents in science.
But peer review is an important and mostly successful way to weed out
bad ideas. Peer review works precisely because it’s
elitist! Those most qualified to criticize a piece of work are
those experienced in just that sort of work. They’re familiar
with the data, and are familiar with the methods best suited for that
sort of analysis. Science is not an egalitarian world, where
everyone’s ideas are given equal respect. Even great scientists
have bad ideas from time to time. Their work is not automatically
correct, just because they’ve done great work in the past.
If you haven’t done work in any scientific field that’s good enough to
pass peer review, then your opinions about science have little
credibility. Imagine how little credibility your ideas about bass
fishing would be afforded if you’ve never actually fished for and
caught a bass! There’s a sort of “Horatio Alger” myth that
outsiders can come in and revolutionize some scientific field despite
not having worked in that field before. Such examples can be
found – for one, the meteorologist Alfred Wegener’s ideas about
continental drift, which eventually revolutionized geology. But
for every such isolated case, there are hundreds, if not thousands, of
worthless ideas proposed by outsiders. Our love for the underdog
shouldn’t cloud our judgment. People outside of a field are
usually too ignorant to make much of a contribution. It’s just
not the case that everyone’s opinion carries the same weight.
If your barroom discussion with Roger turns to science, Roger’s
abilities as a bass fisherman might be marginally relevant in some
areas of science, but mostly don’t merit serious consideration when the
conversation turns to scientific topics. In order to provide
meaningful input, you must first become a peer.
is not a religion but rather a tool for those who wish to think for
themselves about the natural world. Its primary characteristic is
its willingness to entertain questions from those who wish to obtain