Skip to main content

Following on from LoE's excellent diary

When a scientific journal publishes a paper the aim has always been to ensure that it is as close to a statement of fact that is possible. The litmus test of all scientific papers is the Peer Review, where the methodology of the experiment, the veracity of the data and the conclusions are subjected to rigorous and robust critique.

Has the scientific method been rigorously adhered to, has the experiment been correctly deigned, has the data been acquired accurately, what are the tolerances for error, and have the conclusions been based solely on the available data.

A peer review can make or break a career, it can be brutal.

All data produced from the process must be repeatable, all observations accurately noted, all deviations and errors in measurement recorded.

Without the peer review there can be no consensus on the veracity of the study, it is at the heart of what we do, it is the acid test of all we write.

If only politics was subjected to the same rigor.

Science is not a popularity contest, it's aim is not to "win" it aims purely to note and explain and should not to be manipulated via belief.

Prove it, is the cry and not, I believe.

And you better be well prepared for the Peer Review.

So when 90% of scientists declare there is global warming this has nothing to do with belief, or gut feelings, this is based upon solid data that has been subjected to impartial analysis.

Without scientific method the results are flawed.

Without peer review the conclusions suspect.

Without either, your work should never be classed as scientific fact, and should not be published.

There have been abuses in the short term, but in the grand scheme of things, accuracy and methodology have won the day.

EMAIL TO A FRIEND X
Your Email has been sent.
You must add at least one tag to this diary before publishing it.

Add keywords that describe this diary. Separate multiple keywords with commas.
Tagging tips - Search For Tags - Browse For Tags

?

More Tagging tips:

A tag is a way to search for this diary. If someone is searching for "Barack Obama," is this a diary they'd be trying to find?

Use a person's full name, without any title. Senator Obama may become President Obama, and Michelle Obama might run for office.

If your diary covers an election or elected official, use election tags, which are generally the state abbreviation followed by the office. CA-01 is the first district House seat. CA-Sen covers both senate races. NY-GOV covers the New York governor's race.

Tags do not compound: that is, "education reform" is a completely different tag from "education". A tag like "reform" alone is probably not meaningful.

Consider if one or more of these tags fits your diary: Civil Rights, Community, Congress, Culture, Economy, Education, Elections, Energy, Environment, Health Care, International, Labor, Law, Media, Meta, National Security, Science, Transportation, or White House. If your diary is specific to a state, consider adding the state (California, Texas, etc). Keep in mind, though, that there are many wonderful and important diaries that don't fit in any of these tags. Don't worry if yours doesn't.

You can add a private note to this diary when hotlisting it:
Are you sure you want to remove this diary from your hotlist?
Are you sure you want to remove your recommendation? You can only recommend a diary once, so you will not be able to re-recommend it afterwards.
Rescue this diary, and add a note:
Are you sure you want to remove this diary from Rescue?
Choose where to republish this diary. The diary will be added to the queue for that group. Publish it from the queue to make it appear.

You must be a member of a group to use this feature.

Add a quick update to your diary without changing the diary itself:
Are you sure you want to remove this diary?
(The diary will be removed from the site and returned to your drafts for further editing.)
(The diary will be removed.)
Are you sure you want to save these changes to the published diary?

Comment Preferences

  •  Tip Jar. When politics/profit and science become (21+ / 0-)

    confused [a special hi to the fossil fuel lobby] then we have a major problem in trust.

    "Another world is not only possible, she is on her way. On a quiet day, I can hear her breathing." Arundhati Roy

    by LaFeminista on Thu Sep 26, 2013 at 12:06:18 PM PDT

  •  I had a class once (9+ / 0-)

    ... called Data Analysis. A seminar. It covered some things about statistical methods. But the meat of the course was each week, we'd have a couple of papers, and students would be assigned to defend or dissect the paper.

    We got some really good, elegantly designed studies. And some that were flawed. There was one that IIRC was purported to show that salmon used celestial navigation. Two big tanks with salmon. He wrote to the investigator, asking for the data, split out by tank. Turns out that all the data supporting the conclusion were from the tank closer to their campfire, camp lights. Looking at the data only from the farther away tank, and the results were different and did not support their conclusion.

    The professor was one you'd hate to have show up when you had a presentation, or especially an oral exam. He'd have calculator in hand, and if you didn't know everything inside out, you'd be sorry. But anything you did that got by him, and you knew you were pretty solid.

    I liked his class.

    Mark Twain: It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so.

    by Land of Enchantment on Thu Sep 26, 2013 at 12:23:42 PM PDT

  •  Open Source Software is the Same (8+ / 0-)

    From a security standpoint, many thought that open source software would be more vulnerable, but the opposite turned out to be true.  It turns out open scrutiny leads a fastest evolution of development.

    The notion of falsifiability is very powerful, and what elevates the scientific process.  Religious people seem unable to understand why science is different from faith.  The desire to find fault in ones own work is the critical difference.

  •  I have been through the Peer-Review process on (9+ / 0-)

    both ends: author and reviewer. Yes, it can be really rough, but in the end, it makes you a better scholar. I am always grateful to reviewers for helping me to think more deeply about my work and for challenging my ideas.

    Echo chambers are super dangerous for academia.

    In the social sciences our data is more observational than empirical, so there is less repeatability, but the methods must still be sound and the data presented in a clear and convincing fashion.

    "If you don't sin, then Jesus died for nothing!" (on a sign at a Mardi Gras parade in New Orleans)

    by ranger995 on Thu Sep 26, 2013 at 12:38:27 PM PDT

  •  Depends on who the peer reviewers are too (9+ / 0-)

    but it helps.

    Sometimes it seems science is defending consensus ideas. You'll have a bunch of peer reviewed articles then one that looks at it from a different view and all the old ones are tossed in the trash.

    “Conservation… is a positive exercise of skill and insight, not merely a negative exercise of abstinence and caution…” Aldo Leopold

    by ban nock on Thu Sep 26, 2013 at 12:40:42 PM PDT

    •  Then the peer review is flawed, and the (4+ / 0-)
      Recommended by:
      Dr Teeth, ranger995, trumpeter, ban nock

      methodology not adhered to, I have seen it too and got into some nasty correspondence.

      My peer reviews tend to be easy, pilot plant designed and built, several million euros later.....fingers crossed it doesn't fall to pieces since I wrote what it should do on the tin.

      "Another world is not only possible, she is on her way. On a quiet day, I can hear her breathing." Arundhati Roy

      by LaFeminista on Thu Sep 26, 2013 at 12:44:34 PM PDT

      [ Parent ]

      •  With all due respect, this is a crock (6+ / 0-)

        Science is a human process and like every other field - banking, politics, law enforcement, entertainment - there are many many variable that influence what comes out. Peer review is how science "polices" itself. And we have many examples of other fields failing misearble at policing themselves.  Every field has their bad apples (and everyone generally knows who they are!).

        Don't equate the knowledge that arises from scientific inquiry with scientists themselves. Knowledge lasts when it stands the test of time, regardless of who did the work, how it was reivewed, where it was published. Mendel's work could never be repeated and the values in his lab notebook were "too neat". But his theories were correct and validated over time. It's a messy process.

      •  Not necessarily. Sometimes "new shit comes to (8+ / 0-)

        light" and the older way of thinking needs to be rejected.

        That does not mean that the peer review process is flawed, it just means that the first scientists did a great job with the material they had available to them.

        One of my favorite journals is Current Anthropology, because there is an open peer review process. The article is first reviewed for whether or not it belongs in the journal. Then, a number of reviewers write comments that are published and the author responds to them. While it has gotten nasty as times, I usually find the whole discussion fascinating and constructive.

        "If you don't sin, then Jesus died for nothing!" (on a sign at a Mardi Gras parade in New Orleans)

        by ranger995 on Thu Sep 26, 2013 at 12:56:32 PM PDT

        [ Parent ]

        •  The data stands the test of time and hypothesis (5+ / 0-)

          I have spent many hours reading old papers/journals in the British Library, I found it fun and informative.

          You can still repeat the experiments and the data [although to a greater degree of precision] and subject it to new knowledge. A peer review if you like.

          "Another world is not only possible, she is on her way. On a quiet day, I can hear her breathing." Arundhati Roy

          by LaFeminista on Thu Sep 26, 2013 at 01:03:29 PM PDT

          [ Parent ]

          •  I think of the "Cold Fusion" fiasco is actually an (5+ / 0-)

            example of how the peer review process did work. Although the initial articles were published, the standards that require one to include the experimental process allowed other scientists to test the repeatability. Once enough scientists had concluded it was a hoax, it was exposed.

            "If you don't sin, then Jesus died for nothing!" (on a sign at a Mardi Gras parade in New Orleans)

            by ranger995 on Thu Sep 26, 2013 at 01:07:09 PM PDT

            [ Parent ]

          •  Recently some scientists in Santa Cruz (1+ / 0-)
            Recommended by:
            ranger995

            went to court to keep from having to release their data. I forget what their argument was, they won, and they still haven't released it, the bill they were supporting sits on Jerry Brown's desk this afternoon, maybe he's signed it.

            I'm with you though. Peer review is a great process. It forces the scientists signing it to say, "we've reviewed this stuff and it's on the up and up". I sure don't have any better ideas. You don't want to stifle publication by having some sort of difficult publishing process.

            “Conservation… is a positive exercise of skill and insight, not merely a negative exercise of abstinence and caution…” Aldo Leopold

            by ban nock on Thu Sep 26, 2013 at 03:55:49 PM PDT

            [ Parent ]

    •  Thomas Kuhn (0+ / 0-)

      That's just the story of scientific progress a la "The Structure of Scientific Revolution" by Kuhn.   It's not so much defending, as accepting the current best model fitting the known data.  

      Minority rights should never be subject to majority vote.

      by lostboyjim on Fri Sep 27, 2013 at 06:55:04 AM PDT

      [ Parent ]

  •  scientists are human too (4+ / 0-)

    there's a lot of groupthink. People tend to work in what's fashionable, because that's what will get you a job and promotion. There's definitely a hierarchy, and challenging the conventional wisdom can bring down the wrath of the people at the top.

    Scientists are trained to try their best to be objective, to spot contradictions, to gather evidence carefully and thoroughly, etc. But they can only do so much, and sometimes they make mistakes or succumb to peer pressure or prejudice.

    The bigger problem nowadays is that science has become expensive. No longer can Einstein sit by himself at his desk in the patent office and contemplate the nature of space and time during his idle hours. Science these days is a massive enterprise requiring billions of dollars of funding.

    And that means that scientists are more and more at the mercy of the people holding the purse strings. The danger is that the path of scientific inquiry will be more and more determined by factors that are extraneous to science, such as profitability, political dogma, etc. And that could eventually corrupt science as it has corrupted other areas of endeavor.

    Science is not immune to those corrupting forces. If ever there comes a day when fame and fortune are more important to scientists than the pursuit of the truth, then that will be the death of science.

    "In America, the law is king." --Thomas Paine

    by limpidglass on Thu Sep 26, 2013 at 01:22:52 PM PDT

    •  It is time that is the problem (1+ / 0-)
      Recommended by:
      limpidglass
      And that means that scientists are more and more at the mercy of the people holding the purse strings.
      This depends on your field, but often the problem is limited resources.  The competition for use of high end equipment is extreme.  Decisions on who will get a slot are often a function of whose experiment is most likely to generate useful data.

      Computing has helped some fields significantly, but developing algorithms to model an experiment requires a sophisticated programmer's time.

      Funding use to plentiful back, when we were keeping up with the Soviets, but it is no longer a national priority.

      Actually, in cost terms, research is cheaper now than it has been in the past.  Before new technologies would have to be developed, before experiments could be run.  Now there are often more flexible systems, which can run many different experiments.

      The country just stopped caring how the universe works.

    •  Funding is peer-reviewed, too. (2+ / 0-)
      Recommended by:
      engine17, Dr Teeth

      Most of us in senior positions spend a fair amount of time on NIH and other study sections, doing peer review of grant applications. The process is generally pretty thoughtful and helpful, at least in my world (medical research.) Most scientists care a lot about having first-rate medical research - we and our families get sick, too - and they try to remove potential conflicts of interest.

      All that being said, I've occasionally received a weird, less than thoughtful review (but it's rare). The group discussions usually highlight flaws that might have been overlooked but can also correct misapprehensions that might keep something important from being funded.

      The Bush folks put forward some proposals for privatizing NIH peer review but it was obvious that it would cost way more (pay for peer review is minimal) and you wouldn't get the caliber of reviewers. I see it as reciprocity: if you want good people to take time to review your proposals or papers, you have to be willing to contribute, too.

      The biggest problem is that the budget cuts of the last few years mean that many excellent studies go unfunded. It's especially hard for younger researchers with more limited track records. NIH tries hard to compensate for that but we may still lose a big chunk of the next generation of researchers.

      •  Single-digit paylines... (2+ / 0-)
        Recommended by:
        Dr Teeth, Laurel in CA

        Recent National Cancer Institute paylines have been at the 7th percentile for R01s and R21s.

        Its incomprehensible that we train most of the world's best scientists, and then terminate their scientific careers because they did not make it into the top 7% in the peer review process.

        I agree completely with your comment, other than the glaring understatement that "many excellent studies go unfunded." I think it is far more accurate to say that most excellent studies go unfunded.

        •  You are right on single digit paylines. (1+ / 0-)
          Recommended by:
          engine17

          Back in the Good Old Days, anywhere from 25-30% of studies were funded. So it's very likely that most excellent studies are not being funded now. And even first-rate researchers with great track records have to spend far more time writing grants just to keep going. It's very stressful. I have watched a lot of good people leave or be driven out. Sad.

  •  I have been a peer-reviewer for a few articles (1+ / 0-)
    Recommended by:
    Dr Teeth

    (I'm new to the professional role) and have had a few articles reviewed as well.

    While it's wonderful in many ways, be careful of the praise--it's fraught with problems, too.

    1. It's political--science and social science are political--and it does occasionally (probably more than occasionally) happen that established academics/researchers don't like having upstarts shaking the tree--so neophytes may be subject to unfair criticism based on what the reviewer actually sees as personal insult!  (this just happened to me--it cleared two reviewers, but the other one appeared to be quite insulted by what I thought was a pretty fair article.  Fortunately it got through--I addressed all of his concerns and was diplomatic about it, so that probably helped)

    2. Selection bias--exciting, positive results simply sell better than negative results.  So 'we found a link!' can win people over more than 'we didn't find a link!'  This is a huge issue in the medical profession, and one reason that 'major clinical studies' end up being overturned 5, 10 years later all the time.  Both authors and reviewers want to find something.  It's not nefarious--it's almost unconscious, and you really need to pay attention to keep from falling into the trap.

    3. Reviewers often have limited knowledge about the specific topic.  This may be different in the hard sciences--but with all of the theoretical, methodological and interpretative approaches in the social sciences the writer is bound to take a trajectory that falls between two or three reviewers--each of whom comments on what they know about, but may skip out on the rest.  Another personal example--I just reviewed a paper about archaeology in South America (that's sufficiently broad that it doesn't point the finger at anyone :)   I found the methodology innovative and intriguing, if sometimes a bit shakily applied, so I felt that with some minor revisions it could pass (I was the 6th reviewer in the circuit!)  But a seventh reviewer who really know the geographical area really didn't think it would fly, so it got rejected.  

    Now, for this example, you can really say it makes an argument that the Peer Review system really worked--and in this case it did.  But it was only a fluke that it went through 7 reviewers--sometimes it's only 3 or even 2--even at top journals.  Again, this is social science so it may be different in other scientific fields.

    But the bottom line is--be aware of what lies behind peer review!  (this isn't about the climate science comment--I was just talking generally that it's important to be in tune to the process)

    Long post for 2 cents :)

  •  T/R by the way---peer review is an important (1+ / 0-)
    Recommended by:
    Dr Teeth

    topic that people don't think/talk about enough--which they should, considering in many ways it drives both human thought and action...

  •  The Heart of Fundamentalism: The Bible Review (1+ / 0-)
    Recommended by:
    engine17

    (Sorry, LaFem, but the snark is strong today in me, and I am a weak individual.)

    When faith-based scientists (the only true ones) force a paper down the throats of whomever they're after this week, the aim has been to ensure that it as theologically pure as possible.  The litmus test of all Jesus-approved papers is the Bible Review, where the correlation of the data to my interpretation of the Old Testament, the sanctity of the data and the religious dogma are subjected to one-sided critique.

    Has the Bible been rigorously adhered to, has the experiment been cleared with Jesus, was the data influenced by Satan, what are the tolerances for apostasy, and have the conclusions been based solely on Biblical law.

    Jesus can make or break a career, it can be brutal, especially on Rapture Day.

    All data produced from the process must be consistent with Biblical teaching, all appropriate Bible passages accurately noted, all deviations and errors from dogma reported to the authorities.

    Without God-based science, there will be floods, fire, famine, Democrats in the White House, and marriage between dogs and cats of the same gender.  If only science was subjected to the same sort of ignorance.

    True fundamentalist religion is not a factual accuracy contest, it's aim is not to correct theories based on emperical evidence.  It aims purely to note and explain and should not to be manipulated via facts, which have a liberal bias.

    Believe in Jesus, is the cry and not, prove it.

    And you better be well prepared for the Rapture.

    So when 90% of nutcase thocrats declare there is no such thing as global warming this has nothing to do with science or reasonable analysis, this is based upon the Word of God has presented in the version of the Bible I prefer most.

    Without theocracy the results are inconvenient to the value of my stock portfolio.

    Without complete agreement with the Bible, the conclusions nothing but secular humanist deviancy.

    Without either, your work should never be published and should be banned as sacrilegious.

    There have been fact based thoughts in the short term, but in the grand scheme of things, once Rick Santorum becomes president, you "scientists" are toast.

Subscribe or Donate to support Daily Kos.

Click here for the mobile view of the site