Jump to content
Can't remember your login details? Read more... ×
Sign in to follow this  
Leonid

A word of caution

Recommended Posts

This is a word of caution that I've developed thanks to discussions in this thread. If you don't like your religion being challenged, I suggest you click out of this thread now.

 

This thread is going to show a peer-reviewed letter in Nature may not be as conclusive as the abstract shows. I, off course cannot provide the text of the letter as it is behind a paywall, but I can, perhaps email the letter to those with a serious interest.

 

Q. Why am I doing this?

A. Because a lot of people are under the mistaken belief that peer review is a shield against bad science. It is not.

 

Q. Why this specific paper/letter?

A. This is a typical example of alarmist "science" where the abstract has a measure of certainty far higher than the actual meat of the letter/study. It's an adequate representation of why people like greenman3610 and those who use his videos (Psychonaut has been guilty of this) are continuously misinformed by abstracts.

 

Here's the paper. Here's the abstract:

Increases in greenhouse forcing inferred from the outgoing longwave radiation spectra of the Earth in 1970 and 1997

The evolution of the Earth's climate has been extensively studied1, 2, and a strong link between increases in surface temperatures and greenhouse gases has been established3, 4. But this relationship is complicated by several feedback processes—most importantly the hydrological cycle—that are not well understood5, 6, 7. Changes in the Earth's greenhouse effect can be detected from variations in the spectrum of outgoing longwave radiation8, 9, 10, which is a measure of how the Earth cools to space and carries the imprint of the gases that are responsible for the greenhouse effect11, 12, 13. Here we analyse the difference between the spectra of the outgoing longwave radiation of the Earth as measured by orbiting spacecraft in 1970 and 1997. We find differences in the spectra that point to long-term changes in atmospheric CH4, CO2 and O3 as well as CFC-11 and CFC-12. Our results provide direct experimental evidence for a significant increase in the Earth's greenhouse effect that is consistent with concerns over radiative forcing of climate.

Here's the debunking:

The letter focuses on a comparison done from the October 1996 Interferometric Monitor of Greenhouse Gases (IMG) instrument, on board the Japanese ADEOS satellite. It got about 9 months of good outgoing long-wave radiation data. 27 years earlier NASA had flown a similar instrument (IRIS Infrared Interferometric Spectrometer) on the Nimbus 4 spacecraft, between April 1970 and January 1971.

 

The comparison of the two systems is as follows:

 

Characteristic IMG IRIS

.................................................. .................................................. .................................................. ......................

Spectral range (cm-1) 600±3,000 400±1,600

Spatial ®eld of view 8 km ´ 8 km 100 km 100 km

Spectral resolution (cm-1) 0.10±0.25 2.8

Total radiometric uncertainty

(mWm-2 sr-1 per cm-1) 60.23 6(0.5²±1.0³)

Equivalent brightness

temperature uncertainty* (K) 60.15 6(0.3²±0.6³)

.................................................. .................................................. .................................................. ......................

* Defined at 900cm-1 for a brightness temperature of 290 K.

² In centre of band.

³ Towards edge of band.

 

In order to make the two systems comparable, they downgraded IMG by reducing the spectral resolution of the IMG data by degradation, to match that of IRIS, instrument field-of-view effects were taken into account, and land/island areas were masked out.

 

This all seems pretty cool right? Until you start thinking. No amount of degradation for comparison purposes reduces the possibility of instrumentation degradation from an early 1970s satellite.

 

So how did the letter's authors take that into account?:

First, we have examined the central issue of the accuracy and precision of the two data sets, and whether our analysis is justified on this count.We believe it is, for the following reasons.
	(1) We have reported above the agreement between the observed difference spectra and simulated spectra, which are calculated quite independently from basic knowledge of the atmospheric state.
	(2) We have derived difference spectra over a wide variety of regions and times (including the east and west Paci®c, Atlantic, and Indian oceans) and see consistency in the principal absorption features.
	(3) Random error is reduced by averaging: in the extreme case in which all the error in Table 1 is assumed to be random, we obtain a value of 60.058K for the central Pacific difference spectra.
	(4) Absolute accuracy is potentially a more serious issue: if all the error quoted in Table 1 were systematic, this would indicate a maximum absolute peak±peak error for the combined data of about 0.45 K at the centre of the IRIS/IMG pass bands, increasing to about 0.75 K at the edges. Such errors of absolute calibration vary slowly with wavenumber, and could produce small 'ghosts' at the positions of the observed spectral features, though these are very unlikely to be significant. Slowly varying systematic errors may also contribute to the differential window signal, but only at this same level of, 0.45 K.
	(5) Normalization of each averaged spectrum to the intensity at one selected wavenumber indicates no signi®cant multiplicative error between IMG and IRIS above the 0.5 K level.
	(6) Examination of the data shows that neither spatial nor temporal sampling is seriously biased. We conclude that the main features of the observed spectra cannot be accounted for by instrument errors, but that the absolute calibration of the difference spectra might be in error by up to about 0.5 to 0.75K peak±peak.

I hope that's clear. In point 1 they justify their data being right based on the fact that it got the same result band in different region. Self-justification isn't science. Something isn't right because a model says it is.

 

Now in the last point it makes reference to an error of about 0.5K-0.75K. That is repeated elsewhere in the letter.

 

And get this - several observations they made in that letter fall within or just outside that error margin. Compare that to the NOAA dataset, mined from Christy's satellites which have been operating non-stop for 30 years. They show no change whatsoever.

 

The dataset, in graph form can be viewed here. NOAA has the homogenised data available on their site. The point is, two datasets 30 years apart doesn't show change. It shows to separate datasets with no causation between them. The instrumentation is completely different. The sensing capability and design is completely different too. And the sensing periods are different - seasonal patterns affect longwave radiation.

 

And the peer review failed to catch that, allowing the editor to write a freely available abstract that people with no accounts can read and misinterpret as conclusive evidence. I'm no scientist, this may be wrong, but I'm pretty good with stats and as far as I know, degrading observations to match others is not a valid practice. Nor is comparing unlike datasets.

Edited by Leonid

Share this post


Link to post
Share on other sites

I wan't to go ahead and say I like this post because you provide reasoning and some facts to talk on Leo so please don't mistake this as a flame.

 

Providing one example that lets face it you touch on only in the lightest way possible does not discredit an entire processes effectiveness especially given the overwhelming acceptance within the scientific community and a solid track record of success that has been used in some degree in the field of science ever since the enlightenment.

 

No one said peer review is perfect no one said all science is good science but you can't use a minority of cases, which tend to be extreme examples, to try and undermine an established and proven process.

If you truly are good at stats or statistical analysis you should know when assessing trends you usually exclude the extremes called "outliers" as they do not represent accurately the gross majority of results.

 

Oh but to your credit you do something which I like.

When we have science based discussions instead of reading the media analysis's and taking it as fact you should always go back to the source study to assess it's credibility and to make sure it hasn't been misrepresented like we frequently see from the 'pop media'.

Edited by Bundy

Share this post


Link to post
Share on other sites

I know I provide only one example. I can provide plenty others and history is littered with thousands of examples of failed peer reviews.

 

The point is that this example adequately demonstrates that all peer review is flawed in not actually verifying the science. What you see in journals has been peer reviewed but not independently verified - it has not been experimentally tested nor has data been released. All you are being given to read is the opinion of two reviewers that the material is worthy of publishing - not that it is correct.

 

Almost everyone with a smidgen of scientific knowledge or scientific procedural knowledge knows this, but in recent years, thanks to the complete lack of conclusive causation from the climate change industry, peer review has taken on a meaning quite separate from its intent - peer review is not a shield that you can use to ward off accusations of flawed science.

 

All this post shows is that I can take apart a scientific paper and sow enough doubt about it. Doubt that may not be warranted, but because no data has been released, nor procedures, all we have is the word of the author that there is no fraud. Peer review will not pick this up.

Share this post


Link to post
Share on other sites

1.The point is that this example adequately demonstrates that all peer review is flawed in not actually verifying the science.

 

2. peer review is not a shield that you can use to ward off accusations of flawed science.

I strongly disagree with 1. I don't think at all this adequately demonstrates that all peer review is flawed. That is a very weighted statement.

 

I strongly agree with 2, the very premise of a good review process is that publications are open to scrutiny providing the scrutiny is based in science. A study should never reach a point that it is infallible to all criticism providing once again that criticism is relevant and scientific in nature.

Share this post


Link to post
Share on other sites

I strongly disagree with 1. I don't think at all this adequately demonstrates that all peer review is flawed. That is a very weighted statement.

Oh it is. There's actually been peer-reviewed studies done on peer-review :)

 

The fundamental point is that peer review, the way it is done at most journals is actually a popularity contest among established scientists who get to where they are by not rocking the boat.

Share this post


Link to post
Share on other sites

The fundamental point is that peer review, the way it is done at most journals is actually a popularity contest among established scientists who get to where they are by not rocking the boat.

So what you are really saying is not that there is a flaw in peer review process but that the peer review process is not being conducted properly? They are 2 different things with different solutions.

 

In saying that you need to do much more than provide more than one case to highlight a flaw in what is a widely accepted practice within the field of science.

Share this post


Link to post
Share on other sites

So what you are really saying is not that there is a flaw in peer review process but that the peer review process is not being conducted properly? They are 2 different things with different solutions.

There is both actually.

 

In saying that you need to do much more than provide more than one case to highlight a flaw in what is a widely accepted practice within the field of science.

I really don't. I can provide this one and another I provided in the other thread. They are enough to show you the holes in the system, which you've admitted are there in the point where you've agreed with me above.

 

If there's a potential for a flaw and if there are hundreds of documented issues, then a flaw exists.

Share this post


Link to post
Share on other sites

1.The point is that this example adequately demonstrates that all peer review is flawed in not actually verifying the science.

 

2. peer review is not a shield that you can use to ward off accusations of flawed science.

I strongly disagree with 1. I don't think at all this adequately demonstrates that all peer review is flawed. That is a very weighted statement.

 

I strongly agree with 2, the very premise of a good review process is that publications are open to scrutiny providing the scrutiny is based in science. A study should never reach a point that it is infallible to all criticism providing once again that criticism is relevant and scientific in nature.

 

 

You've missed the point. You did in the prior thread, as well.

 

Peer-review is NOT independent verification.

Without independent verification, it isn't following some of the most basic steps to ensure scientific rigour.

 

And, just to back Leo up, I can think of (But would need to research my articles at home for details) several cases in the last years concerning fraudulent results in physics and biochemistry, published under peer review.

 

Your only stay against poor utilization of scientific methodology is independant verification. Peer review *should* include this, but in what I've seen to be an overwhelming number of cases, does not.

Share this post


Link to post
Share on other sites

1.The point is that this example adequately demonstrates that all peer review is flawed in not actually verifying the science.

 

2. peer review is not a shield that you can use to ward off accusations of flawed science.

I strongly disagree with 1. I don't think at all this adequately demonstrates that all peer review is flawed. That is a very weighted statement.

 

I strongly agree with 2, the very premise of a good review process is that publications are open to scrutiny providing the scrutiny is based in science. A study should never reach a point that it is infallible to all criticism providing once again that criticism is relevant and scientific in nature.

 

 

You've missed the point. You did in the prior thread, as well.

 

Peer-review is NOT independent verification.

Without independent verification, it isn't following some of the most basic steps to ensure scientific rigour.

 

Hit the nail on the head.

 

Bundy, peer review is the best that's available because everything else (such as independent verification) takes time and money - putting prices of these journals out of range of mortals. So they take the shortcut of peer review. And because archival of data takes money and resources, they don't do that either.

 

It's picking the best of a bad bunch of options. But it's not a good option. Do you see the difference?

Edited by Leonid

Share this post


Link to post
Share on other sites

The peer review process is a very basic check that someones claim is plausible, and that they have included enough information (correct or not) to support their claim and the journals criteria. There is no checking of results and so incorrect information has a very real change of getting through the review. To compound that matter, some journals allow the author to suggest who should review their work, and suggest who should not. Add to all that, that these journals charge a bunch of money to publish and access, the system looks a little shaky.

 

The good part is that papers don't just drop off the map once published. They are read and cited by many people in the field (and other fields). For example, 51 research papers have cited the above mentioned paper which has itself cited 28 others. When there is an important mistake made, it is picked up on and it is corrected either by the author with an erratum or by another researcher publishing their own work. The field is very competitive and eventually the right information moves forwards.

 

That said, peer review (and journal editing) is an important first step in weeding out repeat work, glaring errors or very small innovations that don't merit their own publication. It is right though, to be hesitant to trust a single conclusion.

Edited by Cummings

Share this post


Link to post
Share on other sites

1.The point is that this example adequately demonstrates that all peer review is flawed in not actually verifying the science.

 

2. peer review is not a shield that you can use to ward off accusations of flawed science.

I strongly disagree with 1. I don't think at all this adequately demonstrates that all peer review is flawed. That is a very weighted statement.

 

I strongly agree with 2, the very premise of a good review process is that publications are open to scrutiny providing the scrutiny is based in science. A study should never reach a point that it is infallible to all criticism providing once again that criticism is relevant and scientific in nature.

 

 

You've missed the point. You did in the prior thread, as well.

 

Peer-review is NOT independent verification.

Without independent verification, it isn't following some of the most basic steps to ensure scientific rigour.

 

And, just to back Leo up, I can think of (But would need to research my articles at home for details) several cases in the last years concerning fraudulent results in physics and biochemistry, published under peer review.

 

Your only stay against poor utilization of scientific methodology is independant verification. Peer review *should* include this, but in what I've seen to be an overwhelming number of cases, does not.

 

Leo explicitly states that " The point is........ all peer review is flawed" In the thread that he started, which I disagreed with.

 

I agreed that simply because peer review exists it is not a shield for study to hide behind.

 

Again having 1, 2 or a handful of examples does not deem all peer review to be flawed simply that it is not 100% accurate or perfect. Your standard of justification is weak at best.

Edited by Bundy

Share this post


Link to post
Share on other sites

The peer review process is a very basic check that someones claim is plausible, and that they have included enough information (correct or not) to support their claim and the journals criteria. There is no checking of results and so incorrect information has a very real change of getting through the review. To compound that matter, some journals allow the author to suggest who should review their work, and suggest who should not. Add to all that, that these journals charge a bunch of money to publish and access, the system looks a little shaky.

 

The good part is that papers don't just drop off the map once published. They are read and cited by many people in the field (and other fields). For example, 51 research papers have cited the above mentioned paper which has itself cited 28 others. When there is an important mistake made, it is picked up on and it is corrected either by the author with an erratum or by another researcher publishing their own work. The field is very competitive and eventually the right information moves forwards.

 

That said, peer review (and journal editing) is an important first step in weeding out repeat work, glaring errors or very small innovations that don't merit their own publication.

That's right. The paper is cited because it claims something and is used thereby to claim something else. But so was the 2006 Stem Cell fraud - it too was cited several dozen times.

 

That's a failure of peer review. One mistake in a paper, cleared by a peer review, can compound to damage a decade of future research. :)

 

You are completely correct regarding weeding out repeat and boring work. However it can also be (and has also been) used to suppress work that should have been published - such as Barbara McClintock's work on Transposons which was against scientific consensus.

Edited by Leonid

Share this post


Link to post
Share on other sites

Firstly, letters aren't scrutinised as closely as proper papers. Letters are generally reports on work in progress not a final presentation of the work.

 

Secondly, you seem to think peer review stops at the editors desk. The most important aspect of the peer review process happens once the research has been published and is subject to the wider scientific community.

 

No scientist should take research on face value, all publications must be read with a critical and sceptical eye.

 

Whether this is crap research I cant tell because I dont have any knowledge of the subject matter.

Share this post


Link to post
Share on other sites

I always thought of "peer review" as what happens when fellow experts read the journal and give their thoughts. Not a basic editorial process.

 

The scientific method isn't useless - but as has been said scientific rigour requires resources which a journal just doesn't have. I guess this means that papers really ought to be judged after half a dozen subsequent experiments have checked their conclusions.

 

That seems to be what happens in the end anyway - the trouble is that without some kind of choke there would be vast volumes of unscientific, agenda-biased shite out there from every Tom, Dick and Harry with an axe to grind. You'd have to wade through so much shit to get to the good stuff it just wouldn't work.

Share this post


Link to post
Share on other sites

That seems to be what happens in the end anyway - the trouble is that without some kind of choke there would be vast volumes of unscientific, agenda-biased shite out there from every Tom, Dick and Harry with an axe to grind. You'd have to wade through so much shit to get to the good stuff it just wouldn't work.

I'm confused.

 

This is what happens now >.>

Share this post


Link to post
Share on other sites

The peer review process is a very basic check that someones claim is plausible, and that they have included enough information (correct or not) to support their claim and the journals criteria. There is no checking of results and so incorrect information has a very real change of getting through the review. To compound that matter, some journals allow the author to suggest who should review their work, and suggest who should not. Add to all that, that these journals charge a bunch of money to publish and access, the system looks a little shaky.

 

The good part is that papers don't just drop off the map once published. They are read and cited by many people in the field (and other fields). For example, 51 research papers have cited the above mentioned paper which has itself cited 28 others. When there is an important mistake made, it is picked up on and it is corrected either by the author with an erratum or by another researcher publishing their own work. The field is very competitive and eventually the right information moves forwards.

 

That said, peer review (and journal editing) is an important first step in weeding out repeat work, glaring errors or very small innovations that don't merit their own publication.

That's right. The paper is cited because it claims something and is used thereby to claim something else. But so was the 2006 Stem Cell fraud - it too was cited several dozen times.

 

That's a failure of peer review. One mistake in a paper, cleared by a peer review, can compound to damage a decade of future research. :)

 

You are completely correct regarding weeding out repeat and boring work. However it can also be (and has also been) used to suppress work that should have been published - such as Barbara McClintock's work on Transposons which was against scientific consensus.

 

Citations themselves are also pretty sketchy. Quite a lot point back to the authors own previous work or of their colleges and many just form part of a literature review rather than used as a basis of the authors own work. The point was that the important stuff comes together in the long term through all those links, as evidenced very clearly by the highly advanced position we are at today. I don't like the system (yet I work in it!) so I can understand the problems you point out when people pick papers for argument. Overall, I wholeheartedly support what it produces.

Share this post


Link to post
Share on other sites

Supam, I'm going to highlight exactly where you are wrong, by quoting those statements that are wrong.

 

Firstly, letters aren't scrutinised as closely as proper papers.

Letters are generally reports on work in progress not a final presentation of the work.

Secondly, you seem to think peer review stops at the editors desk. The most important aspect of the peer review process happens once the research has been published and is subject to the wider scientific community.

 

Leo explicitly states that " The point is........ all peer review is flawed" In the thread that he started, which I disagreed with.

You're disagreeing with a fact?

Share this post


Link to post
Share on other sites

Citations themselves are also pretty sketchy. Quite a lot point back to the authors own previous work or of their colleges and many just form part of a literature review rather than used as a basis of the authors own work. The point was that the important stuff comes together in the long term through all those links, as evidenced very clearly by the highly advanced position we are at today. I don't like the system (yet I work in it!) so I can understand the problems you point out when people pick papers for argument. Overall, I wholeheartedly support what it produces.

We haven't even addressed the major concerns I have with patenting and production of things without independant review >.> :D

 

As I said in another thread, there is no better method in our arsenal, to determine Truth, the capital T kind, than scientific methodology.

I just cry a little every time I see it murdered through faith-based acceptance of its findings.

Share this post


Link to post
Share on other sites

I fail to see the problem.

 

Peer review is supposed to catch stuff like this. However like everything else in life, its not perfect. Mistakes happen. Then they get caught. Wait, isn't that peer review?

Share this post


Link to post
Share on other sites

I fail to see the problem.

 

Peer review is supposed to catch stuff like this. However like everything else in life, its not perfect. Mistakes happen. Then they get caught. Wait, isn't that peer review?

Mostly.

 

You're missing the issue.

 

The issue is when something is published, and, in some cases, for years, despite peer review, no verification has been done, and miscommunication are made. People place faith in the process, where it is not warranted.

 

Scepticism is a vital, healthy, necessary part of science. Nothing is verified till you personally verify it, experimentally or practically.

Share this post


Link to post
Share on other sites

Citations themselves are also pretty sketchy. Quite a lot point back to the authors own previous work or of their colleges and many just form part of a literature review rather than used as a basis of the authors own work. The point was that the important stuff comes together in the long term through all those links, as evidenced very clearly by the highly advanced position we are at today. I don't like the system (yet I work in it!) so I can understand the problems you point out when people pick papers for argument. Overall, I wholeheartedly support what it produces.

We haven't even addressed the major concerns I have with patenting and production of things without independant review >.> :D

 

As I said in another thread, there is no better method in our arsenal, to determine Truth, the capital T kind, than scientific methodology.

I just cry a little every time I see it murdered through faith-based acceptance of its findings.

 

Agreeing or disagreeing with me?

 

To be clear, wholehearted support of what the process produces, not what the individual results produce.

Share this post


Link to post
Share on other sites

Citations themselves are also pretty sketchy. Quite a lot point back to the authors own previous work or of their colleges and many just form part of a literature review rather than used as a basis of the authors own work. The point was that the important stuff comes together in the long term through all those links, as evidenced very clearly by the highly advanced position we are at today. I don't like the system (yet I work in it!) so I can understand the problems you point out when people pick papers for argument. Overall, I wholeheartedly support what it produces.

We haven't even addressed the major concerns I have with patenting and production of things without independant review >.> :D

 

As I said in another thread, there is no better method in our arsenal, to determine Truth, the capital T kind, than scientific methodology.

I just cry a little every time I see it murdered through faith-based acceptance of its findings.

 

Agreeing or disagreeing with me?

 

To be clear, wholehearted support of what the process produces, not what the individual results produce.

 

I was not disagreeing :P

Share this post


Link to post
Share on other sites

I fail to see the problem.

 

Peer review is supposed to catch stuff like this. However like everything else in life, its not perfect. Mistakes happen. Then they get caught. Wait, isn't that peer review?

Mostly.

 

You're missing the issue.

 

The issue is when something is published, and, in some cases, for years, despite peer review, no verification has been done, and miscommunication are made. People place faith in the process, where it is not warranted.

 

Scepticism is a vital, healthy, necessary part of science. Nothing is verified till you personally verify it, experimentally or practically.

 

And thats the foundation of good science.

 

If everything was accepted as gospel then we'd still be wondering why alchemy didn't work.

Share this post


Link to post
Share on other sites

If I understand the description correctly, what we have now is a system that makes an attempt to check basic credibility and plausibility before the paper gets published.

 

Without this, published journals would be full of completely spastic and unscientific experiments involving alien-sightings, ESP, EVP, Voodoo and prophecy.

 

Maybe it looks that way to you already - but IMHO something has to be published sometime, otherwise there can't be anything approaching proper scientific scrutiny.

 

What system would you propose that does away with the problems we have now, without requiring a 15+ year wait (plus resources to re-do the experiments) before a paper is even published?

 

EDIT: the only solution I can think of is to shift the faith of people from peer review to peer verification.

Edited by thesorehead

Share this post


Link to post
Share on other sites

My real answer is not Leo or Arts grant friendly :D

 

The vast majority of scientific verification should be publicly funded, as part of the patents and IP processes. Original research can remain as it currently is.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×