Jump to content


Photo

Bitrot, is it real?


  • Please log in to reply
8 replies to this topic

#1 Master_Scythe

Master_Scythe

    Titan

  • Hero
  • 19,694 posts
  • Location:QLD

Posted 22 February 2017 - 09:08 AM

So I've been researching HOW I want to store my data.

As per my other thread, So far, BTRFS has won (In RAID1, since it's the only reliable type).

 

What I'm having trouble deciphering is how concern over bit rot, and random flips, is:

a. Un-needed (due to firmware CRC etc)

b. Sensible

c. Somewhat Paranoid (this is where I'm comfortable)

d. Beyond Paranoid

e. Just pure bullshit.

 

I read about people fixing significant data on their ZFS pools from bitrot, and "yay thanks ZFS!" threads.
I see similar threads from SnapRAID users, having recovered data from old snapshots that were lost from corruption.

 

But these could fall into the whole "People report when things go wrong, not when they go right" mentality.

 

But then I read articles such as this (and there are MANY others, especially on Reddit):

https://alastairs-pl...t-rot-and-raid/

 

Which make me feel like my "need for checksumming" falls into my 'D. or E." grouping above.

 

I ALSO read conflicting details on 'Linux Raid 5' and whether or not it corrects errors on the fly, since RAID5 needs a form of sum to calculate data if a disk is lost.

apparently some RAID systems do, and some don't? (checksumm on the fly) I can't find solid details.

 

Does anyone actually handle priceless data as a profession here, and might know more than one dude with a few months research does?

Keep in mind, most of mine isnt priceless; that which is, is backed up in triplicate, but the rest is still 'valuable'.

 

 

EDIT: Also, ECC ram is already present. and has had 48 hours of memtest with ECC enabled, and 48 hours with ECC disabled.


Edited by Master_Scythe, 22 February 2017 - 09:21 AM.

Wherever you go in life, watch out for Scythe, the tackling IT support guy.

"I don't care what race you are, not one f*cking bit, if you want to be seen as a good people, you go in there and you f*ck up the people who (unofficially) represent you in a negative light!"


#2 Cybes

Cybes

    Titan

  • Atomican
  • 17,617 posts
  • Location:Where I am

Posted 22 February 2017 - 10:45 AM

That there are any reports at all means it IS something worthy of concern. Trick is, how frequently, and does it matter to YOU.

Even physically carving your data into iridium disks only lasts 100,000 years or so. Long-term archival is hard.

"Reality does not care what you think." - Dr Richard Feynman
"Ideas do not deserve respect.- people do." - Peter Boghossian (paraphrase)


#3 Master_Scythe

Master_Scythe

    Titan

  • Hero
  • 19,694 posts
  • Location:QLD

Posted 22 February 2017 - 10:53 AM

But is it something to worry about like people that get struck by heat-lightning on a sunny day (so never go outside).

Or more like people suffering food poisoning (common, and real risk).

 

Luckily I'm only aiming for probably 10~15 years, before HDD technology advances enough to re-write it to something better.


Edited by Master_Scythe, 22 February 2017 - 10:54 AM.

Wherever you go in life, watch out for Scythe, the tackling IT support guy.

"I don't care what race you are, not one f*cking bit, if you want to be seen as a good people, you go in there and you f*ck up the people who (unofficially) represent you in a negative light!"


#4 Jeruselem

Jeruselem

    Guru

  • Atomican
  • 13,193 posts
  • Location:Not Trump-Land

Posted 22 February 2017 - 11:49 AM

Interesting, wonder if tape suffers from this. Tapes can last for friggin ages.

I don't expect hard drives or SSD drives to last over 10 years, left alone 5.

 

If you want real long term storage

5D_33411ton.jpg


Edited by Jeruselem, 22 February 2017 - 11:54 AM.

Cortana at your service


#5 Cybes

Cybes

    Titan

  • Atomican
  • 17,617 posts
  • Location:Where I am

Posted 22 February 2017 - 12:36 PM

If you want real long term storage


Stone lasts, but it's not good for weathering, wear, or general rough treatment. Which is where the iridium comes in: it's damn near impossible to wreck, short of dropping it in a volcano, and it takes tech good enough to smelt that you're likely going to have microscopes capable of reading the data and realising the puck is worth more than material value.

Google "the Long Now". :)

"Reality does not care what you think." - Dr Richard Feynman
"Ideas do not deserve respect.- people do." - Peter Boghossian (paraphrase)


#6 Master_Scythe

Master_Scythe

    Titan

  • Hero
  • 19,694 posts
  • Location:QLD

Posted 22 February 2017 - 01:02 PM

While on this topic;

 

Surveillance HDD"s are MUCH cheaper than others; is there a problem using these for data storage?


Wherever you go in life, watch out for Scythe, the tackling IT support guy.

"I don't care what race you are, not one f*cking bit, if you want to be seen as a good people, you go in there and you f*ck up the people who (unofficially) represent you in a negative light!"


#7 Rybags

Rybags

    Immortal

  • Super Hero
  • 34,290 posts

Posted 22 February 2017 - 05:42 PM

Surv HDDs?  MSY has them priced about 10% above the general WD Blue and Seagate Barracudas.  Aren't they generally slower rpm?  I'd also assume they're also aimed at long run situations though that said I've run generic desktop drives in machines nonstop since the early 2000s.

 

Long term storage - my feeling is that what was hot 10 years ago is ho-hum now and the price situation is still to the point where you could just go wholesale replacement provided you're in some reasonable range like under 20 TB.

 

Tape - for sure they can be long lasting but the environmental and handling concern for tape is such that you don't want it outside it's comfort zone re temp or humidity for more than a couple of hours every month.  Long term tape/floppy can have the problem of the recording media just flaking off the backing surface, 5.25 floppy has the additional problem that the tissue inside the jacket can go bad and just crumble away, the outer jackets can warp though these can be overcome by just taking the disk out of the jacket and copying to another one.

 

 

Bitrot real - yep, that's why CDs and some magnetic media have you-beaut ECC systems that add 20% extra to the overall data payload.   Combine that with the proper RAID types and it can be double or more that overhead but chances of successful data retention are higher.

What would worry me long term though - put a HDD in the cupboard, all the ECC in the world isn't worth jack shit if the bearing decides to seize up or the spindle motor dies.

Problem is, HDDs are now well and truly "consumer" items and as such the give a shit factor from the manufacturers seems to be declining.



#8 Jeruselem

Jeruselem

    Guru

  • Atomican
  • 13,193 posts
  • Location:Not Trump-Land

Posted 22 February 2017 - 05:47 PM

I have some truly ancient flash and sd cards. Most still work but I never used them heavily. Surprised those haven't died yet.

Cortana at your service


#9 Master_Scythe

Master_Scythe

    Titan

  • Hero
  • 19,694 posts
  • Location:QLD

Posted 23 February 2017 - 08:48 AM

Surv HDDs?  MSY has them priced about 10% above the general WD Blue and Seagate Barracudas.  Aren't they generally slower rpm?  I'd also assume they're also aimed at long run situations though that said I've run generic desktop drives in machines nonstop since the early 2000s.

 

No, the 4TB Surveillance (PIPELINE) drive is about $20 cheaper per drive than any other at MSY.

It might be 'the only one' but it sure is!

 

It's 5900RPM I think, which is more than fine for write once read only videos with a user base of 3......

 

The only thing I heard which might or might not be true, (I can't find any confirmation), is that Surveillance drives have much less firmware based error checking\CRC; because a few milliseconds of corrupted video is less important than keeping a continuous stream of recording data.

But I can't confirm this.


Wherever you go in life, watch out for Scythe, the tackling IT support guy.

"I don't care what race you are, not one f*cking bit, if you want to be seen as a good people, you go in there and you f*ck up the people who (unofficially) represent you in a negative light!"





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users