Jump to content
Sign in to follow this  
Master_Scythe

Bitrot, is it real?

Recommended Posts

So I've been researching HOW I want to store my data.

As per my other thread, So far, BTRFS has won (In RAID1, since it's the only reliable type).

 

What I'm having trouble deciphering is how concern over bit rot, and random flips, is:

a. Un-needed (due to firmware CRC etc)

b. Sensible

c. Somewhat Paranoid (this is where I'm comfortable)

d. Beyond Paranoid

e. Just pure bullshit.

 

I read about people fixing significant data on their ZFS pools from bitrot, and "yay thanks ZFS!" threads.
I see similar threads from SnapRAID users, having recovered data from old snapshots that were lost from corruption.

 

But these could fall into the whole "People report when things go wrong, not when they go right" mentality.

 

But then I read articles such as this (and there are MANY others, especially on Reddit):

https://alastairs-place.net/blog/2014/01/16/bit-rot-and-raid/

 

Which make me feel like my "need for checksumming" falls into my 'D. or E." grouping above.

 

I ALSO read conflicting details on 'Linux Raid 5' and whether or not it corrects errors on the fly, since RAID5 needs a form of sum to calculate data if a disk is lost.

apparently some RAID systems do, and some don't? (checksumm on the fly) I can't find solid details.

 

Does anyone actually handle priceless data as a profession here, and might know more than one dude with a few months research does?

Keep in mind, most of mine isnt priceless; that which is, is backed up in triplicate, but the rest is still 'valuable'.

 

 

EDIT: Also, ECC ram is already present. and has had 48 hours of memtest with ECC enabled, and 48 hours with ECC disabled.

Edited by Master_Scythe

Share this post


Link to post
Share on other sites

That there are any reports at all means it IS something worthy of concern. Trick is, how frequently, and does it matter to YOU.

 

Even physically carving your data into iridium disks only lasts 100,000 years or so. Long-term archival is hard.

Share this post


Link to post
Share on other sites

But is it something to worry about like people that get struck by heat-lightning on a sunny day (so never go outside).

Or more like people suffering food poisoning (common, and real risk).

 

Luckily I'm only aiming for probably 10~15 years, before HDD technology advances enough to re-write it to something better.

Edited by Master_Scythe

Share this post


Link to post
Share on other sites

Interesting, wonder if tape suffers from this. Tapes can last for friggin ages.

I don't expect hard drives or SSD drives to last over 10 years, left alone 5.

 

If you want real long term storage

5D_33411ton.jpg

Edited by Jeruselem

Share this post


Link to post
Share on other sites

If you want real long term storage

Stone lasts, but it's not good for weathering, wear, or general rough treatment. Which is where the iridium comes in: it's damn near impossible to wreck, short of dropping it in a volcano, and it takes tech good enough to smelt that you're likely going to have microscopes capable of reading the data and realising the puck is worth more than material value.

 

Google "the Long Now". :)

  • Like 1

Share this post


Link to post
Share on other sites

Surv HDDs? MSY has them priced about 10% above the general WD Blue and Seagate Barracudas. Aren't they generally slower rpm? I'd also assume they're also aimed at long run situations though that said I've run generic desktop drives in machines nonstop since the early 2000s.

 

Long term storage - my feeling is that what was hot 10 years ago is ho-hum now and the price situation is still to the point where you could just go wholesale replacement provided you're in some reasonable range like under 20 TB.

 

Tape - for sure they can be long lasting but the environmental and handling concern for tape is such that you don't want it outside it's comfort zone re temp or humidity for more than a couple of hours every month. Long term tape/floppy can have the problem of the recording media just flaking off the backing surface, 5.25 floppy has the additional problem that the tissue inside the jacket can go bad and just crumble away, the outer jackets can warp though these can be overcome by just taking the disk out of the jacket and copying to another one.

 

 

Bitrot real - yep, that's why CDs and some magnetic media have you-beaut ECC systems that add 20% extra to the overall data payload. Combine that with the proper RAID types and it can be double or more that overhead but chances of successful data retention are higher.

What would worry me long term though - put a HDD in the cupboard, all the ECC in the world isn't worth jack shit if the bearing decides to seize up or the spindle motor dies.

Problem is, HDDs are now well and truly "consumer" items and as such the give a shit factor from the manufacturers seems to be declining.

  • Like 1

Share this post


Link to post
Share on other sites

I have some truly ancient flash and sd cards. Most still work but I never used them heavily. Surprised those haven't died yet.

Share this post


Link to post
Share on other sites

Surv HDDs? MSY has them priced about 10% above the general WD Blue and Seagate Barracudas. Aren't they generally slower rpm? I'd also assume they're also aimed at long run situations though that said I've run generic desktop drives in machines nonstop since the early 2000s.

 

No, the 4TB Surveillance (PIPELINE) drive is about $20 cheaper per drive than any other at MSY.

It might be 'the only one' but it sure is!

 

It's 5900RPM I think, which is more than fine for write once read only videos with a user base of 3......

 

The only thing I heard which might or might not be true, (I can't find any confirmation), is that Surveillance drives have much less firmware based error checking\CRC; because a few milliseconds of corrupted video is less important than keeping a continuous stream of recording data.

But I can't confirm this.

Share this post


Link to post
Share on other sites

 

Surv HDDs? MSY has them priced about 10% above the general WD Blue and Seagate Barracudas. Aren't they generally slower rpm? I'd also assume they're also aimed at long run situations though that said I've run generic desktop drives in machines nonstop since the early 2000s.

 

No, the 4TB Surveillance (PIPELINE) drive is about $20 cheaper per drive than any other at MSY.

It might be 'the only one' but it sure is!

 

It's 5900RPM I think, which is more than fine for write once read only videos with a user base of 3......

 

The only thing I heard which might or might not be true, (I can't find any confirmation), is that Surveillance drives have much less firmware based error checking\CRC; because a few milliseconds of corrupted video is less important than keeping a continuous stream of recording data.

But I can't confirm this.

 

Hmm,

 

I don't do security much but I do trouble-shoot it for a mate quite a bit - what you are suspecting is pretty much the case, nobody ever notices with slow scan CCTV if a few frames are corrupted.

 

Personally I don't like the drives much, seen too many fail so I usually put in "real" drives :)

 

Cheers

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×