Reliable data storage
I have been reading the "btrfs/ZFS, sans raid and bitrot" thread and a number of thoughts and questions spring to mind. I get the impression that some are looking for a single reliable storage solution to avoid having to do backups. Surely this is impossible, I certainly would ______NEVER______ (excuse the shouting) ever trust my life to a single system if at all possible. When one is doing instrument flying training as a pilot you are constantly told never to rely on a single instrument but scan all of them and come up with an overall coherent picture. If one relies in such circumstances on a single point of failure you __will__ kill yourself. One is told raid or any such thing is a reliabilty strategy __not__ a backup strategy. I personally keep all data I consider important on four separate systems/devices one device (which is in fact duplicate items but differing technolgies) being kept off site. Maintaining this is a bit of a pain but there is no other way as far as I can see. The reason for the number of separate backups is we had in one instance in a large commercial situation managed to destroy two backs trying to restore a system. We only succeeded in the end becuase I had independantly duplicated one of the backups on another system. Lindsay
On 1 July 2014 09:20, <zlinw@mcmedia.com.au> wrote:
I have been reading the "btrfs/ZFS, sans raid and bitrot" thread and a number of thoughts and questions spring to mind.
I get the impression that some are looking for a single reliable storage solution to avoid having to do backups.
Surely this is impossible, I certainly would ______NEVER______ (excuse the shouting) ever trust my life to a single system if at all possible. When one is doing instrument flying training as a pilot you are constantly told never to rely on a single instrument but scan all of them and come up with an overall coherent picture. If one relies in such circumstances on a single point of failure you __will__ kill yourself.
One is told raid or any such thing is a reliabilty strategy __not__ a backup strategy.
Correct and all good points. But even when you've got a good back-up system of a RAID array (btrfs/zfs not withstanding) any bitrot gets backed up too. This happened to me on my media server - 2 disks in a mirrored set had corrupted TV episodes on them which in turn were sent to the back-up in the corrupted state. Software raid didn't know about the corruption and dutifully carried on keeping the corrupted data. Fortunately I have the original DVDs from which I could re-rip the corrupted episodes. So I've switched to zfs and I back-up to two separate places (also on zfs). Weekly scrub ops detect and correct data as best they can. -- Colin Fee tfeccles@gmail.com
On Tue, 1 Jul 2014 09:20:28 zlinw@mcmedia.com.au wrote:
I get the impression that some are looking for a single reliable storage solution to avoid having to do backups.
No that's not the aim. However you do want to reduce the need for external backups. Removable backup (which is necessary to cover lightning strikes etc) isn't something that you are going to do all the time. If you use your removable backup every week then a random restore can be expected to miss 3.5 days of changes. So there's a real benefit in not losing data. ZFS and BTRFS allow taking snapshots of the data. It's easy to write cron jobs that make snapshots regularly. If you have a snapshot every 30 minutes then for the most common uses of backups (deleting the wrong files) you will only lose 15 minutes of data on average.
I personally keep all data I consider important on four separate systems/devices one device (which is in fact duplicate items but differing technolgies) being kept off site. Maintaining this is a bit of a pain but there is no other way as far as I can see.
As Colin noted if you get corruption then it can go to all backups. Also if you have corruption in a backup it can go back to the main system on a restore. Use BTRFS or ZFS for the main data store and BTRFS for the backup media and you won't risk either of those problems. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
Hi, In photography circles I am always told that it doesn't exist unless it is stored in 3 places. I have my a local backup of all my files on an external disk and then I use an online service like crashplan to back my really important stuff to the cloud. This set up has saved me many times. No matter how good your system is anything can cause a malfunction, bad disks, powerfailures so it is always good to keep multiple copies of important files. Gordon. On 1 Jul 2014, at 9:20 am, zlinw@mcmedia.com.au wrote:
I have been reading the "btrfs/ZFS, sans raid and bitrot" thread and a number of thoughts and questions spring to mind.
I get the impression that some are looking for a single reliable storage solution to avoid having to do backups.
Surely this is impossible, I certainly would ______NEVER______ (excuse the shouting) ever trust my life to a single system if at all possible. When one is doing instrument flying training as a pilot you are constantly told never to rely on a single instrument but scan all of them and come up with an overall coherent picture. If one relies in such circumstances on a single point of failure you __will__ kill yourself.
One is told raid or any such thing is a reliabilty strategy __not__ a backup strategy.
I personally keep all data I consider important on four separate systems/devices one device (which is in fact duplicate items but differing technolgies) being kept off site. Maintaining this is a bit of a pain but there is no other way as far as I can see.
The reason for the number of separate backups is we had in one instance in a large commercial situation managed to destroy two backs trying to restore a system. We only succeeded in the end becuase I had independantly duplicated one of the backups on another system.
Lindsay _______________________________________________ luv-main mailing list luv-main@luv.asn.au http://lists.luv.asn.au/listinfo/luv-main
Just to add my 2c considering I started the other thread, I'm not aiming to replace my backups and data is indeed in three places; My main drive (esata) My backup drive (USB) Another backup drive connected to a computer at my parents place, using crashplan over the Internet. The holes in this plan that I can see are: 1. Bitrot (discussed in other thread) 2. All backups are online. Someone malicious on my main pc could potentially wipe the remote backup through the crashplan GUI. This is a hole I'd like to fix but it may be too inconvenient to fix at the moment. Some form of online but write only storage would be stellar. -Noah On Tuesday, July 1, 2014, <zlinw@mcmedia.com.au> wrote:
I have been reading the "btrfs/ZFS, sans raid and bitrot" thread and a number of thoughts and questions spring to mind.
I get the impression that some are looking for a single reliable storage solution to avoid having to do backups.
Surely this is impossible, I certainly would ______NEVER______ (excuse the shouting) ever trust my life to a single system if at all possible. When one is doing instrument flying training as a pilot you are constantly told never to rely on a single instrument but scan all of them and come up with an overall coherent picture. If one relies in such circumstances on a single point of failure you __will__ kill yourself.
One is told raid or any such thing is a reliabilty strategy __not__ a backup strategy.
I personally keep all data I consider important on four separate systems/devices one device (which is in fact duplicate items but differing technolgies) being kept off site. Maintaining this is a bit of a pain but there is no other way as far as I can see.
The reason for the number of separate backups is we had in one instance in a large commercial situation managed to destroy two backs trying to restore a system. We only succeeded in the end becuase I had independantly duplicated one of the backups on another system.
Lindsay _______________________________________________ luv-main mailing list luv-main@luv.asn.au <javascript:;> http://lists.luv.asn.au/listinfo/luv-main
participants (5)
-
Colin Fee -
Gordon Heydon -
Noah O'Donoghue -
Russell Coker -
zlinw＠mcmedia.com.au