On Mon, May 21, 2018 at 05:23:39PM +1000, pushin.linux wrote:

Reply to list wasnt offered in my phone. Apologies.

no problem. I'll reply back to the list, so it goes to the right place.

My photographic data is critical. Music, videos etc are unimportant.

You've probably heard this before but: ****************************************** ****************************************** ** ** ** RAID IS NOT A SUBSTITUTE FOR BACKUP! ** ** ** ****************************************** ****************************************** RAID is convenient, and it allows your system to keep going without having to restore from backup but you will still need to backup your photos and other important data regularly.

I could buy another 2 Tb drive, but what to do with the 1Tb drive. I thought I could have bare system running onthe 1Tb and all storage on a RAID pair.

What you have will work fine, there's nothing wrong with it. I just think that you're better off having your OS disk on some form of RAID as well. The easiest way to do that is to just get another 1TB drive (approx $60). Then you'd have two mirrored drives, one for OS & home dir and other stuff, and one for your photos. If you're using ZFS, you could even set it up so that you have a combined pool with two mirroed pairs (2 x 1TB drives and 2 x 2TB), giving a total of 3TB shared between OS and your photos. This is probably the most flexible setup. LVM would also allow you to combine the storage, but it's quite a bit more work and more complicated to set up. BTW, just to state the obvious - each mirrored pair of drives should be the same size (if they're different, you'll only get the capacity of the smallest drive in the pair), but you can have multiple mirrors of different sizes in a pool. e.g. here's the root zfs pool on my main system. It has the OS, my home directories, and some other stuff on it. Most of my data is on a second 4 TB pool, and this machine also has an 8TB pool called "backup" which has regular (hourly, daily, weekly, monthly) snapshotted backups of every machine on my home network. # zpool status ganesh pool: ganesh state: ONLINE scan: scrub repaired 0B in 0h10m with 0 errors on Sat Apr 28 02:10:27 2018 config: NAME STATE READ WRITE CKSUM ganesh ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ata-Crucial_CT275MX300SSD1_163313AADD8A-part5 ONLINE 0 0 0 ata-Crucial_CT275MX300SSD1_163313AAEE5F-part5 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 ata-Crucial_CT275MX300SSD1_163313AAF850-part5 ONLINE 0 0 0 ata-Crucial_CT275MX300SSD1_163313AB002C-part5 ONLINE 0 0 0 That has two mirrored pairs of drives (roughly equivalent to RAID-10 in mdadm terms), called mirror-0 and mirror-1. BTW, "mirror-0" and "mirror-1" are what are known as "vdev"s or "virtual devices". A vdev can be a mirrored set of drives as above, or a raid-z, or even a single drive (but there's no redundancy for a single drive and adding one to a pool effectively destroys the entire pool's redundancy, so never do that). A ZFS pool is made up of one or vdevs. Also BTW, a mirrored set can be pairs as I have, or (just like RAID-1 mirrors) you can mirror to three or four or more drives if you want extra redundancy (and extra read speed) Anyway, the vdevs here happen to be both the same size because I bought 4 identical SSDs to set it up with (4 x 256 GB was slightly more expensive than 2 x 512GB, but by spreading the IO over 4 drives rather than just two, I get about double the read performance), but there's no reason at all why they couldn't be different sizes. e.g. the easiest and fastest way for me to double the capacity of that pool would be to just add a pair of 512 GB SSDs to it. It's nowhere near full, so I won't be doing that any time in the forseeable future. In fact, that's one of the advantages of using mirrored-pairs - you can upgrade the pool two drives at a time. Either by adding another pair of drives, or by replacing both drives in a pair with larger drives. For comparison, here's the main storage pool "export" of my MythTV box. It has one vdev called "raidz1-0", with 4 x 2TB drives. # zpool status export pool: export state: ONLINE scan: scrub repaired 0B in 15h14m with 0 errors on Sat May 19 18:55:01 2018 config: NAME STATE READ WRITE CKSUM export ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 ata-ST2000DL003-9VT166_5YD1QFAG ONLINE 0 0 0 ata-WDC_WD20EARS-00MVWB0_WD-WCAZA5379164 ONLINE 0 0 0 ata-WDC_WD20EARX-008FB0_WD-WCAZAJ827116 ONLINE 0 0 0 ata-WDC_WD20EARS-00MVWB0_WD-WCAZA5353040 ONLINE 0 0 0 If I wanted to upgrade it, I could either add a second vdev to the pool (a mirrored pair, or another raid-z vdev), OR i could replace each of the 2 TB drives with, say, 4TB drives. I'd only see the extra capacity when ALL drives in the vdev had been replaced. In practice, that would take so long that it would be much faster to just create a new pool with 4 x 4 TB drives and use 'zfs send' to copy everything to the new pool, then retire the old pool. BTW, You can see that I've had to replace one of the Western Digital drives with a Seagate at some time in the past. If I wanted to know when that happened, I could run "zpool history" - ZFS stores a history of every significant thing that happens to the pool. e.g. # zpool history export | grep ata-ST2000DL003-9VT166_5YD1QFAG 2016-06-05.15:00:03 zpool replace -f export ata-WDC_WD20EARX-00PASB0_WD-WCAZA8430027 /dev/disk/by-id/ata-ST2000DL003-9VT166_5YD1QFAG and if I wanted to know when I created the pool: # zpool history export | head -2 History for 'export': 2012-07-15.09:13:43 zpool create -f -o ashift=12 export raidz scsi-SATA_WDC_WD20EARX-00_WD-WCAZA8436337 scsi-SATA_WDC_WD20EARS-00_WD-WCAZA5379164 scsi-SATA_WDC_WD20EARX-00_WD-WCAZA8430027 scsi-SATA_WDC_WD20EARS-00_WD-WCAZA5353040 The ashift=12 option tells 'zpool' to create the pool aligned for 4K sectors (2^12 = 4096) instead of the default 512 byte sectors (ashift=9, 2^9 = 512). And The "scsi-SATA-" and "ata-" prefixes refer to the same drives. I expect that I exported the pool and then re-imported it at some point and the drive names changed slightly. or maybe after an upgrade the kernel stopped caring about the fact that the SATA drives were on a SAS scsi controller. Don't know, don't care, not important...the model and serial numbers identify the drives, and I have sticky labels with the serial numbers on the hot-swap bays. if you carefully compare the zpool create command with the status output above, you'll notice that the drives listed in the create command aren't the same as those in the status output. I've had to replace a few of those WD EARX drives in that pool. # zpool history export | grep replace 2012-07-18.09:27:30 zpool replace -f export scsi-SATA_WDC_WD20EARX-00_WD-WCAZA8430027 scsi-SATA_WDC_WD20EARX-00_WD-WMAZA9502728 2013-01-03.21:15:57 zpool replace -f export scsi-SATA_WDC_WD20EARX-00_WD-WMAZA9502728 scsi-SATA_WDC_WD20EARX-00_WD-WCAZAJ827116 2016-05-18.22:24:28 zpool replace -f export ata-WDC_WD20EARX-00PASB0_WD-WCAZA8436337 /dev/disk/by-id/ata-WDC_WD20EARX-00PASB0_WD-WCAZA8430027 2016-06-05.15:00:03 zpool replace -f export ata-WDC_WD20EARX-00PASB0_WD-WCAZA8430027 /dev/disk/by-id/ata-ST2000DL003-9VT166_5YD1QFAG In fact, you can see that on 18 May 2016, I tried to replace one of the drives (WCAZA8436337) with one i'd previously removed and replaced (WCAZA8430027), then about three weeks later on 6 June 2016 replaced it with a seagate drive. That's what happens when you leave dead/dying drives just lying around without writing "dead" on them.

I really appreciate the enormous amount of support. The only issue now is how to create a roadmap from it all.

Again, no problem. And, like I said, the best thing you can do is to start playing with this stuff in some virtual machines. practice with it until it's completely familiar, and until you understand it well enough to be able to make informed decisions that suit your exact needs. VMs are great for trying stuff out in a safe environment that won't mess with your real system. You can take some stupid risks and learn from them - in fact, that's one of the great things about VMs for learning, you can deliberately do all the things you've read not to do so you understand WHY you shouldn't and also hopefully learn how you can recover from making such disastrous mistakes. In this case, VMs are also a good way to compare the differences between LVM alone, mdadm alone, LVM+mdam, btrfs, and ZFS. and more. Learn what each is capable of and how to use the tools to control them. Learn what happens to an array or pool when you tell KVM to detach one or more of the virtual disks from the running VM. Or if you some garbage data to one or more of the vdisks. I've got a few VMs for doing that. One of them, called "ztest" (because it started out being just for ZFS testing) has a 5GB boot disk (debian sid) plus another 12 virtual disks attached to it, each about 200MB in size. These get combined in various configurations for zfs, btrfs, lvm, mdadm depending on what I want to experiment with at the time. It's still worth doing this even if you've already finished setting up your new drives. as long as you've got somewhere to make a complete fresh backup of your data, you can always rebuild your system if you find a better way to set things up for your needs. And also worth it because the better you know mdadm or lvm or zfs or whatever you end up using, the less likely you are to panic and make a terrible mistake if a drive dies and you have to replace it, or deal with some other problem. the best time to learn system recovery techniques is before you need them, not at the exact moment that you need them :) craig -- craig sanders <cas@taz.net.au>