On Fri, 17 Jan 2020 at 11:36, Andrew Greig via luv-main &lt;luv-main(a)luv.asn.au <mailto:luv-main@luv.asn.au>> wrote: I am thinking, then, for this build, perhaps I should set it up using Ubuntu Server. I will need to get my system to recognise the RAID drives as well. So before I jump in the deep end again, are there any "gotchas" of which I should be aware. Will the server version make life more reliable? Under the hood they're identical i.e. same kernel, same core system apps etc.  They differ in that the desktop versions have GUIs installed by default and come pre-packaged with desktop focused everyday use apps, whereas the server version doesn't. During installation of the server version you usually select a role for the server (or not) e.g. LAMP, mail, print, samba etc.   You don't get a GUI and typically administer it via CLI over SSH.  Nor do you get all the other guff that comes with a desktop version, so in some respect it's relatively clean. You can then add a GUI if you so wish. Each to their own, depends on what *you* want to do with it. -- Colin Fee tfeccles(a)gmail.com <mailto:tfeccles@gmail.com>

On Fri, Jan 17, 2020 at 11:36:29AM +1100, Andrew Greig wrote: You lost your home dir and the data in it when your SSD failed Because your rootfs and /home on the SSD doesn't have any redundancy (i.e. it was a single partition, with no RAID). I strongly recommend setting up a cron job to regularly snapshot it (at least once/day) and do a 'btrfs send' of that snapshot to a sub-volume of your /data filesystem. That way you won't lose much data from that partition if your SSD dies again - you can retrieve it from the last snapshot backup, and will only lose any changes since then. If your / and /home are on separate partitions (or btrfs sub-volumes) you will need to do this for both of them. (if you weren't running btrfs on /, you could do this with rsync instead of 'btrfs send', but rsync would be a lot slower) IME, drives are fragile and prone to failure. It's always best to make plans and backup procedures so that WHEN (not IF) a drive fails, you don't lose anything important...or, at least, minimise your losses. Also, remember that RAID is not a substitute for backup so you should regularly backup your /data filesystem to tape or other drives. Ideally, you should try to have an off-site backup in case of fire/flood/etc (e.g. backup to an external USB drive and store it at your office, lawyer's safe, a friend's house or somewhere. Have at least two of these so you can rotate the offsite backups). If the installer doesn't automatically detect your /data btrfs filesystem and add it to /etc/fstab, it's easy enough to add it yourself. the only significant difference between the server and desktop versions of ubuntu are the packages which are installed by default. e.g. the desktop version installs a whole bunch of desktop stuff (X, desktop environment and GUI apps, etc) that the server version doesn't. Otherwise, they're the same - same kernel, same libc and other standard system libraries, etc. craig -- craig sanders &lt;cas(a)taz.net.au&gt; _______________________________________________ luv-main mailing list luv-main(a)luv.asn.au https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main

Hi, On 18/1/20 2:14 pm, Andrew McGlashan via luv-main wrote: Some links, none of which are new as this occurred some time ago now. https://news.ycombinator.com/item?id=14907771

On Sun, Jan 19, 2020 at 05:38:23PM +1100, russell(a)coker.com.au wrote: Unless you need to make regular backups from workstations or small servers to a "big storage" ZFS backup server. In that case, use zfs so you can use 'zfs send'. Backups will be completed in a very small fraction of the time they'd take with rsync....the time difference is huge - minutes vs hours. That's fast enough to do them hourly or more frequently if needed, instead of daily.

No, it really doesn't. rsyncing **ANY** large set of data is slow, whether it's huge files like VM images or millions of small files (e.g. on a mail server).

rsync has to check at least the file sizes and timestamps, and then the block checksums on every run. On large sets, this WILL take many hours, no matter how much or how little has actually changed.

(a minor benefit of this is that if a file or directory is moved to another directory in the same dataset, the only blocks that actually changed were the blocks containing the directory info, so they're the only blocks that need be sent. rsync, however, would send the entire directory contents

because it's all "new" data. Transparent compression also helps 'zfs send' - compressed data requires fewer blocks to storer it....rsync, though, can't benefit from transparent compression as it has to compare the source file's *uncompressed* data with the target copy)

rsync is still useful as a tool for moving/copying data from one location to another (whether on the same machine or to a different machine), but it's no longer a good choice for backups. it just takes too long - by the time it has finished, the source data will have changed. It's an improved "cp".

I prefer to use the filesystem that's best for all machines on the network. If ZFS is in use on the file-server or backup-server, then that means zfs on everything else. If it's btrfs on the server, then it should be btrfs on everything.

btrfs is not an option here because it just isn't as good as zfs...if i'm

On Sat, Jan 18, 2020 at 02:14:46PM +1100, Andrew McGlashan wrote: If it's physically possible to install a second SSD of the same storage capacity or larger then he absolutely should do so. I vaguely recall suggesting he should get a second SSD for the rootfs ages ago, but my understanding / assumption was that there was only physical space and connectors for one SSD in the machine. The 'btrfs snapshot' + 'btrfs send' suggestion was just a way of regularly backing up a single-drive btrfs filesystem onto his raid-1 btrfs array so that little or nothing was lost in case of another drive failure. It's less than ideal, but a LOT better than nothing. I personally would never use anything less than RAID-1 (or equivalent, such as a mirrored pair on zfs) for any storage. Which means, of course, that I'm used to paying double for my storage capacity - i can't just buy one, I have to buy a pair. Not as a substitute for regular backups, but for convenience when only one drive of a pair has died. Drives die, and the time & inconvenience of dealing with that (and the lost data) cost far more than the price of a second drive for raid-1/mirror. craig -- craig sanders &lt;cas(a)taz.net.au&gt; _______________________________________________ luv-main mailing list luv-main(a)luv.asn.au https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main

On Sat, Jan 18, 2020 at 11:06:50PM +1100, Andrew Greig wrote: Your rootfs and your /data filesystem(*) are entirely separate. Don't confuse them. The /data filesystem needed to be re-balanced when you added the second drive (making it into a raid-1 array). 'btrfs balance' reads and rewrites all the existing data on a btrfs filesystem so that it is distributed equally over all drives in the array. For RAID-1, that means mirroring all the data on the first drive onto the second, so that there's a redundant copy of everything. Your rootfs is only a single partition, it doesn't have a raid-1 mirror, so re-balancing isn't necessary (and would do nothing). BTW, there's nothing being "remembered". 'btrfs balance' just re-balances the existing data over all drives in the array. It's a once-off operation that runs to completion and then exits. All **NEW** data will be automatically distributed across the array. If you ever add another drive to the array, or convert it to raid-0 (definitely NOT recommended), you'll need to re-balance it again. until and unless that happens you don't need to even think about re-balancing, it's no longer relevant. (*) I think you had your btrfs raid array mounted at /data, but I may be mis-remembering that. To the best of my knowledge, you have two entirely separate btrfs filesystems - one is the root filesystem, mounted as / (it also has /home on it, which IIRC you have made a separate btrfs sub-volume for). Anyway, it's a single-partition btrfs fs with no raid. The other is a 2 drive btrfs fs using raid-1, which I think is mounted as /data. /etc/fstab tells the system which filesystems to mount. It gets read at boot time by the system start up scripts. That would be a very good idea. Most modern motherboards will have more than enough NVME and SATA slots for that (e.g. most Ryzen x570 motherboards have 2 or 3 NVME slots for extremely fast SSDs, plus 6 or 8 SATA ports for SATA HDDs and SSDs. They also have enough RAM slots for 64GB DDR-4 RAM, and have at least 2 or 3 PCI-e v4 slots - you'll use one for your graphics card). 2 SSDs for the rootfs including your home dir, and 2 HDDs for your /data bulk storage filesystem. And more than enough drive ports for future expansion if you ever need it. ----------------------- some info on nvme vs sata: NVME SSDs are **much** faster then SATA SSDs. SATA 3 is 6 Gbps (600 MBps), so taking protocol overhead into account SATA drives max out at around 550 MBps. NVME drives run at **up to** PCI-e bus speeds - with 4 lanes, that's a little under 40 Gbps for PCIe v3 (approx 4000 MBps minus protocol overhead), double that for PCIe v4. That's the theoretical maximum speed, anyway. In practice, most NVME SSDs run quite a bit slower than that, about 2 GBps - that's still almost 4 times as fast as a SATA SSD. Some brands and models (e.g. those from samsung and crucial) run at around 3200 to 3500 MBps, but they cost more (e.g. a 1TB Samsung 970 EVO PLUS (MZ-V7S1T0BW) costs around $300, while the 1TB Kingston A2000 (SA2000M8/1000G) costs around $160 but is only around 1800 MBps). AFAIK there are no NVME drives that run at full PCI-e v4 speed (~8 GBps with 4 lanes) yet, it's still too new. That's not a problem, PCI-e is designed to be backwards-compatible with earlier versions, so any current NVME drive will work in pcie v4 slots. NVME SSDs cost about the same as SATA SSDs of the same capacity so there's no reason not to get them if your motherboard has NVME slots (which are pretty much standard these days). BTW, the socket that NVME drives plug into is called "M.2". M.2 supports both SATA & NVME protocols. SATA M.2 runs at 6 Gbps. NVME runs at PCI-e bus speed. So you have to be careful when you buy to make sure you get an NVME M.2 drive and not a SATA drive in M.2 form-factor...some retailers will try to exploit the confusion over this. craig -- craig sanders &lt;cas(a)taz.net.au&gt; _______________________________________________ luv-main mailing list luv-main(a)luv.asn.au https://lists.luv.asn.au/cgi-bin/mailman/listinfo/luv-main

here is the output of blkid /dev/sdb1: LABEL="Data" UUID="73f55e83-2038-4a0d-9c05-8f7e2e741517" UUID_SUB="77fdea4e-3157-45af-bba4-7db8eb04ff08" TYPE="btrfs" PARTUUID="d5d96658-01" /dev/sdc1: LABEL="Data" UUID="73f55e83-2038-4a0d-9c05-8f7e2e741517" UUID_SUB="8ad739f7-675e-4aeb-ab27-299b34f6ace5" TYPE="btrfs" PARTUUID="a1948e65-01" I tried the first UUID for sdc1 and the machine hung but gave me an opportunity to edit the fstab and reboot.

When checking the UUID I discovered that the first entry for both drives were identical.

Should I be using the SUB UUID for sdc1 for the entry in fstab?

I personally would never use anything less than RAID-1 (or equivalent, such as a mirrored pair on zfs) for any storage. Which means, of course, that I'm used to paying double for my storage capacity - i can't just buy one, I have to buy a pair. Not as a substitute for regular backups, but for convenience when only one drive of a pair has died. Drives die, and the time & inconvenience of dealing with that (and the lost data) cost far more than the price of a second drive for raid-1/mirror.

On Sun, Jan 19, 2020 at 05:34:46PM +1100, russell(a)coker.com.au wrote: I don't see how that's the case, how it can help much (if at all). Making a second copy of the data on the same drive that's failing doesn't add much redundancy, but does add significantly to the drive's workload (increasing the risk of failure). It might be ok on a drive with only a few bad sectors or in conjunction with some kind of RAID, but it's not a substitute for RAID.

I haven't either, but I've heard & read of it. Andrew's rootfs SSD seems to have died (or possibly just corrupted so badly it can't be mounted. i'm not sure) I've seen LOTS of HDDs die. Even at home I've had dozens die on me over the years - I've got multiple stacks of dead drives of various ages and sizes cluttering up shelves (mostly waiting for me to need another fridge magnet or shiny coffee-cup coaster :)

If a drive is failing, all the read or write re-tries kill performance on a zpool, and that drive will eventually be evicted from the pool. Lose enough drives, and your pool goes from "DEGRADED" to "FAILED", and your data goes with it.

On Tue, Jan 28, 2020 at 08:02:15PM +1100, russell(a)coker.com.au wrote: it's not rare at all, but a drive doesn't have to be completely non-responsive to be considered "dead". It just has to consistently cause enough errors that it results in the pool being degraded.

To me, that's a dead drive because it's not safe to use. it can not be trusted to reliably store data. it is junk. the only good use for it is to scrap it for the magnets.

a drive that consistently returns bad data is not fit for purpose. it is junk. it is a dead drive.

IMO, two copies of data on a drive you can't trust isn't significantly better or more useful than one copy. It's roughly equivalent to making a photocopy of your important documents and then putting both copies in the same soggy cardboard box in a damp cellar.