On 08/07/2015 5:39 PM, "Brian May" <brian(a)microcomaustralia.com.au> wrote: that matter zfs) instead of software raid and let it manage the raid and file system. BTRFS and Zfs work from the block level up. "RAID" support (currently) offers no guarantee that two copies of everything will be on separate disks. So if one disk fails, you could loose both copies of your data. I can't speak for the case with btrfs but zfs the raid component works as expectec. Last year when i had a disk failure zfs effectively kicked it out of the array but my data was still there as expected. The replacement process was very easy to handle as well with zfs's well thought commands.

On Wed, 8 Jul 2015 at 21:50 Chris Samuel <chris(a)csamuel.org> wrote: Oh, OK, so I misunderstood. So in other words, using BTRFS RAID-1 is fine if you have exactly two disks?

On Thu, 16 Jul 2015 at 18:08 Andrew McGlashan < andrew.mcglashan(a)affinityvision.com.au&gt; wrote: I have always heard people say ZFS is better, Oracle should support ZFS not btrfs, etc. Don't know what to think myself. coreos stopped using btrfs by default. They say they encountered too many problems with it. https://groups.google.com/forum/m/#!topic/coreos-dev/NDEOXchAbuU

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hi, On 16/07/2015 8:28 PM, Brian May wrote: Oracle inherited ZFS, it wasn't their own product; they were already working on btrfs and they also have OCFS2 (Oracle clustered file system 2). Consequently it isn't surprising to see that Oracle has all but abandoned ZFS, there are other reasons too, so it's unfortunate. It /may/ be that ZFS will be the betamax of file systems; but perhaps not, perhaps it just hasn't been given enough love to shine out and become the winner. That forum thread was from over 6 months ago, that's a long time in btrfs speak. I'm not sure, but I think we need appliances that server btrfs file systems via iSCSI or other methods. Having an appliance that is limited to handling all the file system stuff and allowing it to have static kernel (as much as possible), or at least managed independently of other server requirements, such as web / mail / dns and other servers for instance. A.

On Thu, 16 Jul 2015 at 18:08 Andrew McGlashan < andrew.mcglashan(a)affinityvision.com.au&gt; wrote: I had problems looking at that email, it appears Google Mail on android can handle the embedded message, but Google Mail on Firefox can't. As such I looked at the file in a text editor, which gave incorrect links because - I suspect - of quoted-printable encoding. The LCA one is https://www.youtube.com/watch?v=6DplcPrQjvA Just watching it now; I note it says to avoid kernels 3.15 to 3.16.1 - Debian Jessie has 3.16.0... Will watch the NYLUG one next - seems to be at https://www.youtube.com/watch?v=W3QRWUfBua8

On Sat, 18 Jul 2015 at 11:05 Brian May &lt;brian(a)microcomaustralia.com.au&gt; wrote: At 26:25 he said quota support has had significant improvement and "quotas is really something that you can use". Interesting.

On Sat, 18 Jul 2015 at 15:11 Brian May &lt;brian(a)microcomaustralia.com.au&gt; wrote: Got this working. I had to back port nvidia-graphics-driver (from experimental) and libvdpau (from unstable) to Debian Jessie. It seems to work fine. Are there any resources for recommended setup details, e.g. recommended cron jobs for scrubbing btrfs, etc? Anything else recommended with Linux-4.1? Apparently, from videos, manual balancing shouldn't be required any more? Thanks

On Mon, 27 Jul 2015 06:00:40 PM Peter Ross wrote: "BTRFS is a Technology Preview in Red Hat Enterprise Linux 7." You're on an ancient kernel (based on 3.10). Pretty sure what you're seeing has been fixed since then. -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC

Quoting Russell Coker (russell(a)coker.com.au): Just a reminder: ZFS code is licensed under CDDL, which is a weak-copyleft variant of MPL. It is not under BSD or any other permissive licence. (Some permissive-licensing ideologues don't like ZFS in FreeBSD for the same reason they don't like any other copylefted code in that project.)

Quoting Craig Sanders (cas(a)taz.net.au): [ZFS code and other parts of Solaris being under CDDL:] I'm honestly not sure there wasn't a reason derived from Sun's relations with third-party stakeholders -- mostly because that was Netscape Communications's reason for crafting MPL to apply reciprocal obligation or not on a code-module by code-module basis. However, I think we're departing from the scope of luv-main, so I won't elaborate. Yeah, boy, I've sure seen this, and been mystified.

P.S. I added another virtual disk to make it happy. The problem is that I cannot see anything critically close to be exhausted. So I have no reliable way of monitoring and predict when the problem appears. Thanks Peter On Tue, Jul 28, 2015 at 9:54 AM, Peter Ross &lt;petrosssit(a)gmail.com&gt; wrote:

In the meantime, I experimented with btrfs add/delete. btrfs delete was the end of it. The system was stuck and after reset it did not boot anymore. Regards peter On Tue, Jul 28, 2015 at 11:19 AM, Russell Coker &lt;russell(a)coker.com.au&gt; wrote:

Hi Russell, there was enough space (in theory) to keep all data on one disk (less than 5GB on a 8GB partition, from memory), and the "btrfs delete" seems to have succeeded. "btrfs filesystem df /" and device stats looked fine and did not show any reference to the second disk. The trouble only started when I removed the second disk (physically, as in disk on a virtual machine on ESXi). Using a newer kernel is out of question here - I have to use the latest "Enterprise Linux" (CentOS 7) without patches. It looks as I have to abandon all plans to use btrfs and am back in the stone age. Thanks for comments and suggestions Peter On Tue, Jul 28, 2015 at 11:21 PM, Russell Coker &lt;russell(a)coker.com.au&gt; wrote:

Marcus Furlong wrote: I wonder how safe it is to use these kernels, and whether it breaks the userland. Enterprise Linux (with well-tested software in their own "kernel/userland universe") and then throwing out crucial parts of it. I could put a FreeBSD kernel underneath then, it has a Linux kernel ABI, and start deploying CentOS jails on ZFS;-) https://wiki.freebsd.org/VIMAGE/Linux/CentOS55 I don't think I get away with this either;-) Thanks Peter On Wed, Jul 29, 2015 at 12:07 PM, Marcus Furlong &lt;furlongm(a)gmail.com&gt; wrote: > > > Using a newer kernel is out of question here - I have to use the latest > > "Enterprise Linux" (CentOS 7) without patches. > > > http://wiki.centos.org/AdditionalResources/Repositories > > > > In case anyone is interested, openSUSE also has the same type of > stable kernel repo here: > > http://download.opensuse.org/repositories/Kernel:/stable/standard/ > > and the Ubuntu has it's mainline kernel ppa: > > http://kernel.ubuntu.com/~kernel-ppa/mainline/ > > Regards, > Marcus. > -- > Marcus Furlong >

On Thu, 30 Jul 2015 11:47:34 AM Peter Ross wrote: In my bitter experience RHEL has a nasty habit of breaking things on their "minor" point releases. I've seen rsync over ssh broken (no idea how they missed that in testing), at least 5 different Mellanox Infiniband & 10gigE kernel bugs introduced (we had to use RHEL 6.2 kernels on one of our systems until RHEL 6.6 finaly cleared things up and the RHEL5 bug that delivered incoming packets to the wrong interface took about a year to fix by which stage we'd reinstalled with RHEL6) and I suspect there are a few others I've forgotten... The best thing I can say about RHEL is that at least it's not SLES. cheers, Chris -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC

On Thu, 16 Jul 2015 06:08:26 PM Andrew McGlashan wrote: BTRFS doesn't inherently rely on kernel versions (as opposed to things that require binary kernel modules). It is however a project that's in active development with new features being added. If you need a new feature (such as RAID-5/6) then you need a kernel new enough to support it. If you make a filesystem with support for recent features then it can't be mounted on an older kernel (EG a filesystem made with mkfs.btrfs from Jessie can't be mounted with the Wheezy kernel). BTRFS isn't alone in this regard, we had the same issues when Ext3 and Ext4 were released. The difference is that it was always possible to make a filesystem in the older format. I can run mkfs.ext3 on a Debian/Jessie system and mount the filesystem on fairly old kernels without problem. I presume that BTRFS will get similar backwards compatability at some future time. But at the moment the focus is on features. ZFS on Linux has DKMS modules which makes upgrading kernels painful and risky. But it does have better forwards and backwards compatability in other ways. On Thu, 16 Jul 2015 08:28:40 PM Brian May wrote: ZFS is a more mature product, but it's just a matter of time before BTRFS matches that. BTRFS supports more flexibility in devices, EG "RAID-1" arrays of varying numbers of devices of varying sizes. But ZFS has more features for high-end use such as L2ARC and ZIL. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/

Brian May &lt;brian(a)microcomaustralia.com.au&gt; wrote: I hope BTRFS is starting to get the kind of deployment that will enable reliability bugs to be found and fixed. XFS developers have always been very good at running regression tests. If I remember rightly, parts of their test suite were generic and there were plans to apply it to other file systems - exactly what is needed.

On Thu, 23 Jul 2015 08:56:45 AM Jason White wrote: The XFS test suite is being extended to cover bugs in BTRFS too. I think it's more of a generic filesystem test suite than an XFS specific thing nowadays, there was even talk of renaming it. This idea that BTRFS users should upgrade to the latest kernels is a bit silly. If you have a specific bug that is known to be fixed then upgrading it is the right thing to do. But don't expect that newer kernels will always be better. New kernels have new bugs and also expose new bugs by fixing old bugs, in particular performance improvements often expose bugs in older code. I have Debian/Wheezy systems that are running BTRFS without problems and I don't think there's any need for an upgrade. While Debian/Jessie is running a kernel that isn't going to have upstream support for backporting fixes to BTRFS it's also running quite well (surprisingly well really) and again I don't feel a great need to upgrade. My laptop will get kernel 4.0.0 next time I reboot it. This isn't because I expect an benefit from a new kernel (I don't) but because as a DD it's part of my job to try the new versions. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/

I enabled quota's on the default Jessie kernel and it all blew up, so at least that is broken. Disabling quotas was sufficient to put it right again. Running low on space (>10%) also really hurt performance badly (my limited experience with zfs is that it hangs badly when free space gets under about 20%). I upgraded to 4.x at about the same time as I resolved the disk space problem so I can't say if 4.x improves the low-space performance issues though. The problem I had is that I had mythtv storage in a subvolume, and the only control you have over mythtv is that you can tell it to leave a certain amount of GB free, and the maximum that can be is 200GB, so obviously that's a problem on all but the smallest installations. I ended up creating a 1TB file (with nocow enabled) and used ext4 on loopback as my mythtv store. Performance is probably badly impacted but I don't notice it. James

Perhaps. Definitely noticed it on a 26GB ZFS system. As free space got under about 20% there started to be odd problems that I couldn't put my finger on. As free space crept towards 10% it got to the point where deleting a snapshot took a whole day or just would never complete. Deleting files to bring the free space back towards 80% made all the problems go away. That was on a fairly old FreeBSD install though (8.4 maybe). Yes this is true. My two experiences are on my router with a single 60GB SSD (with probably 40GB actually allocated to the filesystem), and my server (2 x 1.5TB, 2 x 2TB). I don't remember what the poor performance threshold was on the former, but on the latter, 200GB of free space caused serious problems. NFS clients would see very frequent kernel messages about delays. As soon as I got back up to about 1TB free all the problems went away, although no measurements were really taken between 200GB and 1TB of free space. That sounds unreasonably fragile. Especially if you are unable to ever do a balance. My first testing of btrfs was on top of bcache, and performance was awesome. I went back to entirely rotating media for production though as I only had a single SSD at my disposal, didn't really need the extreme performance, and had other things to spend money on. Also at the time there were reports of incompatibilities between btrfs and bcache. I expect bcache would out-perform the hot-relocation project, for most workloads. For my server which does lots of streaming writes (mythtv) and lots of random io (other stuff), it would balance things nicely. This guy claims success with bcache + btrfs http://www.spinics.net/lists/linux-btrfs/msg42125.html and raises some interesting points (interesting to me, at least). Btw, when you say 5TB RAID1, what exactly do you mean? Is the 5TB referring to the raw disks or the usable redundant space? I'm never quite sure. James

I thought you needed to run it after adding an additional disk, so that you wouldn't get into a situation where the new disk had space, but all the other available disks were full so there was no second disk to write the copy to? I guess it depends on your performance vs capacity vs power requirements. For performance, you want as many spindles as possible (and preferably 2.5", all other things being equal), for power greenness, you want as few as possible (and again, preferably 2.5"), and for capacity you want a number of disks where $/GB is the best to give you the capacity you want in the number of drive bays you have. Maybe the spindle count isn't as important if you use bcache though, assuming typical access patterns. 3TB still appears to be the best $/GB right now, but obviously that's not the only factor. But if I didn't have a use for 5TB in the next few years, I'd be going with the cheaper disks and buying more of them to fill all my bays (With BTRFS, I just get whatever disks I can get my hands on when I need them, or when surplus disks come my way, and cram them into my server. BTRFS knows what to do with them :) Has anyone ever put the numbers together for the optimal buying strategy? Eg I need x storage now, and my projected storage growth is yTB/year, should I buy the bigger (more expensive) disks now that will last me longer, or smaller (cheaper) disks now and bigger disks later as I need them and when they are cheaper? James

On Fri, 24 Jul 2015 at 21:36 Russell Coker &lt;russell(a)coker.com.au&gt; wrote: That isn't what Chris Mason says: https://youtu.be/W3QRWUfBua8?t=1586 Do you claim to know more then Chris Mason?

On Sun, 26 Jul 2015 12:39:32 AM Brian May wrote: They are still problematic for some people. For instance there is an unfortunate interaction between quotas and snapshots, there was someone having issues on btrfs list recently where the btrfs-cleaner kernel thread would consume a whole core whilst quotas were enabled: http://www.spinics.net/lists/linux-btrfs/msg45652.html # I can confirm that getting rid of the quotas fixed the issue for me. # Just disabling quotas wasn't enough, I had to enable, delete all # qgroups, reboot because disable was hung on one of the filesystems, # then disable quotas. Now when btrfs-cleaner runs it doesn't # completely consume a core, I can see corresponding disk i/o, and the # process goes away after a reasonable amount of time. To give some context this person had 92 subvolumes, including snapshots. All the best, Chris -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC

On Fri, Jul 24, 2015 at 09:35:51PM +1000, Russell Coker wrote: i've used zfs extensively since at least 2011 and i've run into this several times. When the pool gets over about 80% full, performance turns to shit...really, awfully, abysmally bad for both reads and writes. in fact, i ran into it again this week with my backup pool here at home. i'd foolishly allowed it to get to 87% full and performance was abysmal, on the order of kilobytes/second rather than 200+MB/s. i had to replace the 4x1TB drives (in RAIDZ1 configuration) in my backup pool with 4x4TB (configured as two mirrored pairs). i've found that once the pool gets over 80%, it gets slower the more it's used, even fairly light usage over a few hours will make it slow to a crawl...and you can forget about rsync to or from a fs > 80% full (oddly, a zfs send to another pool runs reasonably fast), and daily cron jobs like updating the mlocate db tend to just hang before completion. the only solution is to add more vdevs to the pool, replace all the disks in one or more vdevs with larger disks, or create a new pool with larger/more disks and replicate to it with 'zfs send -R' btrfs probably handles this slightly better because it's easier to add a single disk or two to increase capacity...and, of course, you can rebalance to redistribute your data evenly over all disks in the pool. btw, 'zfs send' is so aweseome that i'm seriously considering converting all my systems here to having root on zfs - i backup some filesystems with zfs send and some with rsync, and backing up a system with zfs send is many orders of magnitude faster than rsync because zfs knows *exactly* which blocks have changed between any two snapshots without having to stat or compare any files on source or destination. same goes for btrfs too (in fact, from what i've read btrfs has more flexible snapshot handling for send/receive) but since my backup pool is zfs, btrfs isn't really an option for me. i could convert to btrfs but i can't afford the extra disks or the time or hassle it would take - rebalancing and resizing are tempting features but not tempting enough. anyway: IMO, snapshots, error detection & correction, and send/receive capabilty are more than enough reasons to use either btrfs or zfs. if you're not already using one of them, you should seriously consider switching. actually, i was wrong earlier when i said there were only three options to fixing a fs over 80% full. You can also delete files to get it back well below 80% again. i had to do this on my mythtv zpool because it got to about 85% full with recordings i either hadn't got around to watching or had little intention of ever watching again. i deleted enough to get it below 60% and performance went back to normal. i've since done a ruthless purge and got it down to 35% full. note: it will be painfully slow if you try to cp or rsync the files to somewhere else before deleting them. zfs send is still fast (and if you send to another pool on the same system you can set the mountpoint of the destination fs so that it mounts in the same place as the source fs used to be). the downside is that send only sends snapshots of entire filesystems, you can't pick and choose which files to move....so, useless for my myth situation but useful for a pool with multiple filesystems on it. and using all but 4% of the pool would mean using 96% of it. the free space issue on zfs is not a "rule of thumb" but a well known fact about it - getting over 80% full is really bad for performance, and just about every tuning or setup guide will mention it. i don't know btrfs as well but it wouldn't surprise me if the <10% free issue that was mentioned is also a hard rule (as in "don't do it") my understanding is that at >80%, zfs changes the algorithm it uses to allocate space to files from "best fit" to "wherever there's some space". this tends to cause massive fragmentation (even more than is common on COW filesystems) with zfs, you'd just do: zfs set secondarycache=metadata <fs> that tells it to use the L2ARC (e.g. ssd or other fast block device cache) to only cache metadata. this can be set on a per-filesystem ("subvolume" in btrfs terms) basis. you can also set primarycache (i.e. ARC in RAM) to the same values - all, none, or metadata with the default being all (actually, inherit from parent with the ultimate parent's default being all) . zfs doesn't do rebalancing, or rseizing, (unfortunately - they're the key features that btrfs has that zfs doesn't) but if it did you wouldn't have to avoid using them so that a kludge like that keeps working. clever tricks can be cool but designed reliable features are better. craig -- craig sanders &lt;cas(a)taz.net.au&gt;