On Fri, 2 Nov 2012, Tim Connors wrote:
On Fri, 2 Nov 2012, Peter Ross wrote:
"Generous memory usage" indeed. My zfs server with 4GB of ram crashes once a week on average with zfs arc backtraces or just livelocks with txg_sync taking 100% CPU usage (with no zfs access, which is bad for a machine with zfs mounted /var by necessity of /boot being a 128MB flash IDE card and / being a 8GB USB nanostick). Each -rc release says "we finally fixed that txg_sync and arc_reclaim issue", but alas it never comes good. My favourite is where sometimes mysteriously arc usage increases tens of times beyond arc_max before the machine grinds to a halt. Someone ballsed up the memory accounting!
(Just saying this just in case anyone thought zfs was preduction ready yet. Apparently the same issues sometimes affects bsd and indiana).
I am using ZFS under FreeBSD in production for ca. 18 months (and used it under OpenSolaris before but there is was more a proof of concept, and then came Oracle..) Amongst the machines are two with 3 or 4GB of RAM but they are doing "low-volume work" (DNS, DHCP, a wiki, a mailing list server, a syslog server etc) The others have 16 GB each, and ARC is limited to 8 GB (vfs.zfs.arc_max=8589934592) because a lot of stuff is more hungry for I/O than memory (proxy servers, file servers, database servers etc.) I haven't seen an issue. At the beginning I was bothering a FreeBSD mailing list with a "ZFS problem" (when I backed up virtual disks via ssh) but at the end it was netstack related, since then my loader.conf has "net.graph.maxdata=65536" in it. Well, it wasn't easy to spot, and it helped to solve older cases. ZFS adds a level of complexity, and I wouldn't mind to have that "outsourced" to the periphery so it does not interfere with the "main system" (imagine a RAID controller with a ZFS API to the main system, and an internal processor+memory to serve it)
I am surprised people find zfs send so useful though. My only use for it would be if I could get data of it onto another filesystem at close to wire speed :)
You will run another stuff as I do.. Anyway, 15 MBit/sec is enough for a high-volume document server system I maintained before, and many other cases. Someone has to produce 15MBit/sec (useful;-) data per second first.
I mean, I thought it was pretty routine back in days of yore to do an: lvm snapshot dd if=/dev/base/home-snap | ssh blah dd of=/dev/base/home
Ok, a couple more commands, but infinitely flexible.
Nothing wrong with that but I don't need the ropes around it anymore, if I take care not to fall in the potential traps (now known to me). Cheers Peter