On Sat, 25 May 2013, Craig Sanders <cas@taz.net.au> wrote:
Except that if you change all your VM's from 4G to 8G it means you can now run half as many... most of my VM's are IO and memory bound.
i thought Russell was talking about RAM on the zfs server, not on the VM?
The Xen server that I run on ZFS now has a lot of RAM spare in the Dom0 to avoid ZFS problems. It's fortunate that RAM is cheap, I was able to arrange to have more purchased than might otherwise be needed, and that the system doesn't need that much RAM for DomUs. But it is a bit of an annoyance. On Sat, 25 May 2013, Craig Sanders <cas@taz.net.au> wrote:
On Sat, May 25, 2013 at 08:12:31PM +1000, Russell Coker wrote:
options zfs zfs_arc_max=536870912
I just checked the system in question, it still has the above in the modules configuration from my last tests. "free" reports that 9G of RAM is used as cache so things seem to be getting cached anyway.
yeah, but it'll be caching everything *except* for ZFS. zfsonlinux doesn't use linux's own caching, it uses ARC and L2ARC. it's not additional to linux's caching, it's instead of.
The system in question has 6.2G of files on the root Ext4 filesystem, a lot of storage mounted as NFS from a NAS, and a ZFS pool. I believe that NFS doesn't get cached for long so the fairly small amount of use of the NFS mount makes it seem that a good portion of the now 10.5G of RAM that is reported as "cached" by free would have to be used by ZFS. There's nothing else for it to be used for.
The system in question doesn't have serious load (it's used as a box I can ssh to to test other systems and for occasional OpenVPN use) and it could be that it gives less performance because of BTRFS. But the fact that it works at all sets it apart from ZFS.
yeah, well, execpt in special circumstances (like a file-server where i'm going to be throwing lots of RAM at the VM anyway), i wouldn't use ZFS as a VM's filesystem. i'd use ext4 or xfs on a zvol.
I'd use NFS root if I wasn't running SE Linux on the DomUs.
Also note that dpkg calls sync() a lot and thus gives poor performance when installing packages on BTRFS. As an aside, I'm proud to have filed the bug report against dpkg which led to this.
dpkg's sync() fetish gives pretty poor performance on most filesystems.
for an exciting time:
# apt-get install eatmydata # ln -s /usr/bin/eatmydata /usr/local/sbin/dpkg
(i used to do this on my home system with an xfs / before i got an SSD. dangerous, but *much* faster. YMMV but i never suffered any data loss or corruption of dpkg's status file etc from it - low risk of a problem but potentially catastrophic if power-failure/crash/whatever does occur)
dpkg was apparently working most of the time for almost everyone before I decided to reproduce a bug I found in the SLES version of RPM and file a bug report about it. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/