Hi Russell, I have a similar problem on FreeBSD where I am using ZFS. Most of the instances are running on jail, the concept similar as the Linux containers. Jails are integrated part of the FreeBSD architecture for years so I trust it security-wise. BTW: "Version 2" (VIMAGE, VNET) is similar as the Crossbar architecture found on Solaris (every jail has its own network stack) Linux containers are not that new either, AFAIK there are webhosting providers offering VPS based on it. It just feels more like an "add-on".. You may use your SE Linux wizardry to increase security if you don't trust it enough. And I have a (commercially licensed) Zimbra server which needs Linux. I am running it in Virtualbox. The disks are files on a "normal" zfs so they profit from snapshoting, zfs send/receive mechanism for off-site backup etc. Inside it is Ubuntu on ext3. It is the mail server for 50 users (and probably 10% of your mentioned mail storage) , and works without problems, as long as the zpool is not short of space (I think I mentioned here the "stand-still" if a 1 TB zpool is going below 50 GB free space). But I don't think it is a really good setup, it is just "good enough" here, and as I have all other stuff running in jails native on FreeBSD, I keep it. (FreeBSD is offering the Linux kernel ABI [with few limitations] so one day I might try to run Ubuntu on it, in a jail). I don't think you win much if you use NFS over ZFS instead. You may increase performance if you use "raw zpool" underneath but then you don't have the "cool stuff" (snapshots, cloning etc.) that wants you to use ZFS in the first place. I could imagine using LVM on Dom0 and giving partitions to the DomUs and running ZFS inside. That way you can snapshot the partitions with LVM outside (to get "disk images") and ZFS management inside. Regards Peter On Mon, 15 Oct 2012, Russell Coker wrote:
I'm looking at converting some Xen servers to ZFS. This includes a couple of servers for a reasonable size mail store (8,000,000 files and 600G of Maildir storage).
For much of the Xen on ZFS stuff I'll just use zvols for block devices and then use regular Linux filesystems such as Ext3 inside them. This isn't particularly efficient but for most DomUs it doesn't matter at all. Most of the DomUs have little disk access as they don't do much writing and have enough cache to cover most reads.
For the mail spool a zvol would be a bad idea, fsck on a 400G Ext3/4 filesystem is a bad thing and having the double filesystem overhead of Ext3/4 on top of a zvol is going to suck for the most disk intensive filesystem.
So it seems that the correct solution is to do one of the following:
1) Run the mail store in the Dom0 which will be good for performance at the cost of management. A server which has direct user access in any form (including POP and IMAP) needs to be running all the latest security patches while a Dom0 can have patches delayed if they don't seem relevant to network issues or virtualisation.
2) Use NFS to mount a ZFS filesystem from the Dom0. This will be good for management but there's the problem of caching. I don't think that NFS caches that aggressively so I'd need to give more RAM to the Dom0 for ZFS caching and I'd probably still lose some read performance.
3) Run ZFS in the DomU separately from the Dom0. This will work well for the DomU as long as there is enough RAM. But having a ZFS filesystem in the Dom0 as well as a separate one in the DomU (which would use different partitions of the same disks) would be difficult (ZFS in the Dom0 will probably want to grab all zpools). Also write performance will take a hit if there are two separate zpools on the same disks as there will be seeks between writes - this will be particularly bad for mail delivery where the message and the log entry will be written to different parts of the disk.
4) Run ZFS for mail storage in the DomU and use something other than ZFS for the Dom0. This has the same performance problems as 3) but without the issue of different ZFS instances fighting about it. Also I would lose the support for hashes on the zvol data, I could use files on BTRFS for similar data integrity (I'm using RAID-1 so RAID-Z isn't an option and therefore the benefits of ZFS over BTRFS are fewer) but that would still give performance issues.
Any suggestions?
-- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/ _______________________________________________ luv-main mailing list luv-main@luv.asn.au http://lists.luv.asn.au/listinfo/luv-main