>
> James Harper <james.harper(a)bendigoit.com.au> wrote:
>
> > CEPH does sound exciting. Is anyone here doing it with Xen?
>
> It was still under heavy development when last I read about it. The kernel
> module entered the mainline and, I assume, has undergone further work
> since
> then.
>
> File systems take a long time to mature. Testing and bug finding efforts
> always help, however.
>
Full of enthusiasm, I've just created 3 xen VM's and created a 3 node CEPH cluster inside. It's amazingly easy to set up, but from what I can see some of the init tools automate a lot of the steps that I assume need to be done manually for ongoing maintenance, so I may go back and set it up the long way around to get a feel for the steps involved, or maybe just add a few more nodes.
For a production CEPH installation, I think I would still use RAID1 underneath and then deploy CEPH on top, as it lowers the risk of a node-down situation, at the expense of requiring additional disks.
Debian appears to have some OCF RA's which I haven't investigated yet, as I'd still want to use pacemaker for managing the actual VM's. For a 5 node cluster with 2 copies of all xen virtual disks I assume it becomes important to run VM's on a node that has a local replica of the data... or is the data striped across all nodes transparently in a "you don't need to know" way? I'd better do some more reading I guess :)
James
Brian May wrote:
> > Dropbox and Gmail to share the files - not what we want. Among other
> > things
> > we'd like to have the files stay within Australian jurisdiction.
>
>
>If the files were securely encrypted, does it matter that they stay in
>Australia?
IANAL, but my understanding is a definite "Yes" to this.
National Privacy Principle 9 applies even to encrypted data, because
there's no hard and fast definition of what "encryption" means.
NPP #4 requires us to take "reasonable steps" to secure information, and
encryption is certainly one of those steps, for what the IT profession
generally considers reasonable.
For example, if I simply password protected an MS Word 97 document, there's
no way in this day and age I could defend that as being "reasonable"
encryption.
I would like to get a file tree representing a specific (or latest)
tag/branch/label from a repository. Trawling the help files shows me
how to manipulate the repository itself, but not to extract the subset
that represents just the code.
I have done the brute force recursive removal of the .git (or .svn) dirs
to get the latest set of files, but there must be a better way.
I originally sent this direct to James resending it to the list
> On Fri, Apr 12, 2013 at 3:17 PM, James Harper <
james.harper(a)bendigoit.com.au> wrote:
>
> > > . Online resize/reconfigure
> >
> > both btrfs and zfs offer this.
> >
>
> Can it seamlessly continue over reboot? Obviously it can't progress while
the system is rebooting like a hardware raid but I'd hope it could pick up
where it left of automatically.
Yes it does.
> This is where a lot of people get this wrong. Once the BIOS has succeeded
in reading the bootsector from a boot disk it's committed. If the
bootsector reads okay (even after a long time on a failing disk) but
anything between the bootsector and the OS fails, your boot has failed.
This 'anything between' includes the grub bootstrap, xen hypervisor, linux
kernel, and initramfs, so it's a substantial amount of data to read from a
disk that may be on its last legs. A good hardware RAID will have long
since failed the disk by this point and booting will succeed.
>
> My last remaining reservation on going ahead with some testing is is
there an equivalent of clvm for zfs? Or is that the right approach for zfs?
My main server cluster is:
>
> 2 machines each running 2 x 2TB disks with DRBD with the primary
exporting the whole disk as an iSCSI volume
> 2 machines each importing the iSCSI volume running lvm (clvm) on top, and
using the lv's as backing stores for xen VM's.
>
> How would this best be done using zfs?
If i was building new infrastructure today with 2 or more machines hosting
VMs i would probably look at using CEPH as the storage layer for the
Virtual machines. this would provide distributed mirrored storage that is
accessible from all machines.
all machines could then be storage and VM hosts.
ref:
http://www.slideshare.net/xen_com_mgr/block-storage-for-vms-with-cephhttp://ceph.com/
On 2013-04-09 02:40, James Harper wrote:
> I have a server that had 4 x 1.5TB disks installed in a RAID5
> configuration (except /boot is a 'RAID1' across all 4 disks). One of
> the disks failed recently and so was replaced with a 3TB disk,
I'd be very wary of running RAID5 on disks >2TB
Remember that, when you have a disk failure, in order to rebuild the array,
it needs to scan every sector of every remaining disk, then write to every
sector of the replacement disk.
Compare those number of read & write operations to the Mean Time Between
Failures of each disk, and you're (statistically) starting to get close to
the point there's a significant risk of a second drive failing prior to the
rebuild finishing. (For a given definition of "significant" risk)
http://www.hetzner.de/en/hosting/produkte_rootserver/poweredge-r720
The above server from Hetzner is an interesting offer. A system with 128G of
RAM for only E299 per month. I'm not aware of any company making a similar
offer at this time (if you know of one then let me know, some of my clients
will be interested).
http://www.hetzner.de/en/hosting/produkte_rootserver/ex4
The problem is storage, E25 per month per disk is a lot of money. They offer
"ex4" servers for E49 per month that have 2*3TB disks which saves E1 per month
and also gives 16G of cache per node.
What would be a good distributed filesystem to run across 4*EX4 nodes with one
R720 as the main server? Would ZFS using RAID-Z2 over iSCSI be a viable
option?
I don't have a real use for such things at this time. But sometimes when you
tell clients about options for cheaply doing things that used to be expensive
then they go for it.
--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/
>
> as you suggest bcache and flashcache seem to offer a way around this for
> mdadm but i've never used either of them - i was already using zfs by
> the time they became available. i don't think the SATA interface speed
> is a deal-breaker for them because the only way around that is spending
> huge amounts of money.
>
There were some not-too-expensive battery backed PCI ramdisks available a while ago. Not anymore though.
>
> > . Battery backed write cache. Bcache/flashcache offer this but they
> > have their shortcomings, in particular that most available cache
> > modules are still on top of the SATA channel.
>
> this is the only real advantage of hardware raid over mdadm. IMO, ZFS's
> ability to use an SSD or other fast block device as cache completely
> eliminates this last remaining superiority of hardware raid over
> software raid.
Yes I've now been enlightened on this subject :)
> > . Online resize/reconfigure
>
> both btrfs and zfs offer this.
>
Can it seamlessly continue over reboot? Obviously it can't progress while the system is rebooting like a hardware raid but I'd hope it could pick up where it left of automatically.
> > . BIOS boot support (see recent thread "RAID, again" by me)
>
> this is a misfeature of a crappy BIOS rather than a fault with software
> raid.
>
> any decent BIOS not only has the ability to choose which disk to boot
> from (rather than hard-code it to only boot from whichever disk is
> plugged into the first disk port) but will also let you specify a boot
> order so that it will try disk 1 followed by disk 2 and then disk 3 or
> whatever. they'll also typically let you press F2 or F12 or whatever at
> boot time to pop up a boot device selection menu.
>
> even server motherboards like supermicro let you choose the boot device
> and have a boot menu option accessible over IPMI.
>
This is where a lot of people get this wrong. Once the BIOS has succeeded in reading the bootsector from a boot disk it's committed. If the bootsector reads okay (even after a long time on a failing disk) but anything between the bootsector and the OS fails, your boot has failed. This 'anything between' includes the grub bootstrap, xen hypervisor, linux kernel, and initramfs, so it's a substantial amount of data to read from a disk that may be on its last legs. A good hardware RAID will have long since failed the disk by this point and booting will succeed.
My last remaining reservation on going ahead with some testing is is there an equivalent of clvm for zfs? Or is that the right approach for zfs? My main server cluster is:
2 machines each running 2 x 2TB disks with DRBD with the primary exporting the whole disk as an iSCSI volume
2 machines each importing the iSCSI volume running lvm (clvm) on top, and using the lv's as backing stores for xen VM's.
How would this best be done using zfs?
Thanks
James
Quoting "Greg Bromage" <greg(a)bromage.org>
> Craig Sanders wrote:
>> one error in 10^14 bits is nothing to worry about with 500GB drives.
>> it's starting to get worrisome with 1 and 2TB drives. It's a guaranteed
>> error with 10+TB arrays....and even a single 3 or 4TB drive has roughly
>> a 30-50% chance of having at least one data error.
>
> *nod* that's the root of my concern. And the "at least one" is the issue,
> because once a disk fails, if there's a second error somewhere then it WILL
> be encountered, because the rebuild has to traverse every sector of every
> remaining disk.
Out of curiosity, I googled for the 10^14 bits.
http://www.zdnet.com/blog/storage/how-data-gets-lost/167
Cause of data loss Perception Reality
Hardware or system problem 78% 56%
Human error 11% 26%
Software corruption or problem 7% 9%
Computer viruses 2% 4%
Disaster 1-2% 1-2%
I find the 26% human error interesting.
IMHO a lot of IT people underestimated quite often the impact of their work and
the errors they make.
Sometimes a simple solution may look worse on paper but it may save the day.
Because it is easy to implement and harder to break;-)
For the 10^14 Bits.. that was five years ago, and I am not sure whether still
valid (a Terbyte was a lot in 2007).
I have overnight synchronisation of ca. 3 TeraByte per night (deltas and
some full-copies after maintenance), all from mirrored disks, and haven't
seen any errors over the last 18 months. ZFS scrubs did not find errors as
well.
Regards
Peter
Craig Sanders wrote:
> one error in 10^14 bits is nothing to worry about with 500GB drives.
> it's starting to get worrisome with 1 and 2TB drives. It's a guaranteed
> error with 10+TB arrays....and even a single 3 or 4TB drive has roughly
> a 30-50% chance of having at least one data error.
*nod* that's the root of my concern. And the "at least one" is the issue,
because once a disk fails, if there's a second error somewhere then it WILL
be encountered, because the rebuild has to traverse every sector of every
remaining disk.
Of course, that might not be enough to cause the rebuild (and thus the
array) to fail. And there's probably another set of probabilities to factor
in there too.
But I can guarantee you a probability of 1.0 (with no deviation) that it
will happen on the array that holds the critical data you need to recover
that your boss accidentally forgot to back up... :)
From: "James Harper" <james.harper(a)bendigoit.com.au>
Sent: Thursday, 11 April, 2013 12:10:37 PM
> Putting the error correction/detection in the filesystem bothers me.
> Actually when I was checking over this email before hitting send it
> occurred to me that maybe I'm wrong about this, knowing next to
> nothing about ZFS as I do. Is a zpool virtual device like an LVM lv,
> and I can use it for things other than running ZFS filesystems on?
I have seen examples of (FreeBSD) UFS partitions on top of Zpools.
http://www.markround.com/archives/37-ZFS-as-a-volume-manager.html
I briefly considered it because of Samba 4 and ACL issues.
ZFS (on FreeBSD, and Solaris, I think) supports NFSv4 ACLs, while UFS
has POSIX ACLs (as Linux ext3/4 has).
NFSv4 ACLs are close to Windows CIFS ACLs (much closer than Posix ACLs).
Still, Samba is using Posix ACLs, and makes it harder for ZFS - on
FreeBSD at least.
The FreeBSD samba 4.0.4 port considers ZFS as underlying filesystem as
untested and it needs some tweaks to get it working.
The reason to use Posix ALs, I believe, is lacking Linux support for
NFSv4 ACLs. There are RichACLs but they are not in the main kernel yet
- OpenSuse has them in their kernel already.
A question to ZFSonLinux - which ACLs are supported here?
Regards
Peter
