I need to move a block device (Xen VM backing store) between sites with a minimum of downtime... my choices so far are: . dd - straightforward and foolproof but will take a while . dd + lvmsync (https://github.com/mpalmer/lvmsync) - untested but sounds plausible . drbd with external metadata - I already use drbd but haven't tried such a thing before I think drbd is going to be the best way to go, especially as it's part of Linux these days. My plan is: 1. Stop the VM 2. create the drbd metadata in a separate lv and initialise it all (without overwriting initial data!) 3. Start the VM 4. create the backing lv and metadata lv at the other site and configure drbd 5. Pick a low-usage time and kick it off then wait for completion 6. Stop the VM at the origin site 7. tear down DRBD 8. Start the VM at the new site Both sites are linked by openvswitch so they are effectively the same lan. Bandwidth is 20Mbits/second so it won't be lightning fast but the VM's are typically only 50GB in size which equates to around 7 hours + overheads. Any comments? I'm about to test with a small and unimportant VM. Thanks James
This may be obvious but while your drbd is in place disk writes will be quite slow. Perhaps there is a way to copy the bulk of the data to the remote site to try and speed up the drbd sync and minimise interruption to the running guest? Good luck!
James Harper <james.harper@bendigoit.com.au> writes:
I need to move a block device (Xen VM backing store) between sites with a minimum of downtime... my choices so far are: [dd/drbd...] Both sites are linked by openvswitch so they are effectively the same lan. Bandwidth is 20Mbits/second so it won't be lightning fast but the VM's are typically only 50GB in size which equates to around 7 hours + overheads.
drbd is probably reasonable if you're intending to leave it around -- for a one-shot move, I probably wouldn't bother (unless I was already a drbd guru). What I probably WOULD try is to lvm snapshot a day before, and sync that. It will be incomplete and incoherent, but you don't care because on the day you rsync --only-write-batch against the snapshot and then upload only the diff and apply it. Since only the changed blocks from the last 24h have changed, that ought to reduce the downtime. It means you'll need to leave the snapshot around beforehand, of course. It it goes tits-up, fall back to bare dd plan. Or, of course, you can do it at the file level instead of the filesystem level, in which case you really *can* just use rsync (modulo usual stuff like bootloader and RDBMSs). From context I'm assuming the guest is windows or something equally horrible from that perspective. end brain dump - sorry about the incoherency
On Tue, 26 Feb 2013, James Harper <james.harper@bendigoit.com.au> wrote:
I think drbd is going to be the best way to go, especially as it's part of Linux these days. My plan is:
Last time I tried DRBD it was killing my systems. It seems that the default configuration for DRBD is to reboot a node if certain failure conditions occur - which can be triggered by network problems. I never managed to get it to stop doing that. In the default mode of operation DRBD writes everything synchronously to the secondary system, if the link between the systems is slow and your primary system is doing synchronous writes (database server or mail server) then it's going to suck. DRBD supports an asynchronous mode which due to bugs was slower than the synchronous mode in my tests with a local GigE network, maybe it's asynchronous support wouldn't look as bad with a slow network. http://etbe.coker.com.au/2012/02/08/more-drbd-performance-tests/ Also in my tests DRBD when the secondary isn't connected gave performance that's suspiciously similar to the performance of a non-DRBD system with Ext4 mounted with the barrier=0 option. Presumably this means that data on a DRBD system will be at the same risk as with barrier=0 on a power failure. On Tue, 26 Feb 2013, "Trent W. Buck" <trentbuck@gmail.com> wrote:
What I probably WOULD try is to lvm snapshot a day before, and sync that. It will be incomplete and incoherent, but you don't care because on the day you rsync --only-write-batch against the snapshot and then upload only the diff and apply it. Since only the changed blocks from the last 24h have changed, that ought to reduce the downtime.
You mean running rsync on a block device? Is that even possible? -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
On Tue, 26 Feb 2013, James Harper <james.harper@bendigoit.com.au> wrote:
I think drbd is going to be the best way to go, especially as it's part of Linux these days. My plan is:
Last time I tried DRBD it was killing my systems. It seems that the default configuration for DRBD is to reboot a node if certain failure conditions occur - which can be triggered by network problems. I never managed to get it to stop doing that.
There was a debian bug against those default options for exactly the reasons you noted, unless it was an actual code bug and not a default configuration bug that you ran in to?
In the default mode of operation DRBD writes everything synchronously to the secondary system, if the link between the systems is slow and your primary system is doing synchronous writes (database server or mail server) then it's going to suck. DRBD supports an asynchronous mode which due to bugs was slower than the synchronous mode in my tests with a local GigE network, maybe it's asynchronous support wouldn't look as bad with a slow network.
http://etbe.coker.com.au/2012/02/08/more-drbd-performance-tests/
Also in my tests DRBD when the secondary isn't connected gave performance that's suspiciously similar to the performance of a non-DRBD system with Ext4 mounted with the barrier=0 option. Presumably this means that data on a DRBD system will be at the same risk as with barrier=0 on a power failure.
I'm looking at either using bcache or moving the metadata to an ssd to try and avoid these performance problems. drbd barrier and flush are configurable, and there are lots of warnings about turning them off.
On Tue, 26 Feb 2013, "Trent W. Buck" <trentbuck@gmail.com> wrote:
What I probably WOULD try is to lvm snapshot a day before, and sync that. It will be incomplete and incoherent, but you don't care because on the day you rsync --only-write-batch against the snapshot and then upload only the diff and apply it. Since only the changed blocks from the last 24h have changed, that ought to reduce the downtime.
You mean running rsync on a block device? Is that even possible?
rsync can do block devices with a few patches here and there. lvmsync (https://github.com/mpalmer/lvmsync if you missed it the first time) appears to be a much better option than this if you are actually using lvm - the procedure is: . take a snapshot . dd the snapshot to the destination, at your leisure . take the vm offline . use lvmsync to copy the changes lvmsync looks at the snapshot, figures out what extents the snapshot holds (which are by definition in the snapshot because they have changed in the original) and copies the matching original extents to the destination. I tested drbd last night and the performance of the vm dropped to the point where it may as well have been offline, so it looks like I'll be doing my next test with lvmsync. drbd have a ($$$) proxy option that would make this much better in that it allows a fairly huge buffer to build up in the case where there is congestion between the primary and secondary, meaning things don't slow down to the speed of the link. The value of $$$ isn't particularly small though, especially for a one-off migration. James
On Wed, 27 Feb 2013, James Harper <james.harper@bendigoit.com.au> wrote:
Last time I tried DRBD it was killing my systems. It seems that the default configuration for DRBD is to reboot a node if certain failure conditions occur - which can be triggered by network problems. I never managed to get it to stop doing that.
There was a debian bug against those default options for exactly the reasons you noted, unless it was an actual code bug and not a default configuration bug that you ran in to?
It was an issue of the default configuration that was hard coded into the utilities or the kernel (can't remember which).
http://etbe.coker.com.au/2012/02/08/more-drbd-performance-tests/
Also in my tests DRBD when the secondary isn't connected gave performance that's suspiciously similar to the performance of a non-DRBD system with Ext4 mounted with the barrier=0 option. Presumably this means that data on a DRBD system will be at the same risk as with barrier=0 on a power failure.
I'm looking at either using bcache or moving the metadata to an ssd to try and avoid these performance problems.
drbd barrier and flush are configurable, and there are lots of warnings about turning them off.
If you read my above blog post you'll notice that default Ext4 performance was 1663, Ext4 barrier=0 was 2875, and a default configuration of DRBD with the secondary disconnected was 2409. I conclude that the data-loss protection that barriers offer resulted in a performance drop from 2875 to 1663 and that the use of DRBD by restoring the performance to 2409 reduced that protectio0n. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
James Harper <james.harper@bendigoit.com.au> writes:
rsync can do block devices with a few patches here and there. lvmsync (https://github.com/mpalmer/lvmsync if you missed it the first time) appears to be a much better option than this if you are actually using lvm - the procedure is:
Ah, apologies. I saw the mention in the OP, but I assumed it was from <idiot who never heard of rsync>, not <clever person who tried rsync, ran into problems, and fixed them>. A glance at the README shows I was wrong.
participants (5)
-
Andrew Spiers -
James Harper -
Piers Rowan -
Russell Coker -
trentbuck＠gmail.com