Performance surprise: partitions - raw vs. on top of ZFS
Hi all, I am doing some research about "the best way" to give a partition to Oracle. I found this surprising and you may too, it is quite amusing: http://jrs-s.net/2013/05/17/kvm-io-benchmarking/ But here’s the big surprise – if we set up a ZVOL, then format it with ext4 and put a .qcow2 on top of that… it performs as well, and in some cases better than, the raw zvol itself did! As odd as it sounds, this leaves qcow2-on-ext4-on-zvol as one of our best performing overall storage methods, with the most convenient options for management. It sounds like it’d be a horrible Rube Goldberg, but it performs like best-in-breed. Who’d’a thunk it? According to the benchmark you have this layering: - qcow2 (virtual disk) as a file on an - ext4 filesystem formatted on a - zvol created on a - raw partition performing better than a raw partition! But - zvol created on a - raw partition is "not worth the hassle". Any thoughts and experiences? I want to give a disk via iSCSI to Oracle (Avi knows more;-) and look for the best setup. The iSCSI target is FreeBSD which can export files as block devices or zvols or raw disks or mirrored disks or other exotic GEOM devices on top of raw disks. According to the benchmark above, a file exported as a blockdevice on top of a zvol could be better than an exported disk.. Could this be true? Regards Peter
Hi, On 5 Sep 2014, at 2:28 pm, Peter Ross <Petros.Listig@fdrive.com.au> wrote:
I am doing some research about "the best way" to give a partition to Oracle.
You want to provide an entire disk to Oracle Automatic Storage Management (ASM) and let it do all the heavy lifting for you. :) Whether this is DAS, SAN or iSCSI provided LUN.
Who’d’a thunk it?
Partitions do sector alignment which filesystems do not. Thus, this is not surprising given that most VM systems (and Oracle RDBMS) do DIRECT_IO to the underlying filesystem.
According to the benchmark above, a file exported as a blockdevice on top of a zvol could be better than an exported disk..
I doubt it would be for Oracle. You might want to check out ORION[1] or SLOB[2] as testing tools that reproduce Oracle-style disk I/O and report performance. You could create an iSCSI LUN from each of the options and do a comparison test to see which one performs best. Note that any statically assigned backing store should always outperform a dynamically sized store. Cheers, Avi [1] http://docs.oracle.com/cd/E11882_01/server.112/e16638/iodesign.htm#PFGRF9522... [2] http://kevinclosson.net/slob/
On Fri, Sep 05, 2014 at 02:33:11PM +1000, Avi Miller wrote:
You want to provide an entire disk to Oracle Automatic Storage Management (ASM) and let it do all the heavy lifting for you. :) Whether this is DAS, SAN or iSCSI provided LUN.
+1 I couldn't agree with this more, this becomes even more apparent when you start doing things like multipathed I/O
On 5 September 2014 14:28, Peter Ross <Petros.Listig@fdrive.com.au> wrote:
But here’s the big surprise – if we set up a ZVOL, then format it with ext4 and put a .qcow2 on top of that… it performs as well, and in some cases better than, the raw zvol itself did! As odd as it sounds, this leaves qcow2-on-ext4-on-zvol as one of our best performing overall storage methods, with the most convenient options for management. It sounds like it’d be a horrible Rube Goldberg, but it performs like best-in-breed. Who’d’a thunk it?
According to the benchmark you have this layering: - qcow2 (virtual disk) as a file on an - ext4 filesystem formatted on a - zvol created on a - raw partition
performing better than a raw partition!
Have you checked whether important things like write barriers and synchronisation are being honoured all the way down the stack? I suspect you'll find they're not, and that is where the performance is coming from.. but will be interested to hear if you can confirm or deny that after investigation. -Toby
participants (4)
-
Avi Miller -
Joel W. Shea -
Peter Ross -
Toby Corkindale