I don't suppose anyone has ported arc_summary and zilstat to read the linux spl structures? http://constantin.glez.de/blog/2011/02/frequently-asked-questions-about-flas... I suspect I'd benefit greatly from l2arc in my usage (not so much zil). But I don't wanna go out and buy this yet, until I *know* it will help improve upon the 40iops I currently suffer from :) http://www.ozbargain.com.au/node/69269 -- Tim Connors
On Thu, 3 May 2012, Tim Connors wrote:
I don't suppose anyone has ported arc_summary and zilstat to read the linux spl structures?
http://constantin.glez.de/blog/2011/02/frequently-asked-questions-about-flas...
I suspect I'd benefit greatly from l2arc in my usage (not so much zil). But I don't wanna go out and buy this yet, until I *know* it will help improve upon the 40iops I currently suffer from :)
Under FreeBSD I have a lot of sysctls, see "sysctl -a | grep zfs" Below the parts from a running machine that may interest you. Can't you find similar sysctls or /proc/sys/whatever entries under zfsonlinux? I have to admit I don't look at them too much at the moment because the systems work without any problems, AFAIK (and more generic nagios baed monitoring tells me). Regards Peter kstat.zfs.misc.xuio_stats.onloan_read_buf: 0 kstat.zfs.misc.xuio_stats.onloan_write_buf: 0 kstat.zfs.misc.xuio_stats.read_buf_copied: 0 kstat.zfs.misc.xuio_stats.read_buf_nocopy: 0 kstat.zfs.misc.xuio_stats.write_buf_copied: 0 kstat.zfs.misc.xuio_stats.write_buf_nocopy: 11165 kstat.zfs.misc.zfetchstats.hits: 85378 kstat.zfs.misc.zfetchstats.misses: 185376 kstat.zfs.misc.zfetchstats.colinear_hits: 3 kstat.zfs.misc.zfetchstats.colinear_misses: 185373 kstat.zfs.misc.zfetchstats.stride_hits: 84498 kstat.zfs.misc.zfetchstats.stride_misses: 0 kstat.zfs.misc.zfetchstats.reclaim_successes: 27 kstat.zfs.misc.zfetchstats.reclaim_failures: 185346 kstat.zfs.misc.zfetchstats.streams_resets: 1 kstat.zfs.misc.zfetchstats.streams_noresets: 880 kstat.zfs.misc.zfetchstats.bogus_streams: 0 kstat.zfs.misc.arcstats.hits: 237195675 kstat.zfs.misc.arcstats.misses: 22729224 kstat.zfs.misc.arcstats.demand_data_hits: 118184178 kstat.zfs.misc.arcstats.demand_data_misses: 9842053 kstat.zfs.misc.arcstats.demand_metadata_hits: 117600831 kstat.zfs.misc.arcstats.demand_metadata_misses: 11956487 kstat.zfs.misc.arcstats.prefetch_data_hits: 177481 kstat.zfs.misc.arcstats.prefetch_data_misses: 384059 kstat.zfs.misc.arcstats.prefetch_metadata_hits: 1233185 kstat.zfs.misc.arcstats.prefetch_metadata_misses: 546625 kstat.zfs.misc.arcstats.mru_hits: 69647071 kstat.zfs.misc.arcstats.mru_ghost_hits: 1660561 kstat.zfs.misc.arcstats.mfu_hits: 166494142 kstat.zfs.misc.arcstats.mfu_ghost_hits: 3040839 kstat.zfs.misc.arcstats.allocated: 27937038 kstat.zfs.misc.arcstats.deleted: 20024653 kstat.zfs.misc.arcstats.stolen: 14838092 kstat.zfs.misc.arcstats.recycle_miss: 2495533 kstat.zfs.misc.arcstats.mutex_miss: 1226 kstat.zfs.misc.arcstats.evict_skip: 1997947 kstat.zfs.misc.arcstats.evict_l2_cached: 0 kstat.zfs.misc.arcstats.evict_l2_eligible: 1518224476672 kstat.zfs.misc.arcstats.evict_l2_ineligible: 45310906368 kstat.zfs.misc.arcstats.hash_elements: 200165 kstat.zfs.misc.arcstats.hash_elements_max: 299703 kstat.zfs.misc.arcstats.hash_collisions: 43153207 kstat.zfs.misc.arcstats.hash_chains: 49489 kstat.zfs.misc.arcstats.hash_chain_max: 13 kstat.zfs.misc.arcstats.p: 2280418432 kstat.zfs.misc.arcstats.c: 4000000000 kstat.zfs.misc.arcstats.c_min: 519421184 kstat.zfs.misc.arcstats.c_max: 4000000000 kstat.zfs.misc.arcstats.size: 3999886576 kstat.zfs.misc.arcstats.hdr_size: 48381456 kstat.zfs.misc.arcstats.data_size: 3527264768 kstat.zfs.misc.arcstats.other_size: 424240352 kstat.zfs.misc.arcstats.l2_hits: 0 kstat.zfs.misc.arcstats.l2_misses: 0 kstat.zfs.misc.arcstats.l2_feeds: 0 kstat.zfs.misc.arcstats.l2_rw_clash: 0 kstat.zfs.misc.arcstats.l2_read_bytes: 0 kstat.zfs.misc.arcstats.l2_write_bytes: 0 kstat.zfs.misc.arcstats.l2_writes_sent: 0 kstat.zfs.misc.arcstats.l2_writes_done: 0 kstat.zfs.misc.arcstats.l2_writes_error: 0 kstat.zfs.misc.arcstats.l2_writes_hdr_miss: 0 kstat.zfs.misc.arcstats.l2_evict_lock_retry: 0 kstat.zfs.misc.arcstats.l2_evict_reading: 0 kstat.zfs.misc.arcstats.l2_free_on_write: 0 kstat.zfs.misc.arcstats.l2_abort_lowmem: 0 kstat.zfs.misc.arcstats.l2_cksum_bad: 0 kstat.zfs.misc.arcstats.l2_io_error: 0 kstat.zfs.misc.arcstats.l2_size: 0 kstat.zfs.misc.arcstats.l2_hdr_size: 0 kstat.zfs.misc.arcstats.memory_throttle_count: 0 kstat.zfs.misc.arcstats.l2_write_trylock_fail: 0 kstat.zfs.misc.arcstats.l2_write_passed_headroom: 0 kstat.zfs.misc.arcstats.l2_write_spa_mismatch: 0 kstat.zfs.misc.arcstats.l2_write_in_l2: 0 kstat.zfs.misc.arcstats.l2_write_io_in_progress: 0 kstat.zfs.misc.arcstats.l2_write_not_cacheable: 880679 kstat.zfs.misc.arcstats.l2_write_full: 0 kstat.zfs.misc.arcstats.l2_write_buffer_iter: 0 kstat.zfs.misc.arcstats.l2_write_pios: 0 kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned: 0 kstat.zfs.misc.arcstats.l2_write_buffer_list_iter: 0 kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter: 0 kstat.zfs.misc.vdev_cache_stats.delegations: 0 kstat.zfs.misc.vdev_cache_stats.hits: 0 kstat.zfs.misc.vdev_cache_stats.misses: 0
On Fri, 4 May 2012, Peter Ross wrote:
On Thu, 3 May 2012, Tim Connors wrote:
I don't suppose anyone has ported arc_summary and zilstat to read the linux spl structures?
http://constantin.glez.de/blog/2011/02/frequently-asked-questions-about-flas...
I suspect I'd benefit greatly from l2arc in my usage (not so much zil). But I don't wanna go out and buy this yet, until I *know* it will help improve upon the 40iops I currently suffer from :)
Under FreeBSD I have a lot of sysctls, see "sysctl -a | grep zfs"
Below the parts from a running machine that may interest you.
Can't you find similar sysctls or /proc/sys/whatever entries under zfsonlinux?
I have to admit I don't look at them too much at the moment because the systems work without any problems, AFAIK (and more generic nagios baed monitoring tells me).
There is /proc/spl/kstat/zfs and /proc/spl/kmem/slab but they are missing a lot of the entries that various scripts require (eg, various flavours of nagios check_zfs scripts, the scripts above, etc). A lot of entries in common with just some simple translation, but enough missing that they don't work. Specifically, to work out whether I will benefit from a large L2ARC cache, I'm probably going to want to interpret: # cat arcstats | grep ghost mru_ghost_hits 4 6473894 mfu_ghost_hits 4 27021711 mru_ghost_size 4 130259968 mru_ghost_evict_data 4 0 mru_ghost_evict_metadata 4 130259968 mfu_ghost_size 4 130259968 mfu_ghost_evict_data 4 0 mfu_ghost_evict_metadata 4 130259968 4G of ram (not expandable - a mini 4 port thecus NAS running debian (because the hacked up scripts that thecus ship in their embedded OS are seriously dodgy)). -- Tim Connors
On Fri, 4 May 2012, Peter Ross wrote:
On Thu, 3 May 2012, Tim Connors wrote:
I don't suppose anyone has ported arc_summary and zilstat to read the linux spl structures?
http://constantin.glez.de/blog/2011/02/frequently-asked-questions-about-flas...
I suspect I'd benefit greatly from l2arc in my usage (not so much zil). But I don't wanna go out and buy this yet, until I *know* it will help improve upon the 40iops I currently suffer from :)
Woot! iops were 40 or so per disk, except for the commit every 5 seconds where it went up to 500 or so (so munin's idea of average iops is mostly counting the writes). Added 16GB of usb key (a $16 usb3 key in a usb2 slot - but at least it thoroughly maxes out the usb2 bandwidth) first as secondarycache=all, and an hour ago as secondarycache=metadata (since bandwidth constrained, I just wanted to have it handle all the seeky stuff, and let the throughputty stuff go through to the grownups). And munin now tells me I've doubled the iops (writes) to the big disks because they're not spending all their time seeking anymore (got 70% hits going to my l2arc since warming up overnight)! It's consistently only using 1G of the usb key - it's allocated 15G so far, but most of that is obviously stale because most of the metadata is eventually being updated (I'm rsyncing my backuppc pool full of hardlinks from the old xfs partition, and now rsync is sitting at 20% cpu usage instead of bugger-all).
Under FreeBSD I have a lot of sysctls, see "sysctl -a | grep zfs"
Below the parts from a running machine that may interest you.
Can't you find similar sysctls or /proc/sys/whatever entries under zfsonlinux?
I have to admit I don't look at them too much at the moment because the systems work without any problems, AFAIK (and more generic nagios baed monitoring tells me).
Maybe I should setup myself a github thingy. Oh, look at that: https://github.com/spacelama/arcstat -- Tim Connors
On Sun, May 06, 2012 at 12:46:01AM +1000, Tim Connors wrote:
I have to admit I don't look at them too much at the moment because the systems work without any problems, AFAIK (and more generic nagios baed monitoring tells me).
Maybe I should setup myself a github thingy.
Oh, look at that: https://github.com/spacelama/arcstat
cool, thanks for that. the munin zfs stuff is useful. have installed it on both my zfs boxes. kmemsum doesn't seem to work, though. # ./kmemsum ./kmemsum: line 33: kldstat: command not found sysctl: cannot stat /proc/sys/vm/kmem_size_max: No such file or directory text.value data.value 4189792256 total.value 4189792256 max.value dunno what kldstat is, and /proc/sys/vm/kmem_size_max isn't on my system (linux 3.2.15, using the debian kernel package linux-image-3.2.0-2-amd64) craig -- craig sanders <cas@taz.net.au>
On Sun, 6 May 2012, Craig Sanders wrote:
On Sun, May 06, 2012 at 12:46:01AM +1000, Tim Connors wrote:
I have to admit I don't look at them too much at the moment because the systems work without any problems, AFAIK (and more generic nagios baed monitoring tells me).
Maybe I should setup myself a github thingy.
Oh, look at that: https://github.com/spacelama/arcstat
cool, thanks for that. the munin zfs stuff is useful.
have installed it on both my zfs boxes.
kmemsum doesn't seem to work, though.
Yep, I probably should have removed that when I couldn't figure out what it was trying to do, and didn't make an attempt to port it (it was 1am by this stage, and I had to get up early this morning to go see 100 old bonnevilles and triumphs and other lovely historical bikes at Bayles) Other things somewhat only partially work. For instance, take a look at zfsarc-l2 - only gets 2 of the 4 values it was trying to obtain. and ./arc_summary.pl (oops, broke between me testing it and committing it to git - just the #! line), which only prints out a few of the 7 pages, because the eval dies at some divide by zeros. Some of these were ported directly from the original solaris versions, some went via freebsd first. At least freebsd somewhat looks a tiny bit like linux - pity the proc values were put under /proc/spl/kstat instead of /proc/sys/... - then you could just use sysctl instead of fudging around with /proc and filtering out the second column containing the typeinfo. -- Tim Connors
On Sun, May 06, 2012 at 03:37:42PM +1000, Tim Connors wrote:
Other things somewhat only partially work. For instance, take a look at zfsarc-l2 - only gets 2 of the 4 values it was trying to obtain.
and ./arc_summary.pl (oops, broke between me testing it and committing it to git - just the #! line),
yeah. fixed that myself. some err msg about 'env -S'. I just replaced it with #!/usr/bin/perl /usr/bin/env is broken, anyway. you can't pass args to the interpreter on the #! line (except on Mac OS. their version supports it apparently).
which only prints out a few of the 7 pages, because the eval dies at some divide by zeros.
i didn't even notice that. when i get time, i'll have a look at that. and zfsarc-l2 as well. craig -- craig sanders <cas@taz.net.au> BOFH excuse #288: Hard drive sleeping. Let it wake up on it's own...
participants (3)
-
Craig Sanders -
Peter Ross -
Tim Connors