I noticed a disk failure just now on one of the drives in my main pool - heard the clicking sound of repeated access attempts so I looked at what 'zpool status' said: pool: export state: DEGRADED status: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the faulted device, or use 'zpool clear' to mark the device repaired. scan: scrub repaired 0 in 8h12m with 0 errors on Sat Aug 31 10:09:50 2013 config: NAME STATE READ WRITE CKSUM export DEGRADED 0 0 0 raidz1-0 DEGRADED 0 0 0 ata-WDC_WD10EACS-00ZJB0_WD-WCASJ2114122 ONLINE 0 0 0 ata-WDC_WD10EACS-00ZJB0_WD-WCASJ2195141 ONLINE 0 0 0 ata-WDC_WD10EARS-00Y5B1_WD-WMAV50817803 FAULTED 1 2 0 too many errors ata-ST31000528AS_9VP18CCV ONLINE 0 0 0 logs mirror-1 ONLINE 0 0 0 ata-OCZ-VECTOR_OCZ-0974C023I4P2G1B8-part7 ONLINE 0 0 0 ata-OCZ-VECTOR_OCZ-8RL5XW08536INH7R-part7 ONLINE 0 0 0 cache ata-OCZ-VECTOR_OCZ-0974C023I4P2G1B8-part6 ONLINE 0 0 0 ata-OCZ-VECTOR_OCZ-8RL5XW08536INH7R-part6 ONLINE 0 0 0 errors: No known data errors the fix is easy, insert replacement disk and run: # zpool replace -f export ata-WDC_WD10EARS-00Y5B1_WD-WMAV50817803 ata-WDC_WD10EARS-00Y5B1_WD-WMAV50933036 # zpool status -v export pool: export state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Wed Sep 4 10:13:09 2013 109G scanned out of 2.78T at 75.9M/s, 10h15m to go 27.3G resilvered, 3.84% done config: NAME STATE READ WRITE CKSUM export DEGRADED 0 0 0 raidz1-0 DEGRADED 0 0 0 ata-WDC_WD10EACS-00ZJB0_WD-WCASJ2114122 ONLINE 0 0 0 ata-WDC_WD10EACS-00ZJB0_WD-WCASJ2195141 ONLINE 0 0 0 replacing-2 FAULTED 0 0 0 ata-WDC_WD10EARS-00Y5B1_WD-WMAV50817803 FAULTED 1 2 0 too many errors ata-WDC_WD10EARS-00Y5B1_WD-WMAV50933036 ONLINE 0 0 0 (resilvering) ata-ST31000528AS_9VP18CCV ONLINE 0 0 0 logs mirror-1 ONLINE 0 0 0 ata-OCZ-VECTOR_OCZ-0974C023I4P2G1B8-part7 ONLINE 0 0 0 ata-OCZ-VECTOR_OCZ-8RL5XW08536INH7R-part7 ONLINE 0 0 0 cache ata-OCZ-VECTOR_OCZ-0974C023I4P2G1B8-part6 ONLINE 0 0 0 ata-OCZ-VECTOR_OCZ-8RL5XW08536INH7R-part6 ONLINE 0 0 0 errors: No known data errors craig -- craig sanders <cas@taz.net.au>
On Wed, Sep 04, 2013 at 10:39:00AM +1000, Craig Sanders wrote:
scan: resilver in progress since Wed Sep 4 10:13:09 2013 109G scanned out of 2.78T at 75.9M/s, 10h15m to go 27.3G resilvered, 3.84% done
nice, the resilver has got up to a decent speed now. scan: resilver in progress since Wed Sep 4 10:13:09 2013 1.39T scanned out of 2.78T at 133M/s, 3h2m to go 357G resilvered, 50.12% done 133M/s isn't bad at all for some WD Green drives in raidz. craig -- craig sanders <cas@taz.net.au>
Craig Sanders <cas@taz.net.au> writes:
I noticed a disk failure just now on one of the drives in my main pool - heard the clicking sound of repeated access attempts so I looked at what 'zpool status' said:
pool: export state: DEGRADED
I read this as "it didn't email me about its degraded state, so if it was racked, I'd never have known until it was too late" :-)
Trent W. Buck <trentbuck@gmail.com> wrote:
I read this as "it didn't email me about its degraded state, so if it was racked, I'd never have known until it was too late" :-)
Did it log the change of state? Zfs is excellent - now if only Oracle would release it under a GPL2-compatible license...
On Wed, Sep 04, 2013 at 04:07:34PM +1000, Jason White wrote:
Trent W. Buck <trentbuck@gmail.com> wrote:
I read this as "it didn't email me about its degraded state, so if it was racked, I'd never have known until it was too late" :-)
Did it log the change of state?
unfortunately, no. can't see any specific zfs error message in my kern.log, just the usual kernel scsi/sata/sd retry errors. IMO, that's a problem. hopefully will get fixed in a future release. at minimum, it should log a degraded event in the zpool history - it has logged the 'zpool replace ...' command there.
Zfs is excellent - now if only Oracle would release it under a GPL2-compatible license...
can't see that happening. it is, however, open source - just not GPL so can't be distributed with the kernel. fortunately, there's no impediment to compiling and installing the module yourself, or even having a dkms package to automate the process. some distros (gentoo and arch iirc) even distribute binary modules - i'm not convinced that that is entirely safe for them to do, but it's their necks on the line, not mine. craig -- craig sanders <cas@taz.net.au>
On Wed, 4 Sep 2013, Craig Sanders wrote:
Zfs is excellent - now if only Oracle would release it under a GPL2-compatible license...
can't see that happening.
it is, however, open source - just not GPL so can't be distributed with the kernel. fortunately, there's no impediment to compiling and installing the module yourself, or even having a dkms package to automate the process.
some distros (gentoo and arch iirc) even distribute binary modules - i'm not convinced that that is entirely safe for them to do, but it's their necks on the line, not mine.
From the mailing list, the 0.6.2 .deb packages haven't been built this time around because supposedly it is now in debian official upstream (experimental), although several people report it's not quite there yet. Obviously built by the same legal and Linus-condoned dkms methods. Given that I'm about to spend 6 weeks overseas, I'm not about to try installing it though :)
(My fileserver has for the first time ever managed to stay up for 31 days without my raspberry pi watchdog rebooting it. I don't really know what changed other than moving / from a very slow SD card to a faster real SSD. I also changed backuppc to fill in incrementals, so maybe there's a lot less metadata (we already know zfs fails on metadata heavy workloads - current discussion is how some 320 byte objects are actually taking up 1024 bytes, but only counting as 320 bytes in the ARC accounting) being accessed in the daily incrementals - not having to seek back through 15 or so incrementals to work out what's changed in a given directory. At the expense of a huge number of hardlinks and inodes being consumed instead, and slower backups) -- Tim Connors
On Wed, Sep 04, 2013 at 04:03:54PM +1000, Trent W. Buck wrote:
I read this as "it didn't email me about its degraded state, so if it was racked, I'd never have known until it was too late" :-)
why would i want a filesystem driver to have email capabilities? it's not zfs' job to email me, any more than it is ext4's job. problem notifications are better handled by some monitoring daemon - nagios or icinga, for example. craig -- craig sanders <cas@taz.net.au>
On Wed, 4 Sep 2013, Craig Sanders wrote:
On Wed, Sep 04, 2013 at 04:03:54PM +1000, Trent W. Buck wrote:
I read this as "it didn't email me about its degraded state, so if it was racked, I'd never have known until it was too late" :-)
why would i want a filesystem driver to have email capabilities?
it's not zfs' job to email me, any more than it is ext4's job.
problem notifications are better handled by some monitoring daemon - nagios or icinga, for example.
It didn't log to syslog? That's poor form. But yes, I do have a cron.hourly check script for zpool. It hasn't yet been excercised in anger (I kinda want to see whether it lights the correct red LED in my little NAS box though - I'm proud of that little script and kernel module :). -- Tim Connors
Craig Sanders <cas@taz.net.au> writes:
On Wed, Sep 04, 2013 at 04:03:54PM +1000, Trent W. Buck wrote:
I read this as "it didn't email me about its degraded state, so if it was racked, I'd never have known until it was too late" :-)
why would i want a filesystem driver to have email capabilities?
it's not zfs' job to email me, any more than it is ext4's job.
problem notifications are better handled by some monitoring daemon - nagios or icinga, for example.
Erm, mdadm emails me when a RAID array fails. I didn't have to write my own script, because that's something everybody running an array should want (more or less). If I had migrated from md to zfs, I'd have been bitten by that. A mismatch of expectations, I guess.
On Fri, 6 Sep 2013, "Trent W. Buck" <trentbuck@gmail.com> wrote:
Craig Sanders <cas@taz.net.au> writes:
On Wed, Sep 04, 2013 at 04:03:54PM +1000, Trent W. Buck wrote:
I read this as "it didn't email me about its degraded state, so if it was racked, I'd never have known until it was too late" :-)
why would i want a filesystem driver to have email capabilities?
it's not zfs' job to email me, any more than it is ext4's job.
problem notifications are better handled by some monitoring daemon - nagios or icinga, for example.
Erm, mdadm emails me when a RAID array fails. I didn't have to write my own script, because that's something everybody running an array should want (more or less). If I had migrated from md to zfs, I'd have been bitten by that. A mismatch of expectations, I guess.
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=658701 Unfortunately mdadm doesn't necessarily email you about problems. Although in the case of RAID-1 a scrub seems to always report problems even when nothing is wrong so this is often an issue of a false alarm. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
Russell Coker <russell@coker.com.au> writes: twb> I read this as "it didn't email me about its degraded state, so if it twb> was racked, I'd never have known until it was too late" :-) twb> [...] twb> Erm, mdadm emails me when a RAID array fails. I didn't have to write my twb> own script, because that's something everybody running an array should twb> want (more or less). If I had migrated from md to zfs, I'd have been twb> bitten by that. A mismatch of expectations, I guess.
Unfortunately mdadm doesn't necessarily email you about problems.
The original discussion was about degraded arrays, which AFAIK mdadm always tries to email me about. That appears to be about sum mismatches, which I readily agree mdadm is crap at.
Although in the case of RAID-1 a scrub seems to always report problems even when nothing is wrong so this is often an issue of a false alarm.
I noticed this too, under Ubuntu 10.04 / 2.6.32. We thought it was the swap partitions, so we removed them but the problem remained. IIRC we never resolved it, so we just ignore the mismatch warnings we get from the monthly check.
trentbuck@gmail.com (Trent W. Buck) writes:
Craig Sanders <cas@taz.net.au> writes:
On Wed, Sep 04, 2013 at 04:03:54PM +1000, Trent W. Buck wrote:
I read this as "it didn't email me about its degraded state, so if it was racked, I'd never have known until it was too late" :-)
why would i want a filesystem driver to have email capabilities?
it's not zfs' job to email me, any more than it is ext4's job.
problem notifications are better handled by some monitoring daemon - nagios or icinga, for example.
Erm, mdadm emails me when a RAID array fails. I didn't have to write my own script, because that's something everybody running an array should want (more or less). If I had migrated from md to zfs, I'd have been bitten by that. A mismatch of expectations, I guess.
PS: I'd accept it freaking out to syslog(3) instead of sendmail(8), since reading the logfiles is something I expect a server's babysitter to do at least daily (cf. checking for a degraded array).
On 4 September 2013 16:52, Craig Sanders <cas@taz.net.au> wrote:
it's not zfs' job to email me, any more than it is ext4's job.
Agreed, however ZFS is also a volume-manager and RAID, but that shouldn't necessitate it also act as MUA.
problem notifications are better handled by some monitoring daemon - nagios or icinga, for example.
Also agreed, and in the instance of ZFS on Solaris, faults can be queried using fmadm, (or optionally logged to syslog) I'm unaware of any Linux equivalent to Solaris FMA (Fault Management Architecture), that could potentially handle and report ZFS status changes. -- Joel Shea <jwshea@gmail.com>
participants (6)
-
Craig Sanders -
Jason White -
Joel W Shea -
Russell Coker -
Tim Connors -
trentbuck＠gmail.com