27 May
2017
27 May
'17
1:09 p.m.
Is it possible to find out when errors occurred on a RAID-Z other than just
monitoring the output of "zpool status" regularly and looking for changes?
I have a RAID-Z that I just discovered has between 3 and 7 checksum errors on
each of 7 disks. I want to know why disks that had worked without errors on
ZFS since 6TB was a big disk have got such errors in the past couple of weeks.
If I knew the date and time of the errors it might give me a clue. The system
in question has 9*6TB and 9*10TB disks in 2 RAID-Z arrays. None of the 10TB
disks had a problem while 7/9 of the 6TB disks reported errors. The 6TB disks
are a recent addition to the pool and the 9*10TB RAID-Z was almost full before
I added them, so maybe the checksum errors are related to which disks had the
most data written.
If I knew which day the errors happened on I might be able to guess at the
cause. But ZFS doesn't seem to put anything in the kernel log.
Any suggestions about what I can do?
--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/
3 Jun
3 Jun
4:09 p.m.
On Sat, May 27, 2017 at 11:09:53PM +1000, Russell Coker via luv-main wrote:
Is it possible to find out when errors occurred on a
RAID-Z other than just
monitoring the output of "zpool status" regularly and looking for changes?
I have a RAID-Z that I just discovered has between 3 and 7 checksum errors on
each of 7 disks. I want to know why disks that had worked without errors on
ZFS since 6TB was a big disk have got such errors in the past couple of weeks.
If I knew the date and time of the errors it might give me a clue. The system
in question has 9*6TB and 9*10TB disks in 2 RAID-Z arrays. None of the 10TB
disks had a problem while 7/9 of the 6TB disks reported errors. The 6TB disks
are a recent addition to the pool and the 9*10TB RAID-Z was almost full before
I added them, so maybe the checksum errors are related to which disks had the
most data written.
If I knew which day the errors happened on I might be able to guess at the
cause. But ZFS doesn't seem to put anything in the kernel log.
Any suggestions about what I can do?
anything in 'zpool history'?
we're using https://calomel.org/zfs_health_check_script.html
individual SMART errors in drives should also list power on hours next
to the errors. perhaps date and time too.
BTW RAIDZ isn't really sufficient for such large arrays of large
drives. you should probably be using Z2 or Z3.
eg. depending on your drive type, you may have a 55% chance of losing
a Z on a rebuild reading 80TB. chance of success:
% bc -l
scale=20
e(l(1-10^-14)*(8*10*10^12))
.44932896411722159143
https://lwn.net/Articles/608896/
better if they're 10^-15 drives - only an 8% chance of a fail. still
not great.
or is ZFS smart and just fails to rebuild 1 stripe (8*2^zshift bytes)
of the data and the rest is ok?
cheers,
robin
8:39 p.m.
On Saturday, 3 June 2017 12:09:14 PM AEST Robin Humble via luv-main wrote:
2017-05-03.10:48:38 zpool import -a [user 0 (root) on tank:linux]
2017-06-01.20:00:03 [txg:5525955] scan setup func=1 mintxg=0 maxtxg=5525955
[on tank]
2017-06-01.20:00:10 zpool scrub pet630 [user 0 (root) on tank:linux]
Thanks for the suggestion but the above are the most recent entries from
"zpool history -il". No mention of the errors that happened after booting on
the 3rd of May or the errors that were found in that scrub.
Any suggestions
about what I can do?
anything in 'zpool history'? we're using
https://calomel.org/zfs_health_check_script.html
Apart from checking for a recent script it has similar features to the mon
script I wrote.
individual SMART errors in drives should also list
power on hours next
to the errors. perhaps date and time too.
The drives didn't have errors as such. There were some "correctable" errors
logged, but not as many as ZFS found. ZFS reported no errors reading the
disk, just checksum errors. The disks are returning bad data and saying it's
good.
BTW RAIDZ isn't really sufficient for such large
arrays of large
drives. you should probably be using Z2 or Z3.
eg. depending on your drive type, you may have a 55% chance of losing
a Z on a rebuild reading 80TB. chance of success:
% bc -l
scale=20
e(l(1-10^-14)*(8*10*10^12))
.44932896411722159143
https://lwn.net/Articles/608896/
better if they're 10^-15 drives - only an 8% chance of a fail. still
not great.
or is ZFS smart and just fails to rebuild 1 stripe (8*2^zshift bytes)
of the data and the rest is ok?
That link refers to a bug in Linux Software RAID which has been fixed and also
didn't apply to ZFS.
Most of the articles about the possibility of multiple drive failues falsely
assume that drives will entirely fail. In the last ~10 years the only drive
I've been responsible for which came close to entirely failing had about
12,000 errors out of 1.5TB. If you had 2 drives fail in that manner
(something I've never seen - the worst I've seen for multiple failures is 2
drives having ~200 errors) then you would probably still get a reasonable
amount of data off. Especially as ZFS makes an extra copy of metadata so on a
RAID-Z1 2 disks getting corrupted won't lose any metadata.
With modern disks you are guaranteed to lose data if you use regular RAID-6.
I've had lots of disks return bad data and say it's good. With RAID-Z1 losing
data is much less likely.
--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/
4 Jun
4 Jun
4:52 p.m.
On Sun, Jun 04, 2017 at 06:39:34AM +1000, Russell Coker wrote:
sorry - I meant to say you need to read down a bit and look for comments
by Neil Brown. some good explanations of drive statistics. the above
calculations come from there. the stats apply to Z1 and Z2 as much as
R5 and R6.
On Saturday, 3 June 2017 12:09:14 PM AEST Robin Humble
via luv-main wrote:
could it be bad RAM or SATA cables?
do you have ECC ram?
kinda sounds like something transient, so power supply or RAM would
be my guess if there's no errors and no red flags (Uncorrecable,
Pending, >few Remapped etc.) in SMART.
however drives lie about SMART quite a lot. errors may just not show
up or may come and go randomly. drives are insane.
individual SMART errors in drives should also
list power on hours next
to the errors. perhaps date and time too.
The drives didn't have errors as
such. There were some "correctable" errors
logged, but not as many as ZFS found. ZFS reported no errors reading the
disk, just checksum errors. The disks are returning bad data and saying it's
good. BTW RAIDZ
isn't really sufficient for such large arrays of large
drives. you should probably be using Z2 or Z3.
eg. depending on your drive type, you may have a 55% chance of losing
a Z on a rebuild reading 80TB. chance of success:
% bc -l
scale=20
e(l(1-10^-14)*(8*10*10^12))
.44932896411722159143
https://lwn.net/Articles/608896/
better if they're 10^-15 drives - only an 8% chance of a fail. still
not great.
or is ZFS smart and just fails to rebuild 1 stripe (8*2^zshift bytes)
of the data and the rest is ok?
That link refers to a bug in Linux Software RAID which has been fixed and also
didn't apply to ZFS. Most of the articles about the possibility of multiple
drive failues falsely
assume that drives will entirely fail. In the last ~10 years the only drive
if md hits a problem during a rebuild that it can't recover from then
it will stop and give up (at least last time I tried it), so it is
essentially a whole drive fail. it's a block device, it can't do
much more... (*)
our hardware raid systems also fail about a disk or two a week. so these
are 'whole disk fails' regardless of whatever metrics they are using.
so whole disk fails definitely do happen.
we had IIRC 8% of drives dying and being replaced per year on one system.
with ZFS I'm assuming (I haven't experienced it myself) it's more common
to have a per-object loss and if eg. drives in a Z1 have dozens of bad
sectors, then as long as 2 of those sectors don't line up then you
won't lose data? (I'm a newbie to ZFS so happy to be pointed at docs.
I can't see any obviously on this topic.)
but of course in that situation you will definitely lose some data if
one drive totally fails. then there'll be sectors with errors and no
redundancy. this is why we all do regular scrubs - so there are no
obvious bad sector of disks around.
the statistics above address what happens after a drive fails in an
otherwise clean Z1 array. they say that 1 new error occurs with
such-and-such a probability during a rebuild of a whole new drive full
of data. that's statistically at a 55% likelihood level for your array
configuration.
so if you're convinced you'll never have a drive fail that's fine & good
luck to you, but I'm just saying that the stats aren't on your side if
you ever do :)
With modern disks you are guaranteed to lose data if
you use regular RAID-6.
every scheme is guaranteed to lose data. it's just a matter of time and
probabilities.
I've had lots of disks return bad data and say
it's good. With RAID-Z1 losing
data is much less likely.
in my experience (~17000 Enterprise SATA drive-years or so, probably
half that of true SAS and plain SATA drive-years), disks silently
giving back bad data is extremely rare. I can't recall seeing it, ever.
once an interposer did return bad data, but that wasn't the disk's fault.
we did have data corruption once and noticed it, and tracked it down,
and it wasn't the fault of disks. so it's not as if we just didn't
notice 'wrong data' at the user level even though (at the time)
everything wasn't checksummed like in ZFS.
I'm not saying silent data corruption doesn't happen (I heard of one
incident in a company recently which hastened their transition to ZFS
on linux), but perhaps there is something else wrong if you are seeing
it "lots".
what sort of disks are you using?
cheers,
robin
(*) in actuality it's often possible (and I have) got around this and
recovered all but a tiny amount of data when it shouldn't be possible
because there is no redundancy left and a drive has errors. it's a very
manual proces and far from easy - find the sector from smart or scsi
errors or the byte where a dd read fails, dd write to it to remap the
sector in the drive, cross fingers, attempt the raid rebuild again,
repeat.
5 Jun
5 Jun
4:23 a.m.
On Sunday, 4 June 2017 12:52:33 PM AEST Robin Humble via luv-main wrote:
The server has ECC RAM. It has 9*8TB disks for which ZFS has never reported
an error. It has 9*6TB disks of which 1 is Seagate and has never had an error
reported. Of the 8*6TB WD disks 7 have ZFS errors reported against them.
The server is a Del PowerEdge Tower T630 which has hot-swap drive bays.
Having damaged SATA cables would be very unlikely. Having SATA cable issues
that match the model of disk is extremely unlikely. The disks are in 3 rows
of 6 disks, so we have disks in 2 rows failing and now correlation with
position other than the fact that the 6TB disks are in the first row and half
the second row.
I've seen that too. Apparently that's supposed to be fixable with a bad
blocks list that you can add WHEN YOU CREATE THE ARRAY. So all of us who have
existing Linux Software RAID arrays can't fix it - fortunately that's mostly
limited to /boot and sometimes / on my systems.
Guaranteed to lose data in a fairly small amount of time if you don't have
checksums on everything.
I've seen it lots of times in all manner of hardware, from old IDE disks to
modern SATA. I don't use SAS much. The problem with SAS is that disks are
more expensive and that leads the people paying towards cutting corners
elsewhere. SATA with adequate redundancy and backups is much better than SAS
with all corners cut.
The drives
didn't have errors as such. There were some "correctable"
errors logged, but not as many as ZFS found. ZFS reported no errors
reading the disk, just checksum errors. The disks are returning bad data
and saying it's good.
could it be bad RAM or SATA cables?
do you have ECC ram? kinda sounds like something transient, so power supply
or RAM would
be my guess if there's no errors and no red flags (Uncorrecable,
Pending, >few Remapped etc.) in SMART.
however drives lie about SMART quite a lot. errors may just not show
up or may come and go randomly. drives are insane.
Yes, drives are insane. I'm pretty sure the PSU is up to the task, it's
designed for high performance SAS disks that draw more power.
sorry - I meant to say you need to read down a bit and
look for comments
by Neil Brown. some good explanations of drive statistics. the above
calculations come from there. the stats apply to Z1 and Z2 as much as
R5 and R6.
I've read all that. But the thing that is missing here is that RAID-Z is
fundamentally not the same as RAID-5 and ZFS is not like a regular filesystem.
https://www.illumos.org/issues/3835
ZFS has redundant copies of metadata. This means that if you have enough
corruption in a RAID-Z to lose some data it probably won't be enough to lose
metadata. Basically a default RAID-Z configuration protects metadata as well
as RAID-6 with checksums would (of which BTRFS RAID-6 is the only strict
implementation I'm aware of). As an aside I think that NetApp WAFL does
something similar with metadata, but as I can't afford it I haven't looked
into it much.
The errors that I have just encountered (over 100 corrupted sectors so far)
would not have been detected on any other RAID configuration. The design of
Linux software RAID is that if redundancy on RAID-5 or RAID-6 doesn't match
the data then it's rewritten. It's theoretically possible to find which block
on a RAID-6 stripe was corrupted, but the assumption is that a changed block
was written and the redundancy was lost. There are 2 ways of fixing this
problem of RAID-5/RAID-6 partial stripe writes. One option is to have an
external journal, EG the battery-backed write-back caches that vendors such as
HP and Dell offer as an expensive option. The other option is to do what
BTRFS, ZFS, and WAFL do and write the new data in a different location. That
gets you the most fragmented filesystem possible but gives greatest
reliability and also snapshots are easy to add.
Most of the
articles about the possibility of multiple drive failues
falsely assume that drives will entirely fail. In the last ~10 years the
only drive
if md hits a problem during a rebuild that it can't recover from
then
it will stop and give up (at least last time I tried it), so it is
essentially a whole drive fail. it's a block device, it can't do
much more... (*) our hardware raid systems also fail about a disk or
two a week. so these
are 'whole disk fails' regardless of whatever metrics they are using.
If these are cases where you have some dozens of errors you don't want the
disk just kicked out, you want it to stay in service until you get the
replacement online.
As an aside a disk error of any kind during a drive replacement can be more of
a problem than it should be. If you are replacing a disk for a non-error
reason (EG you replace disks 1 at a time with bigger disks until you can grow
the array) there's no reason why a failure of another disk at the same time
should be a big deal. But I've seen it bring down a NAS based on Linux
Software RAID. I don't think that would bring down ZFS.
with ZFS I'm assuming (I haven't experienced
it myself) it's more common
to have a per-object loss and if eg. drives in a Z1 have dozens of bad
sectors, then as long as 2 of those sectors don't line up then you
won't lose data? (I'm a newbie to ZFS so happy to be pointed at docs.
I can't see any obviously on this topic.)
The design of ZFS is based on the assumption that errors aren't entirely
independent and I believe it aims to put multiple copies of metadata on
different parts of disks. This plus the extra copy of metadata means when you
do hit data loss you are most likely to see only file data lost. Then at
least you know which files to restore from backup or regenerate.
the statistics above address what happens after a
drive fails in an
otherwise clean Z1 array. they say that 1 new error occurs with
such-and-such a probability during a rebuild of a whole new drive full
of data. that's statistically at a 55% likelihood level for your array
configuration.
It's a fairly low probability in my experience given that I've had systems go
for years running a monthly ZFS scrub without such errors. I've had BTRFS
systems go for years with weekly scrubs without such errors.
so if you're convinced you'll never have a
drive fail that's fine & good
luck to you, but I'm just saying that the stats aren't on your side if
you ever do :)
I'm looking into Ceph now. I think an ideal solution to large scale storage
might be something like Ceph with RAID-1 SSD for metadata storage and either
RAID-Z or BTRFS RAID-0 for data storage depending on how much you rely on Ceph
for data redundancy.
With modern
disks you are guaranteed to lose data if you use regular
RAID-6.
every scheme is guaranteed to lose data. it's just a matter of time
and
probabilities. I've had
lots of disks return bad data and say it's good. With RAID-Z1
losing data is much less likely.
in my experience (~17000 Enterprise SATA drive-years or so, probably
half that of true SAS and plain SATA drive-years), disks silently
giving back bad data is extremely rare. I can't recall seeing it, ever.
once an interposer did return bad data, but that wasn't the disk's fault. we did have data corruption once and noticed it, and
tracked it down,
and it wasn't the fault of disks. so it's not as if we just didn't
notice 'wrong data' at the user level even though (at the time)
everything wasn't checksummed like in ZFS.
I've seen that too. One PC without ECC RAM had BTRFS errors, the BTRFS
developers said that it wasn't the pattern of errors they would expect from
storage failure or kernel errors and that I should check the RAM. The RAM
turned out to be faulty. With Ext4 the system would probably have run for a
lot longer in service and the corrupted data would have ended up on backup
devices...
I'm not saying silent data corruption doesn't
happen (I heard of one
incident in a company recently which hastened their transition to ZFS
on linux), but perhaps there is something else wrong if you are seeing
it "lots".
what sort of disks are you using?
I've explained the disks for this incident above. But in the past I've seen
it on a variety of others. I haven't kept a journal, I probably should.
(*) in actuality it's often possible (and I have)
got around this and
recovered all but a tiny amount of data when it shouldn't be possible
because there is no redundancy left and a drive has errors. it's a very
manual proces and far from easy - find the sector from smart or scsi
errors or the byte where a dd read fails, dd write to it to remap the
sector in the drive, cross fingers, attempt the raid rebuild again,
repeat.
I've done that before too. I even once wrote a personal version of ddrescue
that read from 2 disks and took whichever one would give me data.
--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/
2117
days inactive
2126
days old
4 comments
2 participants
participants (2)
-
Robin Humble -
Russell Coker