bad ext3 filesystem

I have an unusual situation with an ext3 filesystem on a Xen DomU. I get the following on boot: [ 18.153177] blkfront: barrier: empty write xvda1 op failed [ 18.153195] blkfront: xvda1: barrier or flush: disabled [ 18.153206] end_request: I/O error, dev xvda1, sector 20975896 [ 18.153214] end_request: I/O error, dev xvda1, sector 20975896 [ 18.153222] Buffer I/O error on device xvda1, logical block 2621987 [ 18.153229] lost page write due to I/O error on xvda1 [ 18.153253] Aborting journal on device xvda1. [ ok ] Cleaning up temporary files.... [ 19.041362] EXT3-fs (xvda1): error: ext3_journal_start_sb: Detected aborted journal [ 19.041382] EXT3-fs (xvda1): error: remounting filesystem read-only The sequence of events that lead to this was: 1. Filesystem ran out of space (debugging options on for something I was testing and it filled up within hours) 2. Stopped the VM 3. resize2fs 4. Started VM e2fsck in Dom0 on the VM's filesystem (while the VM isn't running!!) shows no problems. I've tried removing and re-creating the journal. The underlying block device was resized from 10G to 20G, so sector 20975896 is well within the bounds of the disk. Should I cut my losses and create a new fs and copy all the files over or can someone suggest a fix? (FYI, the sequence of events in the logs above is the DomU trying to establish if the underlying vbd interface supports barriers, which it appears to not) Thanks James

On Tue, 30 Oct 2012, James Harper <james.harper@bendigoit.com.au> wrote:
e2fsck in Dom0 on the VM's filesystem (while the VM isn't running!!) shows no problems.
Can you mount it in the Dom0 without problems? Are the Dom0 and DomU running the same kernel? -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/

On Tue, 30 Oct 2012, James Harper <james.harper@bendigoit.com.au> wrote:
e2fsck in Dom0 on the VM's filesystem (while the VM isn't running!!) shows no problems.
Can you mount it in the Dom0 without problems?
Yes.
Are the Dom0 and DomU running the same kernel?
No. 2.6.32 in Dom0 and 3.2.0 in DomU. The kernel in DomU hasn't changed though, only the disk size. I have added rootflags=nobarrier to the boot config and it's working now. I guess I'll upgrade when I have a chance. Thanks James

On 30 October 2012 21:13, James Harper <james.harper@bendigoit.com.au> wrote:
[ 18.153206] end_request: I/O error, dev xvda1, sector 20975896 [ 18.153214] end_request: I/O error, dev xvda1, sector 20975896 [ 18.153222] Buffer I/O error on device xvda1, logical block 2621987 [ 18.153229] lost page write due to I/O error on xvda1
Are you sure it isn't a dying disk? e2fsck won't check for disk write errors. Use smartctl and see what it reports for the disk. -- Brian May <brian@microcomaustralia.com.au>

On 30 October 2012 21:13, James Harper <james.harper@bendigoit.com.au> wrote:
[ 18.153206] end_request: I/O error, dev xvda1, sector 20975896 [ 18.153214] end_request: I/O error, dev xvda1, sector 20975896 [ 18.153222] Buffer I/O error on device xvda1, logical block 2621987 [ 18.153229] lost page write due to I/O error on xvda1
Are you sure it isn't a dying disk?
e2fsck won't check for disk write errors.
Use smartctl and see what it reports for the disk.
xvda1 is a block device running on a Xen VM (DomU), so no smartctl support. Also, I'd be seeing the errors in Dom0's logs, and in the HP RAID controller logs. James
participants (3)
-
Brian May
-
James Harper
-
Russell Coker