Filesystem becomes readonly.
Hi LUVers 1.Linux (Debian Wheezy) running on a cloud server 2.Everything working fine for weeks/months 3.Last nite, for some reason all disk writes to server stops 4.Even root cannot write to anywhere on the disk, and plenty of space left on disk (~80%) All file permissions look OK. 5.After reboot all return to normal.(ie problem is fixed) My questions: Is it fair to assume that the problem was caused by something that the cloud provider has done? Or could it be something on server OS side? What can cause this? (I am thinking the virtual disk hosting the VM has become readonly somehow, but how? ) Cheers Daniel.
On Tue, Mar 25, 2014 at 12:27:04PM +1100, Daniel Jitnah wrote:
Is it fair to assume that the problem was caused by something that the cloud provider has done? Or could it be something on server OS side?
it's probably not your VM but it's more likely to be a fault or outage than something the cloud provider has deliberately done. breaking running VMs is something that most providers would try to avoid.
What can cause this?
most likely the VM's disk image became unavailable temporarily - possibly due to network problems, or a server being rebooted. it's hard to be more specific than that because there are countless ways of setting up a cloud server.
(I am thinking the virtual disk hosting the VM has become readonly somehow, but how? )
assuming you are using ext2/3/4 on your VM's disk - the mount option "errors=remount-ro" says to remount the fs as read-only if the kernel has any errors accessing the filesystem (e.g. if a disk is dead/dying or the cable is loose etc). debian at least, and probably other distros, routinely adds "errors=remount-ro" to /etc/fstab for ext filesystems when you build the system. craig -- craig sanders <cas@taz.net.au> BOFH excuse #67: descramble code needed from software company
On 25/03/14 12:58, Craig Sanders wrote:
On Tue, Mar 25, 2014 at 12:27:04PM +1100, Daniel Jitnah wrote:
Is it fair to assume that the problem was caused by something that the cloud provider has done? Or could it be something on server OS side?
it's probably not your VM but it's more likely to be a fault or outage than something the cloud provider has deliberately done. breaking running VMs is something that most providers would try to avoid.
What can cause this?
most likely the VM's disk image became unavailable temporarily - possibly due to network problems, or a server being rebooted.
it's hard to be more specific than that because there are countless ways of setting up a cloud server.
(I am thinking the virtual disk hosting the VM has become readonly somehow, but how? )
assuming you are using ext2/3/4 on your VM's disk - the mount option "errors=remount-ro" says to remount the fs as read-only if the kernel has any errors accessing the filesystem (e.g. if a disk is dead/dying or the cable is loose etc).
debian at least, and probably other distros, routinely adds "errors=remount-ro" to /etc/fstab for ext filesystems when you build the system.
If it is not set in fstab, look at the superblock with tune2fs -l <device> and see what is set for "Errors behavior:"
On Tue, 25 Mar 2014 15:42:06 Allan Duncan wrote:
On 25/03/14 12:58, Craig Sanders wrote:
On Tue, Mar 25, 2014 at 12:27:04PM +1100, Daniel Jitnah wrote:
What can cause this?
most likely the VM's disk image became unavailable temporarily - possibly due to network problems, or a server being rebooted.
Also corruption of storage and RAM are possibilities. The cheaper cloud servers don't have ECC RAM, in the past I've had RAM corruption cause filesystem corruption. I suspect that filesystems such as BTRFS and ZFS can get more messed up in the case of corrupt RAM than simpler filesystems like Ext*. If the system was running for a while "working fine for weeks/months" then it could have been due for a FSCK on the next boot anyway in which case whatever errors there were might have been fixed already. Also in these situations the kernel message log usually has good information about the problem but it can't be written to local disk. So if you don't have syslog over the network to another machine then logging in via ssh and running "dmesg" is the only way to get the data. So by rebooting you probably permanently lost the ability to determine the cause - unless it happens again.
assuming you are using ext2/3/4 on your VM's disk - the mount option "errors=remount-ro" says to remount the fs as read-only if the kernel has any errors accessing the filesystem (e.g. if a disk is dead/dying or the cable is loose etc).
debian at least, and probably other distros, routinely adds "errors=remount-ro" to /etc/fstab for ext filesystems when you build the system.
If it is not set in fstab, look at the superblock with tune2fs -l <device> and see what is set for "Errors behavior:"
Note that you probably don't want to change this. Having the filesystem go read-only in case of problems is a good thing. As you have read access you can login and run programs like dmesg to diagnose it and as you have no write access you don't get further corruption. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
Craig Sanders <cas@taz.net.au> wrote:
assuming you are using ext2/3/4 on your VM's disk - the mount option "errors=remount-ro" says to remount the fs as read-only if the kernel has any errors accessing the filesystem (e.g. if a disk is dead/dying or the cable is loose etc).
debian at least, and probably other distros, routinely adds "errors=remount-ro" to /etc/fstab for ext filesystems when you build the system.
I would suggest running fsck on that file system to repair any metadata corruption that may have occurred.
Craig Sanders <cas@taz.net.au> writes:
(I am thinking the virtual disk hosting the VM has become readonly somehow, but how? )
assuming you are using ext2/3/4 on your VM's disk - the mount option "errors=remount-ro" [...]
...so check your dmesg logs, which after a reboot would probably be something like /var/log/kern.log*.
participants (6)
-
Allan Duncan -
Craig Sanders -
Daniel Jitnah -
Jason White -
Russell Coker -
trentbuck＠gmail.com