I'd like to report a strange problem. I was backing up my desktop last night to an external usb disk last night. The backup failed to complete because I completely filled up the disk. df -h /dev/sdf1 1.4T 1.4T 0 100% /media/dz/5540d70d-... So this morning I deleted (using sudo rm -rf) an old directory on the usb disk which had about 30 GB of data in it. Now the problem is that df still reports the disk is completely full -- even after rebooting. /dev/sdf1 1.4T 1.4T 0 100% /media/dz/5540d70d-... Can someone please explain to what is going on here? Thanks David Zuccaro BSc (Chemistry, University Of Melbourne), GradDipCompSci (Monash), GradDipEd (Melbourne)
On Thu, 20 Aug 2015 02:36:03 PM David Zuccaro wrote:
So this morning I deleted (using sudo rm -rf) an old directory on the usb disk which had about 30 GB of data in it. Now the problem is that df still reports the disk is completely full -- even after rebooting.
/dev/sdf1 1.4T 1.4T 0 100% /media/dz/5540d70d-...
Can someone please explain to what is going on here?
The first what crossed my mind is that deleted directory was only moved to trash. Did you checked it? Petr -- <pb-luv@baum.com.au> Petr Baum, P.O.Box 2364, Rowville 3178 fax +61-3-97643342 This message was created in naturally virus-free operating system: Linux
Try looking for a hidden folder on the drive Nic On 20/08/15 14:51, David Zuccaro wrote:
On 20/08/15 14:45, Petr Baum wrote:
The first what crossed my mind is that deleted directory was only moved to trash. Did you checked it?
Petr There is no trash directory on the usb disk and I emptied the trash folder on my desktop.
luv-main mailing list luv-main@luv.asn.au http://lists.luv.asn.au/listinfo/luv-main
that is what I meant On 20/08/15 15:19, David Zuccaro wrote:
On 20/08/15 14:55, Nic Baxter wrote:
Try looking for a hidden folder on the drive
If you mean a directory name prefixed with a dot then I have done that at the root level at least. _______________________________________________ luv-main mailing list luv-main@luv.asn.au http://lists.luv.asn.au/listinfo/luv-main
David Zuccaro wrote:
On 20/08/15 14:45, Petr Baum wrote:
The first what crossed my mind is that deleted directory was only moved to trash. Did you checked it?
Petr There is no trash directory on the usb disk and I emptied the trash folder on my desktop.
I take it there is no TRIM issue where the USB controller is failing to report , deleted file space as free-space to the OS ? : regards Rohan McLeod
On 20/08/15 15:06, Rohan McLeod wrote:
I take it there is no TRIM issue where the USB controller is failing to report , deleted file space as free-space to the OS ? : regards Rohan McLeod
It could be a TRIM issue Rohan. I just would have thought that if I delete a directory containing 30GB it would be reflected by the df command or has something changed somewhere along the way?
David Zuccaro wrote:
On 20/08/15 15:06, Rohan McLeod wrote:
I take it there is no TRIM issue where the USB controller is failing to report , deleted file space as free-space to the OS ? : regards Rohan McLeod
It could be a TRIM issue Rohan. I just would have thought that if I delete a directory containing 30GB it would be reflected by the df command or has something changed somewhere along the way?
Checking back to your original post I notice you don't mention the OS distribution or version; presumably no current versions would have TRIM issues ? regards Rohan McLeod
On 20/08/15 15:31, Rohan McLeod wrote:
Checking back to your original post I notice you don't mention the OS distribution or version; presumably no current versions would have TRIM issues ? uname -a Linux capua 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1 (2015-05-24) x86_64 GNU/Linux
On Thu, 20 Aug 2015 03:14:54 PM David Zuccaro wrote:
On 20/08/15 15:06, Rohan McLeod wrote:
I take it there is no TRIM issue where the USB controller is failing to report ,
deleted file space as free-space to the OS ?
regards Rohan McLeod
It could be a TRIM issue Rohan. I just would have thought that if I delete a directory containing 30GB it would be reflected by the df command or has something changed somewhere along the way?
TRIM has no affect on the output of df. df reports disk space allocated by the filesystem, TRIM is below the filesystem level. Flash devices have their own "filesystem" to present what looks like a regular block device to the OS, TRIM frees space on that "filesystem" which is not seen by the OS. If you have flash hardware that doesn't have such a "filesystem" then you need to use a Linux filesystem like JFFS2 which knows how flash hardware works. rm -rf doesn't use a trash folder unless you have something unusual going on. Deleting files doesn't free space if the files are hard-linked somewhere else. rsync can preserve hard-links if the -H option is used. Also when I'm doing a rsync backup of a new system I usually start with "cp -rl" or "cp -a --reflink=always" of the backup of another system running the same distribution. That initially saves a few gig of disk space and network transfer, if the 2 systems in question are running a stable release of Linux then over the next few years there probably won't be many changes made so the space savings will continue. If you do this then rm -rf of one backup won't free the space that was shared with another backup. The Ext2/3/4 filesystems reserve some space for root. This space isn't reported as free by df but can be used by root not regular users. You probably run backups as root so this space would be used. tune2fs(8) says that 5% of disk space is reserved by default, 5% of 1.4T is 70G so if you delete 30G of files from a 1.4T Ext2/3/4 filesystem that was really full then df wouldn't report any free space. Run tune2fs -m0 /dev/whatever to fix this. While reserved space is good for the case where home directories are on the root filesystem it does no good for a backup filesystem. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
What does du -sh /media/dz shows ? Regards Majoosh On Thu, Aug 20, 2015 at 2:36 PM, David Zuccaro < david.zuccaro@optusnet.com.au> wrote:
I'd like to report a strange problem. I was backing up my desktop last night to an external usb disk last night. The backup failed to complete because I completely filled up the disk.
df -h /dev/sdf1 1.4T 1.4T 0 100% /media/dz/5540d70d-...
So this morning I deleted (using sudo rm -rf) an old directory on the usb disk which had about 30 GB of data in it. Now the problem is that df still reports the disk is completely full -- even after rebooting.
/dev/sdf1 1.4T 1.4T 0 100% /media/dz/5540d70d-...
Can someone please explain to what is going on here?
Thanks
David Zuccaro
BSc (Chemistry, University Of Melbourne), GradDipCompSci (Monash), GradDipEd (Melbourne)
_______________________________________________ luv-main mailing list luv-main@luv.asn.au http://lists.luv.asn.au/listinfo/luv-main
On Thu, 20 Aug 2015 at 15:16 David Zuccaro <david.zuccaro@optusnet.com.au> wrote:
du -sh /media/dz/5540d70d-f601-4d1a-a241-80e02ce439f3 1.4T /media/dz/5540d70d-f601-4d1a-a241-80e02ce439f3
What does the following command show? i.e. remove the -s summary flag. du -h /media/dz If there was a hidden .Trash directory, this would show it. Seems unlikely however if you used rm to remove the files.
On Thu, 20 Aug 2015 02:36:03 PM David Zuccaro wrote:
I'd like to report a strange problem. I was backing up my desktop last night to an external usb disk last night. The backup failed to complete because I completely filled up the disk.
What does this say? grep /media/dz /proc/mounts -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC
On 20/08/15 19:55, Chris Samuel wrote:
On Thu, 20 Aug 2015 02:36:03 PM David Zuccaro wrote:
I'd like to report a strange problem. I was backing up my desktop last night to an external usb disk last night. The backup failed to complete because I completely filled up the disk. What does this say?
grep /media/dz /proc/mounts
grep /media/dz /proc/mounts /dev/sdf1 /media/dz/5540d70d-f601-4d1a-a241-80e02ce439f3 ext3 rw,nosuid,nodev,relatime,data=ordered 0 0 /dev/sdg1 /media/dz/37d7146b-476b-4c6f-b3d9-454352623a3d ext3 rw,nosuid,nodev,relatime,data=ordered 0 0
On Thu, 20 Aug 2015 09:40:28 PM David Zuccaro wrote:
/dev/sdf1 /media/dz/5540d70d-f601-4d1a-a241-80e02ce439f3 ext3 rw,nosuid,nodev,relatime,data=ordered 0 0
OK, so it's a standard ext3 filesystem. Have you tried unmounting it and running fsck on it, just in case? cheers! Chris -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC
On 20/08/15 21:48, Chris Samuel wrote:
On Thu, 20 Aug 2015 09:40:28 PM David Zuccaro wrote:
/dev/sdf1 /media/dz/5540d70d-f601-4d1a-a241-80e02ce439f3 ext3 rw,nosuid,nodev,relatime,data=ordered 0 0 OK, so it's a standard ext3 filesystem. Have you tried unmounting it and running fsck on it, just in case? cheers! Chris Seems ok fsck /dev/sdf1 fsck from util-linux 2.25.2 e2fsck 1.42.12 (29-Aug-2014) /dev/sdf1 has been mounted 874 times without being checked, check forced. Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/sdf1: 1117706/91578368 files (6.2% non-contiguous), 366230616/366283776 blocks
On Thu, 20 Aug 2015 10:37:34 PM David Zuccaro wrote:
On 20/08/15 21:48, Chris Samuel wrote:
OK, so it's a standard ext3 filesystem. Have you tried unmounting it and running fsck on it, just in case? cheers! Chris
Seems ok
Did that change the space available at all after remounting it? Anything in the lost+found directory on it? cheers, Chris -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC
David Zuccaro writes:
df -h /dev/sdf1 1.4T 1.4T 0 100% /media/dz/5540d70d-...
[I deleted] 30 GB of data [...] df still reports the disk is completely full
By default mke2fs reserved 5% of the filesystem for uid 0. This is also used as a staging area to reduce fragmentation. df does not include this area in the "free" size, so if you fill a disk *as root*, you must remove >5% for df to show any change. 30G is about 2% of 1.4T, so that's not enough. You can remove this reserved area using tune2fs, but that is a bad idea.
On 21/08/15 12:24, Trent W. Buck wrote:
By default mke2fs reserved 5% of the filesystem for uid 0. This is also used as a staging area to reduce fragmentation.
df does not include this area in the "free" size, so if you fill a disk *as root*, you must remove >5% for df to show any change.
30G is about 2% of 1.4T, so that's not enough.
You can remove this reserved area using tune2fs, but that is a bad idea.
Thanks Trent, that would explain the anomaly. I did not know that that the 5% reserved area was accessible to root (I was running the back up as root). Would it be ok to reduce the reserved area to say 1%?
David Zuccaro <david.zuccaro@optusnet.com.au> writes:
On 21/08/15 12:24, Trent W. Buck wrote:
By default mke2fs reserved 5% of the filesystem for uid 0. This is also used as a staging area to reduce fragmentation.
df does not include this area in the "free" size, so if you fill a disk *as root*, you must remove >5% for df to show any change.
30G is about 2% of 1.4T, so that's not enough.
You can remove this reserved area using tune2fs, but that is a bad idea.
Thanks Trent, that would explain the anomaly. I did not know that that the 5% reserved area was accessible to root (I was running the back up as root). Would it be ok to reduce the reserved area to say 1%?
It won't hurt. Without reserved space, you will get more fragmentation (if you repeatedly fill the filesystem). BUT you're running the backup as root, so it effectively ignores that reserved space ANYWAY. e2fsck -f reports fragmentation as a percentage, but I'm not sure how believable that number is. * * * You could just put up with it. IME fragmentation isn't a big deal on ext. You could change the reserved user ID from 0 (using tune2fs), so the root-owned rsync can't use the reserved space. You could run a defrag program occasionally. This mostly consists of writing a new copy of each file, along these lines (untested, pre-coffee): nice ionice -c3 find -O3 /srv/backup -xdev -type f -links 1 -size +1M \ -exec cp -v {} {}~ \; \ -exec mv -v {}~ {} \; ...but it is easy to break unusual files (e.g. hard-linked, sparse) without realizing until it's too late. Last time I looked, all of the ext defrag programs were written by recent Windows refugees and I didn't trust them to get edge cases right any better than the above.
BTW: I am reducing it to 2% since.. long long time. I haven't seen an issue arising from this at all. regards Peter On Mon, Aug 24, 2015 at 10:17 AM, Trent W. Buck <trentbuck@gmail.com> wrote:
David Zuccaro <david.zuccaro@optusnet.com.au> writes:
On 21/08/15 12:24, Trent W. Buck wrote:
By default mke2fs reserved 5% of the filesystem for uid 0. This is also used as a staging area to reduce fragmentation.
df does not include this area in the "free" size, so if you fill a disk *as root*, you must remove >5% for df to show any change.
30G is about 2% of 1.4T, so that's not enough.
You can remove this reserved area using tune2fs, but that is a bad idea.
Thanks Trent, that would explain the anomaly. I did not know that that the 5% reserved area was accessible to root (I was running the back up as root). Would it be ok to reduce the reserved area to say 1%?
It won't hurt.
Without reserved space, you will get more fragmentation (if you repeatedly fill the filesystem).
BUT you're running the backup as root, so it effectively ignores that reserved space ANYWAY.
e2fsck -f reports fragmentation as a percentage, but I'm not sure how believable that number is.
* * *
You could just put up with it. IME fragmentation isn't a big deal on ext.
You could change the reserved user ID from 0 (using tune2fs), so the root-owned rsync can't use the reserved space.
You could run a defrag program occasionally. This mostly consists of writing a new copy of each file, along these lines (untested, pre-coffee):
nice ionice -c3 find -O3 /srv/backup -xdev -type f -links 1 -size +1M \ -exec cp -v {} {}~ \; \ -exec mv -v {}~ {} \;
...but it is easy to break unusual files (e.g. hard-linked, sparse) without realizing until it's too late.
Last time I looked, all of the ext defrag programs were written by recent Windows refugees and I didn't trust them to get edge cases right any better than the above.
_______________________________________________ luv-main mailing list luv-main@luv.asn.au http://lists.luv.asn.au/listinfo/luv-main
Peter Ross <petrosssit@gmail.com> writes:
BTW: I am reducing it to 2% since.. long long time. I haven't seen an issue arising from this at all.
Hm, for the purposes of avoiding fragmentation, I suppose the reserve amount needs to be proportional to <bytes written per interval> rather than the disk size. I assume the disk size has grown faster that write speed, and the 5% default was conservative even when it was introduced. So your result makes intuitive sense to me. OOC, Peter, what fragmentation does e2fsck -f report for your filesystems? With repeated filling as root or with 0% reserve, I've seen it go up to 20% or 30% (IIRC). NB: you can edit mke2fs to make new (ext) fs's default to 2%.
At the moment I have only XFS in production (it is CentOS default) so I cannot give you current information related to ext filesystems. $ sudo xfs_db -r -c frag /dev/sda1 actual 335, ideal 331, fragmentation factor 1.19% Regards Peter On Tue, Aug 25, 2015 at 10:21 AM, Trent W. Buck <trentbuck@gmail.com> wrote:
Peter Ross <petrosssit@gmail.com> writes:
BTW: I am reducing it to 2% since.. long long time. I haven't seen an issue arising from this at all.
Hm, for the purposes of avoiding fragmentation, I suppose the reserve amount needs to be proportional to <bytes written per interval> rather than the disk size.
I assume the disk size has grown faster that write speed, and the 5% default was conservative even when it was introduced.
So your result makes intuitive sense to me.
OOC, Peter, what fragmentation does e2fsck -f report for your filesystems? With repeated filling as root or with 0% reserve, I've seen it go up to 20% or 30% (IIRC).
NB: you can edit mke2fs to make new (ext) fs's default to 2%.
_______________________________________________ luv-main mailing list luv-main@luv.asn.au http://lists.luv.asn.au/listinfo/luv-main
participants (10)
-
Brian May -
Chris Samuel -
David Zuccaro -
majoosh chacko -
Nic Baxter -
Peter Ross -
Petr Baum -
Rohan McLeod -
Russell Coker -
trentbuck＠gmail.com