[luv-main] deleting files on NFS
I have a Debian/Squeeze NFS server running kernel 2.6.32-5-xen-amd64 and a Debian/Unstable NFS client. I just did "rm -rf" on a directory containing 4.2G of files that was mounted via NFS. It was a directory with three large files which were MP4 files captured from digital TV, I had watched them all recently with mplayer. df didn't report any more free space. To get the space back I ended up commenting out the line from /etc/exports, running "exportfs -r", then umounting and mounting it. I didn't check whether the space was free after running "exportfs -r", it probably was. It seems that I was caught by the issue of open file handles. NFS is stateless so it doesn't necessarily process deletion for some time to handle the case of a NFS client that is running processes with open file handles. How long is this supposed to go for? Can I force NFS to give up such caching when I need the space sooner than NFS will free it? -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
On 26.10.11 22:32, Russell Coker wrote:
To get the space back I ended up commenting out the line from /etc/exports, running "exportfs -r", then umounting and mounting it. I didn't check whether the space was free after running "exportfs -r", it probably was.
Wouldn't it have been enough to unmount & remount? That's long [1] been enough to clear stale NFS file handles for me. (And so I'd expect df to then report OK for you.)
It seems that I was caught by the issue of open file handles. NFS is stateless so it doesn't necessarily process deletion for some time to handle the case of a NFS client that is running processes with open file handles.
How long is this supposed to go for? Can I force NFS to give up such caching when I need the space sooner than NFS will free it?
Don't know. I gave up waiting, and whumped it when the users complained. Erik [1] Used to use NFS a lot more nearly 20 years back. -- It is wrong for a man to say he is certain of the objective truth of a proposition unless he can provide evidence which logically justifies that certainty. - Thomas Henry Huxley
On Wed, 26 Oct 2011, Erik Christiansen <dvalin@internode.on.net> wrote:
On 26.10.11 22:32, Russell Coker wrote:
To get the space back I ended up commenting out the line from /etc/exports, running "exportfs -r", then umounting and mounting it. I didn't check whether the space was free after running "exportfs -r", it probably was.
Wouldn't it have been enough to unmount & remount? That's long [1] been enough to clear stale NFS file handles for me. (And so I'd expect df to then report OK for you.)
umount/mount on client? No, umounting on the client didn't work and couldn't work unless the server kept track of which clients had previously used the files in question. It should be possible for the server to do such tracking when the server has enough uptime for client caches to have expired (which would surely happen in less than 2 weeks). But I guess that they don't do such things. Does NFSv4 fix such things? -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
On 26.10.11 23:44, Russell Coker wrote:
umount/mount on client?
No, umounting on the client didn't work and couldn't work unless the server kept track of which clients had previously used the files in question.
True. Now if I could just rattle decades-old memory to recall what I did on the server. It certainly didn't involve editing configs. Ah, shoulda rummaged in extra-cranial memory before the last post. In those _Solaris days_, I made a note: » nfsd may remain oblivious to changes, until: /etc/init.d/nfs-kernel-server restart # Or sufficient?: /etc/init.d/nfs-kernel-server reload # which simply does: /usr/sbin/exportfs -r # Sync /var/lib/nfs/xtab # with /etc/exports. « Sleep beckons, so I won't install NFS, and try it on linux just now, but it ought to be reasonably similar, I expect.
It should be possible for the server to do such tracking when the server has enough uptime for client caches to have expired (which would surely happen in less than 2 weeks). But I guess that they don't do such things.
True too. In a half-ideal *nix world, the period ought to be settable in the config. Anyway, the space should have been free immediately, my brain tells me (at 1 a.m. admittedly). The server was in need of a whack before the file deletion, I think. Erik -- Give a man a computer program and you give him a headache, but teach him to program computers and you give him the power to create headaches for others for the rest of his life. - R. B. Forest
On Wed, 26 Oct 2011, Russell Coker wrote:
On Wed, 26 Oct 2011, Erik Christiansen <dvalin@internode.on.net> wrote:
On 26.10.11 22:32, Russell Coker wrote:
To get the space back I ended up commenting out the line from /etc/exports, running "exportfs -r", then umounting and mounting it. I didn't check whether the space was free after running "exportfs -r", it probably was.
Wouldn't it have been enough to unmount & remount? That's long [1] been enough to clear stale NFS file handles for me. (And so I'd expect df to then report OK for you.)
umount/mount on client?
No, umounting on the client didn't work and couldn't work unless the server kept track of which clients had previously used the files in question. It should be possible for the server to do such tracking when the server has enough uptime for client caches to have expired (which would surely happen in less than 2 weeks). But I guess that they don't do such things.
Does NFSv4 fix such things?
I have never come across this issue, and we use nfs *a lot*. Create files on one computer, delete them on a different computer. Never any issue. Are you absolutely sure any client hasn't still got the file open, causing the files to be sillyrename()d to .nfs???????? (as a sysadmin, I never use bare "ls". I'm always interested in dotfiles. Which is why HP-UX's treatment of "-A" is a pox upon something poxy) The files weren't hard linked somewhere else and you didn't notice? Mind you, redhat's second last kernel ( 2.6.18-274.el5 ) has intersting and impossible nfs behaviour. I believe default metadata caching time is 2 seconds and data for 60 seconds (all controlled by the client). When our webserver's nfs servers are updated by rsync, a new temporary filename is created then renamed to overwrite the original file. New inode, new metadata. But just occasionally (a few hours after upgrade, and then 2 months later), some of the webservers keep serving out old copies of a file, and some of them serve out new copies. At best, an old file might be expected to persist for 2 seconds until the metadata is refreshed. If somehow rsync screwed up (I've checked its flags, and no it doesn't overwrite files in place) and rewrote the file in place, the old data might be expected to hang around for a minute. But 10 minutes later, having to manually remove and recopy a file on the nfs server? I've tried doing loops of creation/deletion/rename etc but never reproduced the result, so tracking these down or even declaring it "solved" at some future kernel upgrade is going to be extremely difficult. I haven't even bothered reporting a bug, because where do I start? We've got 1.5 million files on those webservers, and have had 2 detected instances so far of old files being served out. -- Tim Connors
Quoting "Russell Coker" <russell@coker.com.au>:
I have a Debian/Squeeze NFS server running kernel 2.6.32-5-xen-amd64 and a Debian/Unstable NFS client.
I just did "rm -rf" on a directory containing 4.2G of files that was mounted via NFS. It was a directory with three large files which were MP4 files captured from digital TV, I had watched them all recently with mplayer. df didn't report any more free space.
To get the space back I ended up commenting out the line from /etc/exports, running "exportfs -r", then umounting and mounting it. I didn't check whether the space was free after running "exportfs -r", it probably was.
It seems that I was caught by the issue of open file handles. NFS is stateless so it doesn't necessarily process deletion for some time to handle the case of a NFS client that is running processes with open file handles.
How long is this supposed to go for? Can I force NFS to give up such caching when I need the space sooner than NFS will free it?
As long as it isn't locked by another client (and then you're delete wouldn't have worked), the file should have been deleted on the server side, more or less immediately. ([1]depending on the asynchronous or synchronous mount option the server returns before or after local deletion [2] exception, it isn't deleted if it is opened locally by a process on the server) Other clients would get a stale file handle afterwards, probably not immediately if they cache on their end and do not write to it. At least that's how I remember correct "NFS manners" but the reality of NFS implementations interfered quite often with the idea how NFS _should_ work. Anyway, after looking through the Linux manpages, I cannot see any tuning parameter that could impact the server to work in the way you experienced. It looks more like a bug than a feature to me. Regards Peter
participants (4)
-
Erik Christiansen -
Peter Ross -
Russell Coker -
Tim Connors