On Sat, 24 Mar 2012, Anthony Shipman wrote:
On Sat, 24 Mar 2012 01:56:05 pm Craig Sanders wrote:
BTW, the reason why you see NUL bytes in unflushed sectors after a crash is to avoid leaking data from whatever those sectors were used for before. e.g. say those sectors used to hold the data from /etc/shadow, or a script or config file containing passwords, or some other privacy/security-sensitive data....you wouldn't want that data exposed in, say, a user-owned file after a crash.
after a crash, those sectors may contain some or all of the data you wanted/expected it to, some or all of the data from any previous use, or (most likely) some combination of both. either way, you can't trust that the data is good.
I'm not seeing NUL bytes in the file. I am seeing files with size=0. The most common case is that after rebooting, my kmailrc file has size=0 so all of my e-mail accounts, folders, mailing list configuration etc have been deleted. I've taken to keeping kmail shut down most of the time to reduce the risk of this happening. The same has happened to some other kde configuration files. After each crash I have had to do a "find /home -size 0" to find destroyed files to be restored from backup. This doesn't find all of them though since some are restored to default values after reboot.
kde (and gnome) programs have a long history of failing to do even the most basic of atomic file operations (I'm not talking of failing to do fsync() here, I'm talking simply open(),write(),close() as being their entire file writing sequence) perhaps kmail is frequently updating its dotfile? Get a better desktop environment and mailer :)
On Friday I downloaded a zip file and unpacked it. I then spent some time browsing the HTML documentation in the unpacked directory. Hours later, after the crash, many or most of the HTML files I had read had size=0.
XFS is perfect. You and the multitude of other people over the years that have reported this very same problem of 12 hour old files disappearing, must all be mistaken. QED. XFS has recently taken the place of my backuppc server with its 14million inodes[1]. I like the fact that it has 20GB more space available to it, because somehow it simultaneously uses less space for the metadata, dynamically allocates the inodes, but only uses 7% of the available inodes, whereas the previous ext4 incarnation has been using 25% of the available inodes, and still took 20GB more to store them. But it's slower, I'm worried about the NFS lockup problem Craig mentioned, and I don't like disappearing files, even (especially?) on a backup/archival server. Am I going to spend another month moving it all back to ext4? [1] The transfer took almost a month because it was only making hard links at the rate of about 10 a second[2] [2] hard links are atomic operations as far as the filesystem is concerned, so perhaps every single link involved at least a seek - but god knows why the filesystem operations weren't sorted[3] so it couldn't get slightly better tps, and being atomic at the filesystem level doesn't imply that the inodes need to be forced to disk as barrier calls - and as far as I'm aware, BackupPC_tarPCCopy doesn't make any calls to fsync(). [3] yes, I tried different elevators -- Tim Connors