24 Jan
2014
24 Jan
'14
10:03 a.m.
Assembled cognoscenti;
a general question about file-systems and file fragmentation.
As the price of storage of bytes drops, do file systems really need the
economy of fragmenting files to fill fragmented empty disk space?
That is if there is insufficient space between two existing files,
the new file is just written in the first interstitial space which is
large enough;
or the unfragmented space starting at the last file;
and small interstitial empty spaces are just wasted.
Do any file-systems do this and if not is the reason simply storage
economy ?
regards Rohan McLeod
24 Jan
24 Jan
11:01 a.m.
Rohan McLeod wrote:
a general question about file-systems and file
fragmentation.
What file fragmentation?
It should be negligible unless you're using a legacy fs like FAT, an
unusual workload, or you are routinely filling ext's reserved space
(either by setting it to 0%, or filling the disk as root).
Likewise, naïve defragmentation on ext is trivial:
for each file f
copy f to f'
move f' to f
(For btrfs, zfs and log-oriented filesystems, there is dedicated
infrastructure to coalesce/rebalance/scrub/resilver/&c for you, which
will do a better job.)
If you're asking "why are we wasting code even having this feature",
consider copying a DVD to a disk with 1TB free, but no contiguous free
segment exceeding 4GB. Should the user just be told "just buy a
bigger disk, they're really cheap"? That seems ludicrous to me.
12:16 p.m.
Trent W. Buck wrote:
..........snip
If you're asking "why are we wasting code even having this feature",
No, the question I think I am asking is,
is it really necessary to fragment the file during writing, in the
first place?
Then efficient use of disk space just amounts to defragmenting the free
space; ie arranging the files contiguously.
I don't doubt your contention that FAT is particularly evil in this respect;
or that defragmentation is quite simple in ext or automatic in btrfs and
zf..
consider copying a DVD to a disk with 1TB free, but no
contiguous free
segment exceeding 4GB. Should the user just be told "just buy a
bigger disk, they're really cheap"? That seems ludicrous to me.
On second
thought; I am not even convinced that 'non-fragmented' writes
implies any more than
a temporary loss of space. Can't the OS just write the 4GB DVD to RAM
till the FS makes the free-space contiguous ?
But on the subject of economy , we already accept a trade-off between
disk space utilisation and speed of access etc.;
it seems to be widely accepted that once a disk gets down to about 20%
free-space, it is time to get a bigger drive ?
regards Rohan McLeod
12:42 p.m.
Rohan McLeod <rhn(a)jeack.com.au> wrote:
No, the question I think I am asking is,
is it really necessary to fragment the file during writing, in the
first place?
Then efficient use of disk space just amounts to defragmenting the free
space; ie arranging the files contiguously.
It does, but that's surely where the performance deficit lies, incurred
whenever defragmentation is required.
To make the case, you would have to show that having contiguously allocated
files outweighs the cost of performing the defragmentation. This is most
obviously so if the files are written once, never modified, and read often,
but clearly, most file system usage doesn't follow that pattern.
1:34 p.m.
Jason White wrote:
..............snip
To make the case, you would have to show that having contiguously allocated
files outweighs the cost of performing the defragmentation. This is most
obviously so if the files are written once, never modified, and read often,
but clearly, most file system usage doesn't follow that pattern.
So to your
knowledge:
1/ no FS does this 'contiguous allocation' ?;
2/ the reason being that ' the cost of re-arranging the files
contiguously ',
is greater than 'the cost defragmentation' ?;
thanks for clarifying that , Rohan McLeod
_______________________________________________
luv-talk mailing list
luv-talk(a)lists.luv.asn.au
http://lists.luv.asn.au/listinfo/luv-talk
2:11 p.m.
Rohan McLeod <rhn(a)jeack.com.au> wrote:
So to your knowledge:
1/ no FS does this 'contiguous allocation' ?;
I'm not aware of any.
2/ the reason being that ' the cost of
re-arranging the files
contiguously ',
is greater than 'the cost defragmentation' ?;
I'm confident it would be done if there were a performance benefit.
XFS performs delayed writes to reduce, but not eliminate, fragmentation, as
does ext4 if I remember correctly.
7:06 p.m.
Jason White wrote:
Rohan McLeod <rhn(a)jeack.com.au> wrote:
Jason thanks for replying;
No doubt you are right; but it is far from intuitive.
If I imagine a disk space containing say n groups of contiguous files,
comprising say m bytes in total;
separated by n-1 free-spaces comprising f bytes in total;
then making those groups contiguous, would seem to simply involve,
moving m bytes of n contiguous files,
toward the front of the disk,
so the free-space is moved to the other end of the disk.
Whereas in a file system which allows fragmentation;
if those n-1 free-spaces were occupied by a fragmented file say;
then to get to a contiguous, set of contiguous files would seem to involve
1/ moving the f bytes of the fragmented file to the end of the disk to
defragment it
2/ moving the m bytes of the contiguous files to the front of the disk
3/ moving the f bytes of the defragmented file again, up to the end of
the rest of the contiguous files ?
ie m bytes in the first case and m + 2.f bytes in the second !
regards Rohan McLeod
So to your knowledge:
1/ no FS does this 'contiguous allocation' ?;
I'm not aware of any.
2/ the reason being that ' the cost of
re-arranging the files
contiguously ',
is greater than 'the cost defragmentation' ?;
I'm confident it
would be done if there were a performance benefit. 
9:58 p.m.
In many systems the bottleneck is writing not reading. For example in a typical mail
server writes will outnumber reads because if you have adequate cache then most mail is
read from cache as the users who poll mail the most frequently tend to be the ones who
receive the most mail.
When writes are the bottleneck you don't want to do much to defragment disk allocation
which is good for reads. You want to interleave writes for multiple files to defragment
writes. WAFL, ZFS, and I believe BTRFS do this.
Automatic defragment on BTRFS is new and experimental, I won't use it for a while
except on the most experimental systems. Also I think that the balance bug that affects
systemd systems would also get autodefragment.
--
Sent from my Samsung Galaxy Note 2 with K-9 Mail.
12:27 p.m.
On Sat, 25 Jan 2014 12:25:00 Trent W. Buck wrote:
Russell Coker wrote:
There was a mention in one of the list discussions.
But it's something you can verify yourself. Just run a balance on a filesystem
that has a systemd journal and it will report corruption. Apparently systemd
itself will also report corruption so it's actually corrupting the file data.
--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/
Also I think that the balance bug that affects
systemd systems [...]
I didn't hear about that. Have you a cite?
3352
days inactive
3354
days old
9 comments
4 participants
participants (4)
-
Jason White -
Rohan McLeod -
Russell Coker -
Trent W. Buck