27 Sep
2013
27 Sep
'13
12:13 p.m.
I want to create a filesystem to store my on-disk backups (from Bacula) on a new server.
These backup files will be few (less than 10000) and mostly huge (>1GB). Because I will
have multiple files being written out at once, a large data per inode ratio seems to make
sense as it will greatly reduce fragmentation, and wasted space would be low because of
the small number of files. Also because the write pattern is exclusively streaming writes,
I can go against my normal rule and use RAID5.
I've chosen a 4MB of data per inode ratio based on some rough calculations, but while
my mkfs.ext3 <dev> -i 4194304 just raced through initially, when it got to
"Writing superblocks and filesystem accounting information:" it just seemed to
hang. Strace says it's doing seek, write 4k, seek, write 4k, over and over again. I
hit ^C and the process is now [mkfs.ext3], but the system is still pegged at 100% disk
utilisation.
Any suggestions as to how I could make this go faster? The filesystem is around 8TB (RAID5
of 4 x 3TB disks), so it's not exactly small, and the disks are only 7200RPM SATA, but
I know xfs would complete pretty quick. I'd use xfs but over the years I've used
xfs and ext3 in roughly equal proportions, and I've lost 3 xfs filesystems and no ext3
filesystems, so I'm a little reluctant to commit to it.
Thanks
James
27 Sep
27 Sep
1:33 p.m.
On 27 September 2013 22:13, James Harper <james.harper(a)bendigoit.com.au> wrote:
I've chosen a 4MB of data per inode ratio based on some rough calculations, but while
my mkfs.ext3 <dev> -i 4194304 just raced through initially, when it got to
"Writing superblocks and filesystem accounting information:" it just seemed to
hang. Strace says it's doing seek, write 4k, seek, write 4k, over and over again. I
hit ^C and the process is now [mkfs.ext3], but the system is still pegged at 100% disk
utilisation.
( sorry James for the double reply )
Hi, I am no expert but in case it helps I have used mkfs.ext3 -T
largefile4 in the past without problems on a single 1TB drive and it
was a little faster, not slower, than other mkfs. According to
/etc/mke2fs.conf that is equivalent to
inode_ratio = 4194304
blocksize = -1
so if I was seeing your symptoms I wouldn't be blaming that. Maybe
test it on an individual drive/s first?
28 Sep
28 Sep
8:15 a.m.
On Fri, 27 Sep 2013, James Harper <james.harper(a)bendigoit.com.au> wrote:
I want to create a filesystem to store my on-disk
backups (from Bacula) on
a new server. These backup files will be few (less than 10000) and mostly
huge (>1GB). Because I will have multiple files being written out at once,
a large data per inode ratio seems to make sense as it will greatly reduce
fragmentation, and wasted space would be low because of the small number
of files. Also because the write pattern is exclusively streaming writes,
I can go against my normal rule and use RAID5.
I've chosen a 4MB of data per inode ratio based on some rough calculations,
but while my mkfs.ext3 <dev> -i 4194304 just raced through initially, when
it got to "Writing superblocks and filesystem accounting information:" it
just seemed to hang. Strace says it's doing seek, write 4k, seek, write
4k, over and over again. I hit ^C and the process is now [mkfs.ext3], but
the system is still pegged at 100% disk utilisation.
One of the features of Ext4 is uninit_bg which if enabled at mkfs time should
reduce creation time. I don't have a convenient large storage device to test
this with.
Any suggestions as to how I could make this go faster?
The filesystem is
around 8TB (RAID5 of 4 x 3TB disks), so it's not exactly small, and the
disks are only 7200RPM SATA, but I know xfs would complete pretty quick.
I'd use xfs but over the years I've used xfs and ext3 in roughly equal
proportions, and I've lost 3 xfs filesystems and no ext3 filesystems, so
I'm a little reluctant to commit to it.
I wouldn't use Ext3 for a device that big. Why can't you use ZFS?
I'm using BTRFS for all my backups nowadays so any data corruption will be
flagged.
--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/
29 Sep
29 Sep
12:19 a.m.
I'm using
BTRFS for all my backups nowadays so any data corruption will be
flagged.
And I'm planning on getting familiar with BTRFS at some point.
1:09 a.m.
On Sun, 29 Sep 2013, James Harper <james.harper(a)bendigoit.com.au> wrote:
Actually, revisiting the feature list of BTRFS the
'some point' might be
now. Compression seems particularly attractive as that would take some
load off of the backup client machines
I'm running Debian Wheezy, so I assume I want the 3.10 kernel from
backports.
I don't think that you can ever expect filesystem compression to compete well
with application level compression. Filesystem compression is based around
the expectation of random access while much application compression is based
around entirely compressing large files due to the application/user knowing
that random access isn't required. But if you want random access and
compression then filesystem compression apparently works well.
I'm not even sure if I have compression enabled on my systems, the data I'm
storing on BTRFS isn't going to compress well anyway.
For Debian you want a newer kernel than is available in Wheezy. The Wheezy
kernel does work and if you avoid getting the filesystem anywhere near full
then it will work well. But later kernels have many bug fixes.
Don't use BTRFS RAID-5, that's no-where near usable. If you want data
integrity and RAID-5 then there is no option other than ZFS at this time.
--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/
1:49 a.m.
On Sun, 29 Sep 2013, James Harper
<james.harper(a)bendigoit.com.au>
wrote:
The reason I want to do it on the filesystem is that bacula's compression sucks the
performance out of the machine being backed up. It has LZO now which is better, but in the
little testing I've done, btrfs is faster still.
If the end result is a compression ratio that really sucks then I'll revisit it.
Actually, revisiting the feature list of BTRFS
the 'some point' might be
now. Compression seems particularly attractive as that would take some
load off of the backup client machines
I'm running Debian Wheezy, so I assume I want the 3.10 kernel from
backports.
I don't think that you can ever expect filesystem compression to compete
well
with application level compression. Filesystem compression is based around
the expectation of random access while much application compression is
based
around entirely compressing large files due to the application/user knowing
that random access isn't required. But if you want random access and
compression then filesystem compression apparently works well.
I'm not even sure if I have compression enabled on my systems, the data I'm
storing on BTRFS isn't going to compress well anyway.
For Debian you want a newer kernel than is available in Wheezy. The
Wheezy
kernel does work and if you avoid getting the filesystem anywhere near full
then it will work well. But later kernels have many bug fixes.
wheezy-backports has 3.10
Don't use BTRFS RAID-5, that's no-where near
usable. If you want data
integrity and RAID-5 then there is no option other than ZFS at this time.
I'm testing RAID5 now but I can't find anything wrt stability beyond the original
post announcing it in 3.8. I'm hoping it is now considered more stable but haven't
found anything to back that up yet, so I may be switching to RAID10, then upgrading to
RAID5 when it is considered stable.
I'll be pulling the power and pulling a disk out on the machine soon anyway to see
what happens.
Thanks
James
1:02 a.m.
On Sat, Sep 28, 2013 at 06:15:02PM +1000, Russell Coker wrote:
I wouldn't use Ext3 for a device that big. Why
can't you use ZFS?
yep, for disks and array that size, error-detection and correction are
essential. which means either zfs or btrfs.
I'm using BTRFS for all my backups nowadays so any
data corruption will be
flagged.
how's the raid5 support in btrfs now? last i read (back in june, iirc)
it was still very experimental. one of the problems i read about was
that you couldn't replace a failed drive in a raid5 btrfs.
btrfs raid-0/1/10 modes work fine - and btrfs does offer a way of
auto-converting an array from raid1 to raid5/6, so starting with raid10
and converting to raid5 when its "ready" may be a safer choice.
or zfs, with the -dkms packages available for most distros, installing
zfs and setting it up is easy.
craig
--
craig sanders <cas(a)taz.net.au>
1 Oct
1 Oct
2:25 a.m.
Craig Sanders <cas(a)taz.net.au> writes:
On Sat, Sep 28, 2013 at 06:15:02PM +1000, Russell
Coker wrote:
Some fedora-using guy on #btrfs was running btrfs as a DKMS, with his
stock stable fedora kernel. Dunno if that was a hand-rolled DKMS; I
haven't seen standard btrfs-dkms packages in Debian experimental.
I wouldn't use Ext3 for a device that big.
Why can't you use ZFS?
or zfs, with the -dkms packages available for most
distros, installing
zfs and setting it up is easy.
2:43 a.m.
Craig Sanders <cas(a)taz.net.au> writes:
I've done the same thing before when backporting other things (normally xen related).
It normally requires very little effort, but assumes that the code is entirely
self-contained and doesn't require patches to other areas of the kernel. Given the
size and complexity of btrfs I suspect this might be a bit trickier and probably
impossible to go port back too far from the kernel of origin.
Fortunately wheezy-backports has a 3.10 kernel which should be new enough.
James
On Sat, Sep 28, 2013 at 06:15:02PM +1000, Russell
Coker wrote:
Some fedora-using guy on #btrfs was running btrfs as a DKMS, with his
stock stable fedora kernel. Dunno if that was a hand-rolled DKMS; I
haven't seen standard btrfs-dkms packages in Debian experimental.
I wouldn't use Ext3 for a device that big.
Why can't you use ZFS?
or zfs, with the -dkms packages available for most
distros, installing
zfs and setting it up is easy.
4:37 a.m.
On Tue, Oct 01, 2013 at 12:25:37PM +1000, Trent W. Buck wrote:
Craig Sanders <cas(a)taz.net.au> writes:
huh?
you don't need a dkms package for btrfs - it's built-in to the mainline
kernel, and has been for quite a while.
zfs, though, is not (and probably never will be due to license
incompatibility between CDDL and GPL) part of the standard linux kernel
so you do need to compile an extra module.
installing a zfs-dkms package is the easiest (and completely automated)
way to do that.
Oracle as the copyright holder on the Sun-developed ZFS could solve the
license problem by re-licensing it as GPL or BSD (preferably BSD so that
FreeBSD and Illumos etc could use it too) but that's extremely unlikely
to happen.
They're the main copyright holder for both of the current
modern/advanced filesystems for unix & linux - zfs and btrfs - and seem
to have lost interest in both of them. btrfs is GPL and development
is within the linux kernel (and fusion-io, where the btrfs author now
works). zfs is CDDL and illumos etc have forked it.
craig
--
craig sanders <cas(a)taz.net.au>
BOFH excuse #218:
The UPS doesn't have a battery backup.
On Sat, Sep 28, 2013 at 06:15:02PM +1000, Russell
Coker wrote:
Some fedora-using guy on #btrfs was running btrfs as a DKMS, with his
stock stable fedora kernel. Dunno if that was a hand-rolled DKMS; I
haven't seen standard btrfs-dkms packages in Debian experimental. I wouldn't use Ext3 for a device that big.
Why can't you use ZFS?
or zfs, with the -dkms packages available for most
distros, installing
zfs and setting it up is easy. 
4:43 a.m.
Hi,
On 30/09/2013, at 9:37 PM, Craig Sanders <cas(a)taz.net.au> wrote:
They're the main copyright holder for both of the
current
modern/advanced filesystems for unix & linux - zfs and btrfs - and seem
to have lost interest in both of them.
This is completely untrue. Both ZFS and btrfs remain at the top of our development
priorities. Oracle employs several developers for both, including several of the top btrfs
developers (who were hired by Chris Mason before he moved to fusion-IO). I can't speak
for the ZFS development as that's not part of my sphere of responsibilities, but btrfs
was the single largest piece of development done for our new Unbreakable Enterprise Kernel
Release 2 (which is currently in beta).
Just wanted to clear that up.
Cheers,
Avi
5:08 a.m.
On 30/09/2013, at 9:43 PM, Avi Miller <avi.miller(a)gmail.com> wrote:
I can't speak for the ZFS development as
that's not part of my sphere of responsibilities, but btrfs was the single largest
piece of development done for our new Unbreakable Enterprise Kernel Release 2 (which is
currently in beta).
Typo: should've been Release 3.
1:10 p.m.
On Mon, Sep 30, 2013 at 09:43:53PM -0700, Avi Miller wrote:
On 30/09/2013, at 9:37 PM, Craig Sanders
<cas(a)taz.net.au> wrote:
then they're doing a damn good job of making sure that they have little
control or even influence over the direction that future development
takes - oracle's actions have pretty much forced illumos to fork zfs,
and much of btrfs development seems to be taking place outside of oracle
too.
which is a shame, because funding and supporting the work on btrfs is
one of the really good things that oracle has done.
my suspicion is that oracle execs have no idea how to monetise either
zfs or btrfs or use them as leverage to control linux, so don't see
either as a priority and don't have a clue what to do with either of
them.
oracle geeks are probably different, but geeks don't make
important decisions in corporates like oracle. suits do.
craig
--
craig sanders <cas(a)taz.net.au>
They're the main copyright holder for both of
the current
modern/advanced filesystems for unix & linux - zfs and btrfs - and
seem to have lost interest in both of them.
This is completely untrue. Both ZFS and btrfs remain at the top of our
development priorities.
2:24 p.m.
On 01/10/2013, at 6:10 AM, Craig Sanders <cas(a)taz.net.au> wrote:
my suspicion is that oracle execs have no idea how to
monetise either
zfs or btrfs or use them as leverage to control linux, so don't see
either as a priority and don't have a clue what to do with either of
them.
Or, perhaps Oracle execs don't wish to monetise either or use them to leverage Linux
and would rather just pay for the work to be done without exerting what others see as
excessive control?
oracle geeks are probably different, but geeks
don't make
important decisions in corporates like oracle. suits do.
Actually, the Senior Vice-President of Linux Engineering at Oracle is a massive geek. He
hacks on the kernel in his spare time (under a pseudonym) and has code in almost every
product he manages. He's also 2-down from Larry and reports directly to our Chief
Corporate Architect. So, we don't have a suit in charge of Oracle Linux, we have a
geek. Hence, we have no desire to monetise our contributions. Hell, we're the only
enterprise Linux that gives away the ISO, the updates, the errata, the bug fixes and the
feature development for free.
Say what you will about Oracle as a whole (and I've heard most of it all), but the
Oracle Linux/VM product teams and associated upstream mainline developers are not typical
of the rest of the organisation at all.
Cheers,
Avi
3 Oct
3 Oct
5:05 a.m.
On Tue, 1 Oct 2013 07:24:56 Avi Miller wrote:
You should probably give a LUV talk about how these things work.
--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/
oracle geeks
are probably different, but geeks don't make
important decisions in corporates like oracle. suits do.
Actually, the Senior Vice-President of Linux Engineering at Oracle is a
massive geek. He hacks on the kernel in his spare time (under a pseudonym)
and has code in almost every product he manages. He's also 2-down from
Larry and reports directly to our Chief Corporate Architect. So, we don't
have a suit in charge of Oracle Linux, we have a geek. Hence, we have no
desire to monetise our contributions. Hell, we're the only enterprise Linux
that gives away the ISO, the updates, the errata, the bug fixes and the
feature development for free.
Say what you will about Oracle as a whole (and I've heard most of it all),
but the Oracle Linux/VM product teams and associated upstream mainline
developers are not typical of the rest of the organisation at all.
2 Oct
2 Oct
12:50 a.m.
Craig Sanders <cas(a)taz.net.au> writes:
On Tue, Oct 01, 2013 at 12:25:37PM +1000, Trent W.
Buck wrote:
Except then if you have an old (stable) kernel, you also have an old
btrfs.
Some fedora-using guy on #btrfs was running btrfs
as a DKMS, with his
stock stable fedora kernel. Dunno if that was a hand-rolled DKMS; I
haven't seen standard btrfs-dkms packages in Debian experimental.
huh? you don't need a dkms package for btrfs - it's built-in to the
mainline kernel, and has been for quite a while.
2:21 a.m.
On Wed, Oct 02, 2013 at 10:50:34AM +1000, Trent W. Buck wrote:
Craig Sanders <cas(a)taz.net.au> writes:
that's a problem with an easy answer: "Don't Do That, Then!"
really, just compile the latest kernel or use a backported kernel
package for the distro of your choice.
messing around with dkms just to avoid doing that strikes me as
being somewhat insane (perhaps due to excessive exposure to overly
bureaucratic change management processes - you're not allowed to
change the kernel so you have to pretend you're not actually changing
the kernel, you're just updating one filesystem driver in it and then
pretending that nothing else could possibly be affected)
craig
--
craig sanders <cas(a)taz.net.au>
huh? you don't need a dkms package for btrfs
- it's built-in to the
mainline kernel, and has been for quite a while.
Except then if you have an old (stable) kernel, you also have an old
btrfs.
2:42 a.m.
On Wed, Oct 02, 2013 at 10:50:34AM +1000, Trent W. Buck wrote:
Using a kernel that you know works perfectly well and backporting a few specific
changes/fixes into it is an entirely reasonable thing to do.
James
Craig Sanders <cas(a)taz.net.au> writes:
that's a problem with an easy answer: "Don't Do That, Then!"
really, just compile the latest kernel or use a backported kernel
package for the distro of your choice.
messing around with dkms just to avoid doing that strikes me as
being somewhat insane (perhaps due to excessive exposure to overly
bureaucratic change management processes - you're not allowed to
change the kernel so you have to pretend you're not actually changing
the kernel, you're just updating one filesystem driver in it and then
pretending that nothing else could possibly be affected)
huh? you don't need a dkms package for btrfs
- it's built-in to the
mainline kernel, and has been for quite a while.
Except then if you have an old (stable) kernel, you also have an old
btrfs.
3:02 p.m.
On Wed, Oct 02, 2013 at 02:42:54AM +0000, James Harper wrote:
Using a kernel that you know works perfectly well and
backporting a
few specific changes/fixes into it is an entirely reasonable thing to
do.
replacing an entire file-system driver with one that's about 5 or 6
kernel versions newer is not "backporting a few specific changes/fixes".
it's not a trivial change, and there's a good chance that fixes/changes
in the new btrfs rely on other fixes and changes in the rest of the
kernel.
worse, you have a kernel+btrfs combo that's unique - which means
difficult for anyone else to help you debug or even duplicate any
problem, because nobody else has the exact same combination of code, so
reporting bugs is almost useless. this is not just a btrfs issue, it's a
general problem with backporting instead of upgrading.
sometimes it's worth doing anyway - but as a last resort, when there are
no other good or reasonable options.
craig
--
craig sanders <cas(a)taz.net.au>
1 Oct
1 Oct
2:22 a.m.
Russell Coker <russell(a)coker.com.au> writes:
One of the features of Ext4 is uninit_bg which if
enabled at mkfs time
should reduce creation time.
This can be turned on for ext3: "man mke2fs.conf" & /etc/mke2fs.conf for
default profiles, or mke2fs -O to override it for a single run.
I did this at least once for a large filesystem either with ext3, or
with ext4 before lazy init became the default on ext4. Worked for me.
30 Sep
30 Sep
3:26 a.m.
On 27 September 2013 22:13, James Harper <james.harper(a)bendigoit.com.au> wrote:
I want to create a filesystem to store my on-disk
backups (from Bacula) on a new server. These backup files will be few (less than 10000)
and mostly huge (>1GB). Because I will have multiple files being written out at once, a
large data per inode ratio seems to make sense as it will greatly reduce fragmentation,
and wasted space would be low because of the small number of files. Also because the write
pattern is exclusively streaming writes, I can go against my normal rule and use RAID5.
I've chosen a 4MB of data per inode ratio based on some rough calculations, but while
my mkfs.ext3 <dev> -i 4194304 just raced through initially, when it got to
"Writing superblocks and filesystem accounting information:" it just seemed to
hang. Strace says it's doing seek, write 4k, seek, write 4k, over and over again. I
hit ^C and the process is now [mkfs.ext3], but the system is still pegged at 100% disk
utilisation.
Why aren't you using ext4? It has improvements for handling large
files (extents), among other things.
Although I would have chosen zfs or btrfs for that task myself, unless
I was stuck on something like RHEL :/
3352
days inactive
3358
days old
22 comments
7 participants
participants (7)
-
Avi Miller -
Craig Sanders -
David -
James Harper -
Russell Coker -
Toby Corkindale -
trentbuck＠gmail.com