Does it make sense to compress log files nowadays? BTRFS and ZFS have compression built in, that compression probably won't be as good as gzip on a large file because the filesystems will use smaller block sizes. But it will still provide some benefit. On modern systems writes are more of an issue. Rewriting every log file every night contributes to the wear of SSDs for no real benefit (any system which has log files any fraction of the size of a 60G minimal SSD will probably use ZFS or BTRFS on some big disks for main storage). It seems to me that the default log compression has more problems than benefits with today's technology. It made a lot of sense when 1G was a big hard drive, but nowadays a 60G SSD is about as small as you can buy. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
Russell Coker <russell@coker.com.au> wrote:
It seems to me that the default log compression has more problems than benefits with today's technology. It made a lot of sense when 1G was a big hard drive, but nowadays a 60G SSD is about as small as you can buy.
The old log files are eventually deleted (under standard configurations), hence conserving storage need not be the highest priority. For very high-traffic servers, one simply needs appropriately sized storage for /var, or, alternatively, remote logging.
Does it make sense to compress log files nowadays?
BTRFS and ZFS have compression built in, that compression probably won't be as good as gzip on a large file because the filesystems will use smaller block sizes. But it will still provide some benefit.
On modern systems writes are more of an issue. Rewriting every log file every night contributes to the wear of SSDs for no real benefit (any system which has log files any fraction of the size of a 60G minimal SSD will probably use ZFS or BTRFS on some big disks for main storage).
It seems to me that the default log compression has more problems than benefits with today's technology. It made a lot of sense when 1G was a big hard drive, but nowadays a 60G SSD is about as small as you can buy.
Consider virtual hosting where disk size matters a bit more, because you pay for what you use. To take an example of one of my spam filtering mail relays which sees around 500MB of email a day, and on which I keep 365 days of logs, the logfiles take 380MB compressed but would take 2.3GB uncompressed. If they weren't compressed I'd have to buy more disk than the 10GB I have now. If you're talking desktop systems then I'm not the right person to answer, but around 90% of the Linux servers I maintain are virtual and with only as much disk as they need. I like to keep logfiles for more (sometimes much more) time than the default, so compression makes sense for me. Maybe a Linux desktop which rotates it's logfiles off in a short time and doesn't accumulate much in them anyway would be better configured without logfile compression? I can't see that logfiles would contribute much to overall SSD wear on such a system though. How to SSD's fare with continually appended data anyway? I can't imagine that it would be that kind to the media. Do any filesystems allow deferring the write until you hit an erase-sized block, without deferring more important data? Obviously you risk losing data in the event of a crash, and this is precisely the time that logs are useful :) James
James Harper <james.harper@bendigoit.com.au> wrote:
Consider virtual hosting where disk size matters a bit more, because you pay for what you use. To take an example of one of my spam filtering mail relays which sees around 500MB of email a day, and on which I keep 365 days of logs, the logfiles take 380MB compressed but would take 2.3GB uncompressed. If they weren't compressed I'd have to buy more disk than the 10GB I have now.
True, but Russell's comparison was between compressing the logs with, e.g., gzip or xz, and relying on a file system such as Btrfs to perform the compression (albeit less efficiently). Thus the strength of the above example depends on the size difference between these two cases. The conclusion might still hold, of course, but this isn't obvious without some assumptions and calculations.
James Harper <james.harper@bendigoit.com.au> wrote:
Consider virtual hosting where disk size matters a bit more, because you pay for what you use. To take an example of one of my spam filtering mail relays which sees around 500MB of email a day, and on which I keep 365 days of logs, the logfiles take 380MB compressed but would take 2.3GB uncompressed. If they weren't compressed I'd have to buy more disk than the 10GB I have now.
True, but Russell's comparison was between compressing the logs with, e.g., gzip or xz, and relying on a file system such as Btrfs to perform the compression (albeit less efficiently). Thus the strength of the above example depends on the size difference between these two cases. The conclusion might still hold, of course, but this isn't obvious without some assumptions and calculations.
His opening paragraph was "Does it make sense to compress log files nowadays?" which I took to mean "compress at all by any means" so I answered that. Following that was some theoretical musings about not-yet-ready-for-production-use[1] and not-license-compatible-with-Linux[2] filesystems, but they apply more to a possible future, not "nowadays". Filesystem compression is interesting but as I also commented, the biggest problem with logs on an SSD is going to be the continual appending which is going to be terrible for SSD wear. In theory it should be possible to write to an SSD in such a way that you can fill up the "still erased" tail of the sector with the additional logs without allocating a new sector because the unwritten part of the sector is still writable[3], but I doubt this is possible unless the SSD's support it and they mask so much behind their I/O frontend channel that it would be impossible to tell. This would be completely impossible with a compressed filesystem though, I think (I'm assuming a compressed filesystem compresses in per-inode chunks... I don't really know how they work) Put it another way, an example logfile on my system over a 24 hour period has 34000 lines (each line has, on average, 117 characters). That logfile is an exim4 mainlog so lets say that we are writing 4 lines every 10 seconds, on average (4 lines constitute 1 email worth of logs). To append to the data on a sector in a harddisk you read in the existing data, modify it (stick your new data on the end), then write it back. For a dumb flash device you would have to read in the entire block (~1MB I think), erase it, then write back the block, including your modified data. SSD's are much smarter than that and do all sorts of funky things to ease the erase requirements but you still need to continually write an inode worth of sectors each time you modify it. In any case, by my (very rough) calculations every 10 seconds a new 4K (8 sector) inode of data is re-written to append the data, so ~8640 inode re-writes over the course of the 24 hour period (I know I've overlooked a lot of things here and made a lot of assumptions but the order of magnitude should be about right). When the logfile is zipped up, it consumes 179 inodes. Using a compressed filesystem wouldn't save anything on the daily re-write churn which is per-write based, but would save on the write of the 179 inodes of zipped data, or 2% of total. James [1] https://btrfs.wiki.kernel.org/index.php/FAQ#Is_btrfs_stable.3F "Is btrfs stable? Short answer: No, it's still considered experimental." [2] http://zfsonlinux.org/faq.html#WhatAboutTheLicensingIssue "In a nutshell, the issue is that the Linux kernel which is licensed under the GNU General Public License is incompatible with ZFS which is licensed under the Sun CDDL" [3] no reference here. I'm basing it on my experience with writing to flash memory. Erased state is all binary 1's. Writing can turn the 1 into a 0 but the only way to go from a 0 to a 1 is to erase the entire block.
participants (3)
-
James Harper -
Jason White -
Russell Coker