Hi all,
I want to reorganize some backups and use ZFS snapshots copied to a
"backup NAS" (I am having).
Traditionally some zfs were sent as a .tgz to the NAS.
I want to use ZFS compression on the box instead and look for the best
method to compress it. Space savings are important, speed isn't.
Below the man page describing the compression methods available.
Do you have experience to recommend a method working well?
I do not want to "muck around" and change too much because it is Terabytes
worth of data and I do not want to reconfigure too often.
I do not use compression on my production systems a lot so I do not have
too much experience with it.
I have a few "not frequently used/non-critical" filesystems with lz4 - but
that's it. Speed-wise it did not seem to matter much compared to
uncompressed filesystems (but I did not benchmark properly)
Thanks for any advice
Peter
compression=on | off | lzjb | gzip | gzip-N | zle | lz4
Controls the compression algorithm used for this dataset. The lzjb
compression algorithm is optimized for performance while providing
decent data compression. Setting compression to on uses the lzjb
com-
pression algorithm. The gzip compression algorithm uses the same
com-
pression as the gzip(1) command. You can specify the gzip level by
using the value gzip-N where N is an integer from 1 (fastest) to 9
(best compression ratio). Currently, gzip is equivalent to gzip-6
(which is also the default for gzip(1)). The zle compression algo-
rithm compresses runs of zeros.
The lz4 compression algorithm is a high-performance replacement for
the lzjb algorithm. It features significantly faster compression and
decompression, as well as a moderately higher compression ratio than
lzjb, but can only be used on pools with the lz4_compress feature
set
to enabled. See zpool-features(7) for details on ZFS feature flags
and the lz4_compress feature.
Hi All,
This is a strange one.
Ages ago we suddenly had NZ visitors complaining about time outs and
white screens for our web apps. Noting else was broken except the people
across the ditch (and a few other overseas customers).
After testing and trying a vast number of things we put up varnish and
told them to connect on port :6xxx and hey presto it all went back to
normal.
We did this for a few months and then decided to move everything on :80
through varnish.
The speed improvement is great. Love it. Amazing.
And now we have the NZ clients complaining that things are dropping out
and whitescreening again.
Is there a great firewall of Australia on port 80? How can one service
on port :6xxx work flawlessly and then when moved to :80 kill overseas
users?
Really this is doing my head in. I am faced with changing everything
back (and removing varnish and the speed it gives) and getting the NZ
folks to use a weird port again.
Clues appreciated.
Cheers
Piers
The old LUV server has now been replaced. Due to multiple hardware problems
(drive errors and CPU/RAM problems) I had to do a rush job.
If you see this then the mailing lists are now working. The web site isn't
working yet, I'm still trying to get it going. Any suggestions appreciated.
--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/
Hi all,
I found an older ASUS EEE 701. The Ubuntu login on it was unknown and the
person who used it on company travel left a while ago.
I decided to put Debian 7.6.0 on it (BTW: FreeBSD 10 does not recognize
the WLAN). I installed "standard" (text) and all seemed fine.
Just grub refused to install.
So I installed the grub packages manually (using the "live" USB stick) and
run "grub install /dev/sda" in a chroot environment. All worked but it
still does not boot.
"grub-setup /dev/sda" seems to be working too.. I also used
"grub-mkconfig" successfully.
The Asus EEE 701 has a 8GB SSD drive.
Is there something "special" about the Asus I should do? Workarounds?
Thanks for ideas
Peter
Hi Toby and all,
Before you ask;-)
> I can make a test on a 32 GB RAM, 2 Xeon processor machine but this is a
> bit of limited value, I think.
I can actually test it now and will see the space savings. And then I hope
I do not "kill" the 1GB machine.
Regards
Peter
Hi Toby,
thanks for the answer.
> So.... what's stopping you doing a quick test to find out what is best
> for you?
I have the appliance not available yet but I will get there next week.
Actually I forgot one condition.. the 1 GB RAM.
I installed PC-BSD on a laptop with 1GB of RAM before (with ZFS only) and
realized that it works in general (with Flukebox, no Gnome/KDE..) So I
hope this works for this appliance too (no GUI needed here)
I can make a test on a 32 GB RAM, 2 Xeon processor machine but this is a
bit of limited value, I think.
[For historical reasons - this problem is solved]
The other issue was a "start". ZFS is only compressing new data. So I do
not have an issue to send data from incremental snapshots from an
uncompressed filesystem to a compressed one.
But if I have a few hundred GB already? The "zfs receive" will create the
ZFS..
I figured out a way, you can create the parent filesystem with compression
on and the new one inherits this value:-) I may need some smarts in a
script to get the path names right.
> I ran a quick test using (non-zfs) equivalents of various compression
tools, over a 2.0G filesystem image. (ie. hoping that represents a fair
variety of binary+text files)
> lz4 1.7s 221M
> lz4 -6 10s 189M
> gzip 25s 151M
> bzip2 54s 135M
> 7z 147s 102M
> xz 253s 103M
Thanks for it. I will see whether it matches the ZFS compression.
> So if space is really at a premium, you're better off using an archive
tool to compress everything, rather than zfs' built-in compression.
The fun comes if you can keep a bunch of older copies thanks to snapshots.
At the moment the backup is keeping six weekly backups online (compressed
full backups) plus a year worth of quarterly backups. I hope to push it
further back with snapshots.
> But otherwise, gzip is better than lz4, but at significantly slower
performance.
Agreed
and thanks
Peter
Hello,
Yesterday I tried to disable commercial flagging in mythtv
0.27.4+fixes.2014 (or at least change the default for new recordings)[1].
According to the documentation, e.g.
https://www.mythtv.org/wiki/User_Manual:Detailed_configuration_Frontend,
under "Setup->TV Settings->General" there should be a screen "General
(Jobs)" that has the "Commercial Flagging Method" setting. It doesn't.
In fact a number of settings appear to be missing.
So I went to change the setting in mythweb, only it isn't there either.
So I looked at the mysql database, where I noticed I have duplicate
settings.
mysql> select * from settings where value='JobAllowCommFlag';
+------------------+------+----------+
| value | data | hostname |
+------------------+------+----------+
| JobAllowCommFlag | 0 | gandra |
| JobAllowCommFlag | 0 | gandra |
+------------------+------+----------+
2 rows in set (0.00 sec)
In fact just about all the Job* settings have duplicates. While they are
the same, am wondering if this is why I can't edit the values.
mysql> select * from settings where hostname='gandra' order by value;
...
| JobAllowCommFlag | 0
| gandra |
| JobAllowCommFlag | 0
| gandra |
| JobAllowMetadata | 1
| gandra |
| JobAllowTranscode | 1
| gandra |
| JobAllowTranscode | 1
| gandra |
| JobAllowUserJob1 | 0
| gandra |
| JobAllowUserJob1 | 0
| gandra |
| JobAllowUserJob2 | 0
| gandra |
| JobAllowUserJob2 | 0
| gandra |
| JobAllowUserJob3 | 0
| gandra |
| JobAllowUserJob3 | 0
| gandra |
| JobAllowUserJob4 | 0
| gandra |
| JobAllowUserJob4 | 0
| gandra |
| JobQueueCheckFrequency | 60
| gandra |
| JobQueueCheckFrequency | 60
| gandra |
| JobQueueCPU | 0
| gandra |
| JobQueueCPU | 0
| gandra |
| JobQueueMaxSimultaneousJobs | 1
| gandra |
| JobQueueMaxSimultaneousJobs | 1
| gandra |
| JobQueueWindowEnd | 23:59
| gandra |
| JobQueueWindowEnd | 23:59
| gandra |
| JobQueueWindowStart | 00:00
| gandra |
| JobQueueWindowStart | 00:00
| gandra |
...
Just wondering if it is possible to delete the duplicate entries? Or do I
have to delete everything and recreate new entries?
Unfortunately, I have shown all the fields above, there are no extra
fields, now does there appear to be a unique constraint on value,hostname
which would seem like a good idea (possibly because it is a MyISAM table -
not sure if a new installation of mythtv would have the same issue or not).
Thanks
Notes:
[1] Sometimes mythtv hangs for large periods of time before playing a
recording or in middle of the recording, and I am working on the theory
that the commercial detection, despite being "nice", is the cause.
--
Brian May <brian(a)microcomaustralia.com.au>
Hello,
Am having two strange problems that are starting to eerk me with this
(relatively new) computer, running Debian wheezy:
1. If I boot anything later then 3.12 kernel, I don't get any display. As
in the monitors display black. No cursors of any sort. Changing to a
virtual console doesn't help. Booting in rescue mode doesn't help (I think
this rules out X-Windows being a problem). If I go back to the 3.12 kernel,
everything works perfectly. I also tried plugging monitors into alternative
ports just in case it is going to the wrong place, but get nothing - in any
case, under 3.12 the computer seems pretty good at automatically working
out what ports are active under X-Windows.
01:00.0 VGA compatible controller: NVIDIA Corporation G96 [GeForce 9500 GT]
(rev a1)
Am currently using the non-free nvidia drivers. Had exactly the same
symptoms when I installed the latest kernel without the non-free nvidia
kernel modules. I think the problem is occurring before X starts.
Computer seems to be up and running, and responsive to crl+alt+del despite
not having a display.
2. There seems to be some weird performance problem. e.g. save a 2 kilobyte
file in vim, and the computer can completely freeze (all other windows,
including xterms, stop responding to user input) for, say 30 seconds, while
it is writing that file. Chromium takes ages to load with several tabs, and
pages can fail to start properly while it is doing so.
Computer has 16GB RAM and is not using any swap. It has 11GB of
buffer/cache space:
total used free shared buffers cached
Mem: 15G 13G 2.1G 0B 1.1G 7.9G
-/+ buffers/cache: 4.6G 11G
Swap: 3.8G 0B 3.8G
Problems occurred before starting chromium, previously I wondered if it was
chromium's fault.
This is moving disk + RAID1 + LVM + ext4. Bonnie++ results seem to be
pretty good, better in fact then my work computer, which doesn't suffer
from similar problems.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
falidae 31904M 1228 94 107313 4 47471 2 +++++ +++ 137886 3
441.8 2
Latency 19141us 12435ms 251ms 19913us 88073us
283ms
Version 1.96 ------Sequential Create------ --------Random
Create--------
falidae -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++
+++
Latency 36us 223us 226us 36us 10us
24us
1.96,1.96,falidae,1,1404290654,31904M,,1228,94,107313,4,47471,2,+++++,+++,137886,3,441.8,2,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,19141us,12435ms,251ms,19913us,88073us,283ms,36us,223us,226us,36us,10us,24us
I am currently working on a new theory that the performance problems only
occur when the computer is cold and first turned on. I think I have seen
evidence to disprove this, but guess I should run bonnie++ as soon as I
turn the computer on, just to be sure.
Any other ideas?
Thanks
--
Brian May <brian(a)microcomaustralia.com.au>
Hi All,
Toshiba Satellite 32 bit, has been running under OpenSUSE very nicely
since 11. Tried an Upgrade to 13.2 from 13.1 KDE splash screen indicated
that it failed to load the Display. In 13.1 it was a Gnome desktop. So I
selected CTRL +ALT +F2 and logged in there to try again, this time with
a Gnome desktop. Worse outcome. Clean shut down from the Alt F2 screen
and decided to run the install with LVM btrfs and encryption to force a
format of /home for the next attempt. Now I know that openSUSE has moved
to systemd for this release, but is that the problem, or do I look
elsewhere?
I still have the isos for 13.1 so regression is an option. I am
seriously apprehensive about upgrading my production machine.
Thanks
Andrew Greig
I'm writing this message as a Linux user and a member of LUV - not as a member
of the LUV committee.
Avi, you have a history of making good technical contributions to LUV and of
representing your employer well. However I think that Oracle has been under-
represented given their technical work and their importance in the industry.
I think that it would be good to have some Oracle employees give talks at LUV
meetings as they are doing a lot that LUV members could learn about. Avi you
are obviously well qualified to give such talks, but I think it would be good
to get other Oracle employees with different skills involved too. A lecture
about running the Oracle Database on Linux would be of interest.
Also Red Hat has been running free public events teaching people about their
products and services which I have found to be good value (and I have
recommended some on this list). Does Oracle run such events? If so please
inform the LUV committee and we can forward to the list if appropriate (I
don't think it's appropriate for Linux companies to advertise on this list,
but it's OK for LUV members who aren't associated with companies in question
to recommend them).
Some years ago there was an Oracle database install-fest, that was a useful
event and many people learned a lot there. I think it would be good if Oracle
ran another of those, if that happens I think it would be appropriate for LUV
to promote that.
--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/
