- luv-main - luv.asn.au

RE: Mail Server Really Slow
by James Harper 21 Jan '16

21 Jan '16

(it seems that "reply-all" no longer includes luv-main (from ms outlook at least), so I have to include it manually... what's with that?) > > Of course a RAID-1 of SSDs will massively outperform the RAID-5 you have. > > Given the size I guess it's one of the older HP servers that only takes ~70G disks. If you buy a cheap > Dell PowerEdge server and put a couple of Intel SSDs (not > bought from Dell because Dell charges > heaps for storage) in a RAID-1 configuration it will massively outperform the old HP server. If you use SSDs for any sort of intensive storage, do keep an eye on the SMART "media wearout" values, and replace them before the counter hits 0 (or 1). For the disks we were using (Intel DataCentre SSD's), the docs say that while the disk may well keep running for a long time after the counter hits 1, it is considered worn out and is no longer covered by warranty. SMART does not consider this a failure/old age (the threshold is 0 but the counter never goes below 1), so you have to actually monitor the counter. The RAID controller probably won't tell you this either (that the disk has worn out), and in our case the performance went to crap sometime after the counter hit 1, causing considerable frustration to all involved. Different models and manufacturers obviously differ in this respect too. I'm seeing time-to-replacement of about 12 months on high load system where the SSD's are used for a RAID cache (ZFS, Intel RAID controllers, etc). At home, my little router that is just a laptop running Squid, has used up 2 of 100 SMART units in the ~12 months it has been running. Not particularly relevant to the discussion at hand, but with suggestions of "put in SSD's and all your trouble will go away", it is something you need to consider. James

4 6

Mail Server Really Slow
by Piers Rowan 20 Jan '16

20 Jan '16

Hi there, The server runs: - dovecot - apache (roundcube webmail) - sendmail - clamav-milter - amavisd-milter RAM: 4.10 GB CPU: QEMU Virtual CPU version (cpu64-rhel6), 5 cores The server is a VM on a host server that also provides http / mysql services. The host server runs cron jobs to poll the email server (importing data from mail boxes into the CRM) so - to clutch at straws - I am not sure if the host and guest are competing for the disk IO at the same time with these calls. Contrary to that is that the host server does not experience any slow downs. Before the holidays we added another 30 users to the servers. Below is some of the output of some commands. At the moment I am thinking of moving the VM services into a cloud based host but this it not ideal in the short term. Any ideas? Thanks P CPU load averages 8.74 (1 min) 8.05 (5 mins) 5.63 (15 mins) [root]# vmstat 5 10 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 5 211056 141268 42948 2472992 1 1 246 47 2 2 2 1 93 5 0 0 3 211056 127876 43208 2486312 0 0 2710 46 1212 895 2 1 66 31 0 0 3 211104 141252 41136 2470440 0 10 3164 45 1109 804 2 1 74 23 0 0 3 211104 128228 41412 2484500 0 0 2859 18 566 640 0 0 60 39 0 0 3 211180 143320 40464 2464324 0 15 3122 70 1254 1030 2 1 57 40 0 Tasks: 502 total, 1 running, 501 sleeping, 0 stopped, 0 zombie Cpu(s): 7.7%us, 2.1%sy, 0.0%ni, 48.3%id, 41.6%wa, 0.0%hi, 0.0%si, 0.2%st Mem: 4296356k total, 4169448k used, 126908k free, 41144k buffers Swap: 2621436k total, 211980k used, 2409456k free, 2484540k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7067 clam 20 0 1771m 361m 1584 S 29.5 8.6 265:23.91 clamd 6450 kylier 20 0 19300 2828 1404 D 1.0 0.1 0:00.03 imap 6544 root 20 0 15396 1552 888 R 1.0 0.0 0:00.29 top 3848 root 20 0 78752 3024 2244 S 0.7 0.1 0:00.23 auth 6398 vickij 20 0 18528 2072 1400 D 0.7 0.0 0:00.48 imap 6419 uuu.s 20 0 18524 2024 1388 D 0.7 0.0 0:00.42 imap 6424 uuuu 20 0 19300 2812 1404 D 0.7 0.1 0:00.23 imap 23252 jasmin.s 20 0 19004 2652 1572 S 0.7 0.1 0:00.42 imap 23345 jasmin.s 20 0 26136 2472 1688 S 0.7 0.1 0:02.72 imap 29215 abigail. 20 0 18840 2068 1536 S 0.7 0.0 0:00.58 imap 1 root 20 0 19356 872 648 S 0.3 0.0 6:07.28 init 1336 root 20 0 244m 5128 508 S 0.3 0.1 19:20.96 rsyslogd 6293 glenn.ge 20 0 18388 1888 1432 S 0.3 0.0 0:00.01 imap 6567 nic.will 20 0 18408 1996 1512 D 0.3 0.0 0:00.08 imap 7091 root 20 0 19268 568 404 S 0.3 0.0 12:40.65 dovecot 7175 root 20 0 98.6m 364 320 S 0.3 0.0 3:43.93 tail 23388 uuuu 20 0 19000 2568 1544 S 0.3 0.1 0:14.31 imap 23460 jasmin.s 20 0 23248 2756 1756 D 0.3 0.1 0:01.54 imap ---8<---------8<---- # like this from here down 2 root 20 0 0 0 0 S 0.0 0.0 0:28.72 kthreadd

6 16

Re: Mail Server Really Slow
by Russell Coker 20 Jan '16

20 Jan '16

On Tue, 19 Jan 2016 11:24:44 PM James Harper wrote: > If you use SSDs for any sort of intensive storage, do keep an eye on the > SMART "media wearout" values, and replace them before the counter hits 0 > (or 1). For the disks we were using (Intel DataCentre SSD's), the docs say Interesting, I didn't realise that was the case. I've seen various claims of SSD wearout times being 5+ years which means they would be obsolete before they wear out. > disk has worn out), and in our case the performance went to crap sometime > after the counter hit 1, causing considerable frustration to all involved. Do you know of any good software to measure performance changes in disks? Performance often changes before drives develop faults and it would be good to measure that. EG if the number of iops the disk delivers when it's neat 100% according to the iostat algorithm suddenly changes by an order of magnitude then it's probably due for replacement. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/

2 1

Re: Mail Server Really Slow
by Piers Rowan 20 Jan '16

20 Jan '16

On 20/01/16 08:13, Craig Sanders via luv-main wrote: > if you're wondering why Russell and I are asking all these questions > it's because you haven't provided anywhere near enough details to > answer your performance question. I just wanted to chime in that I really appreciate the help and have benefited from Russell, yours and the others' input immensely. I have a few things on so please don't take my lack of response and disinterest. I need to delve into some of the points and check out the server more before I can respond with anything worth while. Cheers Piers

1 0

BTRFS
by Russell Coker 19 Jan '16

19 Jan '16

The LUV VM is now running BTRFS. I've setup a cron job to make read-only snapshots of the root filesystem. Such snapshots are atomic which means that the people who are making offsite backups of the server can rsync the MySQL data files without integrity problems. Also the filesystem has duplicate metadata to cover the case of corruption. The Dom0 is running Linux Software RAID-1 which saves us from a total disk failure but doesn't handle corruption at all. If data corruption hits the data blocks then we have to rely on offsite backups. BTRFS doesn't support duplicate data blocks (in any normal or desirable configuration) unlike ZFS which has the copies= configuration. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/

2 2

When is a symlink not a symlink? :D
by Lev Lafayette 18 Jan '16

18 Jan '16

I've encountered a thoroughly interesting and somewhat perplexing issue with a user login. The basic symptom is that the user can't login - and as root I can't complete an su as that user. # su - username (hangs) ^C /var/log/secure however suggests a successful connection.. Jan 18 12:25:11 machine su: pam_unix(su-l:session): session opened for user username by root(uid=0) Jan 18 12:26:01 machine su: pam_unix(su-l:session): session closed for user username Now we have a system where the several hundred users in /home are actually symlinks to NFS attached storage on /data/user[1..2]. Which is the case for this user. [user2]# ls -lad username/ drwxr-xr-x 46 username groupname 8192 Jan 15 07:53 username/ # pwd /data/user2 However when running an ls on the login node it shows they have a directory in home that is *not* a symlink, unlike others. # ls -lad /home/username drwxr-xr-x 46 username groupname 8192 Jan 15 07:53 /home/username/ # ls -lad /home/lev lrwxrwxrwx 1 lev home 15 Apr 15 2014 /home/lev -> /data/user1/lev *Except* when one does a directory listing from /home itself (where it has a different groupname)! [root@edward home]# ls -la /home/ .. lrwxrwxrwx 1 username home 19 Jul 26 2013 sanujig -> /data/user2/username Note that the two directories have the same inode. # ls -id /home/username/ 1081162059 /home/sanujig/ ]# ls -id /data/user2/username 1081162059 /data/user2/username Fascinating, eh? -- Lev Lafayette, BA (Hons), GradCertTerAdEd (Murdoch), GradCertPM, MBA (Tech Mngmnt) (Chifley) mobile: 0432 255 208 RFC 1855 Netiquette Guidelines http://www.ietf.org/rfc/rfc1855.txt

4 5

Re: IPv6
by Tony Langdon 18 Jan '16

18 Jan '16

On 13/01/2016 8:42 PM, Erik Christiansen via luv-main wrote: > On 13.01.16 20:09, Tony Langdon via luv-main wrote: >> On 13/01/2016 10:31 AM, Brian May wrote: >>> Tony Langdon via luv-main <luv-main(a)luv.asn.au> writes: >>> >>>> On closer observation, it started with the message from you that I >>>> replied to, and it's only you. Another coincidence is that the Subject: >>>> header hasn't been munged in this thread, yet everyone else's posts >>>> still have the Subject: munging. >>> >>> I don't see any subject munging for any posts on luv-main. >> >> I got it wrong, was actually referring to From: address munging. > > Nah, I wouldn't hold my breath. Hints on that one have been going to > /dev/null since last year. It seems we're all supposed to undo the crap > individually at our ends. I'm just mystified as to why the thread that Russell's message started is not being From: munged, whereas all else is. Though your message is munged, and Reply List is working perfectly. It wasn't so much the From: munging that was bothering me, but the inconsistent reply behaviour (I dislike Reply All, because of the extra emails it generates, much prefer Reply List). -- 73 de Tony VK3JED/VK3IRL http://vkradio.com

5 23

Re: IPv6
by Russell Coker 18 Jan '16

18 Jan '16

On Wed, 13 Jan 2016 08:42:23 PM Erik Christiansen via luv-main wrote: > > I got it wrong, was actually referring to From: address munging. > > Nah, I wouldn't hold my breath. Hints on that one have been going to > /dev/null since last year. It seems we're all supposed to undo the crap > individually at our ends. We had a problem of Gmail and other recipients falsely regarding mail from the LUV server as forged and assigning it to the spam folder. We are dealing with a couple of bugs in Mailman, what I consider to be a deficiency in OpenDKIM and some interactions between SPF and DMARC. If you would like to help with some of these technical issues then go for it. http://postmaster.facebook.com/reputation_and_authentication Facebook is now compelling sysadmins to use SPF or DKIM. This isn't going to go away. It's only a matter of time before Internode starts using DKIM to placate Facebook. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/

9 57

Re: SSHD (and steam and windows and other stuff)
by trentbuck＠gmail.com 15 Jan '16

15 Jan '16

Craig Sanders via luv-main <luv-main(a)luv.asn.au> writes: > On Wed, Jan 13, 2016 at 01:28:57PM +1100, Trent W. Buck wrote: >> AFAICT the only reason SSHDs exist are: >> >> * Windows has nothing like bcache/l2arc; or > > it does. it's called ReadyBoost. Compared to bcache/flashcache or L2ARC > for ZFS, it sucks. You can't just tell Windows to use an SSD (or > partition thereof) to cache an arbitrary disk....well, in theory you That sounds like the one they introduced back when USB keys were cool. I don't think anyone seriously intends it for hybrid SSD/HDD usage. "the maximum cache size per device is 4GB (Vista) or 32GB (Windows 7)." > can but in practice Windows itself decides whether that option will be > available by its own inscrutable and undocumented method. It looks like the drive has to NTFS/exFAT/FAT formatted, because ReadyBoost caches to a file, not a disk/partition. > I discovered this just last week after upgrading my win7 games box to > have an SSD as a boot disk, decided to try ReadyBoost for my main 2TB > steam library drive (not an SSHD) so made a 40GB partition for it. No > matter what I tried, I couldn't get windows to make the option available > in its Disk Manager GUI - it was there, just greyed out. See above re 32GB cap. (Aside: the windows types I've asked about this indicate that they actually use the hardware vendor's hybrid drive implementation, e.g. Intel's.) > I gave up and expanded the main partition...will have to manually move > big games to the fast SSD, which works nicely - witcher 3 loads much > faster off the SSD. More importantly, save games load in about 25 > seconds rather than 60-90 seconds (which is really annoying when you die > repeatedly because you thought attacking a royal griffin seemed like a > good idea, 90 seconds to load the save, 15 seconds to die again while > trying to run away, repeat until you succeed or rage-quit) I know at least one guy who netboots his Windows games machine off a linux array, exported as iSCSI or something. I dunno where the bottleneck is there, but it means he can snapshot the disk &c like it was a VM. >> * My computer only has one disk bay. > > (you don't really need a disk bay for an SSD. just a sata port plus data > and power cables. and if you wanna get fancy you can sticky-tape it to > the side of the case :) I was handwaving that detail away; I was specifically thinking of laptops with a single 2.5" bay and no M.2. > but given that the SSHDs i bought were roughly the same price as > non-SSHDs Hrm... $ msy | foldr grep -Fi -- 2tb 3.5 sata3 7200 135 Seagate 3.5" SSHD 2TB ST2000DX001 NAND 8GB SATA3 7200rpm 64MB Hybrid HDD 98 Seagate 3.5" Barracuda 2TB ST2000DM001 SATA3 7200rpm 64MB Cache Hard Disk $ msy | foldr grep -Fi -- 4tb 3.5 sata3 5400 249 WD 3.5" SSHD WD40E31X 4TB 5400RPM SATA3 Hybrid (8G SSDHDD) NAND SSHD 202 WD 3.5" Blue 4TB WD40EZRZ 64M 5400RPM SATA3 HDD $ msy | foldr grep -Fi -- 4tb 3.5 sata3 5900 329 Seagate 3.5" Enterprise NAS 4TB ST4000VN0001 SATA3 5900rpm 64MB Cache HDD 275 Seagate 3.5" SV35 ST4000VX002 4TB 5900rpm Surveillance HDD include Data Rescue Service 259 Seagate 3.5" NAS 4TB ST4000VN003 SATA3 5900rpm 64MB Cache HDD include Data Rescue Service 239 Seagate 3.5" SV35 ST4000VX000 4TB 5900rpm Surveillance HDD 219 Seagate 3.5" SSHD 4TB ST4000DX001 NAND 8GB SATA3 5900rpm 64MB Hybrid HDD 219 Seagate 3.5" NAS 4TB ST4000VN000 SATA3 5900rpm 64MB Cache HDD 185 Seagate 3.5" Barracuda 4TB ST4000DM000 SATA3 5900RPM 64MB Cache Hard Disk That's a ~20% markup on the first two. SSDs start at A$60 for 120GB -- not much more than the difference. >, i don't see any harm and some potential benefit in using them > on linux. 8GB cache isn't much but it "just works" without any hassle > or configuration. it's giving me some SSD caching on my ZFS 'backup' > pool without having to add another SSD to the same andr dedicate an SSD > partition to the task. I guess that's reasonable. What's the failure mode when the NAND wears out? Does it detect that and fall back to being a regular HDD?

5 10

IPv6
by Russell Coker 14 Jan '16

14 Jan '16

I have enabled IPv6 on the LUV server. I would appreciate it if anyone who is interested could test it out. Just do basic stuff like force your web browser to IPv6 and visit the web site or force your MTA to IPv6 and send mail to the list. Andrew please add 2a01:4f8:100:7463::9 to the glue record for itmustbe, then IPv6-only recursive DNS servers (if they happen to exist) can successfully query the LUV server. If your MTA has IPv6 enabled but broken (which is probably more common than having IPv6 enabled and working correctly) then this may make it impossible for you to participate in the LUV lists. This change is probably going to cause some pain for some people (mostly me). But it's something that we need to do. The world needs to move to IPv6 and as an IT organisation operating in the public interest LUV should be in the lead. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/

5 11