- luv-main - example.com

BTRFS
by Russell Coker 19 Jan '16

19 Jan '16

The LUV VM is now running BTRFS. I've setup a cron job to make read-only snapshots of the root filesystem. Such snapshots are atomic which means that the people who are making offsite backups of the server can rsync the MySQL data files without integrity problems. Also the filesystem has duplicate metadata to cover the case of corruption. The Dom0 is running Linux Software RAID-1 which saves us from a total disk failure but doesn't handle corruption at all. If data corruption hits the data blocks then we have to rely on offsite backups. BTRFS doesn't support duplicate data blocks (in any normal or desirable configuration) unlike ZFS which has the copies= configuration. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/

2 2

When is a symlink not a symlink? :D
by Lev Lafayette 18 Jan '16

18 Jan '16

I've encountered a thoroughly interesting and somewhat perplexing issue with a user login. The basic symptom is that the user can't login - and as root I can't complete an su as that user. # su - username (hangs) ^C /var/log/secure however suggests a successful connection.. Jan 18 12:25:11 machine su: pam_unix(su-l:session): session opened for user username by root(uid=0) Jan 18 12:26:01 machine su: pam_unix(su-l:session): session closed for user username Now we have a system where the several hundred users in /home are actually symlinks to NFS attached storage on /data/user[1..2]. Which is the case for this user. [user2]# ls -lad username/ drwxr-xr-x 46 username groupname 8192 Jan 15 07:53 username/ # pwd /data/user2 However when running an ls on the login node it shows they have a directory in home that is *not* a symlink, unlike others. # ls -lad /home/username drwxr-xr-x 46 username groupname 8192 Jan 15 07:53 /home/username/ # ls -lad /home/lev lrwxrwxrwx 1 lev home 15 Apr 15 2014 /home/lev -> /data/user1/lev *Except* when one does a directory listing from /home itself (where it has a different groupname)! [root@edward home]# ls -la /home/ .. lrwxrwxrwx 1 username home 19 Jul 26 2013 sanujig -> /data/user2/username Note that the two directories have the same inode. # ls -id /home/username/ 1081162059 /home/sanujig/ ]# ls -id /data/user2/username 1081162059 /data/user2/username Fascinating, eh? -- Lev Lafayette, BA (Hons), GradCertTerAdEd (Murdoch), GradCertPM, MBA (Tech Mngmnt) (Chifley) mobile: 0432 255 208 RFC 1855 Netiquette Guidelines http://www.ietf.org/rfc/rfc1855.txt

4 5

Re: IPv6
by Tony Langdon 17 Jan '16

17 Jan '16

On 13/01/2016 8:42 PM, Erik Christiansen via luv-main wrote: > On 13.01.16 20:09, Tony Langdon via luv-main wrote: >> On 13/01/2016 10:31 AM, Brian May wrote: >>> Tony Langdon via luv-main <luv-main(a)luv.asn.au> writes: >>> >>>> On closer observation, it started with the message from you that I >>>> replied to, and it's only you. Another coincidence is that the Subject: >>>> header hasn't been munged in this thread, yet everyone else's posts >>>> still have the Subject: munging. >>> >>> I don't see any subject munging for any posts on luv-main. >> >> I got it wrong, was actually referring to From: address munging. > > Nah, I wouldn't hold my breath. Hints on that one have been going to > /dev/null since last year. It seems we're all supposed to undo the crap > individually at our ends. I'm just mystified as to why the thread that Russell's message started is not being From: munged, whereas all else is. Though your message is munged, and Reply List is working perfectly. It wasn't so much the From: munging that was bothering me, but the inconsistent reply behaviour (I dislike Reply All, because of the extra emails it generates, much prefer Reply List). -- 73 de Tony VK3JED/VK3IRL http://vkradio.com

5 23

Re: IPv6
by Russell Coker 17 Jan '16

17 Jan '16

On Wed, 13 Jan 2016 08:42:23 PM Erik Christiansen via luv-main wrote: > > I got it wrong, was actually referring to From: address munging. > > Nah, I wouldn't hold my breath. Hints on that one have been going to > /dev/null since last year. It seems we're all supposed to undo the crap > individually at our ends. We had a problem of Gmail and other recipients falsely regarding mail from the LUV server as forged and assigning it to the spam folder. We are dealing with a couple of bugs in Mailman, what I consider to be a deficiency in OpenDKIM and some interactions between SPF and DMARC. If you would like to help with some of these technical issues then go for it. http://postmaster.facebook.com/reputation_and_authentication Facebook is now compelling sysadmins to use SPF or DKIM. This isn't going to go away. It's only a matter of time before Internode starts using DKIM to placate Facebook. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/

9 57

Re: SSHD (and steam and windows and other stuff)
by trentbuck＠gmail.com 15 Jan '16

15 Jan '16

Craig Sanders via luv-main <luv-main(a)luv.asn.au> writes: > On Wed, Jan 13, 2016 at 01:28:57PM +1100, Trent W. Buck wrote: >> AFAICT the only reason SSHDs exist are: >> >> * Windows has nothing like bcache/l2arc; or > > it does. it's called ReadyBoost. Compared to bcache/flashcache or L2ARC > for ZFS, it sucks. You can't just tell Windows to use an SSD (or > partition thereof) to cache an arbitrary disk....well, in theory you That sounds like the one they introduced back when USB keys were cool. I don't think anyone seriously intends it for hybrid SSD/HDD usage. "the maximum cache size per device is 4GB (Vista) or 32GB (Windows 7)." > can but in practice Windows itself decides whether that option will be > available by its own inscrutable and undocumented method. It looks like the drive has to NTFS/exFAT/FAT formatted, because ReadyBoost caches to a file, not a disk/partition. > I discovered this just last week after upgrading my win7 games box to > have an SSD as a boot disk, decided to try ReadyBoost for my main 2TB > steam library drive (not an SSHD) so made a 40GB partition for it. No > matter what I tried, I couldn't get windows to make the option available > in its Disk Manager GUI - it was there, just greyed out. See above re 32GB cap. (Aside: the windows types I've asked about this indicate that they actually use the hardware vendor's hybrid drive implementation, e.g. Intel's.) > I gave up and expanded the main partition...will have to manually move > big games to the fast SSD, which works nicely - witcher 3 loads much > faster off the SSD. More importantly, save games load in about 25 > seconds rather than 60-90 seconds (which is really annoying when you die > repeatedly because you thought attacking a royal griffin seemed like a > good idea, 90 seconds to load the save, 15 seconds to die again while > trying to run away, repeat until you succeed or rage-quit) I know at least one guy who netboots his Windows games machine off a linux array, exported as iSCSI or something. I dunno where the bottleneck is there, but it means he can snapshot the disk &c like it was a VM. >> * My computer only has one disk bay. > > (you don't really need a disk bay for an SSD. just a sata port plus data > and power cables. and if you wanna get fancy you can sticky-tape it to > the side of the case :) I was handwaving that detail away; I was specifically thinking of laptops with a single 2.5" bay and no M.2. > but given that the SSHDs i bought were roughly the same price as > non-SSHDs Hrm... $ msy | foldr grep -Fi -- 2tb 3.5 sata3 7200 135 Seagate 3.5" SSHD 2TB ST2000DX001 NAND 8GB SATA3 7200rpm 64MB Hybrid HDD 98 Seagate 3.5" Barracuda 2TB ST2000DM001 SATA3 7200rpm 64MB Cache Hard Disk $ msy | foldr grep -Fi -- 4tb 3.5 sata3 5400 249 WD 3.5" SSHD WD40E31X 4TB 5400RPM SATA3 Hybrid (8G SSDHDD) NAND SSHD 202 WD 3.5" Blue 4TB WD40EZRZ 64M 5400RPM SATA3 HDD $ msy | foldr grep -Fi -- 4tb 3.5 sata3 5900 329 Seagate 3.5" Enterprise NAS 4TB ST4000VN0001 SATA3 5900rpm 64MB Cache HDD 275 Seagate 3.5" SV35 ST4000VX002 4TB 5900rpm Surveillance HDD include Data Rescue Service 259 Seagate 3.5" NAS 4TB ST4000VN003 SATA3 5900rpm 64MB Cache HDD include Data Rescue Service 239 Seagate 3.5" SV35 ST4000VX000 4TB 5900rpm Surveillance HDD 219 Seagate 3.5" SSHD 4TB ST4000DX001 NAND 8GB SATA3 5900rpm 64MB Hybrid HDD 219 Seagate 3.5" NAS 4TB ST4000VN000 SATA3 5900rpm 64MB Cache HDD 185 Seagate 3.5" Barracuda 4TB ST4000DM000 SATA3 5900RPM 64MB Cache Hard Disk That's a ~20% markup on the first two. SSDs start at A$60 for 120GB -- not much more than the difference. >, i don't see any harm and some potential benefit in using them > on linux. 8GB cache isn't much but it "just works" without any hassle > or configuration. it's giving me some SSD caching on my ZFS 'backup' > pool without having to add another SSD to the same andr dedicate an SSD > partition to the task. I guess that's reasonable. What's the failure mode when the NAND wears out? Does it detect that and fall back to being a regular HDD?

5 10

IPv6
by Russell Coker 13 Jan '16

13 Jan '16

I have enabled IPv6 on the LUV server. I would appreciate it if anyone who is interested could test it out. Just do basic stuff like force your web browser to IPv6 and visit the web site or force your MTA to IPv6 and send mail to the list. Andrew please add 2a01:4f8:100:7463::9 to the glue record for itmustbe, then IPv6-only recursive DNS servers (if they happen to exist) can successfully query the LUV server. If your MTA has IPv6 enabled but broken (which is probably more common than having IPv6 enabled and working correctly) then this may make it impossible for you to participate in the LUV lists. This change is probably going to cause some pain for some people (mostly me). But it's something that we need to do. The world needs to move to IPv6 and as an IT organisation operating in the public interest LUV should be in the lead. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/

5 11

Re: SSHD (and steam and windows and other stuff)
by Russell Coker 13 Jan '16

13 Jan '16

On Wed, 13 Jan 2016 08:43:28 PM Craig Sanders via luv-main wrote: > On Wed, Jan 13, 2016 at 01:28:57PM +1100, Trent W. Buck wrote: > > AFAICT the only reason SSHDs exist are: > > * Windows has nothing like bcache/l2arc; or > > it does. it's called ReadyBoost. Compared to bcache/flashcache or L2ARC > for ZFS, it sucks. You can't just tell Windows to use an SSD (or > partition thereof) to cache an arbitrary disk....well, in theory you > can but in practice Windows itself decides whether that option will be > available by its own inscrutable and undocumented method. https://en.wikipedia.org/wiki/ReadyBoost ReadyBoost is described as supporting USB Flash, CF, and other storage systems. USB Flash is notoriously unreliable and has given flash storage a bad reputation. It seems to be only read caching, it would be ridiculous to consider devices like USB Flash for write-back caching. > I discovered this just last week after upgrading my win7 games box to > have an SSD as a boot disk, decided to try ReadyBoost for my main 2TB > steam library drive (not an SSHD) so made a 40GB partition for it. No > matter what I tried, I couldn't get windows to make the option available > in its Disk Manager GUI - it was there, just greyed out. As you noted later in your message SSD storage in the TB range is getting quite affordable now. If you are buying an expensive gaming PC then getting 1TB of SSD for all your favourite games isn't going to change the price much. 2TB of SSD is in the range that many middle class people are prepared to pay. There doesn't seem to be a good case for MS to invest in further development of ReadyBoost. I don't like the idea of Bcache. What I would like to do is to be able to tell BTRFS to use certain disks for metadata only. For example I'd like to have a server with the root and /home subvols on a RAID-1 pair of SSDs as well as the metadata for all other subvols. Then have all the bulk data on a pair of large SATA disks. For my home fileserver I have no performance problems with tasks such as NFS serving big MP4 files (like my collection of FullHD music videos downloaded from Youtube), but I do have performance problems with ls. Also my home server has backups of a number of other systems (including the LUV server). When I run rsync without the -c option metadata access is a major bottleneck. > Microsoft seems to have abandoned the ReadyBoost idea. and they seem to > be confused about what SSD caching of hard disks is good for - their > documentation is all about how it might help on low-memory machines > (<4GB), presumably because they don't have enough RAM to cache large > chunks disk in RAM. as bcache etc on linux prove, SSD caching is > beneficial even on machines with lots of RAM. It really depends on what you are doing. A SSD read cache provides no benefits if the working set of the applications that are running fits in RAM once the system has been running for a while. For example the LUV 524M of storage for all MySQL databases and probably less than 100M of other data that is used often. It has 2.5G of RAM so it does hardly any disk access apart from synchronous writes for the mail queue once it's been running for a while. > windows is so damn primitive and restricted. i can't understand how > anyone could possibly think it's any good or that it makes a decent > desktop environment. it's almost unusable in its awful crappiness. i'm > glad i only have to use it as a games launcher. Some people think of a desktop OS as a games launcher, in which case Windows is doing ok in terms of game support. Steam is moving towards Linux which significantly changes things but at the moment Windows is a passable game launcher. > interestingly, i don't even have to use my KVM to switch to the windows > box any more if i don't want to - steam's in-home streaming feature > means that in addition to running native linux games on steam, i can > "stream" windows-only games from my win7 box. that means it executes on > windows but displays on my linux box in either full-screen or windowed > mode. would probably suck on wireless but there's no noticable lag or > performance loss on a wired gigabit network. i'm kind of amazed at how > well it works. That's interesting. > but given that the SSHDs i bought were roughly the same price as > non-SSHDs, i don't see any harm and some potential benefit in using them > on linux. 8GB cache isn't much but it "just works" without any hassle > or configuration. it's giving me some SSD caching on my ZFS 'backup' > pool without having to add another SSD to the same andr dedicate an SSD > partition to the task. For a filesystem like ZFS or BTRFS where the read hot spots move around the benefit of a small amount of read caching is probably small. If they do write caching then that could make a huge difference as the seek patterns for ZFS and BTRFS can cause significant problems. > but if you're not in a hurry to get an SSD, rumour has it that there > will be significant increases in capacity AND reductions in price on > SSDs this year. Just like every other year. ;) -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/

1 0

SSHD
by Russell Coker 13 Jan '16

13 Jan '16

http://cdn.msy.com.au/Parts/PARTS.pdf https://en.wikipedia.org/wiki/Hybrid_drive https://en.wikipedia.org/wiki/Shingled_magnetic_recording Are SSHDs any good? MSY has Seagate 8TB SSHDs for only $309. That's cheaper than a 6TB disk using regular technology, suspiciously cheap. I'm guessing that they use something like Shingled recording to improve the storage:price ratio and use the SSHD caching to allow decent performance without the OS optimising write patterns for it. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/

9 10

IPv6 at beginners' SIG
by Russell Coker 12 Jan '16

12 Jan '16

Is there any interest in an IPv6 training session at the beginners' SIG? I'm not volunteering to run it as I don't consider myself an expert on it (and I'm doing lots of other things). But I can help out, the LUV server isn't the first server I've setup for IPv6. Tony would you like to run such a session if there's interest? -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/

1 0

more about server migration
by Russell Coker 11 Jan '16

11 Jan '16

As mentioned in a previous message (which many people on the list probably didn't get) I've just moved the LUV server to a VM in Germany courtesy of Paul Menzel who pays the rent on the physical hardware. We had a few problems along the way with DNS records not updating when I wanted them to and with the wrong IP address being used for outbound mail - which caused some servers to correctly reject mail on the basis of bad SPF. DNS is now correct and has been correct for long enough that I expect most systems won't have old addresses cached any more (Google DNS has had the new values for a while). Andrew has started the process of gaining control over the DNS (which is registered in the names of people who haven't been committee members for a long time). We need to get that done before VPAC turns things off. My current plan is to use 2 servers that I run for LUV DNS secondaries. I've been a LUV member for 20 years and both the servers in question have been running for that long (one of them hasn't even changed it's IP address). I think that will give some assurance that things will keep working. As an aside if anyone knows how to contact the people running russell.linux.org.au and the creativecontingencies.com DNS servers about changing the entries for LUV then please let me know off-list. When running at VPAC the watchdog table in the Drupal database took up 20G of disk space and as it was in the ibdata1 file there was no good option for freeing the space even after deleting most rows. As part of the migration to the new server I dumped all MySQL databases and imported them. That solved the disk space problem and as I now have MySQL configured for a file per InnoDB table I will always be able to OPTIMIZE TABLE if I have that happen again. echo "delete from watchdog where timestamp < unix_timestamp() - 604800;" | mysql luv_drupal The above command is in a cron job to delete the old entries from the watchdog table. It would be good if we could fix the Drupal problem that's causing those entries (but I don't know how to do it or have time to learn). It would also be good if we could configure Drupal to not store millions of rows in that table (it's supposed to automatically delete old rows but doesn't). Assistance from Drupal experts is welcome. One thing I want to do is to convert the server to BTRFS. BTRFS snapshots is the best way of backing things up. Going forward I want to have much better backups of the server. The fact that I can't visit the server room any more makes them much more important as does the fact that data is crossing jurisdictions. I will resend this message tomorrow to make sure everyone gets it. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/

1 0