April 2012 - luv-main

ZFS vs BTRFS
by Russell Coker 20 Apr '12

20 Apr '12

I've got a few of my systems (including some with data that is important to me) using BTRFS now. It's been going well and one of my systems has had BTRFS for /home for ages (maybe a year) with no problems in recent times (some past problems when running out of space). One of my clients needs to reliably store terabytes of data which is mostly comprised of data files in the 10MB - 15MB size range. The data files will almost never be re-written and I anticiapte that the main bottleneck will be the latency of NFS and other network file sharing protocols. I would hope that saturating a GigE network when sending 10MB data files from SATA disks via NFS, AFS, or SMB wouldn't be a technical challenge. It seems that BTRFS is the way of the future. But it's still rather new and the lack of RAID-5 is a serious issue when you need to store 10TB with today's technology (that would be 8*3TB disks for RAID-10 vs 5*3TB disks for RAID-5). ZFS seems to be a lot more complex than BTRFS. While having more features is a good thing (BTRFS seems to be missing some sysadmin friendly features) complexity means more testing and more potential for making mistakes. Of course it might turn out that RAID-5 is the killer issue. Servers start becoming a lot more expensive if you want more than 8 disks and even 6 disks is a significant price point. An 8 disk RAID-5 gives something like 21TB usable space vs 12TB on a RAID-10 and a 6 disk RAID-5 gives about 15TB vs 9TB on a RAID-10. Anything else I should consider? -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/

10 25

Well-supported USB3 controllers
by Toby Corkindale 19 Apr '12

19 Apr '12

Hi, There was some brief discussion recently about trouble with a specific USB3 controller under Linux. I wondered if anyone can provide anecdotal evidence for controllers which work *well* under Linux? Day in, day out, without hiccups. Also, ideally ones which are supported well in the Ubuntu 12.04 LTS or Debian Wheezy kernels, rather than bleeding edge. (So that's 3.2.0 or earlier) Cheers, Toby

6 13

distributed databases
by Russell Coker 19 Apr '12

19 Apr '12

One of my clients is proposing a project that requires good storage performance and high reliability. It's an entirely new project so there's no legacy code to deal with. The traditional way of doing this would be to have a cluster of systems maybe in an activa/passive configuration with database replication or with MySQL or PostgreSQL clustering. Those solutions are difficult to manage and upgrade. http://en.wikipedia.org/wiki/Apache_Cassandra I think that probably the best thing to do is to use something like Cassandra on a cluster of servers in a DC to run this. The Cassandra feature set seems good (including being able to add new servers at run-time) and developing it from scratch can't be a lot harder than doing MySQL development. Does anyone have any suggestions for planning at this early stage? I had thought of doing something similar with the Amazon EC2 equivalent to Cassandra, but a quick scan of their web site reveals no mention of it. Did Amazon cancel their cloud key-value store service? -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/

7 11

booting from RAID-1
by Russell Coker 18 Apr '12

18 Apr '12

I've got a lot of servers with the root filesystem (including /boot) on a RAID-1 array. For years I've had systems configured such that if one drive fails then the other can boot the system (although in the case of IDE if drive 0 fails then the other drive would have to be changed to the other cable before booting). http://etbe.coker.com.au/2012/04/17/zfs-btrfs-cheap-servers/ Now I'm considering the case of a reasonable size ZFS server. Would it work to have an 8-disk mirror for the root filesystem? I'll probably put /var on ZFS so the root filesystem won't need to be particularly big. In terms of wasting space a 10G root filesystem mirrored across all disks isn't a big deal when you have 3TB disks. It seems that the ability to boot from any disk and having a symmetric layout provide more benefit than saving maybe 60G of raw disk capacity. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/

5 10

Re: [luv-talk] Buying a new server
by Craig Sanders 18 Apr '12

18 Apr '12

some suggestions: On Wed, Apr 18, 2012 at 01:53:01PM +1000, Dave Oxley wrote: > I have an 7 year old dell server that is in need of replacement. It's > uses among others are MythTV backend, Asterisk, Web server, email, IP > routing, various Java web apps. I had spec'ed and almost bought a new > Dell server but got really disenchanted with their crappy sales guys. > [...] > I've since been looking into building my own server but I'm not sure > what to do for a case. I'm after a case with 6+ hot-plug backplane, > preferably redundant PSU and tool-less. I found the Chenbro SR11269 > which is mostly what I need but various places sell different PSU's for > it. There's the C2W-3820V-R and C2W-5820V which differ in price by about > $200 but I can't find any information about what's differences between > them. Also they state '*For Tylersburg' so I'm not sure if they'll work > with the motherboard I've chosen (S2600COE). I've also looked at Intel > cases for this motherboard but they come at a premium. Norco 4224, 24 SAS/SATA hot-swap bays in a 4RU case, $459 from techbuy. http://www.techbuy.com.au/p/149123/CASINGS_SERVER_-_4U/Norco/RPC-4224.asp they have 4220 and 4216 models with less bays, but they cost about $100 more. My ZFS box at work is in one of these cases, and i'm very tempted to get a 4224 for myself at home. I don't have that many drives, but having lots of room for expansion and drive-swapping would be nice. they also have a bracket available for the internal fan mounts that allows you to replace the 4 x 80mm fans (which are extremely loud) with 3 x 120mm fans - there are some very nice quiet ones available. you'll also want to replace the 2 80mm fans at the back of the case. they are meant for server rooms where you wouldn't hear them over the AC anyway. the fans are not at all compatible with home use (if you value your peace and quiet...or your hearing, for that matter) http://www.techbuy.com.au/p/152644/CASINGS_SERVER_-_ACCESSORIES/Norco/120mm… it's only $10.95, not including fans (expect to pay about $30 each for nice quiet fans). get it. it would be worth it if was an extra $100. add an IBM M1015 8-port SAS/SATA 6Gbps card (on ebay for about $69 + $30 postage from the US). you'll need two SAS SFF-8087 cables to connect the card to the host swap bays. You can find these on about for about $10. http://www.servethehome.com/ibm-m1015-part-1-started-lsi-92208i/ http://www.ebay.com.au/itm/IBM-ServeRAID-M1015-SAS-RAID-Controller-FRU-46M0… t_1191 to connect the motherboard's SATA ports to the bays, you'll need a *REVERSE* SATA to SFF-8087 cable. these also cost about $10 on ebay. it is important to get the reverse cables. the normal ones are for connecting a SAS card to a SATA drive, the reverse cables are to connect a SATA port to a SAS or SATA hotswap backplane. techbuy also sell the cables. but they'll cost 3-5 times as much. if you ever needed to use the other 16 bays (minus what the motherboard ports give access to) in the case, you can add another M1015 card or two. just make sure your motherboard has enough 8x PCI-e slots. > So does anyone have any suggestions of where I go from here? I'm happy > to buy a pre-built server but not happy about buying Dell anymore. I'm > also happy to build my own. > > The budget is about $3000 and the specs I'm after are roughly are: > 1x 6 core Intel Xeon processor > Motherboard with 2 processor sockets, 2+ Gigabit LAN, 6+ SATA > 12+ GB or ram upgradable > 6+ hot plug case with SATA backplane techbuy also have several models of supermicro and tyan server motherboards in stock. i'd compare any prices with either shopbot.com.au or staticice.com.au - maybe you'll find a better deal. personally, i prefer Opteron to Xeon. much better value for money, imo. and the latest AMD "interlagos" CPUs currently have up to 16 cores. some of the motherboards have SAS controllers built-in (but you'll need two of the reverse SATA to SFF-8087 cables to connect them to the backplane). it's probably cheaper to just buy an M1015 card and put it in an 8x PCI-e slot, and makes it easier to upgrade/replace the motherboard later. you might also want to look at pre-built servers from either xenon.com.au or ipspty.com.au - the price difference isn't all that great compared to building it yourself....but you'll still have replace the fans to make it quiet enough for home use. pre-built will probably break your $3K budget, though. > redundant PSU redundant PSU will add a lot to the price - at least $800. maybe that money would be better spent on a good sized UPS. > 9+ TB raid 5 Disk space use ZFS as your filesystem, all the benefits of raid and lvm and more without the hassle - raidz-1 is roughly equivalent to RAID-5 (actually, better than RAID-5). add a 120GB or 240GB SSD as either boot/OS + ZIL write cache + read cache, or boot off ZFS and just use the SSD for the ZIL log and the read cache. The ZIL write cache smoothes out random writes, and gives the same effect as the non-volatile write cache in a hardware raid-card (i.e. write performance doesn't suck. it sucks badly in software raid) 4 or 5 x 3TB drives would give you 9 or 12TB of RAID-5/raidz-1 storage. $800 or $1000. note that there is a minor but noticable performance benefit in ZFS (maybe in mdadm too, i can't recall) if the number of data disks (i.e. total disks minus parity disk(s) is a power of two). or if the budget will stretch to it, 8 x 3TB (or perhaps 8 x 2TB) drives in a raidz-2 (raid-6 equivalent) array. 18TB and $1600 for 3TB drives, or 12TB and $1110 for 2TB drives. either would take all 8 ports on an M1015 SAS card. here's a good page: http://www.servethehome.com/midrange-diy-storage-server-buyers-guide-octobe… the entire site is worth reading. some good stuff there, although with a fairly strong windows focus. ignore that, the hardware stuff is mostly generic and applicable to linux too. also, unlikely as it may sound for an overclocking site, but overclockers.com.au has a fairly decent forum on home and semi-professional storage servers. http://forums.overclockers.com.au/forumdisplay.php?f=44 craig -- craig sanders <cas(a)taz.net.au> BOFH excuse #407: Route flapping at the NAP.

1 0

Bigpond Ultimate modem (sierra) ... what can you do with it?
by Jobst Schmalenbach 17 Apr '12

17 Apr '12

Hi I used to have a telstra plan with a Bigpond Ultimate modem (sierra) What can you do with it afterwards? chuck it? Jobst -- Steve Ballmer is a MCMJ (Microsoft Certified Mouse Jockey). | |0| | Jobst Schmalenbach, jobst(a)barrett.com.au, General Manager | | |0| Barrett Consulting Group P/L & The Meditation Room P/L |0|0|0| +61 3 9532 7677, POBox 277, Caulfield South, 3162, Australia

2 1

Re: "green" drives ( was Re: Moving a PV onto raid)
by Craig Sanders 11 Apr '12

11 Apr '12

On Wed, Apr 11, 2012 at 09:36:43AM +1000, Aníbal Monsalve Salazar wrote: > [ Cc-ing madduck as he may be interested on Craig's comments above ] btw, i had already disabled idle spindown with hdparm -S 0 idle3-tools just updates the firmware settings so that it's disabled by default (as should be the recommend setting for WD Green drives on any linux system).. I had experimented with spindown over summer (didn't want my drives to overheat) but found that ZFS didn't like it. at all. kept getting drives being booted from the zpool seemingly at random, so ended up disabling it. of much less importance, I also found that hddtemp woke the drives up anyway, whenever it queried the temperature. [...] echo disabling disk idle spindown for disk in $DEVICES ; do /sbin/hdparm -q -S 0 "/dev/$disk" done [...] > It's a Debian package since yesterday evening. nice :) craig -- craig sanders <cas(a)taz.net.au> BOFH excuse #314: You need to upgrade your VESA local bus to a MasterCard local bus.

4 8

re: fsync (was Re: "green" drives ( was Re: Moving a PV onto raid))
by Peter Ross 11 Apr '12

11 Apr '12

On Wed, 11 Apr 2012, Toby Corkindale wrote: > According to the man page: > flush If set, the filesystem will try to flush to disk more early > than normal. Not set by default. Can you say what the default kernel policies related to file system flushing are these days? My knowledge may be outdated. Personally I don't care that much whether _my_ docs are saved immediatelly or not, I am connected to a 99.something % reliable power grid, and a loss of a byte once in a blue moon isn't the end of the world I guess. I believe in using menu options like "Safely remove storage" - they imply that without it I may be in trouble. I don't know Windows in detail but I know they do file buffering too. So since leaving single task OSes as CP/M or MS/DOS behind I don't expect that "written by app" means "written to disk". The servers have UPS and battery-backed controllers to ensure this. Of course I am in trouble if the file system gets itself in the knot while flushing - one reason to avoid file systems with a bad reputation. Most of the time kernel developers are writing better and safer code than application developers. vi and emacs are a bit of the exception that proves the rule. Regards Peter

1 0

mail storage in a distributed database
by Russell Coker 10 Apr '12

10 Apr '12

Does anyone know of a mail store that uses a distributed database like Cassandra? I want something that has a delivery agent with a similar interface to maildrop or procmail and which has POP and IMAP servers to provide client access. IBM had a research project implementing a mail server on Cassandra but as far as I can tell they released nothing other than a PDF. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/

8 29

iscsi (was RE: drdb and mdadm and more)
by James Harper 10 Apr '12

10 Apr '12

> > The random i/o interspersed with the rebuild i/o has the effect of totally > trashing the rebuild performance. Random i/o over iscsi has sucked on the > stable Debian kernels. (I believe the better-performing iscsi drivers (which > are a totally independent rewrite) have finally made it into wheezy though.) > Are you referring to the target, initiator, or both here? Wheezy has lio target now which is marketed as faster and better than everything else, although there are arguments about that from the other iscsi target camps... I think lio also has an initiator but I haven't seen Debian packages for that yet. I'm using lio after using iscsitarget (iet) without any problems. James

2 1