What's the cheapest way to create a high speed connection between two Linux servers these days? I'm using bundled 1Gb connections but I think they are creating a bit of a bottleneck these days. IB, FC, and 10Gb adapters aren't real cheap, even when you just go point to point without worrying about switches, but maybe there are other technologies I'm not aware of?? Thanks James
What's the cheapest way to create a high speed connection between two Linux servers these days? I'm using bundled 1Gb connections but I think they are creating a bit of a bottleneck these days.
IB, FC, and 10Gb adapters aren't real cheap, even when you just go point to point without worrying about switches, but maybe there are other technologies I'm not aware of??
Following up on myself... there used to be an IP-over-SCSI driver for Linux, is there such a thing for SAS where two servers are directly connected? James
On Sun, May 06, 2012 at 02:36:14AM +0000, James Harper wrote:
What's the cheapest way to create a high speed connection between two Linux servers these days? I'm using bundled 1Gb connections but I think they are creating a bit of a bottleneck these days.
ebay. if you know what you're looking for, you can find some bargains. but DO YOUR RESEARCH so you don't waste money on something that is exactly what the description said but isn't what you assumed it was :)
IB, FC, and 10Gb adapters aren't real cheap, even when you just go point to point without worrying about switches, but maybe there are other technologies I'm not aware of??
IB and 10Gb ethernet cards seem to go for around $600-$800 new. so a pair would get you a 10-40Gbps connection for around $1200-$1600. not cheap. i'm curious, so i'll do a search........hey, neat. options are much better than expected. This stuff's getting to be affordable for home use (at least 2nd hand). here's an ebay shop with Mellanox InfiniBand MHES18-XTC cards for $250, says more than 10 available. http://www.ebay.com.au/itm/Mellanox-InfiniBand-Adapter-Card-PN-MHES18-XTC-/1... and here's another (american) ebay shop selling what looks like re-badged mellanox MHGS18-XSC cards for $US45. again, "more than 10 available". http://www.ebay.com.au/itm/Infiniband-HCA-Hannstar-SuperMicro-Server-Interfa... mellanox cards are quite well supported by linux. There was a major update to the driver in the 3.3.x rc kernel, so you may want to run that or backport the driver to 3.2.x, or wait for 3.4 to be released. specs for both cards: http://www.mellanox.com/pdf/products/adapter_cards/IH3LX.pdf manual: http://www.mellanox.com/pdf/user_manuals/IH3Lx_PCIex8_HCA_user_manual_1_05.p... that PDF says the MHGS18-XSC is a DDR card so should be 20Gps. The MHES18-XTX is SDR, so only 10Gbps. you'll also need appropriate cables, of course. cheapest CX4 to CX4 cable i could find on ebay was $100 plus $34 for postage. This is a 20M DDR-ready fibre-optic. I don't know if it's suitable for the above cards or not. research required. http://www.ebay.com.au/itm/Zarlink-ZLynx-20M-CX4-CX4-Quad-Infiniband-20Gb-DD... NOTE: some supermicro cards are UIO rather than PCI-e. Electrically, they're identical to PCI-e (and will fit in the slot and work perfectly - my supermicro SAS card is one of these). Physically, the bracket is back-to-front compared to PCI-e, so you either have to use it without a bracket to hold it tightly in place (bad idea - jiggling the cable could easily dislodge the card), or you can use some kind of spacer to fit a standard PCI-e bracket. on http://en.wikipedia.org/wiki/PCI_Express , Wikipedia says "Universal IO: A variant from Super Micro Computer Inc designed for use in low profile rack mounted chassis. It has the connector bracket reversed so it cannot fit in a normal PCI Express socket, but is pin compatible and may be inserted if the bracket is removed." except it's not just low-profile. my UIO card is full height. I have NFI why supermicro decided to make a slightly-incompatible card form factor. more convenient layout perhaps. or maybe just vendor lockin bullshit.. craig -- craig sanders <cas@taz.net.au> BOFH excuse #106: The electrician didn't know what the yellow cable was so he yanked the ethernet out.
On Sunday 06 May 2012 14:41:24 Craig Sanders wrote:
IB and 10Gb ethernet cards seem to go for around $600-$800 new. so a pair would get you a 10-40Gbps connection for around $1200-$1600. not cheap.
Remember that for IPoIB the bandwidth you can get is determined by the subnet manager as well as the devices that are connected together, so a common issue with "managed" switches is that the built in subnet manager will only allow 10Gbps unless you find some way to tweak that. You can use "ipdiagnet" to spot that: -I--------------------------------------------------- -I- IPoIB Subnets Check -I--------------------------------------------------- -I- Subnet: IPv4 PKey:0x7fff QKey:0x00000b1b MTU:2048Byte rate:10Gbps SL:0x00 -W- Suboptimal rate for group. Lowest member rate:20Gbps > group-rate:10Gbps The MTU there is for IB in datagram mode, you can go way over that if you swap to connected mode.. cheers, Chris -- Chris Samuel : http://www.csamuel.org/ : Melbourne, VIC This email may come with a PGP signature as a file. Do not panic. For more info see: http://en.wikipedia.org/wiki/OpenPGP
participants (3)
-
Chris Samuel -
Craig Sanders -
James Harper