Hey all, Does anyone have real-world experience of using linux's interface bonding on public networks? (In the bandwidth-aggregation mode, not the redundancy mode) I was wondering how I could make the following setup work: * Rent a VPS in Melbourne with four IP addresses * Get four (or just two) ADSL connections wired up to home * Have your VPS connect four VPN connections from itself back to each of your home IPs. * Bond all four interfaces together * Create a fifth VPN connection, this time going over the bonded-virtual-interface between VPS and home, and then configure your home server to use that link as the default route? It sounds pretty messy and I'm not sure it'd actually work in practice; the routing tables would be hell to get right. Are there any guides already out there? Cheers, Toby
Hey all, Does anyone have real-world experience of using linux's interface bonding on public networks? (In the bandwidth-aggregation mode, not the redundancy mode)
I was wondering how I could make the following setup work: * Rent a VPS in Melbourne with four IP addresses * Get four (or just two) ADSL connections wired up to home * Have your VPS connect four VPN connections from itself back to each of your home IPs. * Bond all four interfaces together * Create a fifth VPN connection, this time going over the bonded-virtual-interface between VPS and home, and then configure your home server to use that link as the default route?
It sounds pretty messy and I'm not sure it'd actually work in practice; the routing tables would be hell to get right.
Are there any guides already out there?
I've use PPP multilink over ADSL before, many years ago, but your telco has to support that and I suspect not many do as these days ADSL is disproportionally cheap compared to anything else these days and they'd want to push you onto their higher SLA (and price) products. PPP multilink reduces latency a bit too because it splits each packet up equally. This more than offsets the additional overheads. Your way will likely increase latency. A lot for small packets. You have your top layer VPN, your next layer VPN, and then the PPPoE that you use for ADSL has a whole load more layers under it too. (the way we implement ADSL here in Australia is dumb) I think you might be able to use multilink via ppp over L2TP though, which would mean you would have just one VPN spread across 4 links, and it may allow you to split up the packets too. This would reduce a few layers, and IPSEC works well in conjunction with L2TP. That would give you redundancy too if everything is working right. A quick google search for linux multilink l2tp shows up a few promising leads, but if nobody else is doing it the relevant code could have suffered a bit of bitrot. I remember having to go bug fixing in ppp when I tried it over ADSL, but that was a long time ago. James
Toby Corkindale <toby@dryft.net> writes:
Does anyone have real-world experience of using linux's interface bonding on public networks? (In the bandwidth-aggregation mode, not the redundancy mode)
I've done aggregation bonding before and it was sucky - weird heisenbugs IIRC. Now I just buy a 10G NIC.
I was wondering how I could make the following setup work: * Rent a VPS in Melbourne with four IP addresses * Get four (or just two) ADSL connections wired up to home * Have your VPS connect four VPN connections from itself back to each of your home IPs. * Bond all four interfaces together * Create a fifth VPN connection, this time going over the bonded-virtual-interface between VPS and home, and then configure your home server to use that link as the default route?
If that can even work, it's pretty awful. Why do you want this? Can you move the office closer instead? ;-)
On Fri, 28 Mar 2014 14:51:21 Toby Corkindale wrote:
Does anyone have real-world experience of using linux's interface bonding on public networks? (In the bandwidth-aggregation mode, not the redundancy mode)
http://simonmott.co.uk/vpn-bonding The above blog post describes how to do this, it's apparently worked for someone but I wouldn't expect it to work easily or want to rely on it. http://etbe.coker.com.au/2007/08/14/ethernet-bonding-and-a-xen-bridge/ Last time I tried bonding I wasn't impressed, for my home network I started using bridges with STP for reliability instead. I'd rather spend my time trying to get the Liberal party voted out than go to such extremes in search of better performance. -- My Main Blog http://etbe.coker.com.au/ My Documents Blog http://doc.coker.com.au/
On 28/03/14 14:51, Toby Corkindale wrote:
Hey all, Does anyone have real-world experience of using linux's interface bonding on public networks? (In the bandwidth-aggregation mode, not the redundancy mode)
I was wondering how I could make the following setup work: * Rent a VPS in Melbourne with four IP addresses * Get four (or just two) ADSL connections wired up to home * Have your VPS connect four VPN connections from itself back to each of your home IPs. * Bond all four interfaces together * Create a fifth VPN connection, this time going over the bonded-virtual-interface between VPS and home, and then configure your home server to use that link as the default route? Two options:
1. Some ISPs (IIRC 'node and Exetel both offered this at one point) offered MLPPP for a fairly nominal price over two ADSL lines 2. Just use Linux's load balancing to choose between several normal ADSL lines You don't need the additional IPs on the VPS for your example, and again could simply ECMP between them, trying to bond one TCP session across multiple links is a really bad idea.
On 28 March 2014 17:33, Trent W. Buck <trentbuck@gmail.com> wrote:
Toby Corkindale <toby@dryft.net> writes:
Does anyone have real-world experience of using linux's interface bonding on public networks? (In the bandwidth-aggregation mode, not the redundancy mode)
I've done aggregation bonding before and it was sucky - weird heisenbugs IIRC. Now I just buy a 10G NIC.
I was wondering how I could make the following setup work: * Rent a VPS in Melbourne with four IP addresses * Get four (or just two) ADSL connections wired up to home * Have your VPS connect four VPN connections from itself back to each of your home IPs. * Bond all four interfaces together * Create a fifth VPN connection, this time going over the bonded-virtual-interface between VPS and home, and then configure your home server to use that link as the default route?
If that can even work, it's pretty awful.
Why do you want this?
Because I get about 3.5 mbit downstream on ADSL on my line at home, and I'm impatient when it comes to waiting for youtube videos to buffer or large updates to download... at least with four lines I might get an aggregate of 12mbit, which is just about acceptable.
Can you move the office closer instead? ;-)
Yes, but relocating my home (since I own it) would easily cost $100k by the time you include stamp duty, real estate agent's fees, movers and probably some minor renovation at both ends, plus the fact that houses inside the NBN zones in Melbourne are more expensive (on average) than where I currently live.
On 28 March 2014 17:46, Russell Coker <russell@coker.com.au> wrote:
On Fri, 28 Mar 2014 14:51:21 Toby Corkindale wrote:
Does anyone have real-world experience of using linux's interface bonding on public networks? (In the bandwidth-aggregation mode, not the redundancy mode)
http://simonmott.co.uk/vpn-bonding
The above blog post describes how to do this, it's apparently worked for someone but I wouldn't expect it to work easily or want to rely on it.
That's roughly what I was theorising.. Good to see someone else has tried it and had it work -- although yeah, it looks and sounds rather convoluted. I found that a company called Fusion sells boxes that sound like they do essentially that -- you plug arbitrary ADSL/4G/whatever WAN connections into it, and it aggregates them all off to a box somewhere in their data centre. Costs $145/month to bond two lines, or arbitrary numbers of lines for $95/month. The router was quite expensive too.
http://etbe.coker.com.au/2007/08/14/ethernet-bonding-and-a-xen-bridge/
I'd rather spend my time trying to get the Liberal party voted out than go to such extremes in search of better performance.
sigh. yeah, but it'll be 2020 by the time we get rid of them, and in the meantime I'd really, really like to have vaguely-usable internet access. Although since Optus or Telstra apparently accidentally unplugged my port on the DSLAM last week and I was without any internet access at all for a while, I suddenly realised that 3-4 mbit is a lot better than none at all!
Toby Corkindale <toby@dryft.net> wrote:
sigh. yeah, but it'll be 2020 by the time we get rid of them, and in the meantime I'd really, really like to have vaguely-usable internet access.
How good is cable (especially DOCSIS 3) in Melbourne now? It used to be unfriendly to Linux, requiring, as I recall, a special login tool for Telstra and restricting "servers" from being run under Optus, and I'm not sure about speed or reliability.
Although since Optus or Telstra apparently accidentally unplugged my port on the DSLAM last week and I was without any internet access at all for a while, I suddenly realised that 3-4 mbit is a lot better than none at all!
An advantage of having a reasonably modern mobile device (phone or tablet, for example) is that it can serve as an emergency Internet connection in case of a fixed-line outage such as you describe.
On Mon, Mar 31, 2014 at 2:53 PM, Jason White <jason@jasonjgw.net> wrote:
Toby Corkindale <toby@dryft.net> wrote:
How good is cable (especially DOCSIS 3) in Melbourne now?
It varies. At my old house, it was very good for downloads (80Mb/s down, ~ 2Mb/s up). Moved less than a km away, and now it's 20Mb/s down, 1.2Mb/s up.
It used to be unfriendly to Linux, requiring, as I recall, a special login tool for Telstra and restricting "servers" from being run under Optus, and I'm not sure about speed or reliability.
Not any more. I use their provided modem, plugged into a dd-wrt rebranded DIR-632. Absolutely stable and no problems at all. Completely OS agnostic. / Brett
Brett Pemberton <brett.pemberton@gmail.com> wrote:
On Mon, Mar 31, 2014 at 2:53 PM, Jason White <jason@jasonjgw.net> wrote:
Toby Corkindale <toby@dryft.net> wrote:
How good is cable (especially DOCSIS 3) in Melbourne now?
It varies. At my old house, it was very good for downloads (80Mb/s down, ~ 2Mb/s up). Moved less than a km away, and now it's 20Mb/s down, 1.2Mb/s up.
Contention?
It used to be unfriendly to Linux, requiring, as I recall, a special login tool for Telstra and restricting "servers" from being run under Optus, and I'm not sure about speed or reliability.
Not any more. I use their provided modem, plugged into a dd-wrt rebranded DIR-632. Absolutely stable and no problems at all.
Completely OS agnostic.
Excellent. The uptake of mobile devices is certainly going to help Linux users in this respect: a service that requires a specific operating system (or one of a handful of systems) will become less attractive over time. Supporting a range of mobile operating systems is more effort than it is worth, when one can simply offer a modem or router that implements standard protocols. Linux in its various forms simply adds to the diversty - not "officially" supported by ISPs, but compatible thanks to networking standards.
How good is cable (especially DOCSIS 3) in Melbourne now? It used to be unfriendly to Linux, requiring, as I recall, a special login tool for Telstra and restricting "servers" from being run under Optus, and I'm not sure about speed or reliability.
100/3mbit speeds for me (outer eastern Melbourne suburbs), outside of one multi-day outage due to something up the street blowing up, no problems. OS agnostic; they provide a Cable modem that'll do DHCP/NAT/etc. R
On 31 March 2014 14:53, Jason White <jason@jasonjgw.net> wrote:
Toby Corkindale <toby@dryft.net> wrote:
sigh. yeah, but it'll be 2020 by the time we get rid of them, and in the meantime I'd really, really like to have vaguely-usable internet access.
How good is cable (especially DOCSIS 3) in Melbourne now? It used to be unfriendly to Linux, requiring, as I recall, a special login tool for Telstra and restricting "servers" from being run under Optus, and I'm not sure about speed or reliability.
If you're willing to pay extra, you can supposedly get 100mbit downstream on Telstra's cable. Annoyingly, it runs down my street, but they won't hook me up because they're phasing it out due to the NBN coming in.. except my street *doesn't* get the NBN. >.<
Jason White <jason@jasonjgw.net> writes:
Toby Corkindale <toby@dryft.net> wrote:
sigh. yeah, but it'll be 2020 by the time we get rid of them, and in the meantime I'd really, really like to have vaguely-usable internet access.
How good is cable (especially DOCSIS 3) in Melbourne now? It used to be unfriendly to Linux, requiring, as I recall, a special login tool for Telstra and restricting "servers" from being run under Optus, and I'm not sure about speed or reliability.
You are remembering "heartbeat". There was a perl script that faked it on Linux ca. 2005, but Telstra stopped needing it a few years later. Can't find a Wikipedia reference, but http://ozcableguy.com/heartbeat.asp Re "contention?" downthread, my vague recollection is that the bandwidth is shared by everyone on the street, so it's fast as long as everyone else on the street has DSL. Cf. DSL runs over a dedicated copper pair between you and the exchange.
On 28 March 2014 23:55, Julien Goodwin <luv-lists@studio442.com.au> wrote:
On 28/03/14 14:51, Toby Corkindale wrote:
Hey all, Does anyone have real-world experience of using linux's interface bonding on public networks? (In the bandwidth-aggregation mode, not the redundancy mode)
I was wondering how I could make the following setup work: * Rent a VPS in Melbourne with four IP addresses * Get four (or just two) ADSL connections wired up to home * Have your VPS connect four VPN connections from itself back to each of your home IPs. * Bond all four interfaces together * Create a fifth VPN connection, this time going over the bonded-virtual-interface between VPS and home, and then configure your home server to use that link as the default route? Two options:
1. Some ISPs (IIRC 'node and Exetel both offered this at one point) offered MLPPP for a fairly nominal price over two ADSL lines
I forgot to mention earlier -- but both Exetel and iiNet (nee Internode) stopped offering bonded DSL in 2014. It sounds like they were having a lot of trouble setting it up or supporting it. (According to whirlpool, the single option of hardware they were using at the client end was both expensive and awfully crap) An ISP called IIG seems to still be offering bonded DSL, so I've put in an enquiry with them. No prices were listed. Fusion Broadband do a thing where you use their appliance at the client end, and it bonds arbitrary WAN connections, apparently with quite good latency. It's moderately pricy at $95 per WAN connection you want to bond -- on top of the cost of getting the WAN connection -- but does sound like it's a successful solution for some. I'm leaning towards trying the Linux DIY approach to link aggregation.. if it works, great, otherwise even just using the round-robin load-balancing mode would present speed-ups for most things. (web pages, torrents, apt, but not linux ISOs, steam downloads or nvidia drivers*) -Toby * seriously WHY ARE THEY SO HUGE? Quarter of a gigabyte for windows display drivers? What's in them??
Hi, Just following up on my rather old post -- but it took about three months to go from wanting it to getting everything provisioned. I'm running two ADSL lines with Linux doing interface bonding on the (bridged) PPPoE ADSL; the ISP is doing some kind of similar bonding at their end, although I suspect on Cisco hardware, not Linux. Performance seems decent -- prior to this I was getting around 4.5 to 5 mbit down, 0.6mbit up via Exetel on Optus hardware. With the two new lines (Telstra) I'm getting total 9.5 mbit down and 1.6 mbit up via IIG on Annex M AAPT hardware. (Performance tested with the not-super-accurate-but-standard speedtest.net) (Refreshing the landlines helped -- definitely gained 500-1000 kbit per line prior to bonding, perhaps just due to the cables being re-crimped at both ends.. who knows) Whether it actually turns out to be reliable in the long run, I'll see... I do get HA in effect, as if one line goes down, all traffic goes over the other after a fraction of a second delay. The actual bonding stuff in Linux (Ubuntu LTS) was a bit fiddly to configure -- the /etc/network/interfaces.d stanzas for bonding are really only designed to work with ethernet ports or similar, that are up instantly. To make it work automatically with pppd, I needed to define the bond0 interface as having zero slaves, and then add a script to /etc/ppp/ip-up.d/ which calls ifenslave -f bond0 $PPP_INTERFACE (if the IP address matches one of those from my ISP) Toby On 28 March 2014 14:51, Toby Corkindale <toby@dryft.net> wrote:
Hey all, Does anyone have real-world experience of using linux's interface bonding on public networks? (In the bandwidth-aggregation mode, not the redundancy mode)
I was wondering how I could make the following setup work: * Rent a VPS in Melbourne with four IP addresses * Get four (or just two) ADSL connections wired up to home * Have your VPS connect four VPN connections from itself back to each of your home IPs. * Bond all four interfaces together * Create a fifth VPN connection, this time going over the bonded-virtual-interface between VPS and home, and then configure your home server to use that link as the default route?
It sounds pretty messy and I'm not sure it'd actually work in practice; the routing tables would be hell to get right.
Are there any guides already out there?
Cheers, Toby
-- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
participants (8)
-
Brett Pemberton -
James Harper -
Jason White -
Julien Goodwin -
Russell Coker -
Ryan Verner -
Toby Corkindale -
trentbuck＠gmail.com