5 ms·

Redundant Array of Flaky Connections

30 points by jakozaur 14y ago

SlipperySlope 14y ago
Looks great for client-side connections! I wonder to what degree this is a solution for office-based servers. How would DNS failover be handled?
- weirdedhimself 14y ago
  Even though we don't run any office-based servers at Sumo Logic, the recommended way is to make the Peplink a DNS server. It will then serve the IP of the currently active link... They also just sent me this in an email: "Our revolutionary SpeedFusion technology is now available. Adding a wealth of innovations that take Peplink/Pepwave gear to the next level, SpeedFusion provides Balance users a cost-effective, high-bandwidth, and reliable alternative to using expensive MPLS connections in their branch VPNs, while SpeedFusion 4G/3G Bonding gives MAX users the ultimate in speedy and dependable networking on the road. Blazing fast, secure, and easy to manage, SpeedFusion is part of Firmware 5.4, so download this new release and become part of the SpeedFusion revolution today." http://www.peplink.com/speedfusion http://www.peplink.com/speedfusion
  - SlipperySlope 14y ago
    Looks great and thanks for the tip.
- omh 14y ago
  There are several tricks you can use for DNS failover, but it's not generally regarded as a reliable option. DNS caching has all kinds of awkward gotchas. You could also host a reverse proxy somewhere reliable which just forwards things to you over the appropriate connection If you need reliable hosting with multiple providers then you can get your own AS number and use BGP routing to provide failover. But that's more expensive by one or two orders of magnitude.
- pedrocr 14y ago
  You could setup a cloud machine to have the fixed public IP that DNS points to and then have that connect to your office through both connections and do the balancing. A cheap 15$/month low-memory VPS from Amazon or Linode would probably be enough. I wonder if tinc[1] is easy to use to do the load-balancing. [1] http://www.tinc-vpn.org/ http://www.tinc-vpn.org/
excuse-me 14y ago
1, "Set up multiple consumer grade internet connections." 2, Discover that all of them are just reselling the same service provider's product and they all go to the same cable in the same junction box under the street. Even with totally independent enterprise grade 'N'-9s uptime guarantees we discovered they are in the same trench under the road outside our site. A lot of major data centers have discovered that it only takes a single idiot with a backhoe.
- jakozaur 14y ago
  Perhaps checking trace route with mtr should give you the answer to 2.
  - excuse-me 14y ago
    Not with DSL. At least in europe the last mile is always BT/Deutsche Telkom etc but the other ISPs put their adapters in their racks. So it shows a different ISP all the way to the host but they are on the same wire and the same PSU in the same cage.
- bradleyland 14y ago
  Sure, but the vast majority of outages aren't related to a cut cable. I've done a ton of telecom work, and the thing I encountered most commonly was billing issues (ha!). Next up is equipment failure at either end of the line; bad card at the CO or bad modem on premisis. A distant third is outside issues like bad interconnects, and waaaaaaay out in forth is cut cables. It really doesn't happen all that often, even in big cities. By going with DSL and cable, they may be in luck even on that side of things. In a lot of places, phone lines (DSL) were run underground way back when the cities were built. Cable came much later, so it is frequently run above ground on utility poles. This means that if someone backhoes your DSL cable, the coax on the pole for cable internet will still be fine.
  - excuse-me 14y ago
    Yes - most of our problems have been our provider = "we decided to process your office move from 6months ago AGAIN and have disconnected you!" But unthinkingly deciding that having `N` systems means your risk of failure is `1/N` when you don't know how they are connected is a classic way of getting on the front page of the dailywtf.com !
losethos 14y ago
God is not flaky. You have to hold-up your end. "Failure to rescue"? God said the holocaust was to "compact" the Jews. As I think about it, Israel couldn't happen without the Holocaust. On Arabs and Jews, God said "Oil funny, hopefully." God says... thou be afraid of destruction when it cometh. 5:22 At destruction and famine thou shalt laugh: neither shalt thou be afraid of the beasts of the earth. 5:23 For thou shalt be in league with the stones of the field: and the beasts of the field shall be at peace with thee. 5:24 And thou shalt know that thy tabernacle shall be in peace; and thou shalt visit thy habitation, and shalt not sin. 5:25 Thou shalt know also that thy seed shall be great, and thine offspring as the grass of the earth. 5:26 Thou shalt come to thy grave in a full age, like as a shock of corn cometh in in his season. 5:27 Lo this, we have searched it, so it is; hear it, and know thou it for thy good. --------- God says... 9:10 And not only this; but when Rebecca also had conceived by one, even by our father Isaac; 9:11 (For the children being not yet born, neither having done any good or evil, that the purpose of God according to election might stand, not of works, but of him that calleth;) 9:12 It was said unto her, The elder shall serve the younger. 9:13 As it is written, Jacob have I loved, but Esau have I hated. 9:14 What shall we say then? Is there unrighteousness with God? God forbid. 9:15 For he saith to Moses, I will have mercy on whom I will have mercy, and I will have compassion on whom I will have compassion. 9:16 So then it is not of him that willeth, nor of him that runneth, but of God that sheweth mercy.
DenisM 14y ago
I could use a RAFC right now for cell-based internet here in Russia. There are 4 providers for wireless internet (3 UMTS and one CDMA450) and none of the three UMTS work reliably. Haven't tested the last one yet.
- keltex 14y ago
  Looks like the peplink guys (cited in the article) have just the thing for you: http://www.peplink.com/max-mobile-router/ http://www.peplink.com/max-mobile-router/
bdb 14y ago
We do basically the same thing at our office, though with IOS Event Manager foo sprinkled in. It works great.
icebraining 14y ago
we have connection outages almost daily Seriously? Is this considered acceptable for residential ISPs in California? Here I'd find one outage every month to be unacceptable, much less daily ones!
- omh 14y ago
  It depends how you define "outage". When we had an ADSL line it rarely went down, but regular ping tests along it often showed packet loss.
- weirdedhimself 14y ago
  Yeah, for consumer level ISPs in California, sadly, this is a reality. I've used Comcast, AT&T U-Verse, sonic.net, Speakeasy, etc over the years, and they all had outages. Amongst the better ones were Speakeasy and sonic.net. Comcast is amongst the worst. Especially here on Castro Street in downtown Mountain View, which is basically littered with tech startups, things are pretty bad.
  - ScottBurson 14y ago
    Huh, that's odd. I live in Sunnyvale and have had very little trouble with Comcast connectivity for most of the last several years. There was one period of time when I was seeing frequent (> daily) connection loss, but that was cured by a cable modem upgrade.
    - weirdedhimself 14y ago
      Let me add some nuance here: We have the "business" offering from Comcast. It was the only offering that had 100MBit/s when we signed up. With the "business" offering, you are forced to use the "business" modem. The modem is made by SMC. It is a full firewall/gateway. Unfortunately, the software on it is not very stable. For about 4-5 months, we worked with Comcast customer support to resolve this. They tried many different things to fix the issue, without success. My conclusion was that the software on the SMC modem just crashes once a day or so, and then hangs. Eventually, I convinced them to let us use the "simple" residential Motorola modem. That has worked much better. It still crashes every 1-2 weeks, but that's almost acceptable.
omh 14y ago
One advantage not mentioned here is when you need to perform maintenance. Being able to take any single device offline without affecting the service means that you can perform software upgrades or hardware replacements with more flexibility (and often within working hours)
InclinedPlane 14y ago
The biggest problem with this is the difference in ToS between consumer grade and business grade connections. If you try to use a consumer comcast cable connection for business you may run into things like bandwidth caps, filtered ports, etc.
- wmf 14y ago
  You should definitely get business service if you're going to do this; it's not that much more expensive.
  - InclinedPlane 14y ago
    Yes, and that's actually what they did here, even though they label them "consumer grade" connections.
rscale 14y ago
We helped a client of ours setup something akin to this, to work around their (essentially unfixable) Internet situation. We rolled our solution using OpenBSD/pf/carp/pfsync due to client expertise and comfort with BSD. It works well enough that outages on their lines have shown up only as nagios alerts, not actual outages, even through many years of service.

30 points by jakozaur 14y ago

Looks great for client-side connections! I wonder to what degree this is a solution for office-based servers. How would DNS failover be handled?

weirdedhimself 14y ago
Even though we don't run any office-based servers at Sumo Logic, the recommended way is to make the Peplink a DNS server. It will then serve the IP of the currently active link... They also just sent me this in an email: "Our revolutionary SpeedFusion technology is now available. Adding a wealth of innovations that take Peplink/Pepwave gear to the next level, SpeedFusion provides Balance users a cost-effective, high-bandwidth, and reliable alternative to using expensive MPLS connections in their branch VPNs, while SpeedFusion 4G/3G Bonding gives MAX users the ultimate in speedy and dependable networking on the road. Blazing fast, secure, and easy to manage, SpeedFusion is part of Firmware 5.4, so download this new release and become part of the SpeedFusion revolution today." http://www.peplink.com/speedfusion http://www.peplink.com/speedfusion
- SlipperySlope 14y ago
  Looks great and thanks for the tip.
omh 14y ago
There are several tricks you can use for DNS failover, but it's not generally regarded as a reliable option. DNS caching has all kinds of awkward gotchas. You could also host a reverse proxy somewhere reliable which just forwards things to you over the appropriate connection If you need reliable hosting with multiple providers then you can get your own AS number and use BGP routing to provide failover. But that's more expensive by one or two orders of magnitude.
pedrocr 14y ago
You could setup a cloud machine to have the fixed public IP that DNS points to and then have that connect to your office through both connections and do the balancing. A cheap 15$/month low-memory VPS from Amazon or Linode would probably be enough. I wonder if tinc[1] is easy to use to do the load-balancing. [1] http://www.tinc-vpn.org/ http://www.tinc-vpn.org/

excuse-me 14y ago

1, "Set up multiple consumer grade internet connections." 2, Discover that all of them are just reselling the same service provider's product and they all go to the same cable in the same junction box under the street. Even with totally independent enterprise grade 'N'-9s uptime guarantees we discovered they are in the same trench under the road outside our site. A lot of major data centers have discovered that it only takes a single idiot with a backhoe.

jakozaur 14y ago
Perhaps checking trace route with mtr should give you the answer to 2.
- excuse-me 14y ago
  Not with DSL. At least in europe the last mile is always BT/Deutsche Telkom etc but the other ISPs put their adapters in their racks. So it shows a different ISP all the way to the host but they are on the same wire and the same PSU in the same cage.
bradleyland 14y ago
Sure, but the vast majority of outages aren't related to a cut cable. I've done a ton of telecom work, and the thing I encountered most commonly was billing issues (ha!). Next up is equipment failure at either end of the line; bad card at the CO or bad modem on premisis. A distant third is outside issues like bad interconnects, and waaaaaaay out in forth is cut cables. It really doesn't happen all that often, even in big cities. By going with DSL and cable, they may be in luck even on that side of things. In a lot of places, phone lines (DSL) were run underground way back when the cities were built. Cable came much later, so it is frequently run above ground on utility poles. This means that if someone backhoes your DSL cable, the coax on the pole for cable internet will still be fine.
- excuse-me 14y ago
  Yes - most of our problems have been our provider = "we decided to process your office move from 6months ago AGAIN and have disconnected you!" But unthinkingly deciding that having `N` systems means your risk of failure is `1/N` when you don't know how they are connected is a classic way of getting on the front page of the dailywtf.com !

losethos 14y ago

God is not flaky. You have to hold-up your end. "Failure to rescue"? God said the holocaust was to "compact" the Jews. As I think about it, Israel couldn't happen without the Holocaust. On Arabs and Jews, God said "Oil funny, hopefully." God says... thou be afraid of destruction when it cometh. 5:22 At destruction and famine thou shalt laugh: neither shalt thou be afraid of the beasts of the earth. 5:23 For thou shalt be in league with the stones of the field: and the beasts of the field shall be at peace with thee. 5:24 And thou shalt know that thy tabernacle shall be in peace; and thou shalt visit thy habitation, and shalt not sin. 5:25 Thou shalt know also that thy seed shall be great, and thine offspring as the grass of the earth. 5:26 Thou shalt come to thy grave in a full age, like as a shock of corn cometh in in his season. 5:27 Lo this, we have searched it, so it is; hear it, and know thou it for thy good. --------- God says... 9:10 And not only this; but when Rebecca also had conceived by one, even by our father Isaac; 9:11 (For the children being not yet born, neither having done any good or evil, that the purpose of God according to election might stand, not of works, but of him that calleth;) 9:12 It was said unto her, The elder shall serve the younger. 9:13 As it is written, Jacob have I loved, but Esau have I hated. 9:14 What shall we say then? Is there unrighteousness with God? God forbid. 9:15 For he saith to Moses, I will have mercy on whom I will have mercy, and I will have compassion on whom I will have compassion. 9:16 So then it is not of him that willeth, nor of him that runneth, but of God that sheweth mercy.

DenisM 14y ago

I could use a RAFC right now for cell-based internet here in Russia. There are 4 providers for wireless internet (3 UMTS and one CDMA450) and none of the three UMTS work reliably. Haven't tested the last one yet.

keltex 14y ago
Looks like the peplink guys (cited in the article) have just the thing for you: http://www.peplink.com/max-mobile-router/ http://www.peplink.com/max-mobile-router/

bdb 14y ago

We do basically the same thing at our office, though with IOS Event Manager foo sprinkled in. It works great.

icebraining 14y ago

we have connection outages almost daily Seriously? Is this considered acceptable for residential ISPs in California? Here I'd find one outage every month to be unacceptable, much less daily ones!

omh 14y ago
It depends how you define "outage". When we had an ADSL line it rarely went down, but regular ping tests along it often showed packet loss.
weirdedhimself 14y ago
Yeah, for consumer level ISPs in California, sadly, this is a reality. I've used Comcast, AT&T U-Verse, sonic.net, Speakeasy, etc over the years, and they all had outages. Amongst the better ones were Speakeasy and sonic.net. Comcast is amongst the worst. Especially here on Castro Street in downtown Mountain View, which is basically littered with tech startups, things are pretty bad.
- ScottBurson 14y ago
  Huh, that's odd. I live in Sunnyvale and have had very little trouble with Comcast connectivity for most of the last several years. There was one period of time when I was seeing frequent (> daily) connection loss, but that was cured by a cable modem upgrade.
  - weirdedhimself 14y ago
    Let me add some nuance here: We have the "business" offering from Comcast. It was the only offering that had 100MBit/s when we signed up. With the "business" offering, you are forced to use the "business" modem. The modem is made by SMC. It is a full firewall/gateway. Unfortunately, the software on it is not very stable. For about 4-5 months, we worked with Comcast customer support to resolve this. They tried many different things to fix the issue, without success. My conclusion was that the software on the SMC modem just crashes once a day or so, and then hangs. Eventually, I convinced them to let us use the "simple" residential Motorola modem. That has worked much better. It still crashes every 1-2 weeks, but that's almost acceptable.

omh 14y ago

One advantage not mentioned here is when you need to perform maintenance. Being able to take any single device offline without affecting the service means that you can perform software upgrades or hardware replacements with more flexibility (and often within working hours)

InclinedPlane 14y ago

The biggest problem with this is the difference in ToS between consumer grade and business grade connections. If you try to use a consumer comcast cable connection for business you may run into things like bandwidth caps, filtered ports, etc.

wmf 14y ago
You should definitely get business service if you're going to do this; it's not that much more expensive.
- InclinedPlane 14y ago
  Yes, and that's actually what they did here, even though they label them "consumer grade" connections.

rscale 14y ago

We helped a client of ours setup something akin to this, to work around their (essentially unfixable) Internet situation. We rolled our solution using OpenBSD/pf/carp/pfsync due to client expertise and comfort with BSD. It works well enough that outages on their lines have shown up only as nagios alerts, not actual outages, even through many years of service.