4 ms·

I can understand that VIP's can sometimes be akward, ARP fails or even worse that VIPs won't be correctly released due to unknown stuff. However I do wonder wh

by merb 8y ago

I can understand that VIP's can sometimes be akward, ARP fails or even worse that VIPs won't be correctly released due to unknown stuff. However I do wonder why they choose such a complicated thing instead of looking into something like BGP that would've replaced their ARP setup. Also they should've probably tried to look into K8s since inside a cluster such problems do not exist. (There you have the problem to have a good LB at the outside of your cluster)

theptip 8y ago
> something like BGP that would've replaced their ARP setup Just a guess, but anecdotally it seems to me that inside a modern mid-size dev org there are a lot more engineers familiar with K/V stores like etcd/consul/ZK than IP routing protocols like OSPF and BGP. It's not necessarily the wrong decision to forgo an existing well-tested tool in favour of a hand-rolled solution, if you don't have any experts on that existing tool in your org.
- mschuster91 8y ago
  > It's not necessarily the wrong decision to forgo an existing well-tested tool in favour of a hand-rolled solution, if you don't have any experts on that existing tool in your org. It will, however, lead to other problems, especially: - reinvention of the wheel (aka: time waste) - starting from scratch: technologies like OSPF, BGP, or (when we're at higher levels) my/pgSQL vs noSQL are sometimes decades old and battle tested. Which means that there are a lot more edge cases that have been discovered and fixed/documented, as well as best practices that have evolved. Same goes for security. Or even fundamental stuff like ACID criteria (which many of the noSQL friends have had to re-discover and learn)... There's a reason yuge enterprises and government offices still run on mainframes (or, mainframe software on modern hardware): the stability and resilience of these beasts with uptimes in year-scales is something the hipster crap crowd is enviously looking at. And there's a reason that redesign projects in these areas fail so spectacularly: there are decades worth of domain knowledge and experience encoded in these systems. It's impossible to recreate this in a year or two, especially when outsourcing the work to body shops. Only thing I wonder is: it should be, by now, common knowledge (or: state of the art) that outsourcing yuge projects to body shops, especially foreign ones, leads to disasters. So why are managers and politicians not prosecuted or otherwise held responsible when screw-ups originate? After all, when one e.g. designs a building that violates fire code and people die as a result, or one grossly ignores security best practices, the responsible persons can get charged, too...
  - zzzcpan 8y ago
    > my/pgSQL vs noSQL are sometimes decades old and battle tested > run on mainframes (or, mainframe software on modern hardware): the stability and resilience of these beasts with uptimes in year-scales is something the hipster crap crowd is enviously looking at Why so much ignorance? Distributed systems are not hipster things, it's an actual science that helps to understand why no amount of battle testing can make this old tech resilient, it's fundamentally broken. There is just no resilience without at least proper distributed algorithms.
justinsaccount 8y ago
https://thenewstack.io/github-goes-kubernetes-tells/ https://thenewstack.io/github-goes-kubernetes-tells/
scurvy 8y ago
+1 for BGP/OSPF-based VIPs in a layer 3 network. It's 2018. No reason why you can't easily run a layer 3 network with rich routing protocols. Super easy. You don't need application level "discovery" when the IP never changes; the network will tell you where it currently lives. I'll take a layer 3 network over raft/consul any day and all night.
- voltagex_ 8y ago
  I don't think I've ever heard BGP described as "easy", but then again it's not really my area. Where could I go to read up on this without being a networking expert? Are there any safe ways to play with BGP in a lab environment? How expensive would said lab be to set up?
  - merb 8y ago
    Are there any safe ways to play with BGP in a lab environment? actually there are multiple ways of running bgp in a lab. there are cheap variants that are routed into the internet with ipv6 and there are ways to just play without without talking to the outside network. How expensive would said lab be to set up? it heavily depends on what you want. for a high available lab you would of course need multiple nodes/routers, depends on what you want to do. for just playing with bgp, actually using bird http://bird.network.cz/ http://bird.network.cz/ is good enough. Actually ripe sometimes publishes articles with BGP. Here is one that sets up a ANYCAST routing with BGP https://labs.ripe.net/Members/samir_jafferali/build-your-own-anycast-network-in-nine-steps https://labs.ripe.net/Members/samir_jafferali/build-your-own...
    - deleted 8y ago
      [deleted]
    - scurvy 8y ago
      Juniper and Cisco have lab versions of their products (Olive, vMX, vSRX) for free. Arista has something, too. BIRD is super easy to setup, too.

by merb 8y ago

theptip 8y ago

> something like BGP that would've replaced their ARP setup Just a guess, but anecdotally it seems to me that inside a modern mid-size dev org there are a lot more engineers familiar with K/V stores like etcd/consul/ZK than IP routing protocols like OSPF and BGP. It's not necessarily the wrong decision to forgo an existing well-tested tool in favour of a hand-rolled solution, if you don't have any experts on that existing tool in your org.

mschuster91 8y ago
> It's not necessarily the wrong decision to forgo an existing well-tested tool in favour of a hand-rolled solution, if you don't have any experts on that existing tool in your org. It will, however, lead to other problems, especially: - reinvention of the wheel (aka: time waste) - starting from scratch: technologies like OSPF, BGP, or (when we're at higher levels) my/pgSQL vs noSQL are sometimes decades old and battle tested. Which means that there are a lot more edge cases that have been discovered and fixed/documented, as well as best practices that have evolved. Same goes for security. Or even fundamental stuff like ACID criteria (which many of the noSQL friends have had to re-discover and learn)... There's a reason yuge enterprises and government offices still run on mainframes (or, mainframe software on modern hardware): the stability and resilience of these beasts with uptimes in year-scales is something the hipster crap crowd is enviously looking at. And there's a reason that redesign projects in these areas fail so spectacularly: there are decades worth of domain knowledge and experience encoded in these systems. It's impossible to recreate this in a year or two, especially when outsourcing the work to body shops. Only thing I wonder is: it should be, by now, common knowledge (or: state of the art) that outsourcing yuge projects to body shops, especially foreign ones, leads to disasters. So why are managers and politicians not prosecuted or otherwise held responsible when screw-ups originate? After all, when one e.g. designs a building that violates fire code and people die as a result, or one grossly ignores security best practices, the responsible persons can get charged, too...
- zzzcpan 8y ago
  > my/pgSQL vs noSQL are sometimes decades old and battle tested > run on mainframes (or, mainframe software on modern hardware): the stability and resilience of these beasts with uptimes in year-scales is something the hipster crap crowd is enviously looking at Why so much ignorance? Distributed systems are not hipster things, it's an actual science that helps to understand why no amount of battle testing can make this old tech resilient, it's fundamentally broken. There is just no resilience without at least proper distributed algorithms.

justinsaccount 8y ago

https://thenewstack.io/github-goes-kubernetes-tells/ https://thenewstack.io/github-goes-kubernetes-tells/

scurvy 8y ago

+1 for BGP/OSPF-based VIPs in a layer 3 network. It's 2018. No reason why you can't easily run a layer 3 network with rich routing protocols. Super easy. You don't need application level "discovery" when the IP never changes; the network will tell you where it currently lives. I'll take a layer 3 network over raft/consul any day and all night.

voltagex_ 8y ago
I don't think I've ever heard BGP described as "easy", but then again it's not really my area. Where could I go to read up on this without being a networking expert? Are there any safe ways to play with BGP in a lab environment? How expensive would said lab be to set up?
- merb 8y ago
  Are there any safe ways to play with BGP in a lab environment? actually there are multiple ways of running bgp in a lab. there are cheap variants that are routed into the internet with ipv6 and there are ways to just play without without talking to the outside network. How expensive would said lab be to set up? it heavily depends on what you want. for a high available lab you would of course need multiple nodes/routers, depends on what you want to do. for just playing with bgp, actually using bird http://bird.network.cz/ http://bird.network.cz/ is good enough. Actually ripe sometimes publishes articles with BGP. Here is one that sets up a ANYCAST routing with BGP https://labs.ripe.net/Members/samir_jafferali/build-your-own-anycast-network-in-nine-steps https://labs.ripe.net/Members/samir_jafferali/build-your-own...
  - deleted 8y ago
    [deleted]
  - scurvy 8y ago
    Juniper and Cisco have lab versions of their products (Olive, vMX, vSRX) for free. Arista has something, too. BIRD is super easy to setup, too.