9 ms·
Do we really need a new BGP?
- smcleod 9y agoSlightly aside of BGP on an internet scale - We’ve recently switched to using BGP internally within our network for kubernetes + load balancer (nginx) routing, (announcement / discovery) amongst other things and I can tell you - it’s one of the best moves we’ve ever made. The simplicity, performance and reliability is brilliant. https://www.projectcalico.org https://www.projectcalico.org
- deleted 9y ago[deleted]
- _nalply 9y agoDo you have loops internally?
- noselasd 9y agoI would think people deploy BGP to automate routing and routing policies, not primarily to avoid loops.
- mrmondo 9y agoIndeed, completely for fast, simple routing (rather than higher level, more complex failover). Slightly off topic, but in my experience most (general) internal networking loops stem from mis configured devices bridging networks (desktop level, network device level or VM / SDN)
- bartvanH 9y agoI may be old, but if you have a network loop inside your own infrastructure wouldn't it be way more sane to handle that on layer 2? for instance with properly configured spanning tree?
- woah 9y agoA routing protocol will avoid loops more efficiently
- jethro_tell 9y agoAlso, it's allows loops to be used as multi paths from different points in the net. Say you have a rack at south and north and there is a loop to a rack in the middle. For failover, you'll take any path that is up. But for normal operation, you'd prefer to take the shortest path.
- deleted 9y ago[deleted]
- lima 9y agoLayer 2 technologies like MC-LAG do that as well (though I agree that it's better to route, if possible)
- scheesman 9y agoRouting is a Layer 3 issue. Loops are a Layer 2 issue. Multiple paths on a Layer 3 network provide redundancy when used with a routing protocol such as BGP. Multiple paths on a Layer 2 network provide redundancy when used with a protocol such as LACP. Without proper design, 2 network segments connected with multiple links will trigger Spanning Tree and shut down one of the links. And depending on the configuration, it may or may not unshut if the active link pair goes down.
- elihu 9y agoSpanning tree just prunes links from the network to eliminate loops. Sometimes the link that got pruned happened to be the fastest path from point A to point B. So, sometimes you can get a more efficient network if you leave those links in, but that requires more sophisticated routing.
- merb 9y agoOT: is this really "simple" to setup or do you mean that once it's running that it's simple? or what do you mean with simplicity. I actually looking forward to use k8s in our network, too. But I think the simplest solution for HA k8s is using keepalived paired with nginx/haproxy on the master level and add a keepalived-vip for every service. (at least for our 3 node master, 3 node worker setup) (I evaluated the calico + bgp option, too but it looked way to complex, as a k8s starter)
- mrmondo 9y agoYes, it was very simple to setup, deploy and understand on an ongoing process, BGP is incredibly simple by design. I’m heading to bed now (AEDST), but check out: - https://docs.projectcalico.org/latest/getting-started/kubernetes/tutorials/simple-policy https://docs.projectcalico.org/latest/getting-started/kubern... - https://kubernetes.io/docs/tasks/administer-cluster/calico-network-policy/ https://kubernetes.io/docs/tasks/administer-cluster/calico-n... And there’s plenty of other resources out there. IMO if you’re designing a highly available platform, BGP is one of the simplest parts of that complex system. In my experience you’re likely to run into much more finicky problems with higher layer systems including failover delays, reliance on casting systems and state drift.
- scurvy 9y agoKubernetes is very complicated, but BGP is not. If the curve for BGP is too steep, you can do OSPF inside your racks with BGP announcements to spine. This is pretty common as OSPF can use area 0 with little to zero configuration. BGP leaf to spine is standard is layer 3 networks.
- murali-reddy 9y agoAs an alternative approach may I suggest to look at the way kube-router does. It basically leverages BGP and ECMP to provide HA. https://cloudnativelabs.github.io/post/2017-11-01-kube-high-available-ingress/ https://cloudnativelabs.github.io/post/2017-11-01-kube-high-...
- SEJeff 9y agoWe did the same thing but opted for the much simpler kube-router, which is more k8s specific and doesn't have nearly as many moving parts (easier to troubleshoot as a result). Bonus point is uses LVS for service proxy load balancing: https://www.kube-router.io/ https://www.kube-router.io/ https://github.com/cloudnativelabs/kube-router https://github.com/cloudnativelabs/kube-router
- deleted 9y ago[deleted]
- lima 9y agoDepends on your use case, the network policy features in Calico are very important for many real-world use cases.
- SEJeff 9y agoWhat does calico do (Ingress and Egress policy) that kube-router does't? https://cloudnativelabs.github.io/post/2017-05-1-kube-network-policies/ https://cloudnativelabs.github.io/post/2017-05-1-kube-networ...
- leecarraher 9y agoi would think BGP is needed now more than ever considering the move to ipv6.
- deleted 9y ago[deleted]
- candiodari 9y agoBGP works fine for IPv6. This is not a good reason.
- tialaramex 9y agoMostly this makes no sense because the things BGP cares about are orthogonal to mere IPv4 or IPv6 addressing. But if there is a difference, IPv6 means less prefixes are needed for any given provider, since the IPv6 prefixes are huge and unfragmented, we can usually give a provider enough address space in their initial allocation to last them forever, and if they exceptionally out-grow it, the larger space is available, whereas with IPv4 there's no chance anybody can give you a /16 even if you warrant one.
- Duckeh 9y agoIt means less prefixes until people start announcing each /48 in their /32 separately...
- zero_intp 9y agoNetwork Operators dislike buying new hardware. As the policies matriculate through tiers of providers, raw numbers of announced prefixes becomes a sticking point.
- jandrese 9y agoOh yeah, nothing makes a mess of your network like someone coming in and saying "we need to remove these IPs from the middle of our advertised blocks for security reasons".
- tialaramex 9y agoTechnology doesn't magically fix human problems, but good uses of technology can play to human strengths so that humans make fewer mistakes, or the mistakes have fewer bad consequences. BGP can and should be implemented so that the human operators are making the fewest possible mistakes and those mistakes have the least bad consequences. I doubt that protocol changes are what is needed, but that doesn't mean sitting on our hands is the right choice.
- csours 9y agoAs I understand BGP, any updates published by any AS may arbitrarily re-route/disrupt traffic, ie it is not sufficient for me to update my systems, everyone must update all systems for the overall routing infrastructure to be protected. That sounds a lot like a protocol change to me.
- lima 9y ago> any updates published by any AS may arbitrarily re-route/disrupt traffic Fortunately, most ISPs are doing strict filtering on announcements on customer ASes. It's when mid-to-large ISPs fuck up (or neglect the filtering part) that it becomes a problem.
- andersonmvd 9y agoI wonder how it relates to SCION (https://www.scion-architecture.net/ https://www.scion-architecture.net/)
- worxli 9y agoIt states: You become immune to current BGP-level attacks, such as prefix hijacking.
- troligare 9y agoWell isn't this just a case of company’s hiring the wrong people... And there should be a better commit and audit system before changing peer routers on a ISP.
- zero_intp 9y agocosts, costs, costs. As your IP service becomes commoditized fewer, not more, engineers make bigger changes. Interestingly, expert systems reduce, not improve, the quality of the operating engineers: http://journals.sagepub.com/doi/full/10.1177/0001839217751692 http://journals.sagepub.com/doi/full/10.1177/000183921775169...
- zero_intp 9y agoOperator and designer mistakes happen, the goals should be to make simple systems easy to build. Complex networks will need thoughtful solutions, and better designs will have fewer operator interventions. I could be petty and pick apart the article for mistakes, but I agree with the gist- If you compose your needs and environment and use the right design you might not hate the protocol(s) you end up with?
- ninegunpi 9y agoYes we do, there are problems which beg to be fixed. Most of current complaints are about security, in one way or another require plenty of CPU horsepower (which most routers don't have - CPU peaks while suddenly switching upling and retriving fullview is still a typical picture in a modern ISP). What it takes to prevent something like NLRI spoofing? Plenty of layers of data security. And it still won't fix the human problem underneath it. Security-aware BGP versions like S-BGP, soBGP and psBGP address most of security concerns, add plenty of router load and still... don't solve some of the human problems. But current publicly made suggestions on replacing it with something completely different even lack proper understanding of design goals of original BGP standard.
- linsomniac 9y agoMany years ago I ran a small hosting ISP, ~10 cabinets. One of the choices I made and still stand by was I used Linux-based routers. We could throw tons of CPU horsepower at it and the fast path and the slow path were the same thing. We had deep Linux expertise, but only passing Cisco expertise, so staying with the Linux networking stack required less specialized knowledge. But, while most people were worrying about the size of the BGP tables, or trimming the announcements they were receiving, we could get full feeds and never had memory problems. We could run strong filtering, to make sure we weren't sending or receiving any junk like bogons or our users weren't sending with spoofed addresses, without worrying about adding one rule running past the ASIC memory and causing everything to go slow path. We had high availability. All for around $2500 in hardware. It always seemed crazy to me that they would put such small CPUs in the high end routers. I know they tried to have that CPU do almost nothing, and the ASICs do all the heavy lifting. But for our needs, having an insanely fast CPU and dumb interfaces was perfectly adequate, and let us do full BGP feeds on a $1200 router. The only real trick we did was we used one network interface and VLANs, and had the multiple network connections terminated at the switch. We were able to get feeds via fiber that we could just terminate in an SPDIF module in the switch. We had 4 core CPUs, so this allowed us plenty of horsepower to manage packet storms or DDoS instead of livelocking the kernel with too many interrupts (it took a while for the kernel to switch from interrupts to polling, and if packets ramped up too quickly the kernel couldn't switch if you didn't have more CPUs than interfaces).
- jgowdy 9y agoI have to disagree with the primary point being offered up in this post. In my career I’ve met quite a few network engineers who don’t think sufficiently outside the box. They typically say “this is how it’s done, this is how it works.” The idea that we can’t come up with a better protocol than BGP to manage the interconnections of the Internet is absurd. We don’t say “well security isn’t a programming problem, it’s a human problem, so there’s no point in making safer programming languages.” BGP is a ridiculous system of complete trust just like many older protocols. If we sat down and started designing a routing protocol to handle the advertisement of routing tables on the internet, would we come up with BGP again? Would the new protocol potentially offer up fixes for some of the most common and obvious problems with BGP? Then there’s potential for improvement. The network industry resists and slows the horribly needed disruption in several key areas through the dogmatic defense of the status quo that we typically hear from its constituents. This is why so many of us pray for SDN. It’s the only way to wrest control of networking away from those who typically have dogmatic vendor based training rather than a “network science and theory” education that leads to innovation.
- d33 9y ago> BGP is a ridiculous system of complete trust just like many older protocols. This sounds related: https://security.stackexchange.com/q/56069/15648 https://security.stackexchange.com/q/56069/15648
- jtiri1883 9y agoI look forward to reading the first draft of your groups proposal
- ra1n85 9y ago>This is why so many of us pray for SDN. It’s the only way to wrest control of networking Why not take the standards approach and draft a protocol spec? At some point, you'll need to interoperate with those horribly dogmatic vendor hugging networks that serve the majority of traffic on the internet. >who typically have dogmatic vendor based training rather than a “network science and theory” education that leads to innovation. I've seen this as well, and it can be frustrating. Part of me believes this is a defense mechanism used to prevent developers from pushing problems into the network. The other part believes that network engineering has drawn individuals to the field based on earnings potential rather than interest in the work, and the talent just isn't there. Both are anecdotal, so grain of salt and all.
- bogomipz 9y agoThe author states: >"From time to time, I run across (yet another) article about why Border Gateway Protocol (BGP) is so bad, and how it needs to be replaced. This one, for instance, is a recent example." Yet if you read that linked article this author references in the last sentence, that article does not suggest replacing BGP at all. That referenced post even states: >"How do we fix this? Well, aside from making sure that anyone touching BGP knows exactly what they’re doing? Not much." And then the referenced article goes on to mention RPKI and the MANRS frameworks which are not routing protocols nor are they meant to replace BGP.
- aplorbust 9y agoReminded me of this: https://media.ccc.de/v/34c3-9072-bgp_and_the_rule_of_custom https://media.ccc.de/v/34c3-9072-bgp_and_the_rule_of_custom One of the better talks I watched from this years conference.
- aplorbust 9y ago"We've somehow forgotten the old maxim that a protocol is not done until we have removed everything that is not needed." Would software and web developers ever consider adopting this maxim? As a user, I would enthusiastically support the experiment. (For example, results might be smaller program size, and less code for interested users and others to audit.)
- rdtsc 9y agoA bit of topic obviously here. But that's how overall I feel about Apple's UI experience or product design in general. The UI interaction if you think about is also a protocol of sorts - between user and the device. Say user scans the the preferences window, finds the checkboxes they were looking for , picks one, clicks on it, it reacts back and is marked as such and so on. Now try to reduce the number of needed actions and decisions needed by the user there. Maybe they don't have to click "Apply" button when done. Or maybe there are only 2 checkboxes. Or they are not even in the preference menus at all.
- maltalex 9y agoThe relevance of "you cannot solve people problems with technology" point is debatable. Sure, some problems you can't solve, but a properly designed system can definitely reduce the risk of human error - we've seen this in fields such as aviation and medicine (to some extent). But what isn't debatable is the fact that BGP is a security nightmare, and there's really no reason for internet routing to be so vulnerable. It's amazingly simple to divert traffic using BGP, and it has been so for years - https://www.youtube.com/watch?v=S0BM6aB90n8 https://www.youtube.com/watch?v=S0BM6aB90n8.
- exabrial 9y agoDoes BGP have the equivalent of DNSSEC?
- madsushi 9y agoYes, RPKI. Not that widely deployed, <20% of ASs, last time I looked.
- deleted 9y ago[deleted]
- chisleu 9y agoIf we don't replace BGP, people with BGP access are going to keep "accidentally" redirecting compute resources from AWS and other data centers to themselves so they can man in the middle crypto mining, and steal millions.
- arca_vorago 9y agoI don't think we need a new bgp, we just need to fix the old bgp. I have yet to see a practical and realistic alternative. Also, don't forget MPLS is very interdependent in it's current state on BGP, and MPLS is really awesome. Yes, BGP has some weaknesses, but perhaps calling for a new bgp is throwing the baby out with the bathwater. You know what I think is more of a problem is vendor market dominance that uses closed source proprietary boxen on proprietary platforms designed for vendor lock in. Also, some have referenced the power of asics in these black boxen over cots hardware, but these days thats not true because there are open source systems with asics in them. Not exactly related to BGP, but I did an entire infrastructure upgrade for a company. I talked to every vendor in existence, extreme, juniper, cisco, brocade, etc etc. I ended up going with Ubiquiti and saving over $50k... These vendors are gouging customers who already have problems with IT infrastructure budgets, and I'm tired of dealing with them. (and their often non-spec implimentations of protocls) On top of that, you would be amazed at how many "network engineers" don't know anything but cisco, and couldn't route their way out of a paper bag in anything else, this result of the vendor lock-in is much more of a problem. Lets also not forget to mention the NSA-cisco, etc backdooring systems.
- no_identd 9y agoWe wouldn't need BGP at all if they hadn't ditched flow routing from IPv6 before it even became a proper draft, and if IP in general wouldn't assume we still use thicknet everywhere.