12 ms·
How Fastly coded their own routing layer for scaling CDN
- d33 10y agoThat's pretty impressive! As a side note, a writeup on BGP security: https://security.stackexchange.com/questions/56069/what-security-mechanisms-are-used-in-bgp-and-why-do-they-fail https://security.stackexchange.com/questions/56069/what-secu...
- jssjr 10y agoThis is really great work. Do you have any plans to open source some (or all) of the code behind Silverton?
- al_fountain24 10y agoextremely cool stuff
- davidu 10y agoThis is awesomely clever. Not surprised to see from the Fastly team.
- francoisLabonte 10y agoAlso note Spotify published their own stuff also on Arista hardware https://labs.spotify.com/2016/01/26/sdn-internet-router-part-1/ https://labs.spotify.com/2016/01/26/sdn-internet-router-part... Podcast where David Barroso talks about it: http://blog.ipspace.net/2015/01/sdn-router-spotify-on-software-gone-wild.html http://blog.ipspace.net/2015/01/sdn-router-spotify-on-softwa... Arista blog post: https://eos.arista.com/spotifys-sdn-internet-router/ https://eos.arista.com/spotifys-sdn-internet-router/ Disclaimer I work at Arista
- NetStrikeForce 10y agoDidn't they move to Google Cloud? I was very surprised at the time to see their move shortly after reading David's articles.
- brazzledazzle 10y agoI think they just moved event data processing.
- deleted 10y ago[deleted]
- fairramone 10y agoAlso note that David Barroso works at Fastly now. (Source: LinkedIn)
- ChuckMcM 10y agoGreat read, love the "hey what does it need?" approach rather than the "how is this done?" approach. Tut Systems had bought one of the first "hotel internet" companies back in the 90's which used a similar approach by subverting the ARP protocol, when you connected any thing you tried to ARP for it would respond "Yup, that's me! Send me your packets" and you would end up at the "Give us your credit card" signup. The nice thing is that at this level networking is really simple. And if you can get access to the internals of switches to craft behaviors at that level, it is a pretty good way to go.
- snowy 10y agoYou mean proxy ARP?
- ChuckMcM 10y agoPretty much, but more like proxy ARP on steroids, sort of proxy DNS, proxy ARP, proxy everything.
- NetStrikeForce 10y agoProxy ARP it's a thing and it is exactly what you've described :) And if by proxy DNS you mean you'll subvert ARP to reach a DNS server... that's proxy ARP. DNS is a few layers above :) Definition: Proxy ARP is the technique in which one host, usually a router, answers ARP requests intended for another machine. By "faking" its identity, the router accepts responsibility for routing packets to the "real" destination.
- deleted 10y ago[deleted]
- dmourati 10y agoAnd it is a super dirty network hack. Unfortunately can't put that genie back in the bottle.
- mikecb 10y agoIf you like this, you'll like the ONS youtube channel.[1] In particular, the keynotes by Vahdat are pretty amazing. [1] https://www.youtube.com/channel/UCHo2uqQqpmE_Cg5b4qiUpUg https://www.youtube.com/channel/UCHo2uqQqpmE_Cg5b4qiUpUg
- pyvpx 10y agoI hope Arista pushes other vendors to open up their hardware and provide more APIs
- pbarry25 10y agoThat'd be great if they could. We started using Arista 5 or 6 years ago at a small startup I was working for because Arista was the only vendor we could find who was comfortable giving us such an amazing amount of access to the inner workings of their high speed switches. Really enjoyed working with their gear and, when the occasional question came up, their engineers (top notch folks).
- scurvy 10y agoI've run Arista for 10 years now (since Arastra days), but I'm a lot more interested in whitebox switches running some sort of open networking software. The silicon designs are all coming from Broadcom/Intel/etc. This is where stuff like Cumulus comes into play, and is a lot more excited than vendor lock-in on "open" technologies provided by Arista.
- amazon_not 10y agoTL;DR: Fastly needs full routing tables on all CDN nodes in order to determine which is the best transit path to push out content through. In order to save money, they used a programmable Arista switch instead of a traditional router. Their solution is to reflect BGP routes via the switch to the nodes and and fake direct connectivity between the nodes and the transit provides, so that nodes can directly push out content to whichever transit provider they determine is best on a per packet basis. Please correct me if I'm wrong. Maybe I'm just obtuse, but I found the blog post confusing about what it really is about and long winded, taking a very long time to come to the point. There is a severe lack of context in the beginning, it would have tremendously benefited from the what and the why of what they are trying to do.
- edwhitesell 10y agoI'd bet the best transit path isn't the primary reason. Having the routes in the application means it choose a path based on cost, customer preference or any number of other business rules.
- scurvy 10y agoRoutes aren't exposed to customers as a configurable option. At least not at my pay grade with them.
- bogomipz 10y agoBut it is, you can probe and route around brownouts in certain AS's in the path. I would say that a CDNs ability to get to the content to the eyeballs is the most important and trumps all else.
- erentz 10y agoThey'll want to be looking at using MPLS and EPE techniques now that there's support for it on their Arista platforms. This L2 technique is arcane, going to be painful to scale and reapply generally to other areas.
- Twirrim 10y agoAmazon, Google, etc. etc. all these companies building their own custom network devices, and so much of it coming back to both "It does too much, most of which we don't need" and "We want to do our own thing at that layer". As James Hamilton noted at the AWS Re:Invent convention, not only is there they overhead and development expense of these unneeded components, just the sheer complexity of the application running is inevitably leading to bugs and unexpected behaviour. By simplifying the device to just do the few things you actually need it to do, you end up more performant, and more reliable. I wonder if the entrenched network appliance providers will wake up?
- jjoe 10y agoInteresting approach. So Fastly is offloading the full routing table to their carriers' router(s). That's because routers that can hold full BGP tables are expensive to purchase and maintain. But to retain some form of control, they're terminating eBGP at the switch and using iBGP to disseminate (inject) the providers' route (next hop). I feel (I don't have direct experience with this setup) like they're just offloading some compute power (therefore cost) to the hosts. So the cost is automatically spread out across their relatively massive edge nodes. A line showing a router plus support costing $100,000 looks bad in expenditures vs showing a server plus integrated routing showing $2500. I'm curious about how this impacts Varnish considering how table look ups can be bus-expensive during odd route changes/flaps (storms). %sys must go through the roof as a result.
- amazon_not 10y ago> So Fastly is offloading the full routing table to their carriers' router(s). The carriers' routers will have full routing tables in any case, so Fastly is not really offloading anything. They just aren't downloading full routes to a big central router and doing routing decisions there, but rather at a host level. > I feel (I don't have direct experience with this setup) like they're just offloading some compute power (therefore cost) to the hosts. This appears to be the case. The routing part isn't very computationally expensive, the biggest problem at larger bitrates is moving the packets with software. But then again they are already limited by what their CDN nodes can push out, so the routing isn't really much more of a burden.
- jrochkind1 10y ago> they're just offloading some compute power (therefore cost) to the hosts I don't know enough about networking to follow the tech, but they seem to say in the post that's exactly what they intended to do: > The idea of dropping several millions of dollars on overly expensive networking hardware wasn’t particularly appealing to us. As systems engineers we’d much rather invest the money in commodity server hardware, which directly impacts how efficiently we can deliver content.
- scurvy 10y agoI didn't see him in the room, so that doesn't mean he wasn't there, but it sounds like Artur was paying attention to Dave Temkin's talk at NANOG (from Netlix). Slides: https://www.nanog.org/sites/default/files/wednesday.general.temkin.panel.pdf https://www.nanog.org/sites/default/files/wednesday.general.... Video: https://www.youtube.com/watch?v=-05xWeYGn4A https://www.youtube.com/watch?v=-05xWeYGn4A Titled: "Help! My Big Expensive Router Is Really Expensive!" Netflix goes even cheaper/simpler and just uses default routes to a pair of transit providers. It may come as a shock to most of you, but no, Netflix is not 100% in AWS (compute, yes; network, oh hell no).
- aram26 10y agohttps://www.youtube.com/watch?v=TLbzvbfWmfY https://www.youtube.com/watch?v=TLbzvbfWmfY