6 ms·
Running BGP in large-scale data centers
- isaack 5y agoAdd (2021) to the title?
- longnguyen 5y agoNot following. Is it 2022 already?
- deadmutex 5y agoNo. It's 2038
- Scoundreller 5y agoIsn’t that 1901 or 1902?
- darkr 5y agoNo, it's 1st Jan 1970
- Scoundreller 5y agoIt’s a signed integer, so when it rolls, you roll ~68 years before 0.
- deleted 5y ago[deleted]
- reilly3000 5y ago(Before the fall)
- wmf 5y agoI can guess why this was posted but it probably has nothing to do with today's outage. Routing inside a datacenter is different from routing to the Internet. This post is probably more relevant: https://engineering.fb.com/2017/08/21/networking-traffic/steering-oceans-of-content-to-the-world/ https://engineering.fb.com/2017/08/21/networking-traffic/ste...
- jcora 5y agoIt seems like it is coupled in Facebook:)
- bennyp101 5y agoI don’t know, the paper[1] describes how they setup their systems so I think it is relevant to what happened [1] https://research.fb.com/publications/running-bgp-in-data-centers-at-scale/ https://research.fb.com/publications/running-bgp-in-data-cen...
- Hikikomori 5y agoBGP on the internet and inside your datacenter isn't any different though. It has different rules for iBGP and eBGP neighbors, but it's common to have multiple AS in your own datacenter, especially at Facebooks scale.
- m0zg 5y ago> ERR_CONNECTION_CLOSED Yeah. Not sure I want to read that "advice" today.
- schleck8 5y agoAwesome info, i will use this config for all servers asap
- yholio 5y agoTo achieve the goals we’d set, we had to go beyond using BGP as a mere routing protocol. Uh-oh.
- sargun 5y agoPetr wrote up this RFC. I believe he started it when he was at Microsoft, but he finished it at Facebook: https://datatracker.ietf.org/doc/html/rfc7938 https://datatracker.ietf.org/doc/html/rfc7938
- dustintrex 5y agoWe know failures happen in any large-scale system — hence, our routing design aims to minimize the impact of any potential failures. Oops!
- martin1975 5y agoSo, who pushed out the configuration changes and is this person still employed at FB?
- high_5 5y agoAlso, don't forget the reviewers who approved the change. And the designers of the whole system/process that enabled this thing to fall through this process.
- can16358p 5y agoI think it's the design. It should have been fault-tolerant to possible issues exactly like this, and should not allow the configuration to go live in the first place, in case the person or team pushing the configuration makes a mistake (or a sabotage attempt).
- sdevonoes 5y agoOf course! That person learnt a valuable lesson yesterday, so they cannot be fired.
- coretx 5y agoNot per se. Depending on the type of error made, that person might have become a pariah no one wants to peer with.
- rocgf 5y agoIf it was something truly idiotic or malevolent, sure. Otherwise, that would be a really crap culture to work in.
- coretx 5y agoMany people love to work at let's say, a nucleair powerplant or certain aspects of the military and experience similair pressure to not fail.One could say such work is not for everyone but to state it's "crap culture" is somewhat disrespectful towards the people who do it and the masses who suffer when things go wrong.