4 ms·

I'm also really confused as to why they insisted on taking full tables at their edge if they also built their own route optimization technology. It seems like i

by scurvy 10y ago

I'm also really confused as to why they insisted on taking full tables at their edge if they also built their own route optimization technology. It seems like it would be much simpler and faster if they just took a default route from each provider and then did route probing on their top 30% prefixes in each PoP. You don't need full edge routes for that. You need a single route reflector (or two for redundancy) to tell you what routes are in the table. Then probe the busiest prefixes (as determined by flow data) via each provider and take the best route. Localpref the best route higher in the route reflector and send it to the Linux hosts. No need for a huge FIB on the edge. This is basically what Noction IRP/Internap FCP platforms already do. This reminds me of the scene in Primer where they ask if they're doing it for a reason or just showing off. FWIW, the Netflix networks just take a default route from a pair of transit networks in each PoP. At their scale, they found it better to negotiate better worldwide transit deals than optimize networking at the PoP level. Different game than Fastly, but another data point.

bogomipz 10y ago
I remember those FCP boxes, its been a while. I think its maybe what they call Miro now, do you know? The Avaya route science boxes did the same thing. It seems both of these have fallen out of usage. I always thought it was odd that there wasn't an opensource project for this. I know Internap originally did it in PERL.
- scurvy 10y ago
  Yep, FCP's successor is Miro. I haven't had a chance to look into it, but their sales people claim it's all new. Noction has pretty much taken over the entire market. Overall their product is good, but the Internet game has changed a lot since the original FCP days (lots more direct peers, lots more exchanges, lots more route servers, lots more paid peering), and I don't think that things like FCP/Miro/IRP do well once you move past paid peering. Things just become too complex. IRP exposes some of this to you, but the implementation is kludgey (routing instance per peer, DSCP bits to identify peers on exchanges, and more).
  - bogomipz 10y ago
    Agreed about how the Internet game has changed. If I remember correctly Internap didn't peer with anybody they simply bought tons of transit and connected the PNAPs with local fiber. I didn't understand your comment about "... once you move past paid peering." Did you mean "once you move to paid peering"? I would think that moving past it would be actual settlement-free peering. But I'm curious what you meant. Thanks.
amazon_not 10y ago
> It seems like it would be much simpler and faster if they just took a default route from each provider and then did route probing on their top 30% prefixes in each PoP. Probing just 30% of the top prefixes is likely not enough for their use case. Probing is also an after the fact approach. Fastly needs a solution which will give best possible path from first packet. > FWIW, the Netflix networks just take a default route from a pair of transit networks in each PoP. At their scale, they found it better to negotiate better worldwide transit deals than optimize networking at the PoP level. Different game than Fastly, but another data point. Different game indeed. Netflix just cares about throughput, Fastly also needs low latency, hence the need for more optimizing.
- bogomipz 10y ago
  "Probing just 30% of the top prefixes is likely not enough for their use case. Probing is also an after the fact approach. Fastly needs a solution which will give best possible path from first packet." That's not how BGP works. There's no such thing as getting the best path from the first packet in terms of latency. There's nothing in BGP that tell you about the path in terms of latency or performance. The only "best" that BGP gives you is AS path length between you and the destination. It tells you nothing about congestion/packet loss upstream. The only way to get that is out of band probing, like taking latency measurements and pref'ing routes. There is no approach beyond "after the fact" There is nothing in this article that talks about optimization of routes.
  - amazon_not 10y ago
    > That's not how BGP works. Of course not. I only stated what Fastly needs, I did not make a statement about how BGP works. By having full BGP feeds from all transit providers the nodes can use the AS paths as part of the heuristics to determine the best possible path (with the available information) from first packet. Obviously the heuristics will also use after the fact probe information to tune the model. The point is that the solution proposed by the grandparent is not better than the one described in the article. > There is nothing in this article that talks about optimization of routes. No, they talk about route selection.

by scurvy 10y ago

bogomipz 10y ago

I remember those FCP boxes, its been a while. I think its maybe what they call Miro now, do you know? The Avaya route science boxes did the same thing. It seems both of these have fallen out of usage. I always thought it was odd that there wasn't an opensource project for this. I know Internap originally did it in PERL.

scurvy 10y ago
Yep, FCP's successor is Miro. I haven't had a chance to look into it, but their sales people claim it's all new. Noction has pretty much taken over the entire market. Overall their product is good, but the Internet game has changed a lot since the original FCP days (lots more direct peers, lots more exchanges, lots more route servers, lots more paid peering), and I don't think that things like FCP/Miro/IRP do well once you move past paid peering. Things just become too complex. IRP exposes some of this to you, but the implementation is kludgey (routing instance per peer, DSCP bits to identify peers on exchanges, and more).
- bogomipz 10y ago
  Agreed about how the Internet game has changed. If I remember correctly Internap didn't peer with anybody they simply bought tons of transit and connected the PNAPs with local fiber. I didn't understand your comment about "... once you move past paid peering." Did you mean "once you move to paid peering"? I would think that moving past it would be actual settlement-free peering. But I'm curious what you meant. Thanks.

amazon_not 10y ago

> It seems like it would be much simpler and faster if they just took a default route from each provider and then did route probing on their top 30% prefixes in each PoP. Probing just 30% of the top prefixes is likely not enough for their use case. Probing is also an after the fact approach. Fastly needs a solution which will give best possible path from first packet. > FWIW, the Netflix networks just take a default route from a pair of transit networks in each PoP. At their scale, they found it better to negotiate better worldwide transit deals than optimize networking at the PoP level. Different game than Fastly, but another data point. Different game indeed. Netflix just cares about throughput, Fastly also needs low latency, hence the need for more optimizing.

bogomipz 10y ago
"Probing just 30% of the top prefixes is likely not enough for their use case. Probing is also an after the fact approach. Fastly needs a solution which will give best possible path from first packet." That's not how BGP works. There's no such thing as getting the best path from the first packet in terms of latency. There's nothing in BGP that tell you about the path in terms of latency or performance. The only "best" that BGP gives you is AS path length between you and the destination. It tells you nothing about congestion/packet loss upstream. The only way to get that is out of band probing, like taking latency measurements and pref'ing routes. There is no approach beyond "after the fact" There is nothing in this article that talks about optimization of routes.
- amazon_not 10y ago
  > That's not how BGP works. Of course not. I only stated what Fastly needs, I did not make a statement about how BGP works. By having full BGP feeds from all transit providers the nodes can use the AS paths as part of the heuristics to determine the best possible path (with the available information) from first packet. Obviously the heuristics will also use after the fact probe information to tune the model. The point is that the solution proposed by the grandparent is not better than the one described in the article. > There is nothing in this article that talks about optimization of routes. No, they talk about route selection.