8 ms·

Argo and the Cloudflare Global Private Backbone

93 points by jgrahamc 7y ago

johnklos 7y ago
Seems more like a sales pitch than anything which conveys real information. And, as they imply (but others posting here dispute), if it's better in all cases, why wouldn't it be on in all cases? Why didn't they mention the downsides to enabling it?
- jgrahamc 7y ago
  There are two parts here: 1. We've started using our own backbone of fibre links to carry traffic. 2. The Argo product for route optimization and working around Internet problems. #2 can take advantage of #1 now that it exists. #2 is a paid product.
danpalmer 7y ago
> As an example, here’s a measurement of jitter (variance in round trip time) between two of our data centers, Chicago and Newark, over a transit provider’s network... Average jitter over the pictured 6 hours is 4ms, with average round trip latency of 27ms. > Here’s the same jitter chart between Chicago and Newark, except this time, transiting the Cloudflare Global Private Backbone... Here we see a jitter measurement of 536μs (microseconds), almost eight times better than the measurement over a transit provider between the same two sites. This is really impressive. Will be interesting to see what's possible when network roundtrips are 2 orders of magnitude more stable.
deleted 7y ago
[deleted]
andersonrkton 7y ago
Stopped using Argo Tunnel because it didn`t work well inside Brazil... we had a huge % of requests to our API behind the tunnel not being completed.
- tmikaeld 7y ago
  I think this is an ongoing problem for any provider, can be bad local peering or even ISP ddos protections kicking in.
- jgrahamc 7y ago
  I'd love to know more. If you're willing email jgc @ cloudflare with details.
napsterbr 7y ago
Argo is great. I temporarily switched on on a customer's domain and we saw 50+% improvements on latency! Question to CF employees, since they usually participate here on HN: Any chance we are ever going to be able to enable Argo on specific domains / paths, similar to Workers?
matsur 7y ago
I’m the product manager for Argo and author of the blog. Happy to answer any questions!
- devy 7y ago
  Is Argo open sourced? If not, will it be open sourced in the future?
  - eastdakota 7y ago
    Rustam can weigh in here, but I'm not exactly sure what with Argo we'd open source. Argo is a collection of a bunch of different technologies that we use to measure the performance across our transit, peering, and backbone connections. We then send different requests down different paths depending on each customer's particular needs and settings. While I'd imagine we will open source parts of the overall system, there's not really a stand-alone package that we could make available for anyone else to use. If there's a particular part of the overall Argo that would be helpful for you or others, let us know and we'll certainly consider it. Here's a list of things we've built at Cloudflare and open sourced already: https://cloudflare.github.io/ https://cloudflare.github.io/
  - jbergstroem 7y ago
    Frequent user of argo/argo tunnel checking in. As far as I know, the network/lower level parts are not yet in the open, but the tunnel bits are: - Tunnel (cloudflared): https://github.com/cloudflare/cloudflared https://github.com/cloudflare/cloudflared - Argo as ingress (k8s): https://github.com/cloudflare/cloudflare-ingress-controller/ https://github.com/cloudflare/cloudflare-ingress-controller/
- lziest 7y ago
  You should consider to remake those graphs comparing the jitters. The non-backbone graph shows a majority of data points are below 0, which is impossible, so that means there are a small long tail at the high >20ms band, and you intentionally cropped them out. So when you said "oh the backbone graph is much better!". No it is not, it is very confusing actually.
lgats 7y ago
Too bad it costs $5 + $0.10 per GB.
- toomuchtodo 7y ago
  What would you consider a fair per-GB price for Cloudflare to route data over their own private backbone (which I hastily assume includes their own leased fiber, routers/switch fabric, and other associated fixed and variable costs)?
  - lgats 7y ago
    I gladly pay $0.10/GB for some of my smaller, lower-bandwidth sites. For these sites, I see a latency improvement of around 25% for around 50% of (smart-routed) traffic. For a couple of my larger sites (PDF and Image heavy), I am unable to justify the Argo cost. On these sites, over half of my monthly traffic is to bots (Googlebot, Bingbot, Yandex, etc). Traffic costs make up of around half of my hosting bill. A couple of things are keeping me from transitioning these sites to Argo: - I'd rather pay to speed up only real-users (not bots) - I'd rather not pay for traffic not routed over Argo (currently all traffic gets billed under Argo for enabled domains, not just the smart-routed traffic) I currently pay the following rates from Google Cloud->CloudFlare (depending on the CF datacenter the data gets routed to). NA: $0.04/GB EU: $0.05/GB APAC: $0.06/GB If I could pay a similar rate for a speedup (for only the Argo-routed traffic), I'd be more likely to sign up for Argo on all of my domains.
- jgrahamc 7y ago
  We provide a lot of value for free through our free service. Argo is an additional paid service that improves performance for customers that want to pay for that.
  - Avamander 7y ago
    Your free service is nice - except that you can't use CF as a registrar for more than one domain without also having to pay for the proxying (even if there's no need). I'd gladly put all my eggs in one basket instead of having them spread out across five different registrars if that were fixed. Though maybe it's by-design?
wcunning 7y ago
I'm curious if this conflicts with the Argo AI trademark. Does anyone have any insight into how the similarity would be determined? Specifically, I'm thinking that routing and self driving cars are near enough to cause some confusion.
bogomipz 7y ago
This looks like little more than marketing fluff and with rather confusing messaging at that. For example the post states: >"One specific example of Argo’s smarts is its ability to distinguish between multiple potential connectivity options as it leaves a given data center. We call this “transit selection” A "global private backbone" and edge routing have zero to do with each other. They are orthogonal. You can "pref" your transit provider's customer routes from just within POP itself. This has zero to do with whether or not you have a "backbone." The way this works is that you pick your top N destination ASNs and continuously monitor ping time, jitter packet loss etc. Then you prefer the more desirable routes via iBGP. This is not new and not innovative either. Internap has been this 20 for years now. This was the whole "secret sauce" of their PNAP architecture back in the day. In fact they even sold a hardware device to do it called the Internap FCP(flow control platform)it has since been named MIRO [1][2]. Of course many people do exactly this themselves with BIRD/Quagga/Exabgp and some Python. And you certainly don't need "machine learning" to do this. As a CDN customer I only care about getting the best path from the POP back to the eyeball networks. Why would it matter whether the CDN provider has fiber between two of their POPs? There should be enough transit diversity at each edge location that fiber between POPs is irrelevant. I should spend as little time on the CDN provider's network as possible i.e "hot potato" routing. This is the whole value proposition of a CDN. Honestly this whole sounds like a cost savings mechanism that they've trying to spin and sell as a product innovation. Also where is the Cloudflare backbone map? It seems to be conspicuously absent for announcement about a "global private backbone." How many route miles are there? Is it an actual backbone or this just some CWDM gear that only connects POPs in the same city over municipal fiber? [1] https://www.inap.com/press-release/internap-rebuilds-patented-routing-software-content-delivery-network-optimize-traffic-delivery-todays-internet-applications/ https://www.inap.com/press-release/internap-rebuilds-patente... [2] https://www.businesswire.com/news/home/20080206005468/en/Teliax-Deploys-Internaps-Flow-Control-Platform-FCP https://www.businesswire.com/news/home/20080206005468/en/Tel...

93 points by jgrahamc 7y ago

Seems more like a sales pitch than anything which conveys real information. And, as they imply (but others posting here dispute), if it's better in all cases, why wouldn't it be on in all cases? Why didn't they mention the downsides to enabling it?

jgrahamc 7y ago
There are two parts here: 1. We've started using our own backbone of fibre links to carry traffic. 2. The Argo product for route optimization and working around Internet problems. #2 can take advantage of #1 now that it exists. #2 is a paid product.

danpalmer 7y ago

> As an example, here’s a measurement of jitter (variance in round trip time) between two of our data centers, Chicago and Newark, over a transit provider’s network... Average jitter over the pictured 6 hours is 4ms, with average round trip latency of 27ms. > Here’s the same jitter chart between Chicago and Newark, except this time, transiting the Cloudflare Global Private Backbone... Here we see a jitter measurement of 536μs (microseconds), almost eight times better than the measurement over a transit provider between the same two sites. This is really impressive. Will be interesting to see what's possible when network roundtrips are 2 orders of magnitude more stable.

deleted 7y ago

[deleted]

andersonrkton 7y ago

Stopped using Argo Tunnel because it didn`t work well inside Brazil... we had a huge % of requests to our API behind the tunnel not being completed.

tmikaeld 7y ago
I think this is an ongoing problem for any provider, can be bad local peering or even ISP ddos protections kicking in.
jgrahamc 7y ago
I'd love to know more. If you're willing email jgc @ cloudflare with details.

napsterbr 7y ago

Argo is great. I temporarily switched on on a customer's domain and we saw 50+% improvements on latency! Question to CF employees, since they usually participate here on HN: Any chance we are ever going to be able to enable Argo on specific domains / paths, similar to Workers?

matsur 7y ago

I’m the product manager for Argo and author of the blog. Happy to answer any questions!

devy 7y ago
Is Argo open sourced? If not, will it be open sourced in the future?
- eastdakota 7y ago
  Rustam can weigh in here, but I'm not exactly sure what with Argo we'd open source. Argo is a collection of a bunch of different technologies that we use to measure the performance across our transit, peering, and backbone connections. We then send different requests down different paths depending on each customer's particular needs and settings. While I'd imagine we will open source parts of the overall system, there's not really a stand-alone package that we could make available for anyone else to use. If there's a particular part of the overall Argo that would be helpful for you or others, let us know and we'll certainly consider it. Here's a list of things we've built at Cloudflare and open sourced already: https://cloudflare.github.io/ https://cloudflare.github.io/
- jbergstroem 7y ago
  Frequent user of argo/argo tunnel checking in. As far as I know, the network/lower level parts are not yet in the open, but the tunnel bits are: - Tunnel (cloudflared): https://github.com/cloudflare/cloudflared https://github.com/cloudflare/cloudflared - Argo as ingress (k8s): https://github.com/cloudflare/cloudflare-ingress-controller/ https://github.com/cloudflare/cloudflare-ingress-controller/
lziest 7y ago
You should consider to remake those graphs comparing the jitters. The non-backbone graph shows a majority of data points are below 0, which is impossible, so that means there are a small long tail at the high >20ms band, and you intentionally cropped them out. So when you said "oh the backbone graph is much better!". No it is not, it is very confusing actually.

lgats 7y ago

Too bad it costs $5 + $0.10 per GB.

toomuchtodo 7y ago
What would you consider a fair per-GB price for Cloudflare to route data over their own private backbone (which I hastily assume includes their own leased fiber, routers/switch fabric, and other associated fixed and variable costs)?
- lgats 7y ago
  I gladly pay $0.10/GB for some of my smaller, lower-bandwidth sites. For these sites, I see a latency improvement of around 25% for around 50% of (smart-routed) traffic. For a couple of my larger sites (PDF and Image heavy), I am unable to justify the Argo cost. On these sites, over half of my monthly traffic is to bots (Googlebot, Bingbot, Yandex, etc). Traffic costs make up of around half of my hosting bill. A couple of things are keeping me from transitioning these sites to Argo: - I'd rather pay to speed up only real-users (not bots) - I'd rather not pay for traffic not routed over Argo (currently all traffic gets billed under Argo for enabled domains, not just the smart-routed traffic) I currently pay the following rates from Google Cloud->CloudFlare (depending on the CF datacenter the data gets routed to). NA: $0.04/GB EU: $0.05/GB APAC: $0.06/GB If I could pay a similar rate for a speedup (for only the Argo-routed traffic), I'd be more likely to sign up for Argo on all of my domains.
jgrahamc 7y ago
We provide a lot of value for free through our free service. Argo is an additional paid service that improves performance for customers that want to pay for that.
- Avamander 7y ago
  Your free service is nice - except that you can't use CF as a registrar for more than one domain without also having to pay for the proxying (even if there's no need). I'd gladly put all my eggs in one basket instead of having them spread out across five different registrars if that were fixed. Though maybe it's by-design?

wcunning 7y ago

I'm curious if this conflicts with the Argo AI trademark. Does anyone have any insight into how the similarity would be determined? Specifically, I'm thinking that routing and self driving cars are near enough to cause some confusion.

bogomipz 7y ago

This looks like little more than marketing fluff and with rather confusing messaging at that. For example the post states: >"One specific example of Argo’s smarts is its ability to distinguish between multiple potential connectivity options as it leaves a given data center. We call this “transit selection” A "global private backbone" and edge routing have zero to do with each other. They are orthogonal. You can "pref" your transit provider's customer routes from just within POP itself. This has zero to do with whether or not you have a "backbone." The way this works is that you pick your top N destination ASNs and continuously monitor ping time, jitter packet loss etc. Then you prefer the more desirable routes via iBGP. This is not new and not innovative either. Internap has been this 20 for years now. This was the whole "secret sauce" of their PNAP architecture back in the day. In fact they even sold a hardware device to do it called the Internap FCP(flow control platform)it has since been named MIRO [1][2]. Of course many people do exactly this themselves with BIRD/Quagga/Exabgp and some Python. And you certainly don't need "machine learning" to do this. As a CDN customer I only care about getting the best path from the POP back to the eyeball networks. Why would it matter whether the CDN provider has fiber between two of their POPs? There should be enough transit diversity at each edge location that fiber between POPs is irrelevant. I should spend as little time on the CDN provider's network as possible i.e "hot potato" routing. This is the whole value proposition of a CDN. Honestly this whole sounds like a cost savings mechanism that they've trying to spin and sell as a product innovation. Also where is the Cloudflare backbone map? It seems to be conspicuously absent for announcement about a "global private backbone." How many route miles are there? Is it an actual backbone or this just some CWDM gear that only connects POPs in the same city over municipal fiber? [1] https://www.inap.com/press-release/internap-rebuilds-patented-routing-software-content-delivery-network-optimize-traffic-delivery-todays-internet-applications/ https://www.inap.com/press-release/internap-rebuilds-patente... [2] https://www.businesswire.com/news/home/20080206005468/en/Teliax-Deploys-Internaps-Flow-Control-Platform-FCP https://www.businesswire.com/news/home/20080206005468/en/Tel...