12 ms·

Pingora, the proxy that connects Cloudflare to the Internet

arberx 4y ago
Is it open source?
- jgrahamc 4y ago
  It will be. There will be a follow up blog post about the open sourcing with all the gory details of how it was built and how it works.
  - latchkey 4y ago
    I think you should remove the part about closed door development as a negative for nginx given the way that this has been developed.
    - 0x457 4y ago
      Why? It's closed door development in nginx from their perspective. This doesn't apply for their in-house project. They not telling you why you would want to switch off Nginx, they are telling you why they've switched.
  - network2592 4y ago
    It might be a couple of months. The open sourcing of Cloudflare workers was announced in May and still has not been released. [1] [1] https://blog.cloudflare.com/workers-open-source-announcement/ https://blog.cloudflare.com/workers-open-source-announcement...
  - Dowwie 4y ago
    You may be interested in knowing that about two years ago, a team of engineers at Dropbox wrote gory details about their use of Rust and it was inspiring. The passion about their work really came through. The team also held an AMA on /r/rust that went well. See here: https://www.reddit.com/r/rust/comments/fjt4q3/rewriting_the_heart_of_our_sync_engine_in_rust/ https://www.reddit.com/r/rust/comments/fjt4q3/rewriting_the_...
boris 4y ago
I wonder how this is deployed to presumably a large number of hosts? Do you build a distribution package out of your Rust build and ship that? If so, what about the Rust standard library? Though I believe some distributions do provide a package for the Rust standard library, but that means one also has to use the packaged rustc/cargo, which tends to lag behind quite a bit.
- pornel 4y ago
  Yes, the distribution package is built with `cargo deb`, which automatically makes a suitable binary package. It doesn't need Rust in production. Rust's standard library is compiled into the executable. Its size is negligible, especially with link-time-optimizations.
- nicoburns 4y ago
  Rust's standard library is statically linked. Rust binaries typically only require libc (and can be compiled with musl to avoid that dependency too).
- 0x457 4y ago
  Rust doesn't have stable ABI, so distributions don't provide "standard library", they provide the toolchain you use to build your rust packages. Emphasis on packages because that's the use-case for toolchain provided by distro and not local development. In the end, you end up with a binary that is statically linked, except for `glibc` (unless you use libc like musl) and whatever C library you choose to dynamically link against. Since this is an in-house project, they can build their packages with whatever toolchain they choose and distribute packages for deployment. So whatever toolchain provided by distro is irrelevant unless this packages is part of the distro's official package repository (which it isn't in this case).
Sytten 4y ago
The post mentions tokio, but I would be curious to see if it uses tower or something similar in house. For our product (caido.io) we also built a custom HTTP parser so if you open source the tool it could be nice to split the parsing in its own crate so we have an alternative to hyper that can understand malformed requests.
totallyunknown 4y ago
We did the same. We've replaced nginx/lua with a cache server (for video) written in Golang - now serving up to 100 Gbit/s per node. It's more CPU and memory efficient and completely tailored to our needs. We are happy that we moved away from nginx.
- BonoboIO 4y ago
  Wow ... 100 Gbit/s. Where do you work? That’s some serious traffic.
  - trillic 4y ago
    100 Gbit/s is only like 3000 concurrent viewers at 5000 KiB/s.
    - mplewis 4y ago
      “Only” :)
      - 988747 4y ago
        Some sport events are live streamed by 10-20 million people in a single country. that means you need 3000-7000 such nodes to serve them.
        nly 4y ago
        It's almost as if on-demand video streaming isnt suitable for broadcasting live events.
    - totallyunknown 4y ago
      100 Gbit/s / 5000 KiB/s is 20000.
      - ImprobableTruth 4y ago
        KiB = Kibibyte, not Kibibit
noncoml 4y ago
Really curious, are they using async/await?
- jhgg 4y ago
  They mentioned they were using tokio, so naturally, yes.
- jgrahamc 4y ago
  I see quite a bit of async fn and .await(); in the source code. What did you want to know?
  - nemothekid 4y ago
    .await(); Hmm does the code compile at all?
    - jgrahamc 4y ago
      Old C habits die hard.
tothrowaway 4y ago
Does anyone know why nginx used separate processes for workers, instead of threads? This post makes it sound like threads are the way to go, but presumably nginx had a reason for using processes back in the day.
- deleted 4y ago
  [deleted]
- vbernat 4y ago
  Share-nothing architecture were deemed more scalable as you don't need synchronization. But then, you can't share stuff, like a connection pool. Also, the architecture was simpler this way. Nginx is also an application server and it was "easy" to develop applications on top of it because of this architecture.
- AgentME 4y ago
  Nginx was written in C. Multithreaded code in a language that doesn't provide any safety rails is hard to get right, and so is async code. They probably figured that the complexity of doing both async and multithreading outweighed the benefits that were predicted to be small. Rust's type system checks for and prohibits many kinds of mistakes that are possible in multithreaded code and in async code, so it's much easier to combine them safely.
- tyingq 4y ago
  Wikipedia says Nginx started in 2004. If you look at the state of things for threads and other "lots of sockets to deal with" things in that timeframe, you can see multiprocess was probably a safer choice, especially if you intended on running well on a variety of Unix or Unix like OSes. This page has some of that state captured pretty well: http://www.kegel.com/c10k.html http://www.kegel.com/c10k.html
- thayne 4y ago
  Haproxy had a similar architecture for a long time, but recently switched to a worker threads model.
- 0x457 4y ago
marune 4y ago
In the 3rd party section, no mention of HAProxy as a candidate, any specific reason for that?
kronololo 4y ago
What are HTTP status codes greater than 599 used for in practice? It'd be interesting to see another Cloudflare blog post that just goes into detail on the weird protocol behaviour they've had to work around over the years. I imagine they have more insight into this than pretty much any other organisation on the planet.
- Matthias247 4y ago
  If you are in the HTTP proxying business, you might not even know. It's just customer workflows using those status codes. And you can pick either to support them, or to lose the ability to handle traffic for those customers.
- jenny91 4y ago
  Presumably custom statuses for app-to-app traffic or someone's weird API, etc.
- masklinn 4y ago
  I concur with custom signals, using a code >= 600 mitigates the risk that it would ever be allocated, or that you'd conflict with an existing unofficial use.
pornel 4y ago
Huge congratulations to the tokio.rs team — the async runtime has proven to work well even in such demanding project.
fasteo 4y ago
Great write up ! Any cloudflarer involved in this project mind sharing some basic metrics like LOCs, team size, how long from design to first deployment. Just curious.
stjohnswarts 4y ago
Any one else immediately do "open source" ctrl-f? That's all I wanted to read but I bookmarked the article and put it on my list of things to peruse later
- mcherm 4y ago
  > That's all I wanted to read While I agree that in selecting a proxy to use whether it is open source is one of the most important considerations, if that's all you look for in this article, you may be missing something. I thought the article did a good job of describing how they went about making the choice of whether to continue contributing to an existing (nominally open source) system or to build a new one. And of course, it did a good job of showcasing the strengths of Rust (reliability guarantees strong enough that they could identify when problems were due to hardware.)
rowin 4y ago
Was Go considered as the language to write Pingora in? If so, why was Rust chosen?
- TheFlyingFish 4y ago
  Not from Cloudflare, but at a guess: * They already have some pretty deep Rust experience on staff * They were already dissatisfied with the performance penalty from Lua's GC, so Go's GC was presumably unattractive as well * Rust is worth more internet points than Go (just kidding, mostly)
  - 0x457 4y ago
    Well, rust's async ecosystem is top-notch as long as you're writing a network load balancer...
    - ekidd 4y ago
      Rust's ecosystem is usually fantastic for CLI tools and specialized network servers. Large REST API servers aren't quite as solid (but perfectly doable for basic cases), and GUIs are nowhere near mature. In several major areas, I actually like the available Rust crates for a given task more than I like the available npm modules. Rust has many fewer third-party libraries available, but the quality is often good.
      - 0x457 4y ago
        Oh, I know. I was just joking that async rust is heavily geared towards build some kind of L7 proxy.
- riobard 4y ago
  Right now the idiomatic Go approach to handle networking is 2-goroutine per socket (one for reading and the other for writing). Goroutines are very lightweight userspace threads, but they're not free: each costs you a small amount of memory. At Cloudflare's scale, this overhead quickly adds up. So resource-wise, Go isn't ideal for very large scale use cases. There is a more-than-6-year-old proposal to introduce non-blocking I/O API (https://github.com/golang/go/issues/15735 https://github.com/golang/go/issues/15735) but so far it's not gaining much traction. Maybe in Go2.
aliljet 4y ago
I'm mildly blown away to read, 'And the NGINX community is not very active, and development tends to be “behind closed doors”.' Is this a reflection of the company, nginx (now owned by F5) going the way of an Oracle-style takeover of WebLogic from another era?
- moderation 4y ago
  Dropbox wrote about their migration from NGINX to Envoy in July 2020 and highlighted a lot of the same concerns [0]. As per this thread [1], NGINX have posted very similar blog posts for the last two years saying they are 'returning to our open source roots', but without much tangible change. And the Cloudflare CEO forecasted this move away from NGINX back in 2018 [2] 0. https://dropbox.tech/infrastructure/how-we-migrated-dropbox-from-nginx-to-envoy https://dropbox.tech/infrastructure/how-we-migrated-dropbox-... 1. https://news.ycombinator.com/item?id=32572153 https://news.ycombinator.com/item?id=32572153 2. https://twitter.com/eastdakota/status/1024515150546493440 https://twitter.com/eastdakota/status/1024515150546493440
- schmichael 4y ago
  IMHO nginx has never been a particularly "open" or friendly open source project. I don't mean to sound rude. I don't think open source contributors "owe" anyone anything in this regard. If you want to throw code over a wall and run away, that's your prerogative. However I do think Cloudflare's assessment is accurate and a real liability for them. Some of the OSS papercuts with nginx: - nginx has always used a "submit a patch to a mailing list" style of contributions. Many contributions, my own attempt a decade ago, just get ghosted: https://mailman.nginx.org/pipermail/nginx-devel/2010-December/000605.html https://mailman.nginx.org/pipermail/nginx-devel/2010-Decembe... - Neither the contributing page (http://nginx.org/en/docs/contributing_changes.html http://nginx.org/en/docs/contributing_changes.html) nor the Mercurial repo (http://hg.nginx.org/ http://hg.nginx.org/) redirect to HTTPS! - Tests were a later addition and in a distinct repo with a bespoke harness. I'm sure it has advantages, but it also takes extra work for contributors to figure out. - They use Trac?! I loved Trac circa 2008 but had no idea it was still a thing. I can't even login to it without it timing out. I don't want to nitpick an excellent project like nginx, but I think it's clear that easing third party contributions has never been a high priority.
TimTheTinker 4y ago
Did you guys consider HAProxy? I've only ever heard good things about it - particularly stability (though it probably can't beat Rust), performance, and configurability.
vgchh 4y ago
They don't say much on why not Envoy. It would be interesting to hear if there were concerns with it.
- chillfox 4y ago
  To me it looked like they didn't have anything technical against Envoy, they just didn't want to be dependent upon someone else for what is a core part of their product anymore. Which is entirely reasonable and a good idea.
  - jgrahamc 4y ago
    That’s right. Long ago we used an open source DNS server for our authoritative service and at some we knew we had to own our our destiny. Same here.
1vuio0pswjnm7 4y ago
Besides comparing this to Nginx plus Lua (OpenResty), has Cloudflare compared it to Haproxy plus Lua or any other similar proxies. The main issue for me with Rust is that it takes significantly more resources (time, space, memory, CPU) to build projects from source. Building Haproxy is comparatively quick and easy. The haproxy plus lua static binary (musl, no pcre) I use is already growing rather large. I will bet that Pingora binaries using shared libraries will be at least twice, maybe three times the size.
- thayne 4y ago
  I really love haproxy, but I have seen segfaults and other memory errors thay wouldn't have happened if it had been written in rust (not that that was an option when haproxy was originally written). If you are going to write a new http software from scratch, I definitely think rust is a good language to use. I would have liked to have seen more explanation of why they decided to build their own rather than use haproxy though.
  - 1vuio0pswjnm7 4y ago
    I have never experienced a segfault or other memory errors with haproxy. How can I reproduce the errors you describe.
    - thayne 4y ago
      I don't know of any current ones. Because when I have encountered them, I've reported them and the haproxy maintainers fixed them quickly, and backported the fixes to still maintained older versions. I don't remember the exact specifics off the top of my head, but in one case a certain kind of invalid config resulted in a crash rather than a normal error message. In another, there was an error in a bounds check, where if you chained multiple converters in the right way, you could end up capturing some additional data from other http headers.
- dcow 4y ago
  Curious: when building software, especially mission critical software, why are things like time, space, memory, and CPU at build time a concern? The concerns that usually surface for me are reliability, and performance (time, space, memory, CUP) of the resulting binary at run time. What you mention is kinda the core tradeoff. With Rust, you pay for high reliability and performance by spending cycles at compile time to ensure as much. Cloudflare certainly isn't looking to ship a new sexy live-reloaded and written in 30 minutes nodejs app to production 12 times a day. Plus I'm sure CF has some beastly build infrastructure.
keepquestioning 4y ago
Why did Google never try to buy Cloudflare?
JetSpiegel 4y ago
Maybe they could find the time to allow RSS readers to read the HTML on this post. I guess using RSS mean you are atacking their infrastructure.
pjmlp 4y ago
> Our Rust code runs more efficiently compared to our old Lua code. What a surprise, replacing an interpred dynamic language with a AOT compiled static language leads to performance improvements. I guess the learnings using Tcl as configuration language for Apache based proxies 20 years ago was lost in newer generations.
- b112 4y ago
  Hi! I'm the tcl bot, you're the 27th person to ever mention tcl on HN!
sullivanmatt 4y ago
For any of the Cloudflare team that frequents HN, curious if you have an eventual plan to open-source Pingora? I recognize it may stay proprietary if you consider it to be a differentiator and competitive advantage, but this blog post almost has a tone of "introducing this new technology!" as if it's in the cards for the future.
- eastdakota 4y ago
  We are planning on open sourcing it. That's mentioned in the post near the end.
  - peterhadlaw 4y ago
  - Dowwie 4y ago
    Do you think that it would be beneficial during analyst conference calls to highlight that Cloudflare is using Rust to build its next-gen critical systems? It shows a strong commitment to building best-in-class technology.
  - latchkey 4y ago
    It is kind of weird to point out nginx doing closed door development as a negative, and then do exactly the same thing yourself.
    - botanicalfriend 4y ago
      The “closed door development” is a problem for their business. They don’t have that problem when they maintain it, free software aside.
  - sullivanmatt 4y ago
    Thanks Matt, not sure how I missed that. Glad to hear it!
xfalcox 4y ago
I share lots of feelings towards NGINX that Cloudflare mention on this blog post. New features like 103 Early Hints and HTTP/3 exist in HAProxy and Caddy but there is nothing coming in NGINX.
- datalopers 4y ago
  nginx was good for a decade or two. they were acquired and doomed to irrelevancy since.
  - qwertox 4y ago
    F5 is not to blame, they didn't change anything for the worse. The Plus-license is the problem where essential things like monitoring are behind a paywall. Back then this wasn't so important because you basically only had Apache and nginx. I think I read two weeks ago what F5 was going to focus more on improving the open source version. Probably because the competition is getting harder and they're noticing it in a market share decline, but whichever the reason is, this was good to hear. Also it was good to see it no longer being part of a Russian company, even though the devs and owners are good people. You never know how a government can enforce some problematic behavior, specially one which is known for liking to throw people out of high rise windows.
  - 0x457 4y ago
    Nginx is an absolute beast when it comes to accepting HTTP/1.1 from clients and talking HTTP/1.0 (or HTTP/1.1) to a fixed list of backends or serving files from disk. Everything else is...hard. Writing plugins - arcane knowledge with next no none documentation. 3rd party plugins, multiple times "acquired" (read: paid the author to work on a new plugin) an open-source plugin and made a much better version available in this plus offering. gRPC support added only in 2018, http2 support was late, proper web sockets support was late. Many features that were available "for free" with OpenResty were only on the paid version - good luck writing your own plugin. They were doomed before F5 probably around round time when they raised series B1 in 2014. All this money and it still had nothing "new" to show. For every Plus plugin, there was most likely a better solution available, and often enough that solution was free or cheaper. Almost nothing has changed since F5 aquired them.
nicoburns 4y ago
> When crashes do occur an engineer needs to spend time to diagnose how it happened and what caused it. Since Pingora's inception we’ve served a few hundred trillion requests and have yet to crash due to our service code. > In fact, Pingora crashes are so rare we usually find unrelated issues when we do encounter one. Recently we discovered a kernel bug soon after our service started crashing. We've also discovered hardware issues on a few machines, in the past ruling out rare memory bugs caused by our software even after significant debugging was nearly impossible. That's quite the endorsement of Rust. A lot of people focus on the fact that Rust can't absolutely guarantee freedom from crashes and memory safety issues. Which I think misses the point that this kind of experience of running high traffic Rust services in production for months almost without a single issue is common in practice.
- brink 4y ago
  I had the same experience when I wrote a camera capture / motion detection / video logging service for a commercial smart refrigerator in Rust. The Swift component crashed at least twice a week, the Rust component ran for months and months without issue.
- jhgg 4y ago
  We had the same experience at work deploying rust services that serve many billions of requests a day as well.
  - petr_tik 4y ago
    does your company have any public information about this? Blogs, job descriptions with numbers, twitter threads?
    - remram 4y ago
      Not GP but I believe this is it: https://discord.com/category/engineering https://discord.com/category/engineering
- drogus 4y ago
VWWHFSfQ 4y ago
Should have waited to post this until it was actually ready to be open sourced. Otherwise this is just kinda like "huh, neat" without anything else to do with it.
- qwertox 4y ago
  In some cases it can be enough to know that it could be worth waiting for the release instead of putting more resources into a stack you're currently using. You might replace it entirely in a few months if the release turns out to be a product which you can and want to switch to, so it's ok to get a heads-up.
  - nextaccountic 4y ago
    Unfortunately, without being able to run the code yourself or at least seeing a benchmark, it's hard to commit to unreleased code like this
    - qwertox 4y ago
      True. But you're not committing to it, you're just waiting for the evaluation instead directing resources into a change which might become obsolete soon, should the new option be a viable one. In this case, where the code is coming from Cloudflare, you can be sure that they know what they're doing, that there is a high chance that it will be very high quality code. Apart from this, it's an interesting article from a technical perspective due to the various topics it touches. Also, in my case, where I'm looking for an new language to program in, where Rust and Go are the my main options, articles like these help me with the decision process on which one to choose. There's much more value to this article than it just being an announcement that sometime in the near future they will be releasing an open source project. It almost sounded like a "give me the code or shut up"-message.
swlkr 4y ago
Wow this is just what I was looking for, a proxy written in a memory safe language like rust with no GC as an alternative to nginx. Looking forward to the open source version!
AtNightWeCode 4y ago
Sounds good. I never encountered any performance issues with Cloudflare. If you have the time for enhancement, then: 1. Option to hit the cache before workers. (Why we never use workers). 2. Rules for blocking traffic during nights (time-based rules). 3. Make sure every product is a replacement. If you offer the same thing as a cloud provider. Don’t make us write a lot of custom code.
- ehPReth 4y ago
  Could you expand on 3?
  - AtNightWeCode 4y ago
    Why would you swap Azure blob storage or S3 for anything in Cloudflare if it comes with running custom code in workers?
- RL_Quine 4y ago
  When is night on the internet?
  - AtNightWeCode 4y ago
    Not uncommon before the intro of edge services to block login on sites during off working hours or during nights. Or at least doing some rate-limiting. We see many attempts at brute force-attacks during the nights. Most sites are not global.
  - yamtaddle 4y ago
    Most—maybe damn near all—sites see significant dips in traffic for at least a few hours a day. Which part of the day, depends on the site. More often than not, it's while the team is asleep and staffing, if any, is at its lowest point, since teams tend to live roughly in the same ~half of the world that their products are most-used in. Plus there's practically no-one in the Pacific until you reach Japan, and not a ton of "Western" sites see much use in Asia, and vice versa, with a few notable exceptions. It's not unusual for e.g. ecommerce sites to crank up automated fraud prevention "at night" because staffing is so much lower. TL;DR Most sites' usage patterns exhibit a pronounced day/night cycle that's not too far off from natural day/night cycles where the bulk of the team lives.

Is it open source?

jgrahamc 4y ago
It will be. There will be a follow up blog post about the open sourcing with all the gory details of how it was built and how it works.
- latchkey 4y ago
  I think you should remove the part about closed door development as a negative for nginx given the way that this has been developed.
  - 0x457 4y ago
    Why? It's closed door development in nginx from their perspective. This doesn't apply for their in-house project. They not telling you why you would want to switch off Nginx, they are telling you why they've switched.
- network2592 4y ago
  It might be a couple of months. The open sourcing of Cloudflare workers was announced in May and still has not been released. [1] [1] https://blog.cloudflare.com/workers-open-source-announcement/ https://blog.cloudflare.com/workers-open-source-announcement...
- Dowwie 4y ago
  You may be interested in knowing that about two years ago, a team of engineers at Dropbox wrote gory details about their use of Rust and it was inspiring. The passion about their work really came through. The team also held an AMA on /r/rust that went well. See here: https://www.reddit.com/r/rust/comments/fjt4q3/rewriting_the_heart_of_our_sync_engine_in_rust/ https://www.reddit.com/r/rust/comments/fjt4q3/rewriting_the_...

I wonder how this is deployed to presumably a large number of hosts? Do you build a distribution package out of your Rust build and ship that? If so, what about the Rust standard library? Though I believe some distributions do provide a package for the Rust standard library, but that means one also has to use the packaged rustc/cargo, which tends to lag behind quite a bit.

pornel 4y ago
Yes, the distribution package is built with `cargo deb`, which automatically makes a suitable binary package. It doesn't need Rust in production. Rust's standard library is compiled into the executable. Its size is negligible, especially with link-time-optimizations.
nicoburns 4y ago
Rust's standard library is statically linked. Rust binaries typically only require libc (and can be compiled with musl to avoid that dependency too).
0x457 4y ago
Rust doesn't have stable ABI, so distributions don't provide "standard library", they provide the toolchain you use to build your rust packages. Emphasis on packages because that's the use-case for toolchain provided by distro and not local development. In the end, you end up with a binary that is statically linked, except for `glibc` (unless you use libc like musl) and whatever C library you choose to dynamically link against. Since this is an in-house project, they can build their packages with whatever toolchain they choose and distribute packages for deployment. So whatever toolchain provided by distro is irrelevant unless this packages is part of the distro's official package repository (which it isn't in this case).

Sytten 4y ago

The post mentions tokio, but I would be curious to see if it uses tower or something similar in house. For our product (caido.io) we also built a custom HTTP parser so if you open source the tool it could be nice to split the parsing in its own crate so we have an alternative to hyper that can understand malformed requests.

totallyunknown 4y ago

We did the same. We've replaced nginx/lua with a cache server (for video) written in Golang - now serving up to 100 Gbit/s per node. It's more CPU and memory efficient and completely tailored to our needs. We are happy that we moved away from nginx.

BonoboIO 4y ago
Wow ... 100 Gbit/s. Where do you work? That’s some serious traffic.
- trillic 4y ago
  100 Gbit/s is only like 3000 concurrent viewers at 5000 KiB/s.
  - mplewis 4y ago
    “Only” :)
    - 988747 4y ago
      Some sport events are live streamed by 10-20 million people in a single country. that means you need 3000-7000 such nodes to serve them.
      - nly 4y ago
        It's almost as if on-demand video streaming isnt suitable for broadcasting live events.
  - totallyunknown 4y ago
    100 Gbit/s / 5000 KiB/s is 20000.
    - ImprobableTruth 4y ago
      KiB = Kibibyte, not Kibibit

noncoml 4y ago

Really curious, are they using async/await?

jhgg 4y ago
They mentioned they were using tokio, so naturally, yes.
jgrahamc 4y ago
I see quite a bit of async fn and .await(); in the source code. What did you want to know?
- nemothekid 4y ago
  .await(); Hmm does the code compile at all?
  - jgrahamc 4y ago
    Old C habits die hard.

tothrowaway 4y ago

Does anyone know why nginx used separate processes for workers, instead of threads? This post makes it sound like threads are the way to go, but presumably nginx had a reason for using processes back in the day.

deleted 4y ago
[deleted]
vbernat 4y ago
Share-nothing architecture were deemed more scalable as you don't need synchronization. But then, you can't share stuff, like a connection pool. Also, the architecture was simpler this way. Nginx is also an application server and it was "easy" to develop applications on top of it because of this architecture.
AgentME 4y ago
Nginx was written in C. Multithreaded code in a language that doesn't provide any safety rails is hard to get right, and so is async code. They probably figured that the complexity of doing both async and multithreading outweighed the benefits that were predicted to be small. Rust's type system checks for and prohibits many kinds of mistakes that are possible in multithreaded code and in async code, so it's much easier to combine them safely.
tyingq 4y ago
Wikipedia says Nginx started in 2004. If you look at the state of things for threads and other "lots of sockets to deal with" things in that timeframe, you can see multiprocess was probably a safer choice, especially if you intended on running well on a variety of Unix or Unix like OSes. This page has some of that state captured pretty well: http://www.kegel.com/c10k.html http://www.kegel.com/c10k.html
thayne 4y ago
Haproxy had a similar architecture for a long time, but recently switched to a worker threads model.
0x457 4y ago

marune 4y ago

In the 3rd party section, no mention of HAProxy as a candidate, any specific reason for that?

kronololo 4y ago

What are HTTP status codes greater than 599 used for in practice? It'd be interesting to see another Cloudflare blog post that just goes into detail on the weird protocol behaviour they've had to work around over the years. I imagine they have more insight into this than pretty much any other organisation on the planet.

Matthias247 4y ago
If you are in the HTTP proxying business, you might not even know. It's just customer workflows using those status codes. And you can pick either to support them, or to lose the ability to handle traffic for those customers.
jenny91 4y ago
Presumably custom statuses for app-to-app traffic or someone's weird API, etc.
masklinn 4y ago
I concur with custom signals, using a code >= 600 mitigates the risk that it would ever be allocated, or that you'd conflict with an existing unofficial use.

pornel 4y ago

Huge congratulations to the tokio.rs team — the async runtime has proven to work well even in such demanding project.

fasteo 4y ago

Great write up ! Any cloudflarer involved in this project mind sharing some basic metrics like LOCs, team size, how long from design to first deployment. Just curious.

stjohnswarts 4y ago

Any one else immediately do "open source" ctrl-f? That's all I wanted to read but I bookmarked the article and put it on my list of things to peruse later

mcherm 4y ago
> That's all I wanted to read While I agree that in selecting a proxy to use whether it is open source is one of the most important considerations, if that's all you look for in this article, you may be missing something. I thought the article did a good job of describing how they went about making the choice of whether to continue contributing to an existing (nominally open source) system or to build a new one. And of course, it did a good job of showcasing the strengths of Rust (reliability guarantees strong enough that they could identify when problems were due to hardware.)

rowin 4y ago

Was Go considered as the language to write Pingora in? If so, why was Rust chosen?

TheFlyingFish 4y ago
Not from Cloudflare, but at a guess: * They already have some pretty deep Rust experience on staff * They were already dissatisfied with the performance penalty from Lua's GC, so Go's GC was presumably unattractive as well * Rust is worth more internet points than Go (just kidding, mostly)
- 0x457 4y ago
  Well, rust's async ecosystem is top-notch as long as you're writing a network load balancer...
  - ekidd 4y ago
    Rust's ecosystem is usually fantastic for CLI tools and specialized network servers. Large REST API servers aren't quite as solid (but perfectly doable for basic cases), and GUIs are nowhere near mature. In several major areas, I actually like the available Rust crates for a given task more than I like the available npm modules. Rust has many fewer third-party libraries available, but the quality is often good.
    - 0x457 4y ago
      Oh, I know. I was just joking that async rust is heavily geared towards build some kind of L7 proxy.
riobard 4y ago
Right now the idiomatic Go approach to handle networking is 2-goroutine per socket (one for reading and the other for writing). Goroutines are very lightweight userspace threads, but they're not free: each costs you a small amount of memory. At Cloudflare's scale, this overhead quickly adds up. So resource-wise, Go isn't ideal for very large scale use cases. There is a more-than-6-year-old proposal to introduce non-blocking I/O API (https://github.com/golang/go/issues/15735 https://github.com/golang/go/issues/15735) but so far it's not gaining much traction. Maybe in Go2.

aliljet 4y ago

I'm mildly blown away to read, 'And the NGINX community is not very active, and development tends to be “behind closed doors”.' Is this a reflection of the company, nginx (now owned by F5) going the way of an Oracle-style takeover of WebLogic from another era?

moderation 4y ago
Dropbox wrote about their migration from NGINX to Envoy in July 2020 and highlighted a lot of the same concerns [0]. As per this thread [1], NGINX have posted very similar blog posts for the last two years saying they are 'returning to our open source roots', but without much tangible change. And the Cloudflare CEO forecasted this move away from NGINX back in 2018 [2] 0. https://dropbox.tech/infrastructure/how-we-migrated-dropbox-from-nginx-to-envoy https://dropbox.tech/infrastructure/how-we-migrated-dropbox-... 1. https://news.ycombinator.com/item?id=32572153 https://news.ycombinator.com/item?id=32572153 2. https://twitter.com/eastdakota/status/1024515150546493440 https://twitter.com/eastdakota/status/1024515150546493440
schmichael 4y ago
IMHO nginx has never been a particularly "open" or friendly open source project. I don't mean to sound rude. I don't think open source contributors "owe" anyone anything in this regard. If you want to throw code over a wall and run away, that's your prerogative. However I do think Cloudflare's assessment is accurate and a real liability for them. Some of the OSS papercuts with nginx: - nginx has always used a "submit a patch to a mailing list" style of contributions. Many contributions, my own attempt a decade ago, just get ghosted: https://mailman.nginx.org/pipermail/nginx-devel/2010-December/000605.html https://mailman.nginx.org/pipermail/nginx-devel/2010-Decembe... - Neither the contributing page (http://nginx.org/en/docs/contributing_changes.html http://nginx.org/en/docs/contributing_changes.html) nor the Mercurial repo (http://hg.nginx.org/ http://hg.nginx.org/) redirect to HTTPS! - Tests were a later addition and in a distinct repo with a bespoke harness. I'm sure it has advantages, but it also takes extra work for contributors to figure out. - They use Trac?! I loved Trac circa 2008 but had no idea it was still a thing. I can't even login to it without it timing out. I don't want to nitpick an excellent project like nginx, but I think it's clear that easing third party contributions has never been a high priority.

TimTheTinker 4y ago

Did you guys consider HAProxy? I've only ever heard good things about it - particularly stability (though it probably can't beat Rust), performance, and configurability.

vgchh 4y ago

They don't say much on why not Envoy. It would be interesting to hear if there were concerns with it.

chillfox 4y ago
To me it looked like they didn't have anything technical against Envoy, they just didn't want to be dependent upon someone else for what is a core part of their product anymore. Which is entirely reasonable and a good idea.
- jgrahamc 4y ago
  That’s right. Long ago we used an open source DNS server for our authoritative service and at some we knew we had to own our our destiny. Same here.

1vuio0pswjnm7 4y ago

Besides comparing this to Nginx plus Lua (OpenResty), has Cloudflare compared it to Haproxy plus Lua or any other similar proxies. The main issue for me with Rust is that it takes significantly more resources (time, space, memory, CPU) to build projects from source. Building Haproxy is comparatively quick and easy. The haproxy plus lua static binary (musl, no pcre) I use is already growing rather large. I will bet that Pingora binaries using shared libraries will be at least twice, maybe three times the size.

thayne 4y ago
I really love haproxy, but I have seen segfaults and other memory errors thay wouldn't have happened if it had been written in rust (not that that was an option when haproxy was originally written). If you are going to write a new http software from scratch, I definitely think rust is a good language to use. I would have liked to have seen more explanation of why they decided to build their own rather than use haproxy though.
- 1vuio0pswjnm7 4y ago
  I have never experienced a segfault or other memory errors with haproxy. How can I reproduce the errors you describe.
  - thayne 4y ago
    I don't know of any current ones. Because when I have encountered them, I've reported them and the haproxy maintainers fixed them quickly, and backported the fixes to still maintained older versions. I don't remember the exact specifics off the top of my head, but in one case a certain kind of invalid config resulted in a crash rather than a normal error message. In another, there was an error in a bounds check, where if you chained multiple converters in the right way, you could end up capturing some additional data from other http headers.
dcow 4y ago
Curious: when building software, especially mission critical software, why are things like time, space, memory, and CPU at build time a concern? The concerns that usually surface for me are reliability, and performance (time, space, memory, CUP) of the resulting binary at run time. What you mention is kinda the core tradeoff. With Rust, you pay for high reliability and performance by spending cycles at compile time to ensure as much. Cloudflare certainly isn't looking to ship a new sexy live-reloaded and written in 30 minutes nodejs app to production 12 times a day. Plus I'm sure CF has some beastly build infrastructure.

keepquestioning 4y ago

Why did Google never try to buy Cloudflare?

JetSpiegel 4y ago

Maybe they could find the time to allow RSS readers to read the HTML on this post. I guess using RSS mean you are atacking their infrastructure.

pjmlp 4y ago

> Our Rust code runs more efficiently compared to our old Lua code. What a surprise, replacing an interpred dynamic language with a AOT compiled static language leads to performance improvements. I guess the learnings using Tcl as configuration language for Apache based proxies 20 years ago was lost in newer generations.

b112 4y ago
Hi! I'm the tcl bot, you're the 27th person to ever mention tcl on HN!

sullivanmatt 4y ago

For any of the Cloudflare team that frequents HN, curious if you have an eventual plan to open-source Pingora? I recognize it may stay proprietary if you consider it to be a differentiator and competitive advantage, but this blog post almost has a tone of "introducing this new technology!" as if it's in the cards for the future.

eastdakota 4y ago
We are planning on open sourcing it. That's mentioned in the post near the end.
- peterhadlaw 4y ago
- Dowwie 4y ago
  Do you think that it would be beneficial during analyst conference calls to highlight that Cloudflare is using Rust to build its next-gen critical systems? It shows a strong commitment to building best-in-class technology.
- latchkey 4y ago
  It is kind of weird to point out nginx doing closed door development as a negative, and then do exactly the same thing yourself.
  - botanicalfriend 4y ago
    The “closed door development” is a problem for their business. They don’t have that problem when they maintain it, free software aside.
- sullivanmatt 4y ago
  Thanks Matt, not sure how I missed that. Glad to hear it!

xfalcox 4y ago

I share lots of feelings towards NGINX that Cloudflare mention on this blog post. New features like 103 Early Hints and HTTP/3 exist in HAProxy and Caddy but there is nothing coming in NGINX.

datalopers 4y ago
nginx was good for a decade or two. they were acquired and doomed to irrelevancy since.
- qwertox 4y ago
  F5 is not to blame, they didn't change anything for the worse. The Plus-license is the problem where essential things like monitoring are behind a paywall. Back then this wasn't so important because you basically only had Apache and nginx. I think I read two weeks ago what F5 was going to focus more on improving the open source version. Probably because the competition is getting harder and they're noticing it in a market share decline, but whichever the reason is, this was good to hear. Also it was good to see it no longer being part of a Russian company, even though the devs and owners are good people. You never know how a government can enforce some problematic behavior, specially one which is known for liking to throw people out of high rise windows.
- 0x457 4y ago
  Nginx is an absolute beast when it comes to accepting HTTP/1.1 from clients and talking HTTP/1.0 (or HTTP/1.1) to a fixed list of backends or serving files from disk. Everything else is...hard. Writing plugins - arcane knowledge with next no none documentation. 3rd party plugins, multiple times "acquired" (read: paid the author to work on a new plugin) an open-source plugin and made a much better version available in this plus offering. gRPC support added only in 2018, http2 support was late, proper web sockets support was late. Many features that were available "for free" with OpenResty were only on the paid version - good luck writing your own plugin. They were doomed before F5 probably around round time when they raised series B1 in 2014. All this money and it still had nothing "new" to show. For every Plus plugin, there was most likely a better solution available, and often enough that solution was free or cheaper. Almost nothing has changed since F5 aquired them.

nicoburns 4y ago

> When crashes do occur an engineer needs to spend time to diagnose how it happened and what caused it. Since Pingora's inception we’ve served a few hundred trillion requests and have yet to crash due to our service code. > In fact, Pingora crashes are so rare we usually find unrelated issues when we do encounter one. Recently we discovered a kernel bug soon after our service started crashing. We've also discovered hardware issues on a few machines, in the past ruling out rare memory bugs caused by our software even after significant debugging was nearly impossible. That's quite the endorsement of Rust. A lot of people focus on the fact that Rust can't absolutely guarantee freedom from crashes and memory safety issues. Which I think misses the point that this kind of experience of running high traffic Rust services in production for months almost without a single issue is common in practice.

brink 4y ago
I had the same experience when I wrote a camera capture / motion detection / video logging service for a commercial smart refrigerator in Rust. The Swift component crashed at least twice a week, the Rust component ran for months and months without issue.
jhgg 4y ago
We had the same experience at work deploying rust services that serve many billions of requests a day as well.
- petr_tik 4y ago
  does your company have any public information about this? Blogs, job descriptions with numbers, twitter threads?
  - remram 4y ago
    Not GP but I believe this is it: https://discord.com/category/engineering https://discord.com/category/engineering
drogus 4y ago

VWWHFSfQ 4y ago

Should have waited to post this until it was actually ready to be open sourced. Otherwise this is just kinda like "huh, neat" without anything else to do with it.

qwertox 4y ago
In some cases it can be enough to know that it could be worth waiting for the release instead of putting more resources into a stack you're currently using. You might replace it entirely in a few months if the release turns out to be a product which you can and want to switch to, so it's ok to get a heads-up.
- nextaccountic 4y ago
  Unfortunately, without being able to run the code yourself or at least seeing a benchmark, it's hard to commit to unreleased code like this
  - qwertox 4y ago
    True. But you're not committing to it, you're just waiting for the evaluation instead directing resources into a change which might become obsolete soon, should the new option be a viable one. In this case, where the code is coming from Cloudflare, you can be sure that they know what they're doing, that there is a high chance that it will be very high quality code. Apart from this, it's an interesting article from a technical perspective due to the various topics it touches. Also, in my case, where I'm looking for an new language to program in, where Rust and Go are the my main options, articles like these help me with the decision process on which one to choose. There's much more value to this article than it just being an announcement that sometime in the near future they will be releasing an open source project. It almost sounded like a "give me the code or shut up"-message.

swlkr 4y ago

Wow this is just what I was looking for, a proxy written in a memory safe language like rust with no GC as an alternative to nginx. Looking forward to the open source version!

AtNightWeCode 4y ago

Sounds good. I never encountered any performance issues with Cloudflare. If you have the time for enhancement, then: 1. Option to hit the cache before workers. (Why we never use workers). 2. Rules for blocking traffic during nights (time-based rules). 3. Make sure every product is a replacement. If you offer the same thing as a cloud provider. Don’t make us write a lot of custom code.

ehPReth 4y ago
Could you expand on 3?
- AtNightWeCode 4y ago
  Why would you swap Azure blob storage or S3 for anything in Cloudflare if it comes with running custom code in workers?
RL_Quine 4y ago
When is night on the internet?
- AtNightWeCode 4y ago
  Not uncommon before the intro of edge services to block login on sites during off working hours or during nights. Or at least doing some rate-limiting. We see many attempts at brute force-attacks during the nights. Most sites are not global.
- yamtaddle 4y ago
  Most—maybe damn near all—sites see significant dips in traffic for at least a few hours a day. Which part of the day, depends on the site. More often than not, it's while the team is asleep and staffing, if any, is at its lowest point, since teams tend to live roughly in the same ~half of the world that their products are most-used in. Plus there's practically no-one in the Pacific until you reach Japan, and not a ton of "Western" sites see much use in Asia, and vice versa, with a few notable exceptions. It's not unusual for e.g. ecommerce sites to crank up automated fraud prevention "at night" because staffing is so much lower. TL;DR Most sites' usage patterns exhibit a pronounced day/night cycle that's not too far off from natural day/night cycles where the bulk of the team lives.