5 ms·

I tried running Go HTTP servers bare to the internet (after Cloudflare promoted doing so in a blog post), but went back to using a reverse proxy the next time.

I tried running Go HTTP servers bare to the internet (after Cloudflare promoted doing so in a blog post), but went back to using a reverse proxy the next time. The main benefit seems to be convenience. I can upgrade and graceful-restart nginx instead of having to rebuild and redeploy the Go server (involving a full app restart). Not having to worry about goroutine leaks because some jerk decided to send the request line @ 1 byte/sec is just an added bonus.

tills13 7y ago
CloudFlare itself offers request buffering, so you should be good on that front.
jrockway 7y ago
You have to worry about the jerk sending requests at 1 byte per second no matter which webserver you use. It's always a problem to let an unlimited number of people ask for an unlimited amount of resources; it's just that things like goroutines are heavier than a file descriptor or a few bytes of RAM, so you'll notice wasted goroutines more quickly than wasted fds or memory. Typically, you need to consider the total amount of memory you want your web server to use, how much of that memory one request can use, and how long a request can use that memory. (File descriptors must also be considered.) Envoy has a section in their documentation about this here: https://www.envoyproxy.io/docs/envoy/latest/configuration/best_practices/edge#best-practices-edge https://www.envoyproxy.io/docs/envoy/latest/configuration/be... nginx similarly has a number of knobs to turn: https://www.nginx.com/blog/tuning-nginx/ https://www.nginx.com/blog/tuning-nginx/ I use Envoy as my web proxy and nginx to serve static content. My envoy configuration is complicated and my nginx configuration is simple, as a result. I imagine that if you are hosting a serious amount of traffic with Nginx as the edge proxy, more tuning is required. I've never tried, so I don't really know.
- zzzcpan 7y ago
  > You have to worry about the jerk sending requests at 1 byte per second no matter which webserver you use. Not necessarily. It's just free webservers don't bother dealing with it, but there are plenty of simple approaches. Like just dropping connections that are sending requests slower than some threshold or dropping the slowest connection when some total number of connections is reached. Or more complicated, which also works to protect from all kinds of attacks, dropping the highest malicious score or the lowest reputation score client when some resource usage threshold is reached. None of these are easy to implement with synchronous multithreaded networking code though, like in Go. Realistically it's only viable with asynchronous single threaded programming models or an actor model.
  - jlokier 7y ago
    > None of these are easy to implement with synchronous multithreaded networking code though, like in Go. Realistically it's only viable with asynchronous single threaded programming models or an actor model. It's hard to see why synchronous multi-threaded code would find these things any more difficult than async or actor models. All three models are equally able to access shared data structures to keep track of resource usage statistics, per-connection statistics, and timers. OS kernels do this routinely, and are essentially multi-threaded on SMP architectures or with kernel pre-emption.
    - zzzcpan 7y ago
      Basically the reason is you can't just kill a thread that shares memory with other threads. Go doesn't even have an ability to kill goroutines, so your only choices is manual context tracking and manual cancellation in every piece of code. But if you are in a an event loop, for example, you can just destroy any client at any point. Same with actors, if you are in an actor, you can just kill other actors.
      - jlokier 7y ago
        Thanks, that's an interesting point of view. Unfortunately, with event loops and async programming, including async-await models, cancellation is just as fiddly and needing to be explicitly handled by client event handlers/awaiters. For example, think of JavaScript and its promises or their async-await equivalent. There is no standard, generic way to cancel those operations in progress, because it's a tricky problem.
        zzzcpan 7y ago
        > cancellation is just as fiddly and needing to be explicitly handled by client event handlers/awaiters That's not true. In event loops to do cancellation you simply remove event handlers for associated client from whatever event notification mechanism you are using and delete (free) client's data structured, including futures, promises or whatever you are using. Since references to all of them are necessary for event loops to be able to even call event handlers, no awareness of any of it on event handlers' side is required.
- earthboundkid 7y ago
  You have to worry about slow requests somewhere in your stack, but with a good network architecture, you can assume the problem is solved at the open internet interface and ignore it inside your trusted zone.
- toast0 7y ago
  File descriptors are mostly a soft limit --- you can usually easily set the os and process limit higher than what your stack can process. The maximum setting for stock FreeBSD is the number of pages divided by four (so one fd per 16kB of ram on x86). Most systems will run out of ram much before FDs if the FD limit is all the way up. If you have a reasonable amount of ram, and a reasonable way to manage the slow connections, chances are a Slowloris attack is going to use more resources on the attackers side and not be effective. Async i/o in C based servers works pretty well. FreeBSD accept filters can work if protocol appropriate; the kernel doesn't return the socket to accept until data matching a pattern has been sent, see accf_http; but that doesn't work if the client sends the handshake quickly and further data slowly. If you really need to use a stack that doesn't work for this, putting a proxy that captures whole requests at whatever speed and then sends them as fast requests works too.
Twirrim 7y ago
One cloud service I worked for had no end of quirks exposed by someone using a 28.8k modem to upload data (if it'd been 36.6k modem it'd have been fine). It wasn't causing impact on other customers (we already had stuff to handle "slow PUTters"), they were the only one getting a bunch of 500s, but it did expose a series of unrealised assumptions in the service components.
- ronsor 7y ago
  Can you share more information? I can't imagine anyone using a 28.8k modem within the past 5 years.
  - Twirrim 7y ago
    There isn't much to share. We made numerous attempts to reach out to the customer but never got a response, so we don't know exactly what they were doing. We don't know for sure it was a 28.8K modem, it just appeared to be, given the speed they uploaded and the slight variances we saw in the speed (if it was a throttled upload, they tend to be pretty rigid in performance). One of the main things it exposed was that certain libraries we used had buffers in them. We'd proxy the data from the customer to another back end service. By default the library would open the connection to the back end, and wait for the small buffer to fill before sending the data. The back end service would terminate a connection if the connection was open but idle for $x number of seconds. The user was on the threshold of that timeout. Probably half the time they PUT, they'd be slow enough to trigger that back-end timeout, resulting in them getting a 500. I believe eventually they put a small buffer on the ingestion path too before pushing along to the back end, but given those PUTs could get really large, we couldn't buffer the entire content before sending along.

tills13 7y ago

CloudFlare itself offers request buffering, so you should be good on that front.

jrockway 7y ago

You have to worry about the jerk sending requests at 1 byte per second no matter which webserver you use. It's always a problem to let an unlimited number of people ask for an unlimited amount of resources; it's just that things like goroutines are heavier than a file descriptor or a few bytes of RAM, so you'll notice wasted goroutines more quickly than wasted fds or memory. Typically, you need to consider the total amount of memory you want your web server to use, how much of that memory one request can use, and how long a request can use that memory. (File descriptors must also be considered.) Envoy has a section in their documentation about this here: https://www.envoyproxy.io/docs/envoy/latest/configuration/best_practices/edge#best-practices-edge https://www.envoyproxy.io/docs/envoy/latest/configuration/be... nginx similarly has a number of knobs to turn: https://www.nginx.com/blog/tuning-nginx/ https://www.nginx.com/blog/tuning-nginx/ I use Envoy as my web proxy and nginx to serve static content. My envoy configuration is complicated and my nginx configuration is simple, as a result. I imagine that if you are hosting a serious amount of traffic with Nginx as the edge proxy, more tuning is required. I've never tried, so I don't really know.

zzzcpan 7y ago
> You have to worry about the jerk sending requests at 1 byte per second no matter which webserver you use. Not necessarily. It's just free webservers don't bother dealing with it, but there are plenty of simple approaches. Like just dropping connections that are sending requests slower than some threshold or dropping the slowest connection when some total number of connections is reached. Or more complicated, which also works to protect from all kinds of attacks, dropping the highest malicious score or the lowest reputation score client when some resource usage threshold is reached. None of these are easy to implement with synchronous multithreaded networking code though, like in Go. Realistically it's only viable with asynchronous single threaded programming models or an actor model.
- jlokier 7y ago
  > None of these are easy to implement with synchronous multithreaded networking code though, like in Go. Realistically it's only viable with asynchronous single threaded programming models or an actor model. It's hard to see why synchronous multi-threaded code would find these things any more difficult than async or actor models. All three models are equally able to access shared data structures to keep track of resource usage statistics, per-connection statistics, and timers. OS kernels do this routinely, and are essentially multi-threaded on SMP architectures or with kernel pre-emption.
  - zzzcpan 7y ago
    Basically the reason is you can't just kill a thread that shares memory with other threads. Go doesn't even have an ability to kill goroutines, so your only choices is manual context tracking and manual cancellation in every piece of code. But if you are in a an event loop, for example, you can just destroy any client at any point. Same with actors, if you are in an actor, you can just kill other actors.
    - jlokier 7y ago
      Thanks, that's an interesting point of view. Unfortunately, with event loops and async programming, including async-await models, cancellation is just as fiddly and needing to be explicitly handled by client event handlers/awaiters. For example, think of JavaScript and its promises or their async-await equivalent. There is no standard, generic way to cancel those operations in progress, because it's a tricky problem.
      - zzzcpan 7y ago
        > cancellation is just as fiddly and needing to be explicitly handled by client event handlers/awaiters That's not true. In event loops to do cancellation you simply remove event handlers for associated client from whatever event notification mechanism you are using and delete (free) client's data structured, including futures, promises or whatever you are using. Since references to all of them are necessary for event loops to be able to even call event handlers, no awareness of any of it on event handlers' side is required.
earthboundkid 7y ago
You have to worry about slow requests somewhere in your stack, but with a good network architecture, you can assume the problem is solved at the open internet interface and ignore it inside your trusted zone.
toast0 7y ago
File descriptors are mostly a soft limit --- you can usually easily set the os and process limit higher than what your stack can process. The maximum setting for stock FreeBSD is the number of pages divided by four (so one fd per 16kB of ram on x86). Most systems will run out of ram much before FDs if the FD limit is all the way up. If you have a reasonable amount of ram, and a reasonable way to manage the slow connections, chances are a Slowloris attack is going to use more resources on the attackers side and not be effective. Async i/o in C based servers works pretty well. FreeBSD accept filters can work if protocol appropriate; the kernel doesn't return the socket to accept until data matching a pattern has been sent, see accf_http; but that doesn't work if the client sends the handshake quickly and further data slowly. If you really need to use a stack that doesn't work for this, putting a proxy that captures whole requests at whatever speed and then sends them as fast requests works too.

Twirrim 7y ago

One cloud service I worked for had no end of quirks exposed by someone using a 28.8k modem to upload data (if it'd been 36.6k modem it'd have been fine). It wasn't causing impact on other customers (we already had stuff to handle "slow PUTters"), they were the only one getting a bunch of 500s, but it did expose a series of unrealised assumptions in the service components.

ronsor 7y ago
Can you share more information? I can't imagine anyone using a 28.8k modem within the past 5 years.
- Twirrim 7y ago
  There isn't much to share. We made numerous attempts to reach out to the customer but never got a response, so we don't know exactly what they were doing. We don't know for sure it was a 28.8K modem, it just appeared to be, given the speed they uploaded and the slight variances we saw in the speed (if it was a throttled upload, they tend to be pretty rigid in performance). One of the main things it exposed was that certain libraries we used had buffers in them. We'd proxy the data from the customer to another back end service. By default the library would open the connection to the back end, and wait for the small buffer to fill before sending the data. The back end service would terminate a connection if the connection was open but idle for $x number of seconds. The user was on the threshold of that timeout. Probably half the time they PUT, they'd be slow enough to trigger that back-end timeout, resulting in them getting a 500. I believe eventually they put a small buffer on the ingestion path too before pushing along to the back end, but given those PUTs could get really large, we couldn't buffer the entire content before sending along.