17 ms·

So you want to expose Go on the Internet

• yumaikas 10y ago
  It's nice to see that Go's http and TLS libraries are getting even better. They're what attracted me to Go in the first place. Also, the coverage of the various timeouts for HTTP requests is mostly new information to me. Is that something that nginx and apache usually take care of?
  • moomin 10y ago
    I'm not a fan of Go, but its libraries seem top-notch.
    • KirinDave 10y ago
      They are, unless you want to know what went wrong. Then the fundamental problems with the library design sorta beat you over the head.
  • kevinburke 10y ago
    you can usually configure them, see for example http://nginx.org/en/docs/http/ngx_http_core_module.html#send_timeout http://nginx.org/en/docs/http/ngx_http_core_module.html#send... and http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout http://nginx.org/en/docs/http/ngx_http_proxy_module.html#pro... The difference is Nginx sets timeouts based on the time between successive byte reads or writes, so for example if you have a 30 second timeout and receive one byte every 29 seconds, you won't trigger Nginx's timeouts: https://kev.inburke.com/slides/reliable-http/#connect-timeout-requests https://kev.inburke.com/slides/reliable-http/#connect-timeou... Go generally prefers wall-clock timeouts for reading or writing the entire response, that is, if you don't get the whole thing in 30 seconds, return an error, regardless of when you received each individual byte. Although you can configure Nginx-style timeouts if you want.
  • ben_jones 10y ago
    My personal experience with the Go stdlib is that it practices "defensive programming" pretty well. For example HttpClient defaults. It's what you would expect from a language with go's objectives, but it's still something I appreciate (even if it forces me to do things right when I don't want to!).
    • zzzcpan 10y ago
      HttpClient has no usable defaults. In fact, you can't even make a usable file downloader because of the way it handles timeouts, you have to write your own.
• jest7325 10y ago
  I wonder if it would make sense to front-end that with nginx. It has nice http2 and the latest SSL/TLS implementation. Just curious what other think about this approach.
  • kevinburke 10y ago
    literally the first sentence of the post: > Back when crypto/tls was slow and net/http young, the general wisdom was to always put Go servers behind a reverse proxy like NGINX. That’s not necessary anymore!
  • toufka 10y ago
    That's currently what is done. However the idea that I could get my entire stack, from HTTP handler, to router, to logic, to database query, back to response into a single 10mb binary is pretty enticing. If I could get there, I'd have full control over every aspect of an API request from packet to server query, back to payload, in a single programming language, in a single conceptual framework. There is a lot to like about that. I'm not sure it's needed - nginx works so well, and perfect settings are just a single config file away. But if I could get to a truly single-binary deployment, I'd be pretty happy too.
• ruslan_talpa 10y ago
  I still live under the assumption that there are oh so many ways to ____ with a web server beyond opening a TCP connection and keeping it open, for all of them there is code in nginx to defend agains. But hearing this from CloudFlare is worth a second look. Can someone with first hand knowledge of the nginx (front line) codebase comment if the things described in this article is all it takes to have a mostly resilient http service?
  • shanemhansen 10y ago
    I can't comment on the nginx codebase, but I've been running golang production-facing golang servers for a long time and I feel safe in saying I have mostly resilient http services. I've worked with more than one company handling over 100k requests/s on the public internet with Go. Go's networking model combined with the work that's gone into fuzzing the stdlib combined with the benefit of hindsight when it comes to data structures and security combined with lots of love from google web people has resulted in an extremely mature web stack.
    • ruslan_talpa 10y ago
      I see what you are saying but that only confirms that your services are fast and stable as long as ppl are using them for their purpose. What i am interested in is if they are resilient when someone is deliberately attacking them and not with trivial scripts.
      • shanemhansen 10y ago
        I think I understand what you are saying too. I've personally worked on 2 alexa top 100 sites for the US that are using go on the public internet. They see a fair amount of malicious traffic. I actually find Go and net/http to be a pretty solid base for defusing layer 7 attacks.
  • zzzcpan 10y ago
    I wouldn't call nginx to be mostly resilient, but at least nginx doesn't leak resources by default and has measures against some known attacks, like the one that prevents DoS from range requests. It also has some ways to make it somewhat more resilient with limit_req and limit_conn modules and handles timeouts for streaming and external requests properly. The only way for Go to get on that level is to write another networking library, the one in the standard library is pretty much broken by design (they've been working on it for many years and it's still doesn't handle even timeouts properly).
• sofaofthedamned 10y ago
  I usually use haproxy in front of docker containers, this is interesting stuff. I wonder what he feels about using Caddy instead of bare net/http ?
  • grey-area 10y ago
    Caddy server pretty much is bare net/http (it uses the stdlib), so it would be doing the same thing.
• brianpgordon 10y ago
  Has someone wrapped all of this in a library? Or are there plans to update the defaults so that they're more hardened? It seems like it shouldn't be this hard.
  • grey-area 10y ago
    There have been and will be improvements. Go 1.8 improves timeouts. They are constrained to some extent by the Go 1 compatibility promise, for example they don't want to change default behaviour on timeouts probably because of that. Hopefully if they have a Go 2 at some point they'll use that opportunity to clean up some APIs and fix a few things like this.
• smegel 10y ago
  Sigh. If you're configuring "Curve Preferences" you're doing it wrong. Crypto either works out of the box or find another tool.
  • tptacek 10y ago
    First, the person writing this article (Filippo Valsorda) has expertise. Second, the point isn't to select crypto that "works", but rather to select crypto that is efficiently supported by Golang.
    • smegel 10y ago
      > However, you should still set PreferServerCipherSuites to ensure safer and faster cipher suites are preferred, and CurvePreferences to avoid unoptimized curves Sounds more like he's giving advice to others on the finer points of elliptic curve cryptography. Programmers should not need to know this stuff.
      • awj 10y ago
        ...no, it's literally saying "here are the settings you need to make sure you only use efficiently implemented algorithms"
        • smegel 10y ago
          Because making safe, efficient settings the hard-to-tamper-with default would be waaay too sensible?
          • awj 10y ago
            More sensible than trying to have a conversation by stating your points as condescending "questions". In case you actually care to discuss it, there's definitely a trade-off between variety of cipher support (which some people want) and efficiency of cipher support (which some people absolutely need). Prioritizing one over the other is not a clear or simple decision.
• nkozyra 10y ago
  NGINX has so many nice reverse proxy tools out of the box that it's still very appealing to just plop it in front of any service (much less a Go service). Performance & failsafe is a big part of the appeal, but so is local caching, traffic splitting (for A/B testing or regional versions), etc. It's hard to ignore that when choosing to expose your server directly or put it behind NGINX. Unless you use none of these things, you'll end up reinventing a bunch of wheels.
  • weberc2 10y ago
    Given that Go's HTTP interfaces are very composable, and assuming there are libraries to do caching and traffic splitting, you wouldn't be reinventing wheels. At that point, it seems that the question is whether you prefer to manage NGINX config files or write your configuration in Go.
    • nkozyra 10y ago
      OK, so 're-implement' the wheel. Using a variety of unrelated, dubiously-updated libraries. I'm still not sure what the advantage of that is over using perhaps the most reliable, certainly most-used web server as a proxy in front of your app. I'm open to convincing, though.
      • weberc2 10y ago
        How is filling out struct fields more complex than filling out config files? The principle advantage is system simplicity--everything deploys in a single file, no network topology to troubleshoot, fewer moving parts, no new highly-configurable tool to master. If the quality of those libraries is as poor as you suppose, then take NGINX by all means. I don't see any reason to make those assumptions, however. I don't mean to overstate the advantages--I think both solutions are fine; neither will make or break your operation.
        • nkozyra 10y ago
          It's more than filling out a few structs, though. Traffic splitting or caching via NGINX can literally be done in a handful of lines. No go gets, no middleware and it's well tested, mature and backed by software that powers a majority of the web. I use go net/http every day, in production. I trust it, but NGINX provides so much more battle tested functionality out of the box.
• avitzurel 10y ago
  Nginx is your best friend. I always have an Nginx proxy in front of services. Whether it's Go, Ruby or Node (or Docker)
  • akerl_ 10y ago
    Can you clarify why?
    • logn 10y ago
      Not OP but if you want an app in production then you want to be able to configure the http parts and script with command line tools. And if you start doing all that then you're basically reinventing nginx. Even comparing Java, nginx still has nicer http features, such as reloading ssl certs and config without dropping connections.
    • avitzurel 10y ago
      @logn clarified pretty well. Anything you do in your service other than your own business logic is reinventing the wheel (I mean in the HTTP level), and not doing it so well as someone before you (nginx/HaProxy etc...). This is a generalization of course, but my strategy of putting Nginx in front of everything didn't fail me so far.
      • Can_Not 10y ago
        You say strategy, I think you meant well established, battle tested and proven, industry standard best practice.
  • edoceo 10y ago
    And Apache/PHP and Tomcat and all the things. Nginx as the Frontline has been my policy since forever. For resilience, HA, proxy routing, static files, masking backend errors, caching. Does HTTP+S and 2 and Websocket so nicely. Nginx all the things.
    • btmiller 10y ago
• stanleydrew 10y ago
  One reason we didn't do this with our messaging service at Charge was that we didn't want code that we wrote to have access to our private TLS keys in production. Not everyone needs that level of protection, but it's helpful to avoid giving your software engineers footguns that can inadvertently lead to decryption of all your production data streams.
  • dispose13432 10y ago
    >we didn't want code that we wrote to have access to our private TLS keys in production. Correct me if I misunderstood you, but you don't want _engineers_ who write your code to have access to private TLS keys which are _used_ in production
    • closeparen 10y ago
      A separate process is overkill for protection from engineers, just have the private keys read from disk, and only have them on production disks. If you compromise a process, you can potentially exfiltrate its memory. You'd need to also compromise the operating system to exfiltrate memory from other processes. So, keys being in nginx means you can only get the keys by breaking nginx (or the OS), not by breaking the in-house application.
      • paulddraper 10y ago
        > A separate process is overkill for protection from engineers From engineers sure. But a separate process helps for other threats, like heartbleed.
      • amorphid 10y ago
        Or don't have the keys on the server at all. Anyone who gets root access can walk right up to the key file and yoink it. Obviously keys have to be stored somewhere. But it doesn't have to be on every server's disk. Also, try to avoid passing in keys as command line arguments. If you can, avoid using environment variables, too. You can pass thet data in using standard in, so the data is never exposed. Example of leaky environment variables: https://gist.github.com/amorphid/db037f03246962959b6a034b2ca3ef1b https://gist.github.com/amorphid/db037f03246962959b6a034b2ca...
• derefr 10y ago
  Does anyone here know whether such a similar article exists for Erlang (or specifically, the Erlang ecosystem's Cowboy HTTPD library)? Even though Cowboy (and frameworks built atop it, like Phoenix) is known to perform well under load (including DDoS-like load), I've always been wary about exposing directly it to the Internet. I know NGINX was explicitly hardened against many classes of web server attack; I haven't ever seen the same claimed about Cowboy. It'd be reassuring just to know of anyone with a large, public-facing web service, who has deployed Erlang in a directly-exposed HTTP server role, weathered attacks, and come out fine. (Heroku, maybe?) But I haven't heard much on that front, either.
• tedunangst 10y ago
  But no method to just specify max connections before old ones start getting closed? Did I miss that? With nginx, I don't mess with timeouts. I just set max connections appropriately and that's it. Don't care about slow connections when descriptors aren't in short supply.
  • dsp1234 10y ago
    I'm not aware that nginx has an option to "specify max connections before old ones start getting closed". The default behavior is that when max connections are hit, it doesn't allow new connections[0]. Then with the default timeouts of 60 seconds for client_body_timeout and client_header_timeout, a connection can be held for at least 2 minutes[1][2]. Note that the body timeout is "The timeout is set only for a period between two successive read operations, not for the transmission of the whole request body.", so it's possible to arbitrarily extend the amount of time a single connection is open. Since these connections are long lived, and normal connections are generally short, the total number of connections gets dominated by the slow ones. If no other mitigations are put into place, then this can cause a server to hit ulimits/max_conns and keep legitimate requests blocked. This is known as the "Slowloris" attack[3], and is mentioned in the nginx DDoS mitigation blog post[4]. [0] - http://nginx.org/en/docs/http/ngx_http_upstream_module.html#server http://nginx.org/en/docs/http/ngx_http_upstream_module.html#... [1] - http://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_timeout http://nginx.org/en/docs/http/ngx_http_core_module.html#clie... [2] - http://nginx.org/en/docs/http/ngx_http_core_module.html#client_header_timeout http://nginx.org/en/docs/http/ngx_http_core_module.html#clie... [3] - https://en.wikipedia.org/wiki/Slowloris_(computer_security) https://en.wikipedia.org/wiki/Slowloris_(computer_security) [4] - https://www.nginx.com/blog/mitigating-ddos-attacks-with-nginx-and-nginx-plus/ https://www.nginx.com/blog/mitigating-ddos-attacks-with-ngin...
    • tedunangst 10y ago
      Hmmm, thanks. It worked for me, but I should look closer.
• Xeoncross 10y ago
  Peter Lambert wrote his own guide on getting a perfect SSL Labs score by tweaking the go HTTP server config (https://blog.bracebin.com/achieving-perfect-ssl-labs-score-with-go https://blog.bracebin.com/achieving-perfect-ssl-labs-score-w...). Both of these articles are a good quick read for Gophers.
• grobbles 10y ago
  It's years old, but I listened to the advice of- https://dennisforbes.ca/index.php/2013/08/07/ten-reasons-you-should-still-use-nginx/ https://dennisforbes.ca/index.php/2013/08/07/ten-reasons-you... -and in re-analysis it all holds completely true. Nginx gives my deploy flexibility, at essentially negligible cost. And no Go development should include a bunch of boilerplate code to do banal stuff like serving static content.
  • insertnickname 10y ago
    Static file server in one line of Go: http.ListenAndServe(":8080", http.FileServer(http.Dir("/usr/share/doc")) https://godoc.org/net/http#FileServer https://godoc.org/net/http#FileServer
• gtrubetskoy 10y ago
  There's still one thing missing - graceful restart: https://grisha.org/blog/2014/06/03/graceful-restart-in-golang/ https://grisha.org/blog/2014/06/03/graceful-restart-in-golan... BTW - to people new to Go this article may make it look like serving HTTP is complicated, but it's actually remarkably easy. And if you consider that it is actually _possible_ to have a complete server with just the standard lib (and with TLS and HTTP/2 to boot) running as a single process - compared to Python or Ruby, where because of the GIL you _must_ place an apache/nginx/haproxy in front of it and also run a bunch of unicorns or something similar on different ports (at which point you need something like Chef/Puppet to manage the config because it gets very complicated very fast) - this is actually pretty amazing.
  • grey-area 10y ago
    You mean missing from the article or from Go stdlib? This was added in Go 1.8 (in beta now), so that link is a bit out of date I think. https://github.com/golang/go/issues/4674 https://github.com/golang/go/issues/4674 Agree that serving http/https with Go is really pretty straightforward - you can do it without all the tweaks in this article, and most people have been. It works remarkably well by default and scales well too without much effort.
    • gtrubetskoy 10y ago
      I meant missing from both, but I didn't realize that it's making its way into the standard lib - this is very cool, thanks for the link!
  • jjawssd 10y ago
    You argue that people want to put Go in front whereas you would not with Python/Ruby because of the GIL. This is a misunderstanding because people use nginx in front for failover and load balancing, and not just because Python/Ruby are comparatively slow. So your argument is fundamentally flawed.
    • krakensden 10y ago
      There's multiple layers of "in front". Python and ruby shops often have dedicated L7 load balancers, and then run nginx on individual hosts to mux to multiple dynamic language processes/threads.
• merb 10y ago
  still misses how to bind port 80 or 443 via non root user: http://serverfault.com/questions/112795/how-to-run-a-server-on-port-80-as-a-normal-user-on-linux http://serverfault.com/questions/112795/how-to-run-a-server-... Has many good answers.
  • voidlogic 10y ago
    You do not need to run as root to bind an application to a low port, instead use setcap (it works for everything not just Go): https://stackoverflow.com/questions/14537045/how-i-should-ru.. https://stackoverflow.com/questions/14537045/how-i-should-ru....
• module0000 10y ago
  Put it behind HAProxy!(or nginx if you just starting *nixing in the last 5 years and don't know what haproxy is) Both are battle tested(HAProxy more so), and like another poster said...if you don't use something like them, you're going to re-invent the wheel in several areas.
• deleted 10y ago
  [deleted]
• nodesocket 10y ago
  I'd still recommend running NGINX in front of Go or Node backends. NGINX gives you the flexibility to add things like gzip, caching, static asset expires headers, load balancing, health checks, etc. (shamless plug) In terms of the TLS, I wrote a short blog post (http://blog.commando.io/the-perfect-nginx-ssl-configuration/ http://blog.commando.io/the-perfect-nginx-ssl-configuration/) on a setting up NGINX to get an A+ rating on Qualys SSL Labs. It is really only a few lines/directives.
  • Xeoncross 10y ago
    Go does native gzip also, and you can load assets into memory easily for instant streaming if you don't want to use the linux file buffer (which should already have a copy in memory). If it helps anyone, I published the recommendations here plus some other resources into a simple Go package for instant A+ server report: https://github.com/Xeoncross/secureserver https://github.com/Xeoncross/secureserver
  • chrismarlow9 10y ago
    I agree. I enjoy go because the apps I build are simple, and all the "special features" like caching, gzip, expire headers, health checks and reverse proxying come with nginx. I picked go as my language of choice because I want to get to the root of the problem and write code for that, and do that 1 thing very well. Its not that I don't think Go can do it, it's just that I don't want to do it in Go. It feels like an abuse of the language paradigm to do everything in go.
• daurnimator 10y ago
  I'm happy to note that my (newly released) library lua-http covers almost all of these concerns in it's default configuration, with even cleaner semantics around the read/write timeouts. The only thing missing is an equivalent of the "Idle" timeout mentioned in the OP: I'm curious how you think it should behave in the HTTP 1.1 pipelining case, I guess only counting the time that the connection is totally idle?
• epynonymous 10y ago
  i use nginx in front of go for production, this article is interesting as certainly it would reduce some level of administrative obligation if i could remove nginx, case in point, i actually just distribute go binaries to my production servers so i don't even need the go compiler, this would simplify deployment somewhat. but then again, i'm not really changing my nginx configuration that often. some questions that come to mind, i also leverage nginx for static file caching, i've seen some sample code for fileserver in net/http, but what kind of algorithm does fileserver use for caching, lru? can you configure the size of the cache? and in terms of scale, i haven't reached this point yet in my project, but from the _olden_ sinatra days, i'd spin up multiple processes and proxy through nginx. in terms of a single (machine) server, could a go process essentially be limited to 1 per server? i'm assuming the go binary could leverage multiple cores automatically so i wouldn't need to do like ruby or python? what are your experiences with go backend services? i run a restful api server that connects to a database and redis, so far, performance seems good enough where i only need 1 go process per machine.