7 ms·

Zero-downtime restarts in Go

141 points by neeee 14y ago

thibaut_barrere 14y ago
If you deployed Go in production, I'd love reading about that since I'm having a closer look at it now (coming from Ruby). Is there a more widely used option currently?
- jlgreco 14y ago
  Looking at the source, and having looked at how nginx does it, this looks like a pretty straightforward implementation with no real frills. I imagine anyone doing this in production with Go already has just rolled their own solution not much unlike this (doing this for some pet projects of mine has been on my TODO for a few weeks, never got around to it though).
- dsl 14y ago
  Unless you are a committer to Ruby now, or plan to invest in learning more powerful languages, Go probably isn't a good choice. I run a pure Go shop and we generally look for people with hardcore C (not C++ or C#) or systems/kernel backgrounds or at least one open source Go library under their belt. If you're just looking for another language to get away from the cesspit that is Ruby, try Python.
  - cmccabe 14y ago
    Hey man... hire people, not years of experience in X. Let's not pidgeonhole people because of what languages they've used in the past.
  - thibaut_barrere 14y ago
    Do you mean you believe that people using Ruby are either Ruby committers, or people with only pure Ruby experience? If this is true, I think you'll miss some great candidates, to be honest. Things I did before, for instance, include: - writing real-time 2D/3D rendering engines, mixing C++, TurboPascal, Asm - low-level assembly code, writing keyboard hardware interrupt handlers, TSR, video-cards tweaking etc - porting C-libraries from Solaris to Windows - writing a full GUI toolkit - writing 2D games - wrapping C and C++ libraries into .Net - writing a cluster of satellite images processing with PVM Etc (apart from using Java, .Net, Ruby, CoffeeScript etc). To make my point clear: the good rubyists I know have a lot of past experience, similar to this. Talking about "cesspit" is not giving good signals about how you consider other technical choices, either :-)
lbarrow 14y ago
We used this for a while at Braintree. It works great for Unicorn-style restarts, but it doesn't ensure that the server shuts down gracefully. So while you'll have uninterrupted connectivity, if you don't ensure that your server shuts down gracefully, you'll still drop the requests that the old process is servicing when the restart is triggered. We wrote a library called manners for managing the graceful shutdown of a single server: https://github.com/braintree/manners https://github.com/braintree/manners. When used in conjunction with manners, GoAgain provides zero-downtime restarts without any dropped requests. (Manners is still under development, so make sure you understand it before using it.) We had an interesting experience with Go. We ended up deciding the language was not a great fit for our needs right now, in part because the community is still pretty immature. (You can read about our experiences here: https://braintreepayments.com/braintrust/gotchas-irritants-and-warts-in-go-web-development https://braintreepayments.com/braintrust/gotchas-irritants-a...). Still -- as more and more libraries like this get released and stabilized, Go will become an increasingly interesting option for developers.
- mortdeus 14y ago
  Go is only ever going to be as useful as the community works towards making it useful. We are trying our best to implement all the boilerplate code and solve all of Go's short comings. Go is a very young programming language and unfortunately there are alot of cases where we have to implement pkg foo, bar, baz, because pkg quux depends on them. There are going to be many places where things break at this stage in Go's life, however that shouldnt discourage using Go for real world projects, because many of yesterday's bug end up being debugged and patched today. Go is a great investment for your company, and I hope you and your team reconsider using go for your projects after go 1.1 comes out. http://tip.golang.org/doc/go1.1 http://tip.golang.org/doc/go1.1 If you guys need help, feel free to join us in #go-nuts on freenode's irc server. IMHO, The best feature of go is, arguably, the helpful evangelist-like gopher community hanging out in IRC. We wouldnt mind helping you guys with your issues, but you have to help us help you.
  - codexon 14y ago
    I don't think this is a reasonable demand for startups or even medium sized businesses. Libraries cost time and money to produce, and if you are relying on free volunteers, development will either be slow or buggy and likely both. I remember when I tried to look into a MySQL library last month. The first one I tried wouldn't even compile. The 2nd one threw a connection error not when I issued the dial function, but when I tried a query on a fake IP. If you want Go to succeed, get Google to start funding it more or get used to having it grow organically like every other language.
justinhj 14y ago
The documentation is very light so I had a look through the source code. Is the purpose of this to remotely restart a go app (that presumably has been updated and deployed) via TCP whilst keeping the TCP connection alive, or is there more to it?
the_mitsuhiko 14y ago
What novel concept :P
kkowalczyk 14y ago
I run 3 websites written in Go (http://blog.kowalczyk.info/article/uvw2/Thoughts-on-Go-after-writing-3-websites.html http://blog.kowalczyk.info/article/uvw2/Thoughts-on-Go-after...) Those are clearly not Facebook scale servers but for majority of people: you don't need this or any other such solution. Here's what I do for deployment. I have a fabric script that: - copies the new executable to the server - kills currently running executable - starts the new executable The worst that can happen is that a few connections will get dropped and that's perfectly fine 99.9% of websites. You don't control internet so the users will get broken connections all the time for reasons you can't eliminate. They shrug and hit reload button. Complicating things to avoid few seconds of downtime was not worth it for me and rationally, probably not worth it for most.
- codexon 14y ago
  If you are running a paid app or making rapid changes your users will notice.
  - bickfordb 14y ago
    In these cases hopefully your business is real enough to be able to afford a load balancer.
davidw 14y ago
I wonder how long it will be before Go gets "enough" of Erlang to make it a suitable replacement for most people who need/want the Erlang goodies.
- bwooce 14y ago
  There appear to be some philosophical differences that mean this won't occur. Specifically the "let it fail" aspect of OTP, which begets supervisor hierarchies and the consequent lack of exception handling required. This has been brought up before and dismissed (not the unix way perhaps? I forget), and supervisors have been implemented in go by others from Erlang backgrounds. It's just not quite the same as being built in. The way goroutines are not linked to their parents or children is related to this. It's a conscious design decision, but I am still adjusting my style to it.
- mortdeus 14y ago
  http://godoc.org/?q=erlang http://godoc.org/?q=erlang
daakus 14y ago
I built something similar which additionally provides graceful termination of established connections as well as systemd socket activation to (optionally) provide lazy startup of servers: https://github.com/daaku/go.grace https://github.com/daaku/go.grace
voidlogic 14y ago
Cool, it seems like this package might work very well with this one: https://news.ycombinator.com/item?id=5443107 https://news.ycombinator.com/item?id=5443107

141 points by neeee 14y ago

If you deployed Go in production, I'd love reading about that since I'm having a closer look at it now (coming from Ruby). Is there a more widely used option currently?

jlgreco 14y ago
Looking at the source, and having looked at how nginx does it, this looks like a pretty straightforward implementation with no real frills. I imagine anyone doing this in production with Go already has just rolled their own solution not much unlike this (doing this for some pet projects of mine has been on my TODO for a few weeks, never got around to it though).
dsl 14y ago
Unless you are a committer to Ruby now, or plan to invest in learning more powerful languages, Go probably isn't a good choice. I run a pure Go shop and we generally look for people with hardcore C (not C++ or C#) or systems/kernel backgrounds or at least one open source Go library under their belt. If you're just looking for another language to get away from the cesspit that is Ruby, try Python.
- cmccabe 14y ago
  Hey man... hire people, not years of experience in X. Let's not pidgeonhole people because of what languages they've used in the past.
- thibaut_barrere 14y ago
  Do you mean you believe that people using Ruby are either Ruby committers, or people with only pure Ruby experience? If this is true, I think you'll miss some great candidates, to be honest. Things I did before, for instance, include: - writing real-time 2D/3D rendering engines, mixing C++, TurboPascal, Asm - low-level assembly code, writing keyboard hardware interrupt handlers, TSR, video-cards tweaking etc - porting C-libraries from Solaris to Windows - writing a full GUI toolkit - writing 2D games - wrapping C and C++ libraries into .Net - writing a cluster of satellite images processing with PVM Etc (apart from using Java, .Net, Ruby, CoffeeScript etc). To make my point clear: the good rubyists I know have a lot of past experience, similar to this. Talking about "cesspit" is not giving good signals about how you consider other technical choices, either :-)

lbarrow 14y ago

We used this for a while at Braintree. It works great for Unicorn-style restarts, but it doesn't ensure that the server shuts down gracefully. So while you'll have uninterrupted connectivity, if you don't ensure that your server shuts down gracefully, you'll still drop the requests that the old process is servicing when the restart is triggered. We wrote a library called manners for managing the graceful shutdown of a single server: https://github.com/braintree/manners https://github.com/braintree/manners. When used in conjunction with manners, GoAgain provides zero-downtime restarts without any dropped requests. (Manners is still under development, so make sure you understand it before using it.) We had an interesting experience with Go. We ended up deciding the language was not a great fit for our needs right now, in part because the community is still pretty immature. (You can read about our experiences here: https://braintreepayments.com/braintrust/gotchas-irritants-and-warts-in-go-web-development https://braintreepayments.com/braintrust/gotchas-irritants-a...). Still -- as more and more libraries like this get released and stabilized, Go will become an increasingly interesting option for developers.

mortdeus 14y ago
Go is only ever going to be as useful as the community works towards making it useful. We are trying our best to implement all the boilerplate code and solve all of Go's short comings. Go is a very young programming language and unfortunately there are alot of cases where we have to implement pkg foo, bar, baz, because pkg quux depends on them. There are going to be many places where things break at this stage in Go's life, however that shouldnt discourage using Go for real world projects, because many of yesterday's bug end up being debugged and patched today. Go is a great investment for your company, and I hope you and your team reconsider using go for your projects after go 1.1 comes out. http://tip.golang.org/doc/go1.1 http://tip.golang.org/doc/go1.1 If you guys need help, feel free to join us in #go-nuts on freenode's irc server. IMHO, The best feature of go is, arguably, the helpful evangelist-like gopher community hanging out in IRC. We wouldnt mind helping you guys with your issues, but you have to help us help you.
- codexon 14y ago
  I don't think this is a reasonable demand for startups or even medium sized businesses. Libraries cost time and money to produce, and if you are relying on free volunteers, development will either be slow or buggy and likely both. I remember when I tried to look into a MySQL library last month. The first one I tried wouldn't even compile. The 2nd one threw a connection error not when I issued the dial function, but when I tried a query on a fake IP. If you want Go to succeed, get Google to start funding it more or get used to having it grow organically like every other language.

justinhj 14y ago

The documentation is very light so I had a look through the source code. Is the purpose of this to remotely restart a go app (that presumably has been updated and deployed) via TCP whilst keeping the TCP connection alive, or is there more to it?

the_mitsuhiko 14y ago

What novel concept :P

kkowalczyk 14y ago

I run 3 websites written in Go (http://blog.kowalczyk.info/article/uvw2/Thoughts-on-Go-after-writing-3-websites.html http://blog.kowalczyk.info/article/uvw2/Thoughts-on-Go-after...) Those are clearly not Facebook scale servers but for majority of people: you don't need this or any other such solution. Here's what I do for deployment. I have a fabric script that: - copies the new executable to the server - kills currently running executable - starts the new executable The worst that can happen is that a few connections will get dropped and that's perfectly fine 99.9% of websites. You don't control internet so the users will get broken connections all the time for reasons you can't eliminate. They shrug and hit reload button. Complicating things to avoid few seconds of downtime was not worth it for me and rationally, probably not worth it for most.

codexon 14y ago
If you are running a paid app or making rapid changes your users will notice.
- bickfordb 14y ago
  In these cases hopefully your business is real enough to be able to afford a load balancer.

davidw 14y ago

I wonder how long it will be before Go gets "enough" of Erlang to make it a suitable replacement for most people who need/want the Erlang goodies.

bwooce 14y ago
There appear to be some philosophical differences that mean this won't occur. Specifically the "let it fail" aspect of OTP, which begets supervisor hierarchies and the consequent lack of exception handling required. This has been brought up before and dismissed (not the unix way perhaps? I forget), and supervisors have been implemented in go by others from Erlang backgrounds. It's just not quite the same as being built in. The way goroutines are not linked to their parents or children is related to this. It's a conscious design decision, but I am still adjusting my style to it.
mortdeus 14y ago
http://godoc.org/?q=erlang http://godoc.org/?q=erlang

daakus 14y ago

I built something similar which additionally provides graceful termination of established connections as well as systemd socket activation to (optionally) provide lazy startup of servers: https://github.com/daaku/go.grace https://github.com/daaku/go.grace

voidlogic 14y ago

Cool, it seems like this package might work very well with this one: https://news.ycombinator.com/item?id=5443107 https://news.ycombinator.com/item?id=5443107