16 ms·

The Way of the Gopher: Making the Switch from Node.js to Golang

260 points by 0xmohit 11y ago

kjksf 11y ago
For other stories of X => Go rewrites, I started keeping a list at https://quicknotes.io/n/vzb7-rewritten-tales-of-rewriting-s https://quicknotes.io/n/vzb7-rewritten-tales-of-rewriting-s
- feylikurds 11y ago
  That list can easily be made longer. Off of the top of my head I know that there are many more projects which have switch languages to Go. What they say most of the time is (from the article): Since our transition to Golang we haven’t looked back. ... we’ve begun the process of modularizing our code base and spinning up microservices to handle specific roles in our system.
  - kkowalczyk 11y ago
    I agree, it's a work in progress. If you have more links, I'll be happy to add them. I have a much longer list of companies that use Go (https://quicknotes.io/n/1XB0-companies-using-go https://quicknotes.io/n/1XB0-companies-using-go) but it's harder to find descriptions of rewrites than find out who's using Go.
- iMiiTH 11y ago
  You can add The New York Times to that list, although they rewrote from PHP, Python, Groovy, Java and C++ so no idea how it'll fit in that list. https://www.youtube.com/watch?v=bAQ9ShmXYLY https://www.youtube.com/watch?v=bAQ9ShmXYLY
justicezyx 11y ago
Another evidence that the language that is narrowly focused and force "more" correct practices tends to work better in practice. Or, coders are getting worse at master the more powerful language...
- XorNot 11y ago
  99% of the time what is called "mastering" is actually a whole bunch of stuff you shouldn't do if you want a maintainable codebase. Tricks are just that: tricks. No one sets up a business based on how you pull a rabbit out of hat, nor should they.
  - justicezyx 11y ago
    No, mastering a complex and powerful language means use the minimal feature for the majority of your needs, and knows how and when to use the "dangerous" part for a small fraction of your needs. I do not know much of JS, but people write C++ with loads of patterns are prominent examples of people who do not master the language. That's not trick. That's real engineering with (often) bitter experience.
kelkes 11y ago
How many instances of node octo where running to handle the load?
- biot 11y ago
  > And those 4 load-balanced instances we were running Octo > on initially? We’re now doing it with 2.
DannyBee 11y ago
How does one live without is-positive-integer and isArray?
- blitzprog 11y ago
  The real question is: How does one live without is-thirteen? https://github.com/jezen/is-thirteen https://github.com/jezen/is-thirteen
  - ysolo 11y ago
    Best post here.
- deleted 11y ago
  [deleted]
- tptacek 11y ago
  Wait what why https://www.npmjs.com/package/is-positive-integer https://www.npmjs.com/package/is-positive-integer
deleted 11y ago
[deleted]
bitwize 11y ago
I lol'd at the bit about blocking Node's event loop. If Unix weren't so committed to doing I/O the stupid way around, maybe Node wouldn't have to provide fragile, ersatz asynchronicity the way that it does?
- fooyc 11y ago
  Care to explain?
  - bitwize 11y ago
    Two iron-clad rules of data processing: * All data has a type * All I/O is asynchronous and interrupt (event) driven At its inception Unix doubled down on stupid, first by encouraging data to be stored and migrated in the form of untyped text streams, which require ad-hoc parsing and unparsing logic at the endpoints; and secondly by failing to provide asynchronous I/O primitives to user space. POSIX later provided an AIO standard, but it was cumbersome to use, bolted on rather than integrated into the OS, and virtually nobody used it. What you want to be able to do is submit I/O requests to the kernel, and have the kernel notify you when they are completed while you wait or do other processing; you could even put the process to sleep until pending I/O is completed and have it take up no CPU during this time. Instead what people do is burn CPU cycles in select/poll loops, which are the I/O equivalent of asking "Are we there yet? Are we there yet? Are we there yet?" over and over. You can fake asynchronicity by doing a bit of processing before asking "are we there yet?" again, but you have to write your processing code in such a manner as to be done piecewise in a loop, and the lower latency you want the more frequently you have to poll (and the more CPU you have to burn polling). This is how Node does "asynchronous" I/O; the VM stops every few instructions to poll for ready FDs, dispatches to callbacks as necessary, and performs parts of pending large I/O operations which can be done non-blocking. (That's the other sucky bit: you can't just call read(2) and write(2) for large chunks of data on an fd that's been opened O_NONBLOCK and expect it all to work; you have to keep reading or writing in a loop when the fd is ready, subtracting the number of bytes successfully written until the entire buffer is processed.) But if something -- a call into a C library for instance -- stops the VM from doing the poll and I/O bits of its main loop, all I/O simply... stops. And your throughput goes into the toilet. Whereas if the runtime had been based on an OS that natively supports interrupt-driven AIO -- like VMS, Windows, or AmigaOS -- it would be extremely difficult to stall the I/O pipeline completely this way. You might be able to stall further I/O calls with a really long operation, but any pending calls already initiated would complete, and the runtime would be properly notified of their completion. And all this stems from the fact that Unix was the Node.js of its day -- an environment designed to make things easy for casual programmers, not to do things correctly.
BogusIKnow 11y ago
When you switch from Node to Go, two languages that are totally different in style, library support, performance, compile/web turnaround, client/server reusability, I always wonder why you chose Node in this case in the first place.
- pron 11y ago
  For the exact same reason they chose Go now, and the reason they'll switch to something else (much better!) in two-three years: because that's the current fashion. Nothing says "I'm keeping up with my professional development" than relearning IO libraries and build tools, and rewriting in-house code, over and over again, every time in the new langue-du-jour.
  - zappo2938 11y ago
    Meanwhile Facebook is enjoying working with a large PHP code base. When asked they say they can write code that takes time to compile that runs fast in C++ or write code fast in PHP that runs slower. Writing features faster gives them they edge. They have an engineering team that does nothing but optimize PHP. For them an optimization is an acre of servers saved. They if something needs to run blazing fast can write code in assembly language. What I'm told about Node is the JavaScript is running in a single thread but all I/O is being passed to native code which is written in C++ and running on multiple threads. It does use all the CPUs.
    - deleted 11y ago
      [deleted]
    - deleted 11y ago
      [deleted]
cryptica 11y ago
I have read so many articles which incentivize switching from Node.js to Go and every single one of them (this one included) blabber on about the Node.js event loop becoming congested - This is completely misguided - It only shows that the engineer didn't understand the problem. A single Node.js instance runs its business logic in a single process. A single Go instance can run its business logic in multiple processes. Of course if you're comparing a single instance of Go running on 4 CPU cores, it's going to be faster than a single instance of Node.js running on 1 CPU core. But the thing the author fails to mention is that there are a lot of Node.js modules which help you easily scale across multiple CPU cores. E.g. PM2 https://www.npmjs.com/package/pm2 https://www.npmjs.com/package/pm2 Had the author of the article actually understood the real cause of the problem, he would have saved himself and his company a lot of time. Nodejs is still outpacing Golang according to Google Trends and yet I have never seen a single article about switching from X language to Node.js. My guess is that the Go community probably wouldn't exist if it wasn't for this constant form of aggressive propaganda/marketing. Node.js on the other hand doesn't need any marketing/propaganda; it just sells itself. If you have a scalability problem with Node.js (or any language for that matter) and you can't find a simple solution that doesn't require rewriting your whole app, you're probably not qualified to manage this project at scale - Because scaling across multiple CPU cores is kids' play compared to scaling across multiple machines. Heck, my grandma could scale this thing across multiple CPU cores. Personally I think Node.js is great because it encourages you to think about these problems sooner rather than later.
- BogusIKnow 11y ago
  This is always the problem when you're hype driven. I'm sure the decision to use Node was also hype driven. And they will rewrite again with the new hype, right until they are bankrupt because they didn't invest enough in new features.
  - jondubois 11y ago
    Totally agree! I would say the same thing if someone had a whole app written on Python and were having performance issues (I definitely wouldn't advise them to switch to Node.js or Golang) - If your ecosystem offers the tools to solve the problem, it's better to use these.
diegorbaquero 11y ago
The EC2 medium instance has 2 cores. Were you running 2 node processes in cluster mode? The only thing I asked myself when reading this. I'm keeping my vote for node.
- dminor 11y ago
  m3.medium only has 1 core.
firasd 11y ago
The crux of the difference seems to be, in the Node.js service: "when any request timeouts happened, the event and its associated callback was put on an already overloaded message queue. While the timeout event might occur at 1 second, the callback wasn’t getting processed until all other messages currently on the queue, and their corresponding callback code, were finished executing (potentially seconds later)." And in Go, "the service has a worker pool and a delegator which passes off incoming jobs to idle workers. Each worker runs on it’s own goroutine, and returns to the pool once the job is done."
- awjr 11y ago
  Node.js has cluster to solve this issue https://nodejs.org/api/cluster.html#cluster_cluster https://nodejs.org/api/cluster.html#cluster_cluster
  - Sanddancer 11y ago
    Which brings us back to 2001 and Apache's prefork model. Threads are pretty battle tested at this point, and allow for different reasoning about the concurrency of a problem. The problem with node is that it pretty much only allows for one type of scaling if you start running into performance problems. Go gives you a much bigger toolbox to work with if, much like the post's author discovered, your problem set does not match up with the answers node provides.
    - qaq 11y ago
      All is well until you have to deal with SEO for SPAs :) and then there is little choice other then node. https://github.com/audreyt/node-webworker-threads https://github.com/audreyt/node-webworker-threads spawn real threads with web workers compatible API.
- lenkite 11y ago
  This is the first solution that a boring 'enterprise' Java programmer would have used and got on with other work.
partycoder 11y ago
Blocking the event loop in node can cause severe issues. If you are juggling with 5 sticks but one stays in your hand for 1 second, your juggling will certainly fail. Instead of flamegraphs I use nodegrind, which I can use with KCachegrind/QCachegrind.
Ericson2314 11y ago
Not gonna lie, one shitty language for another, lol.
- erlich 11y ago
  What do you use?
  - chridal 11y ago
    Must be Erlang.
  - Ericson2314 11y ago
    [Cause I can't comment on my own post directly] I should give some some credit as Go is strictly better than Javascript.
  - Ericson2314 11y ago
    But to answer the question, languages with (at least) type parameters and no data races (without using escape hatches). (Hint: that includes, but is not limited to, Rust).
erlich 11y ago
Long live Node.js! As everyone has mentioned this seems to be a simple case where the rewrite is a completely different architecture, which could have been done in Node.js. I'm interested in why this continues to happen. Seems like Node.js is a victim of its own simplicity. The baked-in non-blocking concepts of Node.js makes everyone feel like single-threaded is the only way to go, and think that Node.js is inherently flawed when it comes to parallelization. Because Go doesn't give you a simple and relatively scalable solution without even thinking about it like Node.js does, instead you have to choose what to parallelize and think about the problem more. With PM2 scaling Node.js over CPUs becomes incredibly trivial. Also, I think programmers naturally like rewrites/starting from scratch, and a new programming language provides an opportunity to rewrite everything (which the OP from this article and every other "switcher" article seem to do).
- douche 11y ago
  > Seems like Node.js is a victim of its own simplicity NodeJS is a victim of basing itself on a terrible scripting language that was thrown together to wire up basic event handlers to HTML buttons.
- zongitsrinzler 11y ago
  Recently I rewrote a medium sized Websocket app from Node to Go. Hoping for better performance and having heard a lot of good about Go. Since the application relies heavily on sharing resources between sockets I found out the hard way that concurrent programming in Go is not much easier than in Java. Sure, it has a better syntax and channels but compared to Node where you never have to think about things like these the complexity is way higher. Long live Node.js!
  - 59nadir 11y ago
    > Since the application relies heavily on sharing resources between sockets... Which resources?
    - zongitsrinzler 11y ago
illumen 11y ago
I think the author made a pretty good decision to choose a solution which was proven to meet the requirements by another author. However Amazon does provide a tool to batch download multiple things from S3 though. Let's pretend for a moment those tools don't exist already... Let's also pretend Amazon doesn't suggest another solution for when there are lots of GET requests [0]. Let's pretend that using something like nginx which is a very well tuned web proxy might be way better because of SPDY, SSL connection pooling and such [1]. Let's also pretend that node.js can not use multiple processes with cluster cluster [2]. Why wouldn't one use python for this job? Since the company already does a lot of python, and this job is actually easy to do in python (I've done something similar). 400 processes with python3.5 is way less than the 4gig on one medium instance (less than 2.5MB each). Just farm the work out to a ProcessPoolExecutor, and have a timeout on the S3 get requests. That would let you have enough resources to match the spike of 20000 requests per minute (334 per second). A lot of an S3 GET request could be the SSL, AWS auth and such. All quite CPU intensive. So using an async framework that doesn't do SSL+AWS auth async is obviously not going to work well once the requests go up. There's even an example in the concurrent.futures of downloading urls [3]. Made with the beautiful python3.5 import concurrent.futures import requests from awsauth import S3Auth ACCESS_KEY = 'ACCESSKEYXXXXXXXXXXXX' SECRET_KEY = 'AWSSECRETKEYXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' URLS = ['http://www.foxnews.com/', 'http://www.cnn.com/', 'http://europe.wsj.com/', 'http://www.bbc.co.uk/', 'http://some-made-up-domain.com/'] def load_url(url, timeout): return requests.get(url, timeout=timeout).data # return requests.get(url, timeout=timeout, auth=S3Auth(ACCESS_KEY, SECRET_KEY)).data with concurrent.futures.ProcessPoolExecutor(max_workers=400) as executor: # Start the load operations and mark each future with its URL future_to_url = {executor.submit(load_url, url, 60): url for url in URLS} for future in concurrent.futures.as_completed(future_to_url): url = future_to_url[future] try: data = future.result() except Exception as exc: print('%r generated an exception: %s' % (url, exc)) else: print('%r page is %d bytes' % (url, len(data))) [0] http://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html#get-workload-considerations http://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-... [1] https://coderwall.com/p/rlguog/nginx-as-proxy-for-amazon-s3-public-private-files https://coderwall.com/p/rlguog/nginx-as-proxy-for-amazon-s3-... [2] https://nodejs.org/api/cluster.html#cluster_cluster https://nodejs.org/api/cluster.html#cluster_cluster [3] https://docs.python.org/dev/library/concurrent.futures.html https://docs.python.org/dev/library/concurrent.futures.html
spullara 11y ago
Switching from one essentially single threaded language to another wasn't likely to solve your issues. Instead of Go you could have chosen any JVM language or even C++ and had similar success.
- sagichmal 11y ago
  Go is not a single threaded language.
  - spullara 11y ago
    Obviously. I was just pointing out that switching to Go wasn't the interesting bit. Maybe everyone misread my comment.
    - _ak 11y ago
      If you didn't want your comment to be misread, you should have been more specific after the "another".
nissimk 11y ago
Originally, the web server was intended to be a very simple program with most of the complications in the browser. Most web app backbends are similar to this where all they do is query a database, cache or other web servers. This type of thing is perfect for node. If you are doing more complex stuff in your backend then go may be more appropriate.
romanovcode 11y ago
2010 - Making switch from PHP to Ruby 2013 - Making switch from Ruby to Node 2016 - Making switch from Node to Go 2019 - Making switch from Go to {hype}
- tptacek 11y ago
  "So what I'm saying is, we should all still be using PHP."
  - mwcampbell 11y ago
    Or Perl, or C. After all, even PHP was once a new hipster language. Of course, it's good to be skeptical of the current hyped flavor of the moment. But some languages really are better than others for specific tasks, and new languages do sometimes improve on their predecessors.
    - collyw 11y ago
      C is still the choice language for many things. Perl is still cool to me, though seems to have been replaced by Ruby for web development.
  - bliti 11y ago
    What is star fighter written in? Curious, no bad intentions.
    - dikaiosune 11y ago
      IIRC, it's got some Go infrastructure and the trading chapters have bots written in Ruby.
    - tptacek 11y ago
      Go, Ruby, and React.
  - brightball 11y ago
    Ironically enough, given the recent improvements, performance and productivity...that's not really out of the question...
forgettableuser 11y ago
I'm disappointed by the overall tone here. It's as if this were some kind of failure that everybody is piling up on. - She made three passes to fix the existing code base. - After that didn't work, with the CTO's involvement, this Go solution was devised. - It took only 2 weeks to ramp up with Go and build a replacement that solved the problem. - The number of instances needed dropped from 4 to 2 This sounds like a resounding success. So congratulations Alexandra and Digg. And thank you for sharing. The article was well written and I learned some things about how the Node.js event system works under the hood.
- deleted 11y ago
  [deleted]
- rgbrgb 11y ago
  I think the disappointment here is that the OP seemed not to understand the root cause of the problem and implemented a solution in a different language (load balanced worker pool) that could have also worked in the original language without a total rewrite. Then the new language is trotted out as the savior. It sounds like the file fetch workers and request handlers were running on the same process, so the longer-running workers ended up blocking requests. The way it's discussed, it seems like pure luck that they stumbled on a solution that solved this (running the workers on a separate queue/process). I totally agree that it's impressive that it only took 2 weeks to build the Go version, it just seems like it would have taken 2 days to try the worker pool implementation with Node.js.
  - zaphar 11y ago
    But using a process pool is far more heavyweight than threads. There is a cost there that you pay in hardware. Using Go allowed them to do this with less hardware cost than Node. That's a win that continuing to use Node would not have been able to provide.
    - ilaksh 11y ago
      Node does actually have threads with modules like webworker-threads.
takno 11y ago
On the face of it this sounds like a problem where the optimal solution would have been to throw more hardware at it while you looked at the underlying choice of datastore.
0xmohit 11y ago
Some comments prompt adding another link here: <https://medium.com/@tjholowaychuk/farewell-node-js-4ba9e7f3e52b> https://medium.com/@tjholowaychuk/farewell-node-js-4ba9e7f3e...
- sah2ed 11y ago
  https://medium.com/@tjholowaychuk/farewell-node-js-4ba9e7f3e52b https://medium.com/@tjholowaychuk/farewell-node-js-4ba9e7f3e... FTFY
  - 0xmohit 11y ago
    Thanks. Didn't realize that HN screws up links within angle brackets.
planetjones 11y ago
> for each request, Octo typically fetches somewhere between 10–100 keys from S3 I'm really curious as to why this is neccessary - what are the 10-100 keys you are fetching per request ? Just reading the high level article it sounds like there is some issue with what this service is doing, irrespective of the language it's doing it in.
- meonkeys 11y ago
  Another idea: stress-test a Node.js process then hard-limit the number of requests each Node.js process handles based on that stress test. Add more Node.js processes (one per core on the same or other servers) until there is sufficient capacity to handle all requests. This assumes the Node.js code is already distributable across cores and servers, which seems to be the case based on the article. Might have been a day-long workaround rather than a weeks-long rewrite.
  - zaphar 11y ago
    Part of the goal was fixing this without having to just throw hardware at it. Every Node process has a non-trivial amount of overhead that requires additional hardware resources. Overhead that the Go app doesn't have. They could have thrown more machines at this but that costs money. The better solution technology wise was Go. It was better suited to their workload.
- jongleberry 11y ago
  My same thought as well. S3 has a rate limit, which is where a lot of these timeouts or errors are probably happening. Seems like they should be using a database instead.
- zimpenfish 11y ago
  If it's anything like the websites I've worked, 10s (sometimes even 100s) of database requests to render a page isn't all that strange.
- CydeWeys 11y ago
  Well, it's Digg, so maybe each key is a comment? That's entirely reasonable. I can think of any number of sample web applications that would require loading dozens of objects from a DB per page load.
bliti 11y ago
Wait until they discover Elixir and Phoenix. It's the power of Go (a bit more I think) without the ugly syntax.
- akshatpradhan 11y ago
  What sets Elixir apart from Go?
  - bliti 11y ago
    IMO, it's simpler to write and easier to read. It's been a bit faster in t he tests I've run but not by a lot. Probably due to my skills. I just find its nicer all around. It does not carry over anything from C.
- 15155 11y ago
  "Power of Go" for me means shared-memory parallelism. What's the answer for that in Erlang or Elixir?
maleck13 11y ago
Just wanted to add my thanks to the author for writing this enjoyable piece. After working with node for 4 yrs & golang for 1yr, I think they both have their place. If forced to choose one, I would currently pick golang due to its excellent tooling, strict typing, excellent std lib and simple deployment.
kseistrup 11y ago
For old farts like me gopher is a protocol.
ysolo 11y ago
I've been coding in Node for a couple of years. I find it interesting. But I'm aware that the async model of Node inverts the flow of control, turning code inside out and makes application logic difficult to scrutinize and reason about. Stack traces in traditional multithreaded languages are easy to understand, whereas in Node a stack trace is necessarily filled with unrelated calls - or perhaps no stack at all in the case of next tick deferred callbacks - due to the single-threaded nature of the model. This necessitates passing around context via closures (or god forbid global variables!). The "context" in a threaded language is typically just the stack. With Node's async Promises and equivalents we can trick ourselves into simulating linear program flow, but it's a pale imitation of the real thing. This got me thinking - why did the single-threaded "green" threading programming model like how the Java JVM was initially implemented all those years ago fall out of favor? It would be as performant as Node presently is but without the difficult async logic for the end user programmer. Async programming and green threads are duals - both use event queues behind the scenes to drive the scheduling. The complexity of the actual I/O and timer event callbacks is hidden from the programmer in green threads. The pseudo "thread" context switches would only be done upon blocking I/O. No need for actual mutexes as the program is still actually single threaded. Stack traces of green threads within a program would be easy to decipher. Computation on a green thread would also block all I/O as it would in Node, but we'd know that going in - no different from Node in that regard. Just the programming model would be simpler.
- tankenmate 11y ago
  Node's javascript engine v8 can only run in one thread, if you want to access it from a different thread you need to take a lock (for example in Chrome / Chromium you need to take an isolate lock; each page runs it's own isolated v8 engine (read only code pages are shared, all other javascript pages (data plus read/write code pages) are private); in this sense it is very similar to the GIL in Python. So although node/v8 allows for concurrency it does not allow for multi-threaded parallel execution.[0] Because of the way tasks are scheduled in node.js you can have head of line blocking; i.e. a task that is scheduled to run first can block all the subsequent tasks, the issue that the OP suffered from. You can sometimes reduce this by calling process.nextTick(), but you need to remember that it is only IO is non-blocking in node.js.[1] [0] https://stackoverflow.com/questions/14409609/does-the-v8-javascript-engine-have-a-gil https://stackoverflow.com/questions/14409609/does-the-v8-jav... [1] http://greenash.net.au/thoughts/2012/11/nodejs-itself-is-blocking-only-its-io-is-non-blocking/ http://greenash.net.au/thoughts/2012/11/nodejs-itself-is-blo...
KirinDave 11y ago
I think this is great engineering, and it shows that the Node runtime needs to consider how to actually address the needs of developers rather than smugly pointing to really frustrating solutions like Cluster that just don't provide equivalent functionality to what Go (and its predecessors in this style such as Erlang and Haskell) are providing. There are actually a whole host of languages that would let you approach the problem with this technique and get there, so the only thing special about Go is that it is syntactically simple (although it's conceptually quite hard to build reusable code because of poor generic support the resulting bad error handling), so it makes for an easy migration. We could see similar stories about going from Node.js to Erlang, Nim, Rust, F#, C#, Haskell, or even the newer Python & Ocaml runtimes. So it's not like Go is special here. But it's timing is special, because it's a harbinger of the industry's slow and reluctant admission that pretending everything is single threaded is okay. Even I/O multiplexing with coroutines simply cannot keep pace with even casual demands on modern infrastructure.
dalyons 11y ago
im obviously missing something here, hopefully someone can explain? I understand the eventloop/nexttick/etc architecture of node, but i dont understand how in this case he was blocking the loop? Shouldnt all the operations to S3 be async (and thus non-blocking, even if waiting for a timeout)? What was the specific part in this scenario that was causing the loop to stall?
- amalantony06 11y ago
  If I understand correctly, the problem was that they were making several thousands of requests to S3. While the requests to S3 themselves were asynchronous, the callbacks for these requests were queued up for (synchronous) execution in the event loop. Due the large number of callbacks in the queue already, new callbacks for the incoming requests were queued up for execution behind the previous callbacks, leading to latencies in serving up responses.
  - dalyons 11y ago
    Ah ok! Perfect, thanks, that's what I was missing. Makes total sense now. In any lang (even in ruby!) you could have separate thread pools(or EM loops, whatever) for the s3 requests & web handlers. But because node only has one EL, and node interprocess comms is awkward, its tricky. Gotcha, cheers. I wonder if using something like async.eachLimit would have helped; it might prevent the s3 batches from flooding the loop & give a chance for web reqs to interleave, but probably at a cost to the median resp time.

260 points by 0xmohit 11y ago

kjksf 11y ago

For other stories of X => Go rewrites, I started keeping a list at https://quicknotes.io/n/vzb7-rewritten-tales-of-rewriting-s https://quicknotes.io/n/vzb7-rewritten-tales-of-rewriting-s

feylikurds 11y ago
That list can easily be made longer. Off of the top of my head I know that there are many more projects which have switch languages to Go. What they say most of the time is (from the article): Since our transition to Golang we haven’t looked back. ... we’ve begun the process of modularizing our code base and spinning up microservices to handle specific roles in our system.
- kkowalczyk 11y ago
  I agree, it's a work in progress. If you have more links, I'll be happy to add them. I have a much longer list of companies that use Go (https://quicknotes.io/n/1XB0-companies-using-go https://quicknotes.io/n/1XB0-companies-using-go) but it's harder to find descriptions of rewrites than find out who's using Go.
iMiiTH 11y ago
You can add The New York Times to that list, although they rewrote from PHP, Python, Groovy, Java and C++ so no idea how it'll fit in that list. https://www.youtube.com/watch?v=bAQ9ShmXYLY https://www.youtube.com/watch?v=bAQ9ShmXYLY

justicezyx 11y ago

Another evidence that the language that is narrowly focused and force "more" correct practices tends to work better in practice. Or, coders are getting worse at master the more powerful language...

XorNot 11y ago
99% of the time what is called "mastering" is actually a whole bunch of stuff you shouldn't do if you want a maintainable codebase. Tricks are just that: tricks. No one sets up a business based on how you pull a rabbit out of hat, nor should they.
- justicezyx 11y ago
  No, mastering a complex and powerful language means use the minimal feature for the majority of your needs, and knows how and when to use the "dangerous" part for a small fraction of your needs. I do not know much of JS, but people write C++ with loads of patterns are prominent examples of people who do not master the language. That's not trick. That's real engineering with (often) bitter experience.

kelkes 11y ago

How many instances of node octo where running to handle the load?

biot 11y ago
> And those 4 load-balanced instances we were running Octo > on initially? We’re now doing it with 2.

DannyBee 11y ago

How does one live without is-positive-integer and isArray?

blitzprog 11y ago
The real question is: How does one live without is-thirteen? https://github.com/jezen/is-thirteen https://github.com/jezen/is-thirteen
- ysolo 11y ago
  Best post here.
deleted 11y ago
[deleted]
tptacek 11y ago
Wait what why https://www.npmjs.com/package/is-positive-integer https://www.npmjs.com/package/is-positive-integer

deleted 11y ago

[deleted]

bitwize 11y ago

I lol'd at the bit about blocking Node's event loop. If Unix weren't so committed to doing I/O the stupid way around, maybe Node wouldn't have to provide fragile, ersatz asynchronicity the way that it does?

fooyc 11y ago
Care to explain?
- bitwize 11y ago
  Two iron-clad rules of data processing: * All data has a type * All I/O is asynchronous and interrupt (event) driven At its inception Unix doubled down on stupid, first by encouraging data to be stored and migrated in the form of untyped text streams, which require ad-hoc parsing and unparsing logic at the endpoints; and secondly by failing to provide asynchronous I/O primitives to user space. POSIX later provided an AIO standard, but it was cumbersome to use, bolted on rather than integrated into the OS, and virtually nobody used it. What you want to be able to do is submit I/O requests to the kernel, and have the kernel notify you when they are completed while you wait or do other processing; you could even put the process to sleep until pending I/O is completed and have it take up no CPU during this time. Instead what people do is burn CPU cycles in select/poll loops, which are the I/O equivalent of asking "Are we there yet? Are we there yet? Are we there yet?" over and over. You can fake asynchronicity by doing a bit of processing before asking "are we there yet?" again, but you have to write your processing code in such a manner as to be done piecewise in a loop, and the lower latency you want the more frequently you have to poll (and the more CPU you have to burn polling). This is how Node does "asynchronous" I/O; the VM stops every few instructions to poll for ready FDs, dispatches to callbacks as necessary, and performs parts of pending large I/O operations which can be done non-blocking. (That's the other sucky bit: you can't just call read(2) and write(2) for large chunks of data on an fd that's been opened O_NONBLOCK and expect it all to work; you have to keep reading or writing in a loop when the fd is ready, subtracting the number of bytes successfully written until the entire buffer is processed.) But if something -- a call into a C library for instance -- stops the VM from doing the poll and I/O bits of its main loop, all I/O simply... stops. And your throughput goes into the toilet. Whereas if the runtime had been based on an OS that natively supports interrupt-driven AIO -- like VMS, Windows, or AmigaOS -- it would be extremely difficult to stall the I/O pipeline completely this way. You might be able to stall further I/O calls with a really long operation, but any pending calls already initiated would complete, and the runtime would be properly notified of their completion. And all this stems from the fact that Unix was the Node.js of its day -- an environment designed to make things easy for casual programmers, not to do things correctly.

BogusIKnow 11y ago

When you switch from Node to Go, two languages that are totally different in style, library support, performance, compile/web turnaround, client/server reusability, I always wonder why you chose Node in this case in the first place.

pron 11y ago
For the exact same reason they chose Go now, and the reason they'll switch to something else (much better!) in two-three years: because that's the current fashion. Nothing says "I'm keeping up with my professional development" than relearning IO libraries and build tools, and rewriting in-house code, over and over again, every time in the new langue-du-jour.
- zappo2938 11y ago
  Meanwhile Facebook is enjoying working with a large PHP code base. When asked they say they can write code that takes time to compile that runs fast in C++ or write code fast in PHP that runs slower. Writing features faster gives them they edge. They have an engineering team that does nothing but optimize PHP. For them an optimization is an acre of servers saved. They if something needs to run blazing fast can write code in assembly language. What I'm told about Node is the JavaScript is running in a single thread but all I/O is being passed to native code which is written in C++ and running on multiple threads. It does use all the CPUs.
  - deleted 11y ago
    [deleted]
  - deleted 11y ago
    [deleted]

cryptica 11y ago

I have read so many articles which incentivize switching from Node.js to Go and every single one of them (this one included) blabber on about the Node.js event loop becoming congested - This is completely misguided - It only shows that the engineer didn't understand the problem. A single Node.js instance runs its business logic in a single process. A single Go instance can run its business logic in multiple processes. Of course if you're comparing a single instance of Go running on 4 CPU cores, it's going to be faster than a single instance of Node.js running on 1 CPU core. But the thing the author fails to mention is that there are a lot of Node.js modules which help you easily scale across multiple CPU cores. E.g. PM2 https://www.npmjs.com/package/pm2 https://www.npmjs.com/package/pm2 Had the author of the article actually understood the real cause of the problem, he would have saved himself and his company a lot of time. Nodejs is still outpacing Golang according to Google Trends and yet I have never seen a single article about switching from X language to Node.js. My guess is that the Go community probably wouldn't exist if it wasn't for this constant form of aggressive propaganda/marketing. Node.js on the other hand doesn't need any marketing/propaganda; it just sells itself. If you have a scalability problem with Node.js (or any language for that matter) and you can't find a simple solution that doesn't require rewriting your whole app, you're probably not qualified to manage this project at scale - Because scaling across multiple CPU cores is kids' play compared to scaling across multiple machines. Heck, my grandma could scale this thing across multiple CPU cores. Personally I think Node.js is great because it encourages you to think about these problems sooner rather than later.

BogusIKnow 11y ago
This is always the problem when you're hype driven. I'm sure the decision to use Node was also hype driven. And they will rewrite again with the new hype, right until they are bankrupt because they didn't invest enough in new features.
- jondubois 11y ago
  Totally agree! I would say the same thing if someone had a whole app written on Python and were having performance issues (I definitely wouldn't advise them to switch to Node.js or Golang) - If your ecosystem offers the tools to solve the problem, it's better to use these.

diegorbaquero 11y ago

The EC2 medium instance has 2 cores. Were you running 2 node processes in cluster mode? The only thing I asked myself when reading this. I'm keeping my vote for node.

dminor 11y ago
m3.medium only has 1 core.

firasd 11y ago

The crux of the difference seems to be, in the Node.js service: "when any request timeouts happened, the event and its associated callback was put on an already overloaded message queue. While the timeout event might occur at 1 second, the callback wasn’t getting processed until all other messages currently on the queue, and their corresponding callback code, were finished executing (potentially seconds later)." And in Go, "the service has a worker pool and a delegator which passes off incoming jobs to idle workers. Each worker runs on it’s own goroutine, and returns to the pool once the job is done."

awjr 11y ago
Node.js has cluster to solve this issue https://nodejs.org/api/cluster.html#cluster_cluster https://nodejs.org/api/cluster.html#cluster_cluster
- Sanddancer 11y ago
  Which brings us back to 2001 and Apache's prefork model. Threads are pretty battle tested at this point, and allow for different reasoning about the concurrency of a problem. The problem with node is that it pretty much only allows for one type of scaling if you start running into performance problems. Go gives you a much bigger toolbox to work with if, much like the post's author discovered, your problem set does not match up with the answers node provides.
  - qaq 11y ago
    All is well until you have to deal with SEO for SPAs :) and then there is little choice other then node. https://github.com/audreyt/node-webworker-threads https://github.com/audreyt/node-webworker-threads spawn real threads with web workers compatible API.
lenkite 11y ago
This is the first solution that a boring 'enterprise' Java programmer would have used and got on with other work.

partycoder 11y ago

Blocking the event loop in node can cause severe issues. If you are juggling with 5 sticks but one stays in your hand for 1 second, your juggling will certainly fail. Instead of flamegraphs I use nodegrind, which I can use with KCachegrind/QCachegrind.

Ericson2314 11y ago

Not gonna lie, one shitty language for another, lol.

erlich 11y ago
What do you use?
- chridal 11y ago
  Must be Erlang.
- Ericson2314 11y ago
  [Cause I can't comment on my own post directly] I should give some some credit as Go is strictly better than Javascript.
- Ericson2314 11y ago
  But to answer the question, languages with (at least) type parameters and no data races (without using escape hatches). (Hint: that includes, but is not limited to, Rust).

erlich 11y ago

Long live Node.js! As everyone has mentioned this seems to be a simple case where the rewrite is a completely different architecture, which could have been done in Node.js. I'm interested in why this continues to happen. Seems like Node.js is a victim of its own simplicity. The baked-in non-blocking concepts of Node.js makes everyone feel like single-threaded is the only way to go, and think that Node.js is inherently flawed when it comes to parallelization. Because Go doesn't give you a simple and relatively scalable solution without even thinking about it like Node.js does, instead you have to choose what to parallelize and think about the problem more. With PM2 scaling Node.js over CPUs becomes incredibly trivial. Also, I think programmers naturally like rewrites/starting from scratch, and a new programming language provides an opportunity to rewrite everything (which the OP from this article and every other "switcher" article seem to do).

douche 11y ago
> Seems like Node.js is a victim of its own simplicity NodeJS is a victim of basing itself on a terrible scripting language that was thrown together to wire up basic event handlers to HTML buttons.
zongitsrinzler 11y ago
Recently I rewrote a medium sized Websocket app from Node to Go. Hoping for better performance and having heard a lot of good about Go. Since the application relies heavily on sharing resources between sockets I found out the hard way that concurrent programming in Go is not much easier than in Java. Sure, it has a better syntax and channels but compared to Node where you never have to think about things like these the complexity is way higher. Long live Node.js!
- 59nadir 11y ago
  > Since the application relies heavily on sharing resources between sockets... Which resources?
  - zongitsrinzler 11y ago

illumen 11y ago

I think the author made a pretty good decision to choose a solution which was proven to meet the requirements by another author. However Amazon does provide a tool to batch download multiple things from S3 though. Let's pretend for a moment those tools don't exist already... Let's also pretend Amazon doesn't suggest another solution for when there are lots of GET requests [0]. Let's pretend that using something like nginx which is a very well tuned web proxy might be way better because of SPDY, SSL connection pooling and such [1]. Let's also pretend that node.js can not use multiple processes with cluster cluster [2]. Why wouldn't one use python for this job? Since the company already does a lot of python, and this job is actually easy to do in python (I've done something similar). 400 processes with python3.5 is way less than the 4gig on one medium instance (less than 2.5MB each). Just farm the work out to a ProcessPoolExecutor, and have a timeout on the S3 get requests. That would let you have enough resources to match the spike of 20000 requests per minute (334 per second). A lot of an S3 GET request could be the SSL, AWS auth and such. All quite CPU intensive. So using an async framework that doesn't do SSL+AWS auth async is obviously not going to work well once the requests go up. There's even an example in the concurrent.futures of downloading urls [3]. Made with the beautiful python3.5 import concurrent.futures import requests from awsauth import S3Auth ACCESS_KEY = 'ACCESSKEYXXXXXXXXXXXX' SECRET_KEY = 'AWSSECRETKEYXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' URLS = ['http://www.foxnews.com/', 'http://www.cnn.com/', 'http://europe.wsj.com/', 'http://www.bbc.co.uk/', 'http://some-made-up-domain.com/'] def load_url(url, timeout): return requests.get(url, timeout=timeout).data # return requests.get(url, timeout=timeout, auth=S3Auth(ACCESS_KEY, SECRET_KEY)).data with concurrent.futures.ProcessPoolExecutor(max_workers=400) as executor: # Start the load operations and mark each future with its URL future_to_url = {executor.submit(load_url, url, 60): url for url in URLS} for future in concurrent.futures.as_completed(future_to_url): url = future_to_url[future] try: data = future.result() except Exception as exc: print('%r generated an exception: %s' % (url, exc)) else: print('%r page is %d bytes' % (url, len(data))) [0] http://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html#get-workload-considerations http://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-... [1] https://coderwall.com/p/rlguog/nginx-as-proxy-for-amazon-s3-public-private-files https://coderwall.com/p/rlguog/nginx-as-proxy-for-amazon-s3-... [2] https://nodejs.org/api/cluster.html#cluster_cluster https://nodejs.org/api/cluster.html#cluster_cluster [3] https://docs.python.org/dev/library/concurrent.futures.html https://docs.python.org/dev/library/concurrent.futures.html

spullara 11y ago

Switching from one essentially single threaded language to another wasn't likely to solve your issues. Instead of Go you could have chosen any JVM language or even C++ and had similar success.

sagichmal 11y ago
Go is not a single threaded language.
- spullara 11y ago
  Obviously. I was just pointing out that switching to Go wasn't the interesting bit. Maybe everyone misread my comment.
  - _ak 11y ago
    If you didn't want your comment to be misread, you should have been more specific after the "another".

nissimk 11y ago

Originally, the web server was intended to be a very simple program with most of the complications in the browser. Most web app backbends are similar to this where all they do is query a database, cache or other web servers. This type of thing is perfect for node. If you are doing more complex stuff in your backend then go may be more appropriate.

romanovcode 11y ago

2010 - Making switch from PHP to Ruby 2013 - Making switch from Ruby to Node 2016 - Making switch from Node to Go 2019 - Making switch from Go to {hype}

tptacek 11y ago
"So what I'm saying is, we should all still be using PHP."
- mwcampbell 11y ago
  Or Perl, or C. After all, even PHP was once a new hipster language. Of course, it's good to be skeptical of the current hyped flavor of the moment. But some languages really are better than others for specific tasks, and new languages do sometimes improve on their predecessors.
  - collyw 11y ago
    C is still the choice language for many things. Perl is still cool to me, though seems to have been replaced by Ruby for web development.
- bliti 11y ago
  What is star fighter written in? Curious, no bad intentions.
  - dikaiosune 11y ago
    IIRC, it's got some Go infrastructure and the trading chapters have bots written in Ruby.
  - tptacek 11y ago
    Go, Ruby, and React.
- brightball 11y ago
  Ironically enough, given the recent improvements, performance and productivity...that's not really out of the question...

forgettableuser 11y ago

I'm disappointed by the overall tone here. It's as if this were some kind of failure that everybody is piling up on. - She made three passes to fix the existing code base. - After that didn't work, with the CTO's involvement, this Go solution was devised. - It took only 2 weeks to ramp up with Go and build a replacement that solved the problem. - The number of instances needed dropped from 4 to 2 This sounds like a resounding success. So congratulations Alexandra and Digg. And thank you for sharing. The article was well written and I learned some things about how the Node.js event system works under the hood.

deleted 11y ago
[deleted]
rgbrgb 11y ago
I think the disappointment here is that the OP seemed not to understand the root cause of the problem and implemented a solution in a different language (load balanced worker pool) that could have also worked in the original language without a total rewrite. Then the new language is trotted out as the savior. It sounds like the file fetch workers and request handlers were running on the same process, so the longer-running workers ended up blocking requests. The way it's discussed, it seems like pure luck that they stumbled on a solution that solved this (running the workers on a separate queue/process). I totally agree that it's impressive that it only took 2 weeks to build the Go version, it just seems like it would have taken 2 days to try the worker pool implementation with Node.js.
- zaphar 11y ago
  But using a process pool is far more heavyweight than threads. There is a cost there that you pay in hardware. Using Go allowed them to do this with less hardware cost than Node. That's a win that continuing to use Node would not have been able to provide.
  - ilaksh 11y ago
    Node does actually have threads with modules like webworker-threads.

takno 11y ago

On the face of it this sounds like a problem where the optimal solution would have been to throw more hardware at it while you looked at the underlying choice of datastore.

0xmohit 11y ago

Some comments prompt adding another link here: <https://medium.com/@tjholowaychuk/farewell-node-js-4ba9e7f3e52b> https://medium.com/@tjholowaychuk/farewell-node-js-4ba9e7f3e...

sah2ed 11y ago
https://medium.com/@tjholowaychuk/farewell-node-js-4ba9e7f3e52b https://medium.com/@tjholowaychuk/farewell-node-js-4ba9e7f3e... FTFY
- 0xmohit 11y ago
  Thanks. Didn't realize that HN screws up links within angle brackets.

planetjones 11y ago

> for each request, Octo typically fetches somewhere between 10–100 keys from S3 I'm really curious as to why this is neccessary - what are the 10-100 keys you are fetching per request ? Just reading the high level article it sounds like there is some issue with what this service is doing, irrespective of the language it's doing it in.

meonkeys 11y ago
Another idea: stress-test a Node.js process then hard-limit the number of requests each Node.js process handles based on that stress test. Add more Node.js processes (one per core on the same or other servers) until there is sufficient capacity to handle all requests. This assumes the Node.js code is already distributable across cores and servers, which seems to be the case based on the article. Might have been a day-long workaround rather than a weeks-long rewrite.
- zaphar 11y ago
  Part of the goal was fixing this without having to just throw hardware at it. Every Node process has a non-trivial amount of overhead that requires additional hardware resources. Overhead that the Go app doesn't have. They could have thrown more machines at this but that costs money. The better solution technology wise was Go. It was better suited to their workload.
jongleberry 11y ago
My same thought as well. S3 has a rate limit, which is where a lot of these timeouts or errors are probably happening. Seems like they should be using a database instead.
zimpenfish 11y ago
If it's anything like the websites I've worked, 10s (sometimes even 100s) of database requests to render a page isn't all that strange.
CydeWeys 11y ago
Well, it's Digg, so maybe each key is a comment? That's entirely reasonable. I can think of any number of sample web applications that would require loading dozens of objects from a DB per page load.

bliti 11y ago

Wait until they discover Elixir and Phoenix. It's the power of Go (a bit more I think) without the ugly syntax.

akshatpradhan 11y ago
What sets Elixir apart from Go?
- bliti 11y ago
  IMO, it's simpler to write and easier to read. It's been a bit faster in t he tests I've run but not by a lot. Probably due to my skills. I just find its nicer all around. It does not carry over anything from C.
15155 11y ago
"Power of Go" for me means shared-memory parallelism. What's the answer for that in Erlang or Elixir?

maleck13 11y ago

Just wanted to add my thanks to the author for writing this enjoyable piece. After working with node for 4 yrs & golang for 1yr, I think they both have their place. If forced to choose one, I would currently pick golang due to its excellent tooling, strict typing, excellent std lib and simple deployment.

kseistrup 11y ago

For old farts like me gopher is a protocol.

ysolo 11y ago

I've been coding in Node for a couple of years. I find it interesting. But I'm aware that the async model of Node inverts the flow of control, turning code inside out and makes application logic difficult to scrutinize and reason about. Stack traces in traditional multithreaded languages are easy to understand, whereas in Node a stack trace is necessarily filled with unrelated calls - or perhaps no stack at all in the case of next tick deferred callbacks - due to the single-threaded nature of the model. This necessitates passing around context via closures (or god forbid global variables!). The "context" in a threaded language is typically just the stack. With Node's async Promises and equivalents we can trick ourselves into simulating linear program flow, but it's a pale imitation of the real thing. This got me thinking - why did the single-threaded "green" threading programming model like how the Java JVM was initially implemented all those years ago fall out of favor? It would be as performant as Node presently is but without the difficult async logic for the end user programmer. Async programming and green threads are duals - both use event queues behind the scenes to drive the scheduling. The complexity of the actual I/O and timer event callbacks is hidden from the programmer in green threads. The pseudo "thread" context switches would only be done upon blocking I/O. No need for actual mutexes as the program is still actually single threaded. Stack traces of green threads within a program would be easy to decipher. Computation on a green thread would also block all I/O as it would in Node, but we'd know that going in - no different from Node in that regard. Just the programming model would be simpler.

tankenmate 11y ago
Node's javascript engine v8 can only run in one thread, if you want to access it from a different thread you need to take a lock (for example in Chrome / Chromium you need to take an isolate lock; each page runs it's own isolated v8 engine (read only code pages are shared, all other javascript pages (data plus read/write code pages) are private); in this sense it is very similar to the GIL in Python. So although node/v8 allows for concurrency it does not allow for multi-threaded parallel execution.[0] Because of the way tasks are scheduled in node.js you can have head of line blocking; i.e. a task that is scheduled to run first can block all the subsequent tasks, the issue that the OP suffered from. You can sometimes reduce this by calling process.nextTick(), but you need to remember that it is only IO is non-blocking in node.js.[1] [0] https://stackoverflow.com/questions/14409609/does-the-v8-javascript-engine-have-a-gil https://stackoverflow.com/questions/14409609/does-the-v8-jav... [1] http://greenash.net.au/thoughts/2012/11/nodejs-itself-is-blocking-only-its-io-is-non-blocking/ http://greenash.net.au/thoughts/2012/11/nodejs-itself-is-blo...

KirinDave 11y ago

I think this is great engineering, and it shows that the Node runtime needs to consider how to actually address the needs of developers rather than smugly pointing to really frustrating solutions like Cluster that just don't provide equivalent functionality to what Go (and its predecessors in this style such as Erlang and Haskell) are providing. There are actually a whole host of languages that would let you approach the problem with this technique and get there, so the only thing special about Go is that it is syntactically simple (although it's conceptually quite hard to build reusable code because of poor generic support the resulting bad error handling), so it makes for an easy migration. We could see similar stories about going from Node.js to Erlang, Nim, Rust, F#, C#, Haskell, or even the newer Python & Ocaml runtimes. So it's not like Go is special here. But it's timing is special, because it's a harbinger of the industry's slow and reluctant admission that pretending everything is single threaded is okay. Even I/O multiplexing with coroutines simply cannot keep pace with even casual demands on modern infrastructure.

dalyons 11y ago

im obviously missing something here, hopefully someone can explain? I understand the eventloop/nexttick/etc architecture of node, but i dont understand how in this case he was blocking the loop? Shouldnt all the operations to S3 be async (and thus non-blocking, even if waiting for a timeout)? What was the specific part in this scenario that was causing the loop to stall?

amalantony06 11y ago
If I understand correctly, the problem was that they were making several thousands of requests to S3. While the requests to S3 themselves were asynchronous, the callbacks for these requests were queued up for (synchronous) execution in the event loop. Due the large number of callbacks in the queue already, new callbacks for the incoming requests were queued up for execution behind the previous callbacks, leading to latencies in serving up responses.
- dalyons 11y ago
  Ah ok! Perfect, thanks, that's what I was missing. Makes total sense now. In any lang (even in ruby!) you could have separate thread pools(or EM loops, whatever) for the s3 requests & web handlers. But because node only has one EL, and node interprocess comms is awkward, its tricky. Gotcha, cheers. I wonder if using something like async.eachLimit would have helped; it might prevent the s3 batches from flooding the loop & give a chance for web reqs to interleave, but probably at a cost to the median resp time.