7 ms·
From Node.js to Go
- deleted 11y ago[deleted]
- fithisux 11y agoThe right decision. If only I could do the same with java and Python.
- kiril-me 11y agoYep, leak of numbers exists in this blog post. Node is certainly not for websocket communication, until it's simple chats or message count is not so huge.
- Gonzih 11y agoThis blog post feels very empty. No code snippets? No actuall before after benchmarks/graphs? How can you state anything without showing numbers?
- rahul286 11y agoYep. I was looking for same.
- lossolo 11y agoI can confirm some of the statements in that Blog as someone using both also in production. Memory footprint is a lot lower than node like 5-10x times what I've observed on similar code. Go has also very efficient GC, that can be used on large heaps with low latency.
- qaq 11y agoI think would be cool to know at least app type. For something that has a ton of ws connections node will generally have smaller memory footprint.
- lossolo 11y agoWe are using Go for a lot of internal tooling, text processing, also for websockets and web API. Can you provide some more data about memory footprint in node with websockets? How much memory per connection you saw? From my experience it's the other way around. But the most important point is that while both can have a lot of connections active, when you start sending messages thru that websocket connections you will see a LOT higher latency in node.
- qaq 11y agoafter looking at it you are right it is higher for node
- swah 11y agoAlways expecting something like https://aphyr.com/posts/329-jepsen-rethinkdb-2-1-5 https://aphyr.com/posts/329-jepsen-rethinkdb-2-1-5
- joshrotenberg 11y agoHoping for? Yeah. Expecting? Rarely.
- Cyph0n 11y agoAgreed. I posted a comparison between Node and Go UDP socket performance a few months back with some code snippets. My submission was completely ignored. HN can be weird sometimes, huh. Edit (and shameless plug): http://assil.me/2015/11/07/roadomatic-node-vs-go.html http://assil.me/2015/11/07/roadomatic-node-vs-go.html You might need to read the previous post (and only other one!) to understand how the GeoJSON geospatial query works.
- andrewstuart2 11y agoThere was an analysis done recently that showed a few votes very early on (I think 4 may have been the number quoted) is often enough to bump you to the front page. Have ten developers on staff, each with an HN account? Some might see that as a free ticket. At best it's only a very temporary traffic boost, though, unless the content is legitimately interesting enough to warrant return visits.
- throwanem 11y ago> Have ten developers on staff, each with an HN account? I think that's what HN calls a "voting ring" and attempts to detect and counteract.
- adrusi 11y agoI don't think it can be detected the first time its attempted with any particular group of users.
- Cyph0n 11y agoYep, that's true. I got around 5000 page views from Reddit over 2 days, but then the traffic died off.
- tedmiston 11y agoI've noticed that posts to the same link can be made just hours apart or with a slightly more interesting headline -- while the former gets just a couple votes, the latter makes it to the front page. It's a bit strange.
- deleted 11y ago[deleted]
- goldbrick 11y agoIn particular I wished they would have elaborated on the claimed instability in x/net/websocket.
- inlined 11y agoI can't speak to instability, but I had to rewrite a massive stack because gorilla gave me callbacks to execute on control packets and x/net/websocket didn't. If you have any sort of long lived mobile connection, you will encounter half closed sockets due to mobile network quirks. At scale you really want to clean these up. To do so, you need an application level timeout and you want to listen to the PING/PONG control frames to reset that timeout.
- zeveb 11y ago> If you have any sort of long lived mobile connection, you will encounter half closed sockets due to mobile network quirks. At scale you really want to clean these up. Could golang.org/x/net/context.WithTimeout be used to handle this?
- thegeekpirate 11y agohttps://github.com/gorilla/websocket#gorilla-websocket-compared-with-other-packages https://github.com/gorilla/websocket#gorilla-websocket-compa...
- whateveracct 11y agoWelcome to the carGo cult :^)
- deleted 11y ago[deleted]
- lossolo 11y agoI use Go and Node in production. Node is used for cold paths and Go for one of hot paths (where GC latency is still not a problem). Go for sure is more readable than Node callback hell and i really like more writing synchronous like code inside gorutines than do everything around callbacks/promises. Static typing and compiler is also big + for Go.
- SonicSoul 11y agocorrect me if I'm wrong but in Node you could use something like a Generator + Promise combination to achieve synchronous looking code with no callbacks https://eladnava.com/write-synchronous-node-js-code-with-es6-generators/ https://eladnava.com/write-synchronous-node-js-code-with-es6...
- drostie 11y agoYou need a wrapper to make it work, but yes, you can do this. The first thing you need is a wrapper to make a Node-style call into a Promise: function node(fn, ...args) { return new Promise((accept, reject) => { args.push((err, ...vals) => err? reject(err) : accept.apply(this, vals)); fn.apply(this, args); }); } What does this do? The argument of `new Promise` is a function which takes two functions, `accept` and `reject`, as its arguments. The function will be immediately run and is meant to do some stuff, possibly asynchronously, and then eventually call the first one of these on success or the second on failure. (One thing that takes a while to get used to about Promises is that generally the asynchronous thing starts when the Promise is created, so execution is not deferred ever.) We just take the args that it was called with, and append our own usual-Node-style callback to those arguments, asking it to pipe the Node error to the reject() function or the successful values to the accept() function. Then you need a sort of "trampoline" to convert a generator-yielding-promises into a single promise. That looks like this: function go(gen_maker) { var gen, goNext; gen = gen_maker(); goNext = val => { var item = gen.next(val); return item.done? item.value : item.value.then(goNext); }; return Promise.resolve(undefined).then(goNext); } Now what does this do? It defines a semirecursive function goNext which will hand itself to any new promises. The goNext function just runs the generator for one step with its argument, and looks at what it got out of that generator. If the generator has completed, it says "I'm done, great, return that last value!" and if it hasn't completed it says "okay, I have a Promise here, I need to tell it that when it's done computing what it's computing, it should goNext() with that value, to hand that value and control back to the generator." Finally after defining this magical function we begin the first gen.next(undefined) run by creating a Promise which immediately resolves to undefined, and telling it to goNext() with it. Of course, `gen` is the output `GeneratorFunction` that is generated by running a function* generator-maker on some arguments. With those two basics, you do eventually de-nest everything to just one layer of asynchronicity. You of course cannot do better than this to get truly synchronous code, but you might not need to if your last expression would have been undefined anyway. For example, here is how you then write something which accepts two arguments and appends the first to the second, using a 10MB buffer in memory. (Yes, I know this is just `fs.createReadStream(process.argv[1]).pipe(fs.createWriteStream(process.argv[2], {flags: 'a'})`, with some messing around with the high-water mark; it's just to prove a point.) // In this comment, we pretend node and go are defined as above, that we have // checked for exactly two args on process.argv and logged some usage // instructions if not, etc. var fs = require('fs'); const buff_size = 10 * 1024 * 1024; go(function* () { var in_file, out_file, buff, count; buff = new Buffer(buff_size); in_file = yield node(fs.openFile, process.argv[1], 'r'); do { out_file = yield node(fs.openFile, process.argv[2], 'a'); count = yield node(fs.read, in_file, buff, 0, buff_size, null); yield node(fs.write, out_file, buff, 0, count); } while (count === buff_size); yield node(fs.close, in_file); yield node(fs.close, out_file); }); In terms of larger libraries that do the same, so that you don't have to roll your own `node` and `go` functions, see e.g. http://taskjs.org/ http://taskjs.org/ and some other similar projects.
- mccolin 11y agoI love the anecdotal account -- especially since I am researching a similar change -- but would truly appreciate more numbers for transition stories like this. Memory usage before and after. Throughput, etc. Also what was the cost in time and man hours of conversion?
- lossolo 11y agoIt really depends on how much experience someone has with Go. It will be more hours for sure than Node, probably more code also. You need to watch out on couple of things also that you don't need in Node. For example default passing by value to functions so you need to use pointers for large structs if you want performance, you need to know that strings are immutable which means that if you do something like "some" + "ting" you are basically creating new string and both strings are copied to that string, if you have a lot of that kind of operations (like really a lot) then use bytes.buffer etc. A lot of things that are not so obvious for beginner in Go. But you got great tools at your disposal by default like pprof which is great profiler, data race checker etc. Standard tooling is really great.
- adamtulinius 11y agoStrings in javascript/nodejs are also immutable.
- Kinnard 11y agoYou will appreciate this: https://medium.com/@theflapjack103/the-way-of-the-gopher-6693db15ae1f https://medium.com/@theflapjack103/the-way-of-the-gopher-669...
- lpinca 11y agoI wonder what was the WebSocket module used in Node. There is this new module https://github.com/alexhultman/uWebSockets https://github.com/alexhultman/uWebSockets which is very promising.
- Bushra8 11y agoI would say number of modules is a really bad metric to judge the worth of a platform on. The truth in this is Go has a really fantastic standard lib, hence not needing millions of modules whereas node.js has tons because it's using a language that has a very weak standard lib and doesn't have things like integers.
- fighting 11y agoGo seems to be still a bit of a black box in terms of performance/memory profiling. On one recent project, the go memory profiler was showing about 35MB allocated while top showed memory usage at 500MB to 1GB. In another instance the performance profiler would show microseconds elapsed while the user experienced tens of minutes with no response from the program. I really think they should consider including a function level profiler instead of just a sampler, as well as provide more insight/control over all the goroutines that are generated. Just saying golang is better than node.js may not be saying all that much.
- lossolo 11y agoCan you provide more details? Because i didn't encounter such problems. Was this 500 MB in top was RSS ? What you describe should not happen, did you report a bug?
- fighting 11y agoThey were more like things the go runtime was doing but which I had no clue or information about, rather than bugs. The top info was indeed rss.
- lossolo 11y agoFor me it seems like you probably leaked gorutines/not closed os resources or you encapsulated some data structures in themselves which then can't be garbage collected. It really looks like memory leak caused by app logic. But as i said you should report this if you are certain that is Go bug.
- Torn 11y agoAre you looking at virtual memory / memory-mapped files?
- donatj 11y agoI've seen similar things happen when using very large maps in Go, memory being eaten and never released.
- qaq 11y agoHow did this make fp? 0 factual content. X is better then Y
- oblio 11y agoIt made first page by having popular values for X and Y.
- josep2 11y agoBeating the living shit out of this dead horse.
- CraigJPerry 11y agoWhy 89 upvotes, there's zero technical details?
- d23 11y agoAnd considering the recent leftpad disaster, it makes this "point" a bit tone deaf: > Node.js has been around the block longer it boasts more than double the modules
- nkozyra 11y agoWell the leftpad disaster wasn't really a symptom of the # of modules. I think it's probably fair (and unsurprising) to say there are a lot more "significant" modules in the Node world than Go, even if we discount the 10-line basic function modules.
- devty 11y agoI don't understand either.
- wooptoo 11y agoThis page is worded like an advertisement.
- KAdot 11y ago> Node.js has been around the block longer it boasts more than double the modules Yes, Node.js has more third-party packages available, but the quality of Go's packages is usually higher. Also, you can do a lot with built-in packages in Go compared to Node.js.
- grigio 11y agoIs this really the best compromise? What about NodeJS + Rust (for the heavy tasks) via Neon? https://github.com/rustbridge/neon https://github.com/rustbridge/neon
- andrewstuart2 11y ago> Another thing I found bothersome were the generated GoDoc pages. They were ordinarily not as easy to understand as hand written Node.js GitHub readme pages and came with very few, if any, examples. I'll definitely agree here. It's not obvious right away how to write examples that get included in the generated documentation, and definitely unintuitive that it's a completely different process than the comment-based approach.
- Twirrim 11y agoa.k.a. from one hotness to the next?
- sspiff 11y agoTL;DR: Using a lower-level, compiled programming language results in lower memory consumption and faster software. Shocker. Besides the lack of data to illustrate their point, their point is well know and has been well documented for decades. The reason people use these higher level, higher overhead languages is generally time-to-market, maintainability and development cost. I don't so much care about the performance difference data, I can measure that myself easily and cheaply. Measuring frustration, development time and bug-hunting time is something that would be expensive to measure (you have to build a real world product in both languages with users and maintain it, which they seem to have done). Yet this part is glossed over and never quantified in the article.
- gbersac 11y ago> their point is well know and has been well documented for decades. Node.js and go are 7 years old, so no it has not been documented for decades :p But your comment is still right.
- sspiff 11y agoThose are just examples of a high-level language and a systems programming language though, and the expected result is the same as comparing Perl/Tcl/Python with C/C++, or whichever decades old variation of this topic you want to pick :)
- untothebreach 11y agothe point that has been known for decades is not, "Switching from Node.js to Go gives you better performance," it is "switching from a high-level language to a low-level language gives you better performance." This point _has_ been known for decades, but new generations of programmers keep re-discovering it. (note: I am not critizing new programmers, I am one of them. in fact, having recently switched from python to rust, you could say I am one of the many re-discovering the performance joys of lower-level languages)
- mort96 11y agoHigher level languages generally being slower than lower level languages has been documented for decades.
- lopatin 11y agoIf anyone is planning on building a semi-ambitious web system soon, you should evaluate Go. It's kind of scary how productive it allows me to be now. All of the following benefits, of course, may not apply to your project but I wanted to share my experience regardless. I started my project as a collection of (micro?) services, all talking to a distributed kv store, containers all orchestrated, etc ... That's just how we build things nowadays right? Well one day I said screw it and rewrote the whole thing as a Go monolith. Including the DB layer, which now runs on BoltDB, an embedded k/v store. So now I have a web analytics system, reverse proxy, rest api, pixel api, license key server, reporting engine, emailing service, and a custom stripped down time series database (uses Bolt as a storage engine) complete with a custom SQL-like language (stripped down version of InfluxQL). It all exists as a single binary, which magically works on Windows, OS X, and linux, with zero run-time dependencies. This has allowed me to provide my customers with a self-hosted version of my software if they don't want to use my cloud service. My cloud service is just a differently configured version of the self hosted binary. Compilation takes just a few seconds. Regarding performance. A single node handles thousands of requests per second. It's possible for me to optimize that further (I suspect by a lot), but I don't need to. And up to 1TB single node storage. Again, there have been larger BoltDB deployments but I simply don't need to push it very far since I can easily run multiple nodes when the time comes, each handling some subset of user accounts. Another delightful thing is my dependency situation. I have 8 library dependencies in total, and have never even had to think about which version I'm using past the initial download. They're mostly lean and mean and stay out of the way. Things like CORS middleware, JWT library, request mux, Bolt, and a few others. And they all work together since they all conform to Go's built-in defined interfaces. What I appreciate about the library ecosystem is that most of the libraries you will use are _done_. Go has no dependency versioning, semver, etc... It sounds concerning at first but I was pleasantly surprised about how much of a non-issue that is in Go. I suspect that a project of similar scale would have resulted in many lost hours in Maven, Gemfile, or package.json. Also, the concurrency makes me happy. I've never been one to multi-thread my programs. But Go makes it easy to build concurrent pipelines and services without needing a team of engineers to review it for correctness. Anyways I'm rambling, I guess I should list some of the frustrations and trade-offs I made. For one, yes you will miss generics at first. It seems like a weird thing to not have in the language. It kind of takes getting used to, but you will figure out how to write the same functionality in terms of interfaces. It's certainly unique, a bit more typing, but in the end, things still get done. But, yes, you will have to shift your mind to a different programming style. Which some find annoying. On the plus side, at least for me, Go is one of those languages that has changed the way that I think about programming. Also, not really speaking to the Go language, but if you're considering Bolt as I mentioned, don't expect it to handle replication or distributed fanciness for you. I'd have to say the biggest tradeoff I made is switching to a scheduled data file backup system in the place of an actual database service. With naive scheduled backups, you might lose a few minutes of data in the case of a total node failure. This can be improved on my end, but that starts to get pretty involved. ... I don't know why I just wrote an essay. Hopefully this encourages someone try out Go for their next project.
- BuckRogers 11y agoIt's hard to make a great analysis because there's no hard data on the memory used, the code creating that memory use or their infrastructure. But if I may, due to them being less than forthcoming, I'd like to speculate. Anger many, please a few. If they needed something better than Node they may simply be doing it wrong and I'd lean that way first before subscribing to Go being the solution. Which has me thinking it's the most convenient band-aid nearby. There are alternatives to the language/platform hopping. My case would be a decent example. Most of my stuff is in Python2. I have a newer Django project that I'm vetting CPython3 with, but will remain writing long running CPU intensive services in Python2 on PyPy. I see CPython + PyPy as a pretty reasonable approach and have been able to scale quite well for most situations which don't involve battling millions of concurrent users and petabytes of real-time data analysis. There's also the advantage of a single language backend and developer productivity. Node.js callbacks never matched gevent or Python3's async I/O and Go is adding yet another language to the stack where PyPy (or Node in their case) would likely handle it. Difficult to say without looking at how they've implemented everything but I have doubts whatever they're doing couldn't be done with resourceful coding. Most likely, these are people who want an excuse to try something else and they're just excited. Given the self-professed extremity of their case, Node, Python, Go, et al would not resolve something so seriously memory-bound well enough that it wouldn't be worth calling out to a non-GC'd library. I would steer away from GC-anything and drive a stake through the heart. Writing a small limited-scope Rust extension to do what they needed and called that from Node.