5 ms·

>I think node.js is a great platform, I'm curious as to why. For large scale applications like this, you have other options that offer higher performance ceil

>I think node.js is a great platform, I'm curious as to why. For large scale applications like this, you have other options that offer higher performance ceilings, have more safety and correctness features, and are likely more productive as well. What is the attraction to node? A guy has to invent a scripting language for browsers in 9 days -> he decides on a lisp -> management says no it has to look like java -> he comes up with something -> its dynamically typed -> lets run a huge banking infrastructure on this wat

a13n 7y ago
> safety and correctness features You can achieve safety and correctness features for node via good lint rules and typescript/flow.
- Nextgrid 7y ago
  You can technically achieve all of that even in an obscure language like Brainfuck but it doesn’t mean it’s a good idea. Why would you duct tape hacks on top of hacks to achieve the result you want instead of using a language that has already all of the functionality built-in?
  - kevstev 7y ago
    I can appreciate your suspicion of a dynamically typed language, but type script isn't really a "hack" its a superset of the language. I am not a huge fan of it, to me its kind of the worst of both worlds- if you need type safety, use something else. However, if you haven't taken a look at Joi- https://hapi.dev/family/joi/?v=16.1.8 https://hapi.dev/family/joi/?v=16.1.8 you probably should. To me this was a very happy medium- you can enforce your "types" at the point of ingress and egress at your api, and even do validations there, while still being able to enjoy the flexibility of type safety within your code.
    - rclayton 7y ago
      Lol - I don’t think I’ve ever met someone as enamored with Joi as I am. I think I abuse it in all the same ways as you do. I even publish my Joi schemas for request validation with my API specifications.
  - bfrydl 7y ago
    I find it really annoying how JavaScript is treated on developer forums like this one. Why is it that when the same exact things are done in a typical language it's called “tooling” but in JavaScript it's “duct tape hacks on top of hacks”? Don't get me wrong, my favorite language is Rust, but pretending the JavaScript ecosystem is unusable doesn't make you cool. I can be extremely productive in TypeScript.
    - Nextgrid 7y ago
      JavaScript itself is not a hack, it works fine for its original purpose - making webpages interactive. What I consider hacks is the hundreds of different tools and dialects of JS that are used to bend JS into doing something it wasn’t really designed for. JS is fine in the browser, and TypeScript is also fine there because you don’t really have the choice to run anything different (although Web Assembly might change the game soon). But on the backend you have the privilege to pick between dozens of different languages that are better suited to the task and support the features you want out of the box without layering hacks on top. Why not just go with one of those?
  - a13n 7y ago
    We chose JavaScript/Node because it's the language we knew best, and because we have to write it in the client as we're building a web-based service. No other language offers us that, unless it compiles to JavaScript (eg TypeScript). This grants us a magnitude of benefits, like being able to do server-side rendering and share code between the client and server. It also means we only need to hire people who can write JS, instead of JS and something else. TypeScript, Flow, and ESLint aren't hacks. These are mature tools used by some of the largest, most sophisticated engineering teams in the world.
kevstev 7y ago
My background was doing low level C++ in HFT/Algorithmic trading for years, with a bit of Java interspersed, before doing this whole complete right turn of doing webdev in js for ecommerce. In C++, doing web stuff was very difficult, build times were long, JSON support existed but was awkward, iterating was just very difficult, even when you had your whole build/deploy setup going, there was still a lot of work if you wanted to build a CRUD app just marshalling and unmarshalling objects to the DB, etc. It was a drag at best and a real pain at worst. Java... was a bit better, and I don't really have a problem with Java as a language. Java programmers, however, seem to really delight in building architectural monuments and get paid by the abstraction. In every java system I have jumped into, you are always neck deep in xml, and massive object hierarchies and factory factories and its just like where the F is the code that actually does stuff?! I remember working on one project, where I was building the "engine" and another guy was building the web interface for it, and was using Spring when it was relatively new, and he happily declared "all I need to do is wire up my configs now, I am pretty much done." Narrator: A week later, he still wasn't done... and this wasn't unusual in my experience, in most java apps "config" was just as complex and problematic as code, and the mindset of "its just config" made config changes more likely to cause a production outage of some sort. Then, you take a look at node. You look at a getting started tutorial. Its javascript on the front, and on the back. The JSON in between is "native" and is convenient and easy to use, easy to read, lightweight, and just makes a lot of intuitive sense- especially when I had found myself neck deep in XML in previous jobs for the same tasks. I had a nice looking HTML5 web app running in a few minutes- my mind was blown. Then you take a look at the frameworks- express and hapi, and the vast module ecosystem- and how easy it was to build a simple CRUD website with leveldb, or mysql, or really an endless array of storage options. And people were using those options! It wasn't just the bog standard RDBMS being used every place, with your only real choice being mysql, postgres, or if you had money, Oracle. Building endpoints with routes in these frameworks made your code so easy to divide up along clear lines, and there just wasn't the endless miles of boilerplate/scaffold code, and ugly syntax and type systems to fight with and plan ahead of. Things Just Worked. Turning around a code change was a matter of seconds, not a minutes long build process- I had never felt so productive- and writing code was fun again! Deploys were easy, restarts were fast. Rollbacks, when necessary, were painless. There was a plugin/module for everything (too much in hindsight). Now, this was 6 years ago. Go was around, but still kind of a blip on the radar, Ruby/Python were probably the closest real contenders. Ruby had lost steam, I honestly took some cursory looks at it, but it didn't seem to have traction. Python, suffered from its single threadedness and GIL, and its popularity with the ML crowd- Flask and such existed, but was pretty rudimentary compared to what Express/Hapi were offering, and no one seemed that interested in those projects. I like Go a lot, and for a pure backend service, it might be my go-to today, as one of the original arguments for Node was "its the same language on the front and the back, no more delineation between FE and BE developers, anyone can jump in and fix the bugs!" Which, along the lines of my original comment, don't really work out in reality, at least not on larger systems. People drawn to FE work usually have never done real systems development and don't understand how things work under the hood- which isn't a problem, until one day it is and then its a huge one. The dynamic typing argument... is somewhat valid, but I found that enforcing api contracts with hapi/joi gave you the equivalence of type safety at your interface borders, while still giving you the flexibility of dynamic typing within your code. In fact, Joi went even farther than just type checking, it could check that your int was within range for the field, that your dates were formatted properly, etc... In mega large codebases, this will come back to bite you, but I found the plugin architecture of Hapi really discouraged that kind of crap from leaking in and it was easy to build truly modularized code. The performance ceilings aren't that different, and not that impactful, at least not until you get to FANG scale, and I mean literally only FANG scale. We were running a billion dollar business with on 8 fairly small VMs for the API layer, which handled all of the ecommerce transaction handling. I remember at one point we encountered a memory leak of some sort in node, and the instances were falling over and dying about once an hour, but restarting and recovering- this was causing a few % error rates to our customers. I was insistent that we get all hands on deck to figure this out ASAP, and our head of Ops type person said "kevstev, we can throw hardware at this problem to meet SLOs until you get it under control. Your monthly server costs are less than my studio apartment cost me per month in Jersey City 15 years ago." You just have to have a basic understanding of whats going on at an architectural level, something a few hours of doing the right reading and experimenting can get you if you have the proper background. The number of gotchas to avoid to get that performance were an order of magnitude, if not more, fewer than in a language like C++ (Which I feel has actually gotten so complicated and difficult to grok its become a parody of itself- and I say that as someone who used it and adored it for 15 years).
JMTQp8lwXL 7y ago
Most bugs encountered in production systems aren't type based issues. Types are more useful for developer productivity (e.g., intellisense) than any other purpose.
- RSZC 7y ago
  Previously worked at a Node company - at one point in an effort to improve code quality we ran statistics on errors we'd seen over the past [period - forget exactly]. Type errors were our most common source of error both by number of total errors and number of distinct errors. I'd love to see any data or case studies that claim the opposite if you have any.
  - kevstev 7y ago
    Just curious- did you use Joi or anything similar to try to at least verify at some point that you had a valid object?
  - JMTQp8lwXL 7y ago
    https://blog.acolyer.org/2017/09/19/to-type-or-not-to-type-quantifying-detectable-bugs-in-javascript/ https://blog.acolyer.org/2017/09/19/to-type-or-not-to-type-q... > Is a 15% reduction in bugs making it all the way through your development pipeline worth it to you?... https://www.reddit.com/r/typescript/comments/aofcik/38_of_bugs_at_airbnb_could_have_been_prevented_by/ https://www.reddit.com/r/typescript/comments/aofcik/38_of_bu... > 38% of bugs at Airbnb could have been prevented by TypeScript according to postmortem analysis I've never seen a number far outside of the 15-30% range. In my experience, most bugs are operator error. Developers didn't code for branching paths that should've been accounted for, etc. Personally, I'm a fan of TypeScript. Just don't expect to remove the majority of your bugs via its usage. The old "no silver bullet" adage.
    - bathtub365 7y ago
      How is JavaScript worth it if it typically adds 15-30% more bugs? That’s an enormous figure.
NohatCoder 7y ago
The real killer feature is async. Since a modern web request typically spend most of the time waiting for database calls, file system requests or similar, a naively coded server in most languages can handle relatively few requests per thread, so you scale up the number of threads to something like 100 per core, and now the overhead of running and switching between these treads is limiting the performance. Being used to Node, I was flabbergasted when writing C for Linux* . The file system commands just leave my thread hanging while the result is being generated, if I use it on a network drive it might hang for a minute before timing out, so I have to make a tread for each file system command, solely so that it can stall without bringing down the whole application. * I have no delusions that Windows is any better, Linux is just what I have first hand experience with.
- cwp 7y ago
  It happens that Windows is better. It has much better kernel support for async IO.
- unlinked_dll 7y ago
  >a naively coded server in most languages can handle relatively few requests per thread, so you scale up the number of threads to something like 100 per core, and now the overhead of running and switching between these treads is limiting the performance. This would be true if you hired someone to write a server in C about 15 years ago. It's not true today. And I hope you're not putting a naively coded server like that in production, or at least doing the hour of research once you notice it's awfully slow to solve the problem. Like if you wrote your backend in Go, Rust, Java or any number of languages (even C/C++ with common dependencies!) and did a little reading while you designed it, this issue wouldn't exist.
DanHulton 7y ago
You're handily skipping over 15 years of improvement and iteration between the last and second-last points there.

a13n 7y ago

> safety and correctness features You can achieve safety and correctness features for node via good lint rules and typescript/flow.

Nextgrid 7y ago
You can technically achieve all of that even in an obscure language like Brainfuck but it doesn’t mean it’s a good idea. Why would you duct tape hacks on top of hacks to achieve the result you want instead of using a language that has already all of the functionality built-in?
- kevstev 7y ago
  I can appreciate your suspicion of a dynamically typed language, but type script isn't really a "hack" its a superset of the language. I am not a huge fan of it, to me its kind of the worst of both worlds- if you need type safety, use something else. However, if you haven't taken a look at Joi- https://hapi.dev/family/joi/?v=16.1.8 https://hapi.dev/family/joi/?v=16.1.8 you probably should. To me this was a very happy medium- you can enforce your "types" at the point of ingress and egress at your api, and even do validations there, while still being able to enjoy the flexibility of type safety within your code.
  - rclayton 7y ago
    Lol - I don’t think I’ve ever met someone as enamored with Joi as I am. I think I abuse it in all the same ways as you do. I even publish my Joi schemas for request validation with my API specifications.
- bfrydl 7y ago
  I find it really annoying how JavaScript is treated on developer forums like this one. Why is it that when the same exact things are done in a typical language it's called “tooling” but in JavaScript it's “duct tape hacks on top of hacks”? Don't get me wrong, my favorite language is Rust, but pretending the JavaScript ecosystem is unusable doesn't make you cool. I can be extremely productive in TypeScript.
  - Nextgrid 7y ago
    JavaScript itself is not a hack, it works fine for its original purpose - making webpages interactive. What I consider hacks is the hundreds of different tools and dialects of JS that are used to bend JS into doing something it wasn’t really designed for. JS is fine in the browser, and TypeScript is also fine there because you don’t really have the choice to run anything different (although Web Assembly might change the game soon). But on the backend you have the privilege to pick between dozens of different languages that are better suited to the task and support the features you want out of the box without layering hacks on top. Why not just go with one of those?
- a13n 7y ago
  We chose JavaScript/Node because it's the language we knew best, and because we have to write it in the client as we're building a web-based service. No other language offers us that, unless it compiles to JavaScript (eg TypeScript). This grants us a magnitude of benefits, like being able to do server-side rendering and share code between the client and server. It also means we only need to hire people who can write JS, instead of JS and something else. TypeScript, Flow, and ESLint aren't hacks. These are mature tools used by some of the largest, most sophisticated engineering teams in the world.

kevstev 7y ago

My background was doing low level C++ in HFT/Algorithmic trading for years, with a bit of Java interspersed, before doing this whole complete right turn of doing webdev in js for ecommerce. In C++, doing web stuff was very difficult, build times were long, JSON support existed but was awkward, iterating was just very difficult, even when you had your whole build/deploy setup going, there was still a lot of work if you wanted to build a CRUD app just marshalling and unmarshalling objects to the DB, etc. It was a drag at best and a real pain at worst. Java... was a bit better, and I don't really have a problem with Java as a language. Java programmers, however, seem to really delight in building architectural monuments and get paid by the abstraction. In every java system I have jumped into, you are always neck deep in xml, and massive object hierarchies and factory factories and its just like where the F is the code that actually does stuff?! I remember working on one project, where I was building the "engine" and another guy was building the web interface for it, and was using Spring when it was relatively new, and he happily declared "all I need to do is wire up my configs now, I am pretty much done." Narrator: A week later, he still wasn't done... and this wasn't unusual in my experience, in most java apps "config" was just as complex and problematic as code, and the mindset of "its just config" made config changes more likely to cause a production outage of some sort. Then, you take a look at node. You look at a getting started tutorial. Its javascript on the front, and on the back. The JSON in between is "native" and is convenient and easy to use, easy to read, lightweight, and just makes a lot of intuitive sense- especially when I had found myself neck deep in XML in previous jobs for the same tasks. I had a nice looking HTML5 web app running in a few minutes- my mind was blown. Then you take a look at the frameworks- express and hapi, and the vast module ecosystem- and how easy it was to build a simple CRUD website with leveldb, or mysql, or really an endless array of storage options. And people were using those options! It wasn't just the bog standard RDBMS being used every place, with your only real choice being mysql, postgres, or if you had money, Oracle. Building endpoints with routes in these frameworks made your code so easy to divide up along clear lines, and there just wasn't the endless miles of boilerplate/scaffold code, and ugly syntax and type systems to fight with and plan ahead of. Things Just Worked. Turning around a code change was a matter of seconds, not a minutes long build process- I had never felt so productive- and writing code was fun again! Deploys were easy, restarts were fast. Rollbacks, when necessary, were painless. There was a plugin/module for everything (too much in hindsight). Now, this was 6 years ago. Go was around, but still kind of a blip on the radar, Ruby/Python were probably the closest real contenders. Ruby had lost steam, I honestly took some cursory looks at it, but it didn't seem to have traction. Python, suffered from its single threadedness and GIL, and its popularity with the ML crowd- Flask and such existed, but was pretty rudimentary compared to what Express/Hapi were offering, and no one seemed that interested in those projects. I like Go a lot, and for a pure backend service, it might be my go-to today, as one of the original arguments for Node was "its the same language on the front and the back, no more delineation between FE and BE developers, anyone can jump in and fix the bugs!" Which, along the lines of my original comment, don't really work out in reality, at least not on larger systems. People drawn to FE work usually have never done real systems development and don't understand how things work under the hood- which isn't a problem, until one day it is and then its a huge one. The dynamic typing argument... is somewhat valid, but I found that enforcing api contracts with hapi/joi gave you the equivalence of type safety at your interface borders, while still giving you the flexibility of dynamic typing within your code. In fact, Joi went even farther than just type checking, it could check that your int was within range for the field, that your dates were formatted properly, etc... In mega large codebases, this will come back to bite you, but I found the plugin architecture of Hapi really discouraged that kind of crap from leaking in and it was easy to build truly modularized code. The performance ceilings aren't that different, and not that impactful, at least not until you get to FANG scale, and I mean literally only FANG scale. We were running a billion dollar business with on 8 fairly small VMs for the API layer, which handled all of the ecommerce transaction handling. I remember at one point we encountered a memory leak of some sort in node, and the instances were falling over and dying about once an hour, but restarting and recovering- this was causing a few % error rates to our customers. I was insistent that we get all hands on deck to figure this out ASAP, and our head of Ops type person said "kevstev, we can throw hardware at this problem to meet SLOs until you get it under control. Your monthly server costs are less than my studio apartment cost me per month in Jersey City 15 years ago." You just have to have a basic understanding of whats going on at an architectural level, something a few hours of doing the right reading and experimenting can get you if you have the proper background. The number of gotchas to avoid to get that performance were an order of magnitude, if not more, fewer than in a language like C++ (Which I feel has actually gotten so complicated and difficult to grok its become a parody of itself- and I say that as someone who used it and adored it for 15 years).

JMTQp8lwXL 7y ago

Most bugs encountered in production systems aren't type based issues. Types are more useful for developer productivity (e.g., intellisense) than any other purpose.

RSZC 7y ago
Previously worked at a Node company - at one point in an effort to improve code quality we ran statistics on errors we'd seen over the past [period - forget exactly]. Type errors were our most common source of error both by number of total errors and number of distinct errors. I'd love to see any data or case studies that claim the opposite if you have any.
- kevstev 7y ago
  Just curious- did you use Joi or anything similar to try to at least verify at some point that you had a valid object?
- JMTQp8lwXL 7y ago
  https://blog.acolyer.org/2017/09/19/to-type-or-not-to-type-quantifying-detectable-bugs-in-javascript/ https://blog.acolyer.org/2017/09/19/to-type-or-not-to-type-q... > Is a 15% reduction in bugs making it all the way through your development pipeline worth it to you?... https://www.reddit.com/r/typescript/comments/aofcik/38_of_bugs_at_airbnb_could_have_been_prevented_by/ https://www.reddit.com/r/typescript/comments/aofcik/38_of_bu... > 38% of bugs at Airbnb could have been prevented by TypeScript according to postmortem analysis I've never seen a number far outside of the 15-30% range. In my experience, most bugs are operator error. Developers didn't code for branching paths that should've been accounted for, etc. Personally, I'm a fan of TypeScript. Just don't expect to remove the majority of your bugs via its usage. The old "no silver bullet" adage.
  - bathtub365 7y ago
    How is JavaScript worth it if it typically adds 15-30% more bugs? That’s an enormous figure.

NohatCoder 7y ago

The real killer feature is async. Since a modern web request typically spend most of the time waiting for database calls, file system requests or similar, a naively coded server in most languages can handle relatively few requests per thread, so you scale up the number of threads to something like 100 per core, and now the overhead of running and switching between these treads is limiting the performance. Being used to Node, I was flabbergasted when writing C for Linux* . The file system commands just leave my thread hanging while the result is being generated, if I use it on a network drive it might hang for a minute before timing out, so I have to make a tread for each file system command, solely so that it can stall without bringing down the whole application. * I have no delusions that Windows is any better, Linux is just what I have first hand experience with.

cwp 7y ago
It happens that Windows is better. It has much better kernel support for async IO.
unlinked_dll 7y ago
>a naively coded server in most languages can handle relatively few requests per thread, so you scale up the number of threads to something like 100 per core, and now the overhead of running and switching between these treads is limiting the performance. This would be true if you hired someone to write a server in C about 15 years ago. It's not true today. And I hope you're not putting a naively coded server like that in production, or at least doing the hour of research once you notice it's awfully slow to solve the problem. Like if you wrote your backend in Go, Rust, Java or any number of languages (even C/C++ with common dependencies!) and did a little reading while you designed it, this issue wouldn't exist.

DanHulton 7y ago

You're handily skipping over 15 years of improvement and iteration between the last and second-last points there.