4 ms·

> Javascript is slowest at around 200KiB/s: I get around 1.56MiB/s with that code. PHP gets 4.04MiB/s. Python gets 4.35MiB/s. > What's also interesting is th

> Javascript is slowest at around 200KiB/s: I get around 1.56MiB/s with that code. PHP gets 4.04MiB/s. Python gets 4.35MiB/s. > What's also interesting is that node crashes after about a minute I believe this is because `while(1)` runs so fast that there is no "idle" time for V8 to actually run GC. V8 is a strange beast, and this is just a guess of mine. The following code shouldn't crash, give it a try: node -e 'function write() {process.stdout.write("1"); process.nextTick(write)} write()' | pv > /dev/null It's slower for me though, giving me 1.18MiB/s. More examples with Babashka and Clojure: bb -e "(while true (print \"1\"))" | pv > /dev/null 513KiB/s clj -e "(while true (print \"1\"))" | pv > /dev/null 3.02MiB/s clj -e "(require '[clojure.java.io :refer [copy]]) (while true (copy \"1\" *out*))" | pv > /dev/null 3.53MiB/s clj -e "(while true (.println System/out \"1\"))" | pv > /dev/null 5.06MiB/s Versions: PHP 8.1.6, Python 3.10.4, NodeJS v18.3.0, Babashka v0.8.1, Clojure 1.11.1.1105

mg 4y ago
Your node version indeed did not crash. Tried for 2 minutes. But using a longer string crashed after 23s here: node -e 'function write() {process.stdout.write("1111111111222222222233333333334444444444555555555566666666667777777777888888888899999999990000000000"); process.nextTick(write)} write()' | pv > /dev/null
- capableweb 4y ago
  Hm, strange. With the same out of memory error as before or a different one? Tried running that one for 2 minutes, no errors here, and memory stays constant. Also, what NodeJS version are you on?
  - mg 4y ago
    Yes, same error as before. Memory usage stays the same for a while, then starts to skyrocket shortly before it crashes. node is v10.24.0. (Default from the Debian 10 repo)
    - capableweb 4y ago
      Huh yeah, seems to be a old memory leak. Running it on v10.24.0 crashes for me too. After some quick testing in a couple of versions, it seems like it got fixed in v11 at least (didn't test any minor/patch versions). By the way, all versions up to NodeJS 12 (LTS) are "end of life", and should probably not be used if you're downloading 3rd party dependencies, as there are bunch of security fixes since then, that are not being backported.
      - captn3m0 4y ago
        I used this exact issue today while pointing out how Debian support dates can be misleading as packages themselves aren’t always getting fixes: https://github.com/endoflife-date/endoflife.date/issues/763#issuecomment-1144494317 https://github.com/endoflife-date/endoflife.date/issues/763#...
marginalia_nu 4y ago
> I believe this is because `while(1)` runs so fast that there is no "idle" time for V8 to actually run GC. V8 is a strange beast, and this is just a guess of mine. Java has (had) weird idiosyncrasies like this as well, well it doesn't crash, but depending on the construct you can get performance degradations depending on how the language inserts safepoints (where the VM is at a knowable state and a thread can be safely paused for GC or whatever). I don't know if this holds today, but I know there was a time where you basically wanted to avoid looping over long-type variables, as they had different semantics. The details are a bit fuzzy to me right now.
wolfgang42 4y ago
>> What's also interesting is that node crashes after about a minute > I believe this is because `while(1)` runs so fast that there is no "idle" time for V8 to actually run GC. V8 is a strange beast, and this is just a guess of mine. Not exactly: the GC is still running; it’s live memory that’s growing unbounded. What’s going on here is that WritableStream is non-blocking; it has advisory backpressure, but if you ignore that it will do its best to accept writes anyway and keep them in a buffer until it can actually write them out. Since you’re not giving it any breathing room, that buffer just keeps growing until there’s no more memory left. `process.nextTick()` is presumably slowing things down enough on your system to give it a chance to drain the buffer. (I see there’s some discussion below about this changing by version; I’d guess that’s an artifact of other optimizations and such.) To do this properly, you need to listen to the return value from `.write()` and, if it returns false, back off until the stream drains and there’s room in the buffer again. Here’s the (not particularly optimized) function I use to do that: async function writestream(chunks, stream) { for await (const chunk of chunks) { if (!stream.write(chunk)) { // When write returns null, stream is starting to buffer and we need to wait for it to drain // (otherwise we'll run out of memory!) await new Promise(resolve => stream.once('drain', () => resolve())) } } } I do wish Node made it more obvious what was going on in this situation; this is a very common mistake with streams and it’s easy to not notice until things suddenly go very wrong. ETA: I should probably note that transform streams, `readable.pipe()`, `stream.pipeline()`, and the like all handle this stuff automatically. Here’s a one-liner, though it’s not especially fast: node -e 'const {Readable} = require("stream"); Readable.from(function*(){while(1) yield "1"}()).pipe(process.stdout)' | pv > /dev/null
- Matthias247 4y ago
  Are there still no async write functions which handle this easier than the old event based mechanism? Waiting for drain also sounds like it might reduce throughout since then there is 0 buffered data and the peer would be forced t Öl pause reading. A „writable“ event sounds more appropriate - but the node docs don’t mention one.

mg 4y ago

Your node version indeed did not crash. Tried for 2 minutes. But using a longer string crashed after 23s here: node -e 'function write() {process.stdout.write("1111111111222222222233333333334444444444555555555566666666667777777777888888888899999999990000000000"); process.nextTick(write)} write()' | pv > /dev/null

capableweb 4y ago
Hm, strange. With the same out of memory error as before or a different one? Tried running that one for 2 minutes, no errors here, and memory stays constant. Also, what NodeJS version are you on?
- mg 4y ago
  Yes, same error as before. Memory usage stays the same for a while, then starts to skyrocket shortly before it crashes. node is v10.24.0. (Default from the Debian 10 repo)
  - capableweb 4y ago
    Huh yeah, seems to be a old memory leak. Running it on v10.24.0 crashes for me too. After some quick testing in a couple of versions, it seems like it got fixed in v11 at least (didn't test any minor/patch versions). By the way, all versions up to NodeJS 12 (LTS) are "end of life", and should probably not be used if you're downloading 3rd party dependencies, as there are bunch of security fixes since then, that are not being backported.
    - captn3m0 4y ago
      I used this exact issue today while pointing out how Debian support dates can be misleading as packages themselves aren’t always getting fixes: https://github.com/endoflife-date/endoflife.date/issues/763#issuecomment-1144494317 https://github.com/endoflife-date/endoflife.date/issues/763#...

marginalia_nu 4y ago

> I believe this is because `while(1)` runs so fast that there is no "idle" time for V8 to actually run GC. V8 is a strange beast, and this is just a guess of mine. Java has (had) weird idiosyncrasies like this as well, well it doesn't crash, but depending on the construct you can get performance degradations depending on how the language inserts safepoints (where the VM is at a knowable state and a thread can be safely paused for GC or whatever). I don't know if this holds today, but I know there was a time where you basically wanted to avoid looping over long-type variables, as they had different semantics. The details are a bit fuzzy to me right now.

wolfgang42 4y ago

>> What's also interesting is that node crashes after about a minute > I believe this is because `while(1)` runs so fast that there is no "idle" time for V8 to actually run GC. V8 is a strange beast, and this is just a guess of mine. Not exactly: the GC is still running; it’s live memory that’s growing unbounded. What’s going on here is that WritableStream is non-blocking; it has advisory backpressure, but if you ignore that it will do its best to accept writes anyway and keep them in a buffer until it can actually write them out. Since you’re not giving it any breathing room, that buffer just keeps growing until there’s no more memory left. `process.nextTick()` is presumably slowing things down enough on your system to give it a chance to drain the buffer. (I see there’s some discussion below about this changing by version; I’d guess that’s an artifact of other optimizations and such.) To do this properly, you need to listen to the return value from `.write()` and, if it returns false, back off until the stream drains and there’s room in the buffer again. Here’s the (not particularly optimized) function I use to do that: async function writestream(chunks, stream) { for await (const chunk of chunks) { if (!stream.write(chunk)) { // When write returns null, stream is starting to buffer and we need to wait for it to drain // (otherwise we'll run out of memory!) await new Promise(resolve => stream.once('drain', () => resolve())) } } } I do wish Node made it more obvious what was going on in this situation; this is a very common mistake with streams and it’s easy to not notice until things suddenly go very wrong. ETA: I should probably note that transform streams, `readable.pipe()`, `stream.pipeline()`, and the like all handle this stuff automatically. Here’s a one-liner, though it’s not especially fast: node -e 'const {Readable} = require("stream"); Readable.from(function*(){while(1) yield "1"}()).pipe(process.stdout)' | pv > /dev/null

Matthias247 4y ago
Are there still no async write functions which handle this easier than the old event based mechanism? Waiting for drain also sounds like it might reduce throughout since then there is 0 buffered data and the peer would be forced t Öl pause reading. A „writable“ event sounds more appropriate - but the node docs don’t mention one.