4 ms·
>> What's also interesting is that node crashes after about a minute > I believe this is because `while(1)` runs so fast that there is no "idle" time for V8 to
by wolfgang42 4y ago
>> What's also interesting is that node crashes after about a minute
> I believe this is because `while(1)` runs so fast that there is no "idle" time for V8 to actually run GC. V8 is a strange beast, and this is just a guess of mine.
Not exactly: the GC is still running; it’s live memory that’s growing unbounded.
What’s going on here is that WritableStream is non-blocking; it has advisory backpressure, but if you ignore that it will do its best to accept writes anyway and keep them in a buffer until it can actually write them out. Since you’re not giving it any breathing room, that buffer just keeps growing until there’s no more memory left. `process.nextTick()` is presumably slowing things down enough on your system to give it a chance to drain the buffer. (I see there’s some discussion below about this changing by version; I’d guess that’s an artifact of other optimizations and such.)
To do this properly, you need to listen to the return value from `.write()` and, if it returns false, back off until the stream drains and there’s room in the buffer again.
Here’s the (not particularly optimized) function I use to do that:
async function writestream(chunks, stream) {
for await (const chunk of chunks) {
if (!stream.write(chunk)) {
// When write returns null, stream is starting to buffer and we need to wait for it to drain
// (otherwise we'll run out of memory!)
await new Promise(resolve => stream.once('drain', () => resolve()))
}
}
}
I do wish Node made it more obvious what was going on in this situation; this is a very common mistake with streams and it’s easy to not notice until things suddenly go very wrong.
ETA: I should probably note that transform streams, `readable.pipe()`, `stream.pipeline()`, and the like all handle this stuff automatically. Here’s a one-liner, though it’s not especially fast:
node -e 'const {Readable} = require("stream"); Readable.from(function*(){while(1) yield "1"}()).pipe(process.stdout)' | pv > /dev/null
- Matthias247 4y agoAre there still no async write functions which handle this easier than the old event based mechanism? Waiting for drain also sounds like it might reduce throughout since then there is 0 buffered data and the peer would be forced t Öl pause reading. A „writable“ event sounds more appropriate - but the node docs don’t mention one.