5 ms·
Node.js, Pipes, and Disappearing Bytes
- moralestapia 2y ago??? Wait, so what's the solution?
- simonbw 2y agoMy understanding is that it's "don't call system.exit() until you have finished writing everything to system.stdout".
- gcommer 2y agoEven better: never use process.exit(). Set process.exitCode instead. https://nodejs.org/api/process.html#processexitcode https://nodejs.org/api/process.html#processexitcode
- o11c 2y agoOr just throw an error. See also `process.on('exit', ...)`; you can conditionally set process.exitCode in it (only if there was no other failure) if you want.
- 3np 2y agoUse Stream.finished. $ node -e "const { Stream } = require('stream'); s = process.stdout; s.write('@'.repeat(128 * 1024)); Stream.finished(s, () => { process.exit(0); })" | wc -c 131072 https://nodejs.org/api/stream.html#streamfinishedstream-options-callback https://nodejs.org/api/stream.html#streamfinishedstream-opti...
- ksr 2y agoI use fs.writeFileSync: $ node -e "fs.writeFileSync(1, Buffer.from('@'.repeat(128 * 1024))); process.exit(0);" | wc -c 131072
- 3np 2y agoIt doesn't work for all streams but for the stdout/stderr case where they can be treated like file descriptors, I prefer this simpler approach over fiddling with the Streams API. (If blocking the process on writing synchronously to stdout is indeed what you actually want...)
- jitl 2y agoHere's how I solved this problem in Notion's internal command line tools: function flushWritableStream(stream: NodeJS.WritableStream) { return new Promise(resolve => stream.write("", resolve)).catch( handleFlushError, ) } /** * In NodeJS, process.stdout and process.stderr behave inconsistently depending on the type * of file they are connected to. * * When connected to unix pipes, these streams are *async*, so we need to wait for them to be flushed * before we exit, otherwise we get truncated results when using a Unix pipe. * * @see https://nodejs.org/api/process.html#process_a_note_on_process_i_o */ export async function flushStdoutAndStderr() { await Promise.all([ flushWritableStream(process.stdout), flushWritableStream(process.stderr), ]) } /** * If `module` is the NodeJS entrypoint: * * Wait for `main` to finish, then exit 0. * Note that this does not wait for the event loop to drain; * it is suited to commands that run to completion. * * For processes that must outlive `main`, see `startIfMain`. */ if (require.main === module) { await main(argv) await flushStdoutAndStderr() setTimeout(() => process.exit(0)) }
- 3np 2y agoHm, that does not solve the issue in TFA and I'm not sure it consistently works like you intend: $ node -e "process.stdout.write('@'.repeat(128 * 1024)); process.stdout.write(''); process.exit(0);" | wc -c 65536 You want to use `Stream.finished`: $ node -e "const { Stream } = require('stream'); s = process.stdout; s.write('@'.repeat(128 * 1024)); Stream.finished(s, () => { process.exit(0); })" | wc -c 131072 https://nodejs.org/api/stream.html#streamfinishedstream-options-callback https://nodejs.org/api/stream.html#streamfinishedstream-opti... If this helps you, consider bothering your infra team to look at opening access back up for tor IPs (;
- ecedeno 2y ago> node -e "process.stdout.write('@'.repeat(128 * 1024)); process.stdout.write(''); process.exit(0);" | wc -c That's missing the part where it waits for the last write to flush before exiting.
- hipadev23 2y agoI’m confused. If process.stdout.write() returns false when the pipe is full, do you not need to loop and call it again or something analogous? Or does it continue operating on the write in the background and that’s why waiting for the .finished() event works? Is there a reason it doesn’t use standard nodejs promise semantics (await process.stdout.write)? So probably best solution is util.promisify()?
- benatkin 2y agoThis is clickbait. The process exiting without unflushed output doesn't mean disappearing bytes. The bytes were there but the program left without them.
- richbell 2y agoYes, that is the conclusion of the investigation. It does not make this clickbait.
- benatkin 2y agoI disagree, which is why I left my comment. The bytes didn't disappear in a way that justifies the headline on HN IMHO because there was a very clear explanation of where they went.
- pyre 2y agoThe title is written from the perspective of the original problem, not from the solution. It then leads the user through the process of discovering why the bytes were 'disappearing.' From the perspective of the original users the bytes were disappearing.
- ptx 2y agoSince the writer never wrote the remaining bytes to the pipe, but only to its own memory buffer, and that memory buffer is destroyed when writer exits, "disappearing bytes" seems accurate. The bytes buffered in the memory of the writer process disappear along with the process when it exits.
- lexicality 2y agoclickbait would be "POSIX pipes corrupt JSON data? You won't believe this footgun NodeJS intentionally points at linux users!" this title is simply a poetic description of the initial bug report.
- fovc 2y agoPOSIX is weird, but NodeJS streams are designed to be misused
- molsson 2y agoprocess.stdout._handle.setBlocking(true) ...is a bit brutal but works. Draining the stream before exiting also kind of works but there are cases where drain will just permanently block. async function drain(stream) { return new Promise((resolve) => stream.on('drain', resolve)) }
- yladiz 2y agoWhich cases will it permanently block?
- jitl 2y agoThe writable stream will only emit 'drain' if the buffer fills past the limit. In that case, a prior call to `writable.write(...)` would return `false` indicating you should wait for drain before calling write again. Even if your code can't access the return value for the last writable.write call, you can check `if (writable.writableNeedDrain) { ...` to decide to wait for drain. This program will run forever, since we never write to stdout, stdout never "drains": setInterval(() => console.error("stderr: ", new Date().toISOString()), 5_000) process.stdout.once("drain", () => { console.error("stderr: stdout.once.drain, exit") process.exit(0) })
- jitl 2y agoWhen you want to await a single instance of a Node EventEmitter, please use `stream.once('drain', (err) => ...)` so you don't leak your listener callback after the promise resolves.
- arctek 2y agofsync doesn't work here right because unix pipes are in memory? I've had luck elsewhere with nodejs and WriteableStreams that refuse to flush their buffers before a process.exit() using fsync on the underlying file descriptors.
- Joker_vD 2y agoWell, yes: EINVAL fd is bound to a special file (e.g., a pipe, FIFO, or socket) which does not support synchronization. Or as POSIX puts it, [EINVAL] The fildes argument does not refer to a file on which this operation is possible.
- userbinator 2y agoWhen I read the title and the first sentence I immediately thought of partial writes, one of those things that a lot of people seem to ignore until things stop working, often intermittently. I don't work with Node.js, but I've had to fix plenty of network code that had the same bug of not handling partial writes correctly. I thought gpt-4o and o1-preview would be able to do this pretty easily, but surprisingly not. You're surprised that AI doesn't work? I'm not.
- mgoetzke 2y agoi never found an AI system which could solve any relevant programming problem. It is good at reading docs and giving a 0-1 start point, but most of everything else is not well understood by them yet.
- Joel_Mckay 2y ago"I don't work with Node.js" Consider yourself blessed, as it is the worst designed code ecosystem I've had stinking up the infrastructure. It is worse than VB in many ways. "You're surprised that AI doesn't work? I'm not." In general, most productivity studies quietly hide the fact ML LLM make zero impact on experienced developers performance. However, if the algorithm is well formed arbitrary nonsense, then an LLM can generate plausible sounding slop all day long. Best of luck =3
- namaria 2y agoFew decisions I took make me happier than steering clear of the whole thing in my early days of learning code. What ticked me off first was the silent failures but I keep getting confirmation I did the right thing.
- pdr94 2y agoreat investigation! This highlights a crucial aspect of Node.js's asynchronous nature when dealing with pipes. It's a reminder that we should always be mindful of how Node handles I/O operations differently based on the output destination. The key takeaway is the behavior difference between synchronous (files/TTYs) and asynchronous (pipes) writes in Node.js. This explains why `process.exit()` can lead to truncated output when piping. For those facing similar issues, remember to handle the `drain` event or use a more robust streaming approach to ensure all data is written before exiting. This post is a valuable resource for debugging similar "mysterious" pipe behavior in Node.js applications.