4 ms·
Here's how I solved this problem in Notion's internal command line tools: function flushWritableStream(stream: NodeJS.WritableStream) { return new Pr
by jitl 2y ago
Here's how I solved this problem in Notion's internal command line tools:
function flushWritableStream(stream: NodeJS.WritableStream) {
return new Promise(resolve => stream.write("", resolve)).catch(
handleFlushError,
)
}
/**
* In NodeJS, process.stdout and process.stderr behave inconsistently depending on the type
* of file they are connected to.
*
* When connected to unix pipes, these streams are *async*, so we need to wait for them to be flushed
* before we exit, otherwise we get truncated results when using a Unix pipe.
*
* @see https://nodejs.org/api/process.html#process_a_note_on_process_i_o
*/
export async function flushStdoutAndStderr() {
await Promise.all([
flushWritableStream(process.stdout),
flushWritableStream(process.stderr),
])
}
/**
* If `module` is the NodeJS entrypoint:
*
* Wait for `main` to finish, then exit 0.
* Note that this does not wait for the event loop to drain;
* it is suited to commands that run to completion.
*
* For processes that must outlive `main`, see `startIfMain`.
*/
if (require.main === module) {
await main(argv)
await flushStdoutAndStderr()
setTimeout(() => process.exit(0))
}
- 3np 2y agoHm, that does not solve the issue in TFA and I'm not sure it consistently works like you intend: $ node -e "process.stdout.write('@'.repeat(128 * 1024)); process.stdout.write(''); process.exit(0);" | wc -c 65536 You want to use `Stream.finished`: $ node -e "const { Stream } = require('stream'); s = process.stdout; s.write('@'.repeat(128 * 1024)); Stream.finished(s, () => { process.exit(0); })" | wc -c 131072 https://nodejs.org/api/stream.html#streamfinishedstream-options-callback https://nodejs.org/api/stream.html#streamfinishedstream-opti... If this helps you, consider bothering your infra team to look at opening access back up for tor IPs (;
- ecedeno 2y ago> node -e "process.stdout.write('@'.repeat(128 * 1024)); process.stdout.write(''); process.exit(0);" | wc -c That's missing the part where it waits for the last write to flush before exiting.
- 3np 2y agoYou might think that would be it? $ node -e "process.stdout.write('@'.repeat(128 * 1024)); process.stdout.write(''); setTimeout(()=>{ process.exit(0); }, 0);" | wc -c 131072 Sure. But not so fast! You're still just racing and have no guarantees. Increase the pressure and it snaps back: $ node -e "process.stdout.write('@'.repeat(1280 * 1024)); process.stdout.write(''); setTimeout(()=>{ process.exit(0); }, 0);" | wc -c 65536 Meanwhile: $ node -e "const { Stream } = require('stream'); s = process.stdout; s.write('@'.repeat(1280 * 1024)); Stream.finished(s, () => { process.exit(0); })" | wc -c 1310720
- deleted 2y ago[deleted]
- Thorrez 2y agoYou're not passing anything into the 2nd parameter of stdout.write()
- jitl 2y agoYour one liner doesn’t do the same thing as the code I posted. You missed passing a callback to stream.write and waiting for the callback before exiting the process. Once we await that callback to resolve, then the buffered data from before that point is known to be flushed. I don’t want to wait for a “finished” event because I want my program to terminate once my main() function and a flush after main() is complete. If I wait for finish, a background “thread” can extend the process lifetime by continuing to log. This is also why we don’t set process.exitCode and wait for the event loop to drain and Node to auto-exit. That might be the “right way”, but if someone leaves a lingering setTimeout(…, 10_MINUTES) you’ll have a bad time. It’s also more clear to reason about. Our rule: if you want the process to wait for your async work to complete, then `await` it.
- 3np 2y ago> Once we await that callback to resolve, then the buffered data from before that point is known to be flushed. This is not true IME. Just more likely. See my comment here: https://news.ycombinator.com/item?id=41829905 https://news.ycombinator.com/item?id=41829905 > If I wait for finish, a background “thread” can extend the process lifetime by continuing to log. Can you not address this by calling .end() on the stream? https://nodejs.org/api/stream.html#writableendchunk-encoding-callback https://nodejs.org/api/stream.html#writableendchunk-encoding... > Our rule: if you want the process to wait for your async work to complete, then `await` it. In general I very much agree. In this specific case, though, the awaits are just buying you a bit more time to race the buffer which is why they appear to help. The "flush" will not wait for completion before exiting the process. Maybe also worth keeping in mind that even if this would be working, stuff like this has historically changed breakingly a few times between major Node releases. If you're relying on unspecified behavior you're obviously in darker waters.
- throwitaway1123 2y ago> See my comment here: https://news.ycombinator.com/item?id=41829905 https://news.ycombinator.com/item?id=41829905 Your comment isn't equivalent to the original code. Your one liner is doing this: `process.stdout.write('')` Jitl's example is doing this: `new Promise(resolve => stream.write("", resolve))` He's passing in a promise resolver as the callback to stream.write (this is basically a `util.promisify` version of the writable.write chunk callback). If the data is written in order (which it should be), then I don't see how the `stream.write` promise could resolve before prior data is flushed. The documentation says: "The writable.write() method writes some data to the stream, and calls the supplied callback once the data has been fully handled". [1] [1] https://nodejs.org/api/stream.html#writablewritechunk-encoding-callback https://nodejs.org/api/stream.html#writablewritechunk-encodi...
- maple3142 2y agoI think this is what @jitl means: node -e "process.stdout.write('@'.repeat(128 * 1024)); process.stdout.write('',()=>process.exit(0)); " | wc -c It writes an empty string and use its callback to detect if it has been flushed, which means previous writes are also flushed.