3 ms·
"Unfortunately, generators and iterators are synchronous. Unless we’re willing to read our entire file into memory, they can’t help us." This doesn't seem to f
by jerf 9y ago
"Unfortunately, generators and iterators are synchronous. Unless we’re willing to read our entire file into memory, they can’t help us."
This doesn't seem to follow to me. Synchronous reading means you have to read a given chunk in before you can process it, which means that if you're not reading out of cache that you're going to stall on the spinning disk, but synchronous code doesn't require you to read the whole file at once. And you can easily write something that will process it into nice lines for you. The "standard library" may not do that because there isn't much standard library for Javascript, but it's not that hard.
The proof is, that's what the Python snippet is doing. It's built into Python in this case, and I would expect it does some magic internal stuff to accelerate this common case (in particular doing some scanning at the C level to see how long of a Python string to allocate once, instead of using Python-level string manipulation), but the API ought to be one you can do in Node, even synchronously, with generators.
- abritinthebay 9y agoThank you. This is not Promises being slow (though they do have inherent overhead) this is just plain bad code! I don't know much about the author, Dan, but this should be able to be written to be a LOT faster. Perhaps not as fast as the synchronous example (reading something into memory then processing it is always going to be very fast if you can do it) but faster than the code example they give.
- Jare 9y agoThe article is correct that promises and async iterators are not a good idea for tight innerloops. The author just doesn't seem to realize that it is possible to read parts of a file synchronously, and thus build a sync iterator that way. fs.readSync() should do it: https://nodejs.org/api/fs.html#fs_fs_readsync_fd_buffer_offset_length_position https://nodejs.org/api/fs.html#fs_fs_readsync_fd_buffer_offs... (I assume there are 100s of npm packages that do this already)
- abritinthebay 9y agoRight, exactly. Very well put :)
- hetman 9y agoI think you've misunderstood the author's point. He would like a fast solution that also looks as simple as the Python example. Even if you have a synchronous file read function that's doing partial file reads, you can not use it inside a synchronous loop to process the returned data because it will hold up the entire Node.js event loop until the whole file has been processed. You can use the synchronous file read function in a way that doesn't hold up the main event loop but to do that you re-introduce all the complexity the author was trying to avoid.
- Jare 9y agoBut you CAN make a loop that looks as simple as the Python example; Python just happens to provide one built in, right? The Python code holds the thread for the entire duration just like the Node one will hold the event loop. Wouldn't the Python loop cause the same problems as the node one in an evented app, say, a Twister app? (sorry my Python is a few years rusty) The ideal would be just as performant as the sync Python code, but effectively async. And his observation that async iterators by design can't do it without severe overhead for such tight inner loops is very good.
- Arnavion 9y ago>Synchronous reading means you have to read a given chunk in before you can process it, which means that if you're not reading out of cache that you're going to stall on the spinning disk, but synchronous code doesn't require you to read the whole file at once. It's implicit that stalling the node.js event loop is not allowed, so a synchronous "line" generator or iterator would block the event loop when it has exhausted the data read so far. The node.js event loop cannnot suspend the current task and process other tasks while waiting for more data to be read. Edit: The point of the author's post is that async iterators generate a Promise (and thus a yield to the event loop) for every element, even when the element is already available synchronously. Even if the generator has read three lines in the latest call to read(), it must still yield three promises of one line each (each involving one yield back to the event loop) before all three lines are processed. Edit 2: That said, it's not hard to fix this, and the author actually comes close to the solution but doesn't realize it. They say the result of the async iterator in the spec should contain an array of values, but it's just as easy to write the "line" iterator to yield an array of lines as its values without changing the spec.
- hetman 9y agoI think the problem with yielding an array of lines is that all the code that uses the result of the function would have to be aware of this grouping of results all the way up the stack. What he was proposing was a solution that looked like you were only returning one value per iteration (so none of the code doing the processing had to be aware there are arrays of values) but underneath the covers was actually being processed in batches on each run through the event loop. It's actually a very elegant solution. The only code that would have to be aware that it's actually dealing with batches of values is the code producing those batches right at the top of the stack of calls.
- Arnavion 9y agoThat's correct, and I understand the author's solution. My point was that changing the spec is not a hard requirement, since it can be accomplished in user code.