8 ms·

A better streams API is possible for JavaScript

• user3939382 7mo ago
  “ The Streams Standard was developed between 2014 and 2016 with an ambitious goal to provide "APIs for creating, composing, and consuming streams of data that map efficiently to low-level I/O primitives." Before Web streams, the web platform had no standard way to work with streaming data.” This is what UDP is for. Everything actually has to be async all the way down and since it’s not, we’ll just completely reimplement the OS and network on top of itself and hey maybe when we’re done with that we can do it a third time to have the cloud of clouds. The entire stack we’re using right down to the hardware is not fit for purpose and we’re burning our talent and money building these ever more brittle towering abstractions.
  • afavour 7mo ago
    UDP is a protocol, not an API
    • mlhpdx 7mo ago
      True. But it’s also true that trying to shoehorn every use case into TCP streams is counter productive. A stream API can layer over UDP as well (reading in order of arrival with packet level framing), but such a stream would a bit weird and incompatible with many stream consumers (e.g. [de]compression). A UDP API is simpler and more naturally event (packet) oriented. The concepts don’t mix well. Still, it would be nice if they browser supported a UDP API instead of the weird and heavy DTLS and QUIC immitations.
      • drysart 7mo ago
        The browser does have a UDP data stream available for applications to send arbitrary bytes over UDP; it's part of WebRTC.
        • mlhpdx 7mo ago
          While Web RTC is built on UDP, it does not allow sending arbitrary UDP. It's DTLS, perhaps encapsulating SCTP, and DCEP.
      • kaoD 7mo ago
        TCP or UDP are orthogonal to this, so the original comment feels like a non sequitur. These streams are not network streams and could be a file, chunks of procedural audio, or whatever.
        • mlhpdx 7mo ago
          I agree, the stream concept should be (and is) very general and ideally cover all these cases - any “bytes producing” source. I was trying to be open minded about that and conceive a stream API over a UDP socket. It’d work IMHO, but be a little odd compared to an event-like API.
  • delaminator 7mo ago
    We're too busy building products while waiting for the perfect system to arrive.
    • user3939382 7mo ago
      I’m building everything from first principles, I’m not climbing the exponential curve with some billionaire that has to finance it.
      • delaminator 7mo ago
        I really doubt you are. you're not visiting the transistor shop every time you want to build a react component
        • user3939382 7mo ago
          Good thing your confidence is a soft requirement :)
• Feathercrown 7mo ago
  [flagged]
• kg 7mo ago
  It's a real shame that BYOB (bring your own buffer) reads are so complex and such a pain in the neck because for large reads they make a huge difference in terms of GC traffic (for allocating temporary buffers) and CPU time (for the copies). In an ideal world you could just ask the host to stream 100MB of stuff into a byte array or slice of the wasm heap. Alas.
  • amluto 7mo ago
    I wonder if you can get most of the benefit BYOB with a much simpler API: for await (const chunk of stream) { // process the chunk stream.returnChunk(chunk); } This would be entirely optional. If you don’t return the chunk and instead let GC free it, you get the normal behavior. If you do return it, then the stream is permitted to return it again later. (Lately I’ve been thinking that a really nice stream or receive API would return an object with a linear type so that you must consume it and possibly even return it. This would make it impossible to write code where task cancellation causes you to lose received data. Sadly, mainstream languages can’t do this directly.)
  • hrmtst93837 7mo ago
    [flagged]
• ai-christianson 7mo ago
  [flagged]
  • slowcache 7mo ago
    > high-performance data processing tools in JS I may be naive in asking this, but what leads someone to building high perf data tools in JS? JS doesn't seem to me like it would be the tool of choice for such things
    • thadt 7mo ago
      Browsers
      • speed_spread 7mo ago
        Since when are browsers themselves built in JavaScript? Mainstream, fast ones?
        • thadt 7mo ago
          Clarification - in the past when I've written high performance data tools in JS, it was almost entirely to support the use case of needing it to run in a browser. Otherwise, there are indeed more suitable environments available. To your question, I was about to point out Firefox[1], but realized you clarified 'mainstream'[2]... [1] https://briangrinstead.com/blog/firefox-webcomponents https://briangrinstead.com/blog/firefox-webcomponents [2] https://gs.statcounter.com/browser-market-share https://gs.statcounter.com/browser-market-share
        • deleted 7mo ago
          [deleted]
    • n_e 7mo ago
      I have a SaaS project where the backend is in JS. I also have some data processing to do with large file (several TB). Doing it is in JS is more convenient as I can reuse code from the backend, and it is also the language I know best. Performance-wise, I get about half the throughput I had with the same processsing done it rust, which doesn't change anything for my use-case. However that's not really relevant to the context of the post as I'm using node.js streams which are both saner and fast. I'm guessing that the post is relevant to people using server-side runtimes that only implement web streams.
• dilap 7mo ago
  > The problems aren't bugs; they're consequences of design decisions that may have made sense a decade ago, but don't align with how JavaScript developers write code today. > I'm not here to disparage the work that came before — I'm here to start a conversation about what can potentially come next. Terrible LLM-slop style. Is Mr Snell letting an LLM write the article for him or has he just appropriated the style?
  • lapcat 7mo ago
    You’ve got it backwards: LLMs were trained on human writing and appropriated our style.
    • have_faith 7mo ago
      Partially true. They've been trained and then aligned towards a preferred style. They don't use em-dashes because they are over-represented in the training material (majority of people don't use them).
      • lapcat 7mo ago
        It seems likely that with the written word, as with most things, a minority of people produce the majority of content. Most people publish relatively few words compared to professional writers. Possibly the LLM vendors could bias the models more toward nonprofessional content, but then the quality and utility of the output would suffer. Skip the scientific articles and books, focus on rando internet comments, and you’ll end up with a lot more crap than you already get.
    • GoblinSlayer 7mo ago
      They converge...
  • jitl 7mo ago
    cloudflare does seem to love ai written everything
  • azangru 7mo ago
• shevy-java 7mo ago
  We deserve a better language than JavaScript. Sadly it will never happen. WebAssembly failed to keep some of its promises here.
  • postalrat 7mo ago
    Where can I find these not kept promises?
    • nindalf 7mo ago
      They haven't yet made languages other than JavaScript first-class languages for the web - https://hacks.mozilla.org/2026/02/making-webassembly-a-first-class-language-on-the-web/ https://hacks.mozilla.org/2026/02/making-webassembly-a-first.... I wouldn't call this a broken promise, but it was something people were hoping would take less than a decade.
  • gejose 7mo ago
    There's always a comment like this in most discussions about javascript.
  • krashidov 7mo ago
    > WebAssembly failed to keep some of its promises here classic case of not using an await before your promise
  • teaearlgraycold 7mo ago
    As wonky as JS is I really like it. Typescript has done such a good job at making it fun to use.
    • tprgoreturnon 7mo ago
      [dead]
• conartist6 7mo ago
  As it happens i have an even better API than this article proposes! They propose just using an async iterator of UInt8Array. I almost like this idea, but it's not quite all the way there. They propose this: type Stream<T> = { next(): Promise<{ done, value: UInt8Array<T> }> } I propose this, which I call a stream iterator! type Stream<T> = { next(): { done, value: T } | Promise<{ done, value: T }> } Obviously I'm gonna be biased, but I'm pretty sure my version is also objectively superior: - I can easily make mine from theirs - In theirs the conceptual "stream" is defined by an iterator of iterators, meaning you need a for loop of for loops to step through it. In mine it's just one iterator and it can be consumed with one for loop. - I'm not limited to having only streams of integers, they are - My way, if I define a sync transform over a sync input, the whole iteration can be sync making it possible to get and use the result in sync functions. This is huge as otherwise you have to write all the code twice: once with sync iterator and for loops and once with async iterators and for await loops. - The problem with thrashing Promises when splitting input up into words goes away. With async iterators, creating two words means creating two promises. With stream iterators if you have the data available there's no need for promises at all, you just yield it. - Stream iterators can help you manage concurrency, which is a huge thing that async iterators cannot do. Async iterators can't do this because if they see a promise they will always wait for it. That's the same as saying "if there is any concurrency, it will always be eliminated."
  • conartist6 7mo ago
    There's one more interesting consequence: you rid yourself of the feedback problem. To see the problem let's create a stream with feedback. Lets say we have an assembly line that produces muffins from ingredients, and the recipe says that every third muffin we produce must be mushed up and used as an ingredient for further muffins. This works OK until someone adds a final stage to the assembly line, which puts muffins in boxes of 12. Now the line gets completely stuck! It can't get a muffin to use on the start of the line because it hasn't made a full box of muffins yet, and it can't make a full box of muffins because it's starved for ingredients after 3. If we're mandated to clump the items together we're implicitly assuming that there's no feedback, yet there's also no reason that feedback shouldn't be a first-class ability of streams.
  • 7mo ago
• murmansk 7mo ago
  For gods sake, finally, somebody have said this!
• ralusek 7mo ago
  I tinkered with an alternative to stream interfaces: https://github.com/ralusek/streamie https://github.com/ralusek/streamie allows you to do things like infiniteRecords .map(item => doSomeAsyncThing(item), { concurrency: 5 }); And then because I found that I often want to switch between batching items vs dealing with single items: infiniteRecords .map(item => doSomeAsyncSingularThing(item), { concurrency: 5 }) .map(groupOf10 => doSomeBatchThing(groupsOf10), { batchSize: 10 }) // Can flatten back to single items .map(item => backToSingleItem(item), { flatten: true });
• animanoir 7mo ago
  [dead]
• z3t4 7mo ago
  I like Node.JS streams. It's very satisfying to rent a 250MB memory machine and let it process GB's of data using streams.
• bikeshaving 7mo ago
  A long time ago, I wrote an abstraction called a Repeater. Essentially, the idea behind it is, what would the Promise constructor look like if it was translated to async iterables. import { Repeater } from "@repeaterjs/repeater"; const keys = new Repeater(async (push, stop) => { const listener = (ev) => { if (ev.key === "Escape") { stop(); } else { push(ev.key); } }; window.addEventListener("keyup", listener); await stop; window.removeEventListener("keyup", listener); }); const konami = ["ArrowUp", "ArrowUp", "ArrowDown", "ArrowDown", "ArrowLeft", "ArrowRight", "ArrowLeft", "ArrowRight", "b", "a"]; (async function() { let i = 0; for await (const key of keys) { if (key === konami[i]) { i++; } else { i = 0; } if (i >= konami.length) { console.log("KONAMI!!!"); break; // removes the keyup listener } } })(); https://github.com/repeaterjs/repeater https://github.com/repeaterjs/repeater It’s one of those abstractions that’s feature complete and stable, and looking at NPM it’s apparently getting 6.5mil+ downloads a week for some reason. Lately I’ve just taken the opposite view of the author, which is that we should just use streams, especially with how embedded they are in the `fetch` proposals and whatever. But the tee critique is devastating, so maybe the author is right. It’s exciting to see people are still thinking about this. I do think async iterables as the default abstraction is the way to go.
  • boilerupnc 7mo ago
    Off topic - But just wanna say - Love the cheat code! 30 Lives added :-) Nostalgia runs deep with that code. So deep - in fact, that I sign many of my emails off with "Sent by hitting Up, Up, Down, Down, Left, Right, Left, Right, B, A"
    • sfink 7mo ago
      Off topic to the off topic, but that logic doesn't look right. It seems like if up is pressed, you might need to reset i to 1 or 2, not 0.
      • bikeshaving 7mo ago
        Not accepting PRs for this. I think the logic is sound (Easter Eggs should be difficult to trigger).
• paulddraper 7mo ago
  Just use AsyncIterator<UIntArray>. The objection is > The Web streams spec requires promise creation at numerous points — often in hot paths and often invisible to users. Each read() call doesn't just return a promise; internally, the implementation creates additional promises for queue management, pull() coordination, and backpressure signaling. But that's 95% manageable by altering buffer sizes. And as for that last 5%....what are you doing with JS to begin with?
• tracker1 7mo ago
  One minor niggle on freeing resources... I'm hoping it becomes more popular with libraries, but there's using/await using with disppse/disposeAsync which works similarly to C#'s use of using. I'm working on a db driver that uses it by convention as part of connection/pool usage cleanup.
• halfmatthalfcat 7mo ago
  The Observables spec should just get merged and implemented. https://github.com/tc39/proposal-observable https://github.com/tc39/proposal-observable
  • bakkoting 7mo ago
    Observables has moved to WHATWG [1] and been implemented in Chrome, although I don't know if the other browsers have expressed any interest (and there's still some issues [2] to be worked through). But Observables really do not solve the problems being talked about in this post. [1] https://github.com/WICG/observable https://github.com/WICG/observable [2] https://github.com/WICG/observable/issues/216 https://github.com/WICG/observable/issues/216
• spankalee 7mo ago
  Async iterables aren't necessarily a great solution either because of the exact same promise and stack switching overhead - it can be huge compared to sync iterables. If you're dealing with small objects at the production side, like individual tag names, attributes, bindings, etc. during SSR., the natural thing to do is to just write() each string. But then you see that performance is terrible compared to sync iterables, and you face a choice: 1. Buffer to produce larger chunks and less stack switching. This is the exact same thing you need to do with Streams. or 2. Use sync iterables and forgo being able to support async components. The article proposes sync streams to get around this some, but the problem is that in any traversal of data where some of the data might trigger an async operation, you don't necessarily know ahead of time if you need a sync or async stream or not. It's when you hit an async component that you need it. What you really want is a way for only the data that needs it to be async. We faced this problem in Lit-SSR and our solution was to move to sync iterables that can contain thunks. If the producer needs to do something async it sends a thunk, and if the consumer receives a thunk it must call and await the thunk before getting the next value. If the consumer doesn't even support async values (like in a sync renderToString() context) then it can throw if it receives one. This produced a 12-18x speedup in SSR benchmarks over components extracted from a real-world website. I don't think a Streams API could adopt such a fragile contract (ie, you call next() too soon it will break), but having some kind of way where a consumer can pull as many values as possible in one microtask and then await only if an async value is encountered would be really valuable, IMO. Something like `write()` and `writeAsync()`. The sad thing here is that generators are really the right shape for a lot of these streaming APIs that work over tree-like data, but generators are far too slow.
  • jauntywundrkind 7mo ago
    I liked conartist6's proposal, type Stream<T> = { next(): { done, value: T } | Promise<{ done, value: T }> } Where T=Uint8Array. Sync where possible, async where not. Engineers had a collective freak out panic back in 2013 over Do not unleash Zalgo, a worry about using callbacks with different activation patterns. Theres wisdom there, for callbacks especially; it's confusing if sometime the callback fires right away, sometimes is in fact async. https://blog.izs.me/2013/08/designing-apis-for-asynchrony/ https://blog.izs.me/2013/08/designing-apis-for-asynchrony/ And this sort of narrow specific control has been with us since. It's generally not cool to use MaybeAsync<T> = T | Promise<T>, for similar "it's better to be uniform" reasons. We've been so afraid of Zalgo for so long now. That fear just seems so overblown and it feels like it hurts us so much that we can't do nice fast things. And go async when we need to. Regarding the pulling multiple, it really depends doesn't it? It wouldn't be hard to make a utility function that lets you pull as many as you want queueing deferrables, allowing one at a time to flow. But I suspect at least some stream sources would be just fine yielding multiple results without waiting. They can internally wait for the previous promise, use that as a cursor. I wasn't aware that generators were far too slow. It feels like we are using the main bit of the generator interface here, which is good enough.
• matheus-rr 7mo ago
  The practical pain with Web Streams in Node.js is that they feel like they were designed for the browser use case first and backported to the server. Any time I need to process large files or pipe data between services, I end up fighting with the API instead of just getting work done. The async iterable approach makes so much more sense because it composes naturally with for-await-of and plays well with the rest of the async/await ecosystem. The current Web Streams API has this weird impedance mismatch where you end up wrapping everything in transform streams just to apply a simple operation. Node's original stream implementation had problems too, but at least `.pipe()` was intuitive. You could chain operations and reason about backpressure without reading a spec. The Web Streams spec feels like it was written by the kind of person who thinks the solution to a complex problem is always more abstraction.
  • zarzavat 7mo ago
    It's news to me that anyone actually uses the web streams in node. I thought they were just for interoperability, for code that needs to run on both client and server.
    • apitman 7mo ago
      You need to use them for things like Cloudflare and Denos HTTP servers, which is actually a fairly common (and nice) pattern: https://blog.val.town/blog/the-api-we-forgot-to-name/ https://blog.val.town/blog/the-api-we-forgot-to-name/
      • ale 7mo ago
        > Two years ago Cloudflare released an API for creating servers in JavaScript. Now every modern JavaScript cloud provider supports it. This is so ridiculously far from the truth lol. Every JS runtime after node has been championing web APIs and that’s how you get the fetch API’s Request/Response outside the browser.
• cogman10 7mo ago
  Seems pretty similar to the design of OKIO in java [1]. With pretty similar goals ultimately. Here's a presentation on the internal details and design decisions. [2] [1] https://github.com/square/okio https://github.com/square/okio [2] https://www.youtube.com/watch?v=Du7YXPAV1M8 https://www.youtube.com/watch?v=Du7YXPAV1M8
• notnullorvoid 7mo ago
  There's a lot I like about this API, mainly the pull-based iterator approach. I don't really see what the value of the sync APIs are though. What's the difference of just using iterators directly for sync streams?
  • jonkoops 7mo ago
    It avoids the overhead of Promises, so I can imagine that this would be quite useful if you know that blocking the thread is fine for a little while (e.g. in a worker).
    • notnullorvoid 7mo ago
      I mean the APIs like `Stream.pullSync` you could do that with a regular (non-async) iterator/generator.
• socalgal2 7mo ago
  Promises should not be a big overhead. If they are, that seems like a bug in JS engines. At a native level (C++/rust), a Promise is just a closure added to a list of callbacks for the event loop. Yes, if you did 1 per streamed byte then it would be huge but if you're doing 1 promise per megabyte, (1000 per gig), it really shouldn't add up 1% of perf.
  • conartist6 7mo ago
    I'm fairly sure it's not Promises that are actually the heavy part but the `await` keyword as used in the `for await` loop. That's because await tries to preserve the call stack for debugging, making it a relatively high-level expensive construct from a perf perspective where a promise is a relatively low-level cheap one. So if you're going to flatten everything into one stream then you can't have a for loop implementation that defensively awaits on every step, or else it'll be slooooooooow. That's my proposal for the change to the language is a syntax like for await? (value of stream) { } which would only do the expensive high-level await when the underlying protocol forced it to by returning a promise-valued step.
    • esprehn 7mo ago
      Async call stacks is an optional feature when the devtools is open. There shouldn't be overhead from await like that?
      • conartist6 7mo ago
        It's awfully hard to know, and I am not myself sure.
  • pornel 7mo ago
    In Rust, a Future can have only exactly one listener awaiting it, which means it doesn't need dynamic allocation and looping for an arbitrary number of .then() callbacks. This allows merging a chain of `.await`ed futures into a single state machine. You could get away with awaiting even on every byte.
• nottorp 7mo ago
  Well, it's also possible to replace JavaScript with a better language, it's just too late for it...
• adamnemecek 7mo ago
  It might be a good idea to look into the research on streams as coalgebras, there is quite a bit, for example here https://cs.ru.nl/~jrot/CTC20/ https://cs.ru.nl/~jrot/CTC20/. Coalgebras might seem too academic but so were monads at some point and now they are everywhere.
• rhodey 7mo ago
  the pull-stream module and its ecosystem is relevant here the idea is basically just use functions. no classes and very little statefulness https://www.npmjs.com/package/pull-stream https://www.npmjs.com/package/pull-stream
• bennettpompi1 7mo ago
  I really enjoyed reading this article however I can't help but feeling that if you need anything described within it probably shouldn't be writing JS in the first place
• szmarczak 7mo ago
  > This pattern has caused connection pool exhaustion in Node.js applications using undici (the fetch() implementation built into Node.js), and similar issues have appeared in other runtimes. That's an inherent flaw of garbage collected languages. Requiring to explicitly close a resource feels like writing C. Otherwise you have a memory leak or resource exhaustion, because the garbage collector may or may not free the resource. Even C++ is better at this, because it does reference counting instead.
• etler 7mo ago
  There are many use cases where having a value stream is very useful. I do agree having a separate simpler byte only stream would make sense though. I think the current capabilities of web streams should be kept and an IOStream could be added for optimizing byte streams. Ideally splitting out the use cases would allow both implementations to be simpler, but that ship has probably sailed.
• steve_adams_86 7mo ago
  I ran into a performance issues a few months ago where native streams were behaving terribly, and it seemed to be due to bad back-pressure implementation. I tried several implementations, tweaked settings, but ultimately couldn't get around it. In some cases I had bizarre drops in activity when the consumer was below capacity. It could have been related to the other issue they mention, which is the cost of using promises. My streams were initiating HEAPS of promises. The cost is immense when you're operating on a ton of data. Eventually I had to implement some complex logic to accomplish batching to reduce the number of promises, then figure out some clever concurrency strategies to manage backpressure more manually. It worked well. Once I was happy with what I had, I ported it from Deno to Go and the result was so stunningly different. The performance improvement was several orders of magnitude. I also built my custom/native solution using the Effect library, and although some people claim it's inefficient and slow, it out-performed mine by something like 15% off the shelf, with no fine-tuning or clever ideas. I wished I'd used it from the start. The difference is likely in that it uses a fiber-based model rather than promises at the execution layer, but I'm not sure.
• socketcluster 7mo ago
  https://socketcluster.io/ https://socketcluster.io/ has had such stream implementation and backpressure management since at least 2019. Here's the WritableConsumableStream module: https://github.com/SocketCluster/writable-consumable-stream https://github.com/SocketCluster/writable-consumable-stream SocketCluster solves the problem of maintaining message order with async processing. This feature is even more useful now with LLMs as you can process data live, transform streams with AI with no risk of mangling the message order. I may have been the first person to use a for-await-of loop in this way with backpressure. At least on an open source project.
• mhh__ 7mo ago
  I've only used them in dotnet, I would be very interested to read any strong opinions about the use of Streams both in practice and as an abstract point in API design.
  • austin-cheney 7mo ago
    Streams are how modern operating systems work, most commonly to transfer audio, video, file system, and network data from hardware to channels available for applications. So a common scenario is to stream data from a file and pipe it to a network interface for transfer to other computers or to a web browser.
• apatheticonion 7mo ago
  What's wrong with a `Read` `Write` interface like every other language? const buffer = new UInt8Array(256) const bytesRead = await reader.read(buffer) if (bytesRead === 0) { // Done return }
• sholladay 7mo ago
  As a maintainer on the Ky team, I give a big thumbs up to this proposal. We have run into many problems with web streams over the years and solving them has always proven to be hairy, including the unbounded memory growth from response.clone(). The Deno team implemented a stream API inspired by Go, which I was happy with, until they ultimately acquiesced to web streams. This proposal shares some of those principles as well.
• yumechan 7mo ago
  [dead]
• esprehn 7mo ago
  Web Streams do feel rather painful compared to other languages. The author ends up basically describing kotlin flows which are great and I wish the web would adopt that model (Observables wanted to be that but the API is much worse than flows in practice). Fwiw the original Streams API could have been simpler even without async iterators. interface Stream<T> { // Return false from the callback to stop early. // Result is if the stream was completed. forEach(callback: (chunk: T) => Promise<boolean | undefined>): Promise<boolean> } Similarly adding a recycleBuffer(chunk) method would have gone a long way towards BYOB without all the ceremony. If we're optimizing allocations we can also avoid all the {done,value} records and return a semaphore value for the end in the proposed API. (Web) API design is really difficult and without a voice in the room pushing really hard on ergonomics and simplicity it's easy to solve all the use cases but end up with lots of awkward corners and costs later.
• nateb2022 7mo ago
  Regarding the benchmarks, "Async iteration (8KB × 1000): ~530 GB/s vs ~35 GB/s": how do you achieve 530 GB/s throughput on an M1 Pro which has a 200GB/s memory bandwidth? The "~275 GB/s" figure for chained transforms has the same problem. I suspect the benchmarks, if not most of this project, suffer from poor quality control on vibecoded implementations.
• yonran 7mo ago
  I don’t know how ReadableStream.tee() got specified to backpressure when the faster branch is not consumed, since this is the opposite of what nodejs does when multiple Writables attached via Readable.pipe() and also the opposite of what the requirements document (https://github.com/whatwg/streams/blob/e9355ce79925947e8eb496563d599c329769d315/Requirements.md#you-must-be-able-to-pipe-a-stream-to-more-than-one-writable-stream https://github.com/whatwg/streams/blob/e9355ce79925947e8eb49...) says: “letting the speed of the slowest output determine the speed of the tee”. I like the idea of the more ergonomic, faster api in new-stream with no buffering except at Stream.push(). NodeJS and web streams put infinitely expandable queues at every ReadableStream and WritableStream so that you can synchronously res.write(chunk) as much as you want with abandon. This API basically forces you to use generators that yield instead of synchronously writing chunks.
• jenkings 7mo ago
  As I’m sure many have, I wrote a wrapper around AyncIterables so I could use them more succinctly. However I wasn’t concerned with performance as I was using it in a lambda handling small batches or scanning items from a DB and streaming pages back. https://github.com/juliantcook/fluent-async-iterator https://github.com/juliantcook/fluent-async-iterator I had hoped we would have a better API by now. It was also very useful for CLI tools utilising unix pipes.
• sillyboi 7mo ago
  The real win here isn’t just performance,it’s convergence. When ReadableStream behaves the same in the browser, Workers, and other runtimes, stream-based code becomes portable and predictable. That reduces subtle backpressure bugs and eliminates “works here but not there” edge cases. Standardization at the streams layer is a big deal for building reliable streaming systems across environments.
  • thehamkercat 7mo ago
    Yep, it isn't just x; it's y
• jnbridge 7mo ago
  The tension between "streams as lazy sequences" vs "streams as async event channels" isn't unique to JavaScript. Every major runtime has hit this wall: - Java went through it with java.util.stream (pull-based, lazy) vs Reactive Streams/Project Reactor (push-based, backpressure-aware). The result was two completely separate APIs that don't compose well. - .NET actually handled this better with IAsyncEnumerable<T> in C# 8 — a single abstraction that's pull-based but async-aware. It composes naturally with LINQ and doesn't require a separate reactive library for most use cases. - Go side-stepped the problem entirely with goroutines and channels, making the whole streams abstraction unnecessary for most cases. What I find interesting about this proposal is it's trying to learn from that prior art. The biggest mistake Java made was bolting async streams on top of a synchronous abstraction and then needing a completely separate spec (Reactive Streams) for the async case. If JavaScript can get a single unified abstraction that handles both sync iteration and async backpressure, that would be a genuine improvement over what exists in most other runtimes.