3 ms·

This specific proposal will only really help with the last part - it will let you write return `data:application/octet-stream;base64,${(value as Uint8Array

This specific proposal will only really help with the last part - it will let you write return `data:application/octet-stream;base64,${(value as Uint8Array).toBase64()}`; but you'll still have to do the work of reading the stream to a buffer yourself. There is a _very_ early stage (as in, it's literally just an idea one person had, which may never happen) proposal [1] to do zero-copy ArrayBuffer concatenation, which would further simplify this - once you'd collected the chunks you could `value = new Uint8Array(ArrayBuffer.of(chunks.map(chunk => chunk.buffer))` instead of manual concatenation. Finally, there's the Array.fromAsync proposal [2] and/or async iterator helpers proposal [3] (which I am also working on), which would make it easier to collect the chunks. Putting these together, you'd get something like if (isReadableStream<Uint8Array>(value)) { const chunks = await Array.fromAsync(value); if (chunks[0].byteLength) { value = new Uint8Array(ArrayBuffer.of(...chunks.map(chunk => chunk.buffer))); } else { throw new Error(`Unrecognized readable stream type: ${ chunks[0].constructor.name }`); } } return `data:application/octet-stream;base64,${(value as Uint8Array).toBase64()}`; [1] https://github.com/jasnell/proposal-zero-copy-arraybuffer-list https://github.com/jasnell/proposal-zero-copy-arraybuffer-li... [2] https://github.com/tc39/proposal-array-from-async https://github.com/tc39/proposal-array-from-async [3] https://github.com/tc39/proposal-async-iterator-helpers https://github.com/tc39/proposal-async-iterator-helpers

bsimpson 3y ago
It's nice that in an imaginary future that code would be shorter, but it's unfortunate that the conceptual understanding needed to write it isn't. You'd still need to know how to juggle a whole bunch of related concepts - ReadableStream chunks, Uint8Arrays, ArrayBuffers - to write that transformation. Why does de-chunking a byte array need to be complicated: new Uint8Array(ArrayBuffer.of(...chunks.map(chunk => chunk.buffer))) esp when chunking is specified by the platform in ReadableStream? ----- You have made me realize I don't even know what the right venue is to vote on stuff. How should I signal to TC39 that e.g. Array.fromAsync is a good idea?
- bakkoting 3y ago
  Yeah, in your case I think most of the complexity is actually on the ReadableStream side, not the base64 side. The thing that I'd actually want for your case is either a TransformStream for byte stream <-> base64 stream (which I expect will come eventually, once the simple case gets done; it's also easy in userland [1]), or something which would let you read the entire stream into a single Uint8Array or ArrayBuffer, which is a long-standing suggestion [2]. --- > Why does de-chunking a byte array need to be complicated Keep in mind the concat proposal is _very_ early. If you think it would be useful to be able to concat Uint8Arrays and have that implicitly concatenate the underlying buffers, [3] is the place to open an issue. --- > You have made me realize I don't even know what the right venue is to vote on stuff. How should I signal to TC39 that e.g. Array.fromAsync is a good idea? Unfortunately, it's different places for different things. Streams are not TC39 at all; the right place for suggestions there is in the WHATWG streams repo [4]. Usually there's already an existing issue and you can add your use case as a comment in the relevant issue. TC39 proposals all have their own Github repositories, and you can open a new issue with your use case. Concrete use cases are much more helpful than just "this is a good idea". Though `fromAsync` in particular everyone agrees is good, and it mostly just needs implementations, which are ongoing; see e.g. [5]. If you _really_ want to advance a stage 3 proposal, you can contribute a PR to Chrome or Firefox with an implementation - but for nontrivial proposals that's usually hard. For TC39 in particular, use cases are only really valuable pre-stage-3 proposals. [1] https://github.com/lucacasonato/base64_streams/blob/7c4ed815bd630bec3c185cf37b64cbb23be127ae/src/whatwg/encoder.ts https://github.com/lucacasonato/base64_streams/blob/7c4ed815... [2] https://github.com/whatwg/streams/issues/1019 https://github.com/whatwg/streams/issues/1019 [3] https://github.com/jasnell/proposal-zero-copy-arraybuffer-list https://github.com/jasnell/proposal-zero-copy-arraybuffer-li... [4] https://github.com/whatwg/streams https://github.com/whatwg/streams [5] https://bugs.chromium.org/p/v8/issues/detail?id=13321 https://bugs.chromium.org/p/v8/issues/detail?id=13321

bsimpson 3y ago

It's nice that in an imaginary future that code would be shorter, but it's unfortunate that the conceptual understanding needed to write it isn't. You'd still need to know how to juggle a whole bunch of related concepts - ReadableStream chunks, Uint8Arrays, ArrayBuffers - to write that transformation. Why does de-chunking a byte array need to be complicated: new Uint8Array(ArrayBuffer.of(...chunks.map(chunk => chunk.buffer))) esp when chunking is specified by the platform in ReadableStream? ----- You have made me realize I don't even know what the right venue is to vote on stuff. How should I signal to TC39 that e.g. Array.fromAsync is a good idea?

bakkoting 3y ago
Yeah, in your case I think most of the complexity is actually on the ReadableStream side, not the base64 side. The thing that I'd actually want for your case is either a TransformStream for byte stream <-> base64 stream (which I expect will come eventually, once the simple case gets done; it's also easy in userland [1]), or something which would let you read the entire stream into a single Uint8Array or ArrayBuffer, which is a long-standing suggestion [2]. --- > Why does de-chunking a byte array need to be complicated Keep in mind the concat proposal is _very_ early. If you think it would be useful to be able to concat Uint8Arrays and have that implicitly concatenate the underlying buffers, [3] is the place to open an issue. --- > You have made me realize I don't even know what the right venue is to vote on stuff. How should I signal to TC39 that e.g. Array.fromAsync is a good idea? Unfortunately, it's different places for different things. Streams are not TC39 at all; the right place for suggestions there is in the WHATWG streams repo [4]. Usually there's already an existing issue and you can add your use case as a comment in the relevant issue. TC39 proposals all have their own Github repositories, and you can open a new issue with your use case. Concrete use cases are much more helpful than just "this is a good idea". Though `fromAsync` in particular everyone agrees is good, and it mostly just needs implementations, which are ongoing; see e.g. [5]. If you _really_ want to advance a stage 3 proposal, you can contribute a PR to Chrome or Firefox with an implementation - but for nontrivial proposals that's usually hard. For TC39 in particular, use cases are only really valuable pre-stage-3 proposals. [1] https://github.com/lucacasonato/base64_streams/blob/7c4ed815bd630bec3c185cf37b64cbb23be127ae/src/whatwg/encoder.ts https://github.com/lucacasonato/base64_streams/blob/7c4ed815... [2] https://github.com/whatwg/streams/issues/1019 https://github.com/whatwg/streams/issues/1019 [3] https://github.com/jasnell/proposal-zero-copy-arraybuffer-list https://github.com/jasnell/proposal-zero-copy-arraybuffer-li... [4] https://github.com/whatwg/streams https://github.com/whatwg/streams [5] https://bugs.chromium.org/p/v8/issues/detail?id=13321 https://bugs.chromium.org/p/v8/issues/detail?id=13321