3 ms·

The proposal for native base64 support for Uint8Arrays is mine. I'm glad to see people are interested in using it. (So am I!) For a status update, for the last

• bsimpson 3y ago
  Thanks for doing this! I had to convert a ReadableSteam to base64 recently, and I was shocked at how much boilerplate it required in 2023: if (isReadableStream<Uint8Array>(value)) { const chunks = []; for await (const chunk of value) { chunks.push(chunk); } if (chunks[0].byteLength) { const length = chunks.reduce( (agg, next) => agg + next.length, 0 ); // Make the same species TypedArray that ReadableStream gave us. value = new (chunks[0].constructor as Uint8ArrayConstructor)(length); for (let i = 0, offset = 0; i < chunks.length; i++) { value.set(chunks[i], offset); offset += chunks[i].length; } } else { throw new Error(`Unrecognized readable stream type: ${ chunks[0].constructor.name }`); } } return `data:application/octet-stream;base64,${ btoa( (value as Uint8Array).reduce( (result, charCode) => result + String.fromCharCode(charCode), '' ) ) }`; I had to learn/implement a lot of little details to simply change the encoding from binary to its most common text representation. That's a day I would have rather spent doing product work.
  • bakkoting 3y ago
    This specific proposal will only really help with the last part - it will let you write return `data:application/octet-stream;base64,${(value as Uint8Array).toBase64()}`; but you'll still have to do the work of reading the stream to a buffer yourself. There is a _very_ early stage (as in, it's literally just an idea one person had, which may never happen) proposal [1] to do zero-copy ArrayBuffer concatenation, which would further simplify this - once you'd collected the chunks you could `value = new Uint8Array(ArrayBuffer.of(chunks.map(chunk => chunk.buffer))` instead of manual concatenation. Finally, there's the Array.fromAsync proposal [2] and/or async iterator helpers proposal [3] (which I am also working on), which would make it easier to collect the chunks. Putting these together, you'd get something like if (isReadableStream<Uint8Array>(value)) { const chunks = await Array.fromAsync(value); if (chunks[0].byteLength) { value = new Uint8Array(ArrayBuffer.of(...chunks.map(chunk => chunk.buffer))); } else { throw new Error(`Unrecognized readable stream type: ${ chunks[0].constructor.name }`); } } return `data:application/octet-stream;base64,${(value as Uint8Array).toBase64()}`; [1] https://github.com/jasnell/proposal-zero-copy-arraybuffer-list https://github.com/jasnell/proposal-zero-copy-arraybuffer-li... [2] https://github.com/tc39/proposal-array-from-async https://github.com/tc39/proposal-array-from-async [3] https://github.com/tc39/proposal-async-iterator-helpers https://github.com/tc39/proposal-async-iterator-helpers
    • bsimpson 3y ago
      It's nice that in an imaginary future that code would be shorter, but it's unfortunate that the conceptual understanding needed to write it isn't. You'd still need to know how to juggle a whole bunch of related concepts - ReadableStream chunks, Uint8Arrays, ArrayBuffers - to write that transformation. Why does de-chunking a byte array need to be complicated: new Uint8Array(ArrayBuffer.of(...chunks.map(chunk => chunk.buffer))) esp when chunking is specified by the platform in ReadableStream? ----- You have made me realize I don't even know what the right venue is to vote on stuff. How should I signal to TC39 that e.g. Array.fromAsync is a good idea?
      • bakkoting 3y ago