7 ms·

Goodbye, Node.js Buffer

217 points by ingve 3y ago

hersko 3y ago
I recently ran into an issue where I needed to store an Uint8Array as a BYTEA in postgres. When i stored it directly, the Uint8Array would be modified and made bigger. When i converted it to a buffer and then stored it, it would work. Still not sure why that was the case.
- TSiege 3y ago
  I wonder if that's a lack of good support for Uint8Array in the client you're using with postgres?
  - hersko 3y ago
    Not sure. I was using Sequalize for the ORM, but don't know if it was a Sequalize or postgres issue.
wayvey 3y ago
It's good to see that people care about native JS standards. I'm a bit concerned how Bun seems to be adding their own proprietary APIs to their JS runtime, and can't help feeling that it could be intentional vendor lock-in. Since they are venture backed I'm sure they want some ROI from the project eventually and if they start charging money it will be harder to switch away due to the proprietary APIs without refactoring your project.
- thomasreggi 3y ago
  It's really because we have some wonderful collaboration between vendors now, and they don't want to have to come up with their own APIs. https://wintercg.org/ https://wintercg.org/
  - pjmlp 3y ago
    Cough. Vercel.
- jokethrowaway 3y ago
  It's still OSS, not proprietary APIs. Feel free to fork it in the future, once they need to make their money back. If it's faster than node and the API to do what I need is better (Bun.FileSystemRouter comes to mind), I don't see why not to switch over. Heck, I would accept even less retro-compatibility to get rid of transpiling, that completely oscene political mess they did with modules and treating TS as a 2nd class citizen. The amount of time I'm still hit by ESM vs CJS is insane. Truly a Python 2vs3 moment for the node.js community.
  - chlorion 3y ago
    The term "proprietary" does not always refer to software licenses, but that's one of the common causes of something to be proprietary.
    - SahAssar 3y ago
      How would you define it? The wiki article for it in the context of software (https://en.wikipedia.org/wiki/Proprietary_software https://en.wikipedia.org/wiki/Proprietary_software) seems to pretty much stand it in opposition to open licenses, but I also have a sort of gut feeling that it could also refer to interfaces in OSS that are not designed to re-implementable. Not sure about that though.
      - cxr 3y ago
        "Nonstandard and controlled by one particular organization." <https://en.wiktionary.org/wiki/proprietary https://en.wiktionary.org/wiki/proprietary> XBL, a precursor to Web Components in use in Firefox for 15+ years before any ordinary webdev was talking about shadow DOM, was open source in every sense, for example. It was also proprietary. See also: "Problems with XUL". > XUL is a proprietary technology developed by Mozilla and only used by Mozilla. <https://mozilla.github.io/firefox-browser-architecture/text/0003-problems-with-xul.html#availability-of-example-code-and-help https://mozilla.github.io/firefox-browser-architecture/text/...>
        3y ago
- brundolf 3y ago
  The accusation of malice goes a bit far, but I agree it's a problem whether it's intentional or not. Bun takes a slapdash approach to shipping features and APIs, which means things get out fast but also rubs me the wrong way re: long-term implications and vision
  - eyelidlessness 3y ago
    I’ll vouch for non-malice. Bun’s creator frequently tweets about hypothetical APIs he may want to introduce, often seeking feedback explicitly. The ideas are always earnest “would you find this helpful?” sorts of things. (For what it’s worth, I frequently reply that these hypotheticals are a bad idea when they break expectations etc, at least if I can fit my reasoning in tweet length. I think this has been at least marginally successful in pushing back in some cases.) The problem, with Bun but really with the ecosystem at large, is that shipping stuff is (still) the de facto way that standards kind of congeal into something resembling actual standards.
    - brundolf 3y ago
      Yeah. I follow him on twitter too, and it's both impressive and distressingly chaotic the way he spitballs and then settles on APIs in the form of twitter surveys
      - tomjakubowski 3y ago
        seems to work for twitter's CEO - though he does sometimes ignore what the survey says
      - Aeolun 3y ago
        That works just fine if you are the main person working on something.
- lemper 3y ago
  > can't help feeling that it could be intentional vendor lock-in. it's always vendor lock-in. can never trust companies, mate!
kev946 3y ago
I find buffer useful for its conversion functions to/from different encodings, e.g., `Buffer.from(data, 'hex')` or `Buffer.toString('base64')`. Is there a good way to do this with `Uint8Array`?
- davidmurdoch 3y ago
  Polyfill Buffer.
  - norman784 3y ago
    I would prefer not install any dependency if possible, I don't see a necessity of installing a polyfill just to not use the built in one because it's node specific.
- ash_gti 3y ago
  You can use ‘atob’ and ‘btoa’ functions for some of that.
  - denzquix 3y ago
    Although those functions operate on "binary strings", not Uint8Arrays, and there is no especially clean way that vanilla JS exposes to convert between the two that I am aware of.
  - deleted 3y ago
    [deleted]
  - esprehn 3y ago
    Those functions are fundamentally broken: https://developer.mozilla.org/en-US/docs/Glossary/Base64#javascript_support https://developer.mozilla.org/en-US/docs/Glossary/Base64#jav... See the whole section on converting arbitrary binary data and the complex ways to do it.
- mofle 3y ago
  People are working on bringing Base64/Hex conversion to JavaScript: https://github.com/tc39/proposal-arraybuffer-base64 https://github.com/tc39/proposal-arraybuffer-base64 I also provide a package to make the transition easier: https://github.com/sindresorhus/uint8array-extras https://github.com/sindresorhus/uint8array-extras (Feel free to copy-paste the code if you don't want another dependency)
davidmurdoch 3y ago
Buffer's `slice` is nice to have in many cases though. The pool makes it generally faster. And `allocUnsafe` is a great feature! I like Buffer.
- mofle 3y ago
  `Buffer.slice` is deprecated: https://nodejs.org/api/buffer.html#bufslicestart-end https://nodejs.org/api/buffer.html#bufslicestart-end
- Etheryte 3y ago
  As discussed in another thread, `subarray()`[0] fills the same purpose. [0] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/TypedArray/subarray https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
supermatt 3y ago
> Buffer also comes with additional caveats. For instance, Buffer#slice() creates a mutable segment linked to the original Buffer, while Uint8Array#slice() creates an immutable copy, resulting in possible unpredictable behavior. I can see how the Buffer behavior here would be preferable in cases where performance is important or memory constrained environments.
- mofle 3y ago
  Uint8Array has this too, but it's called `.subarray()`. The problem is that Buffer is a subclass of Uint8Array, but changes the behavior of the `.slice()` method.
  - deleted 3y ago
    [deleted]
thomasreggi 3y ago
I've been using the native Uint8Array that Deno provides and it's been great overall. From time to time I may need to covert away from a Buffer type and it's a little bit of a headache, but I think the ecosystem is supporting Uint8Array, at least on the server very well.
- deleted 3y ago
  [deleted]
ravenstine 3y ago
I agree with using Uint8Array (although it's really ArrayBuffer that's the key difference), but what is up with the author using an NPM package to do something as trivial as checking if an object is a Uint8Array? `Uint8Array.prototype.isPrototypeOf` should be perfectly adequate and doesn't involve adding an attack vector to your application.
- mofle 3y ago
  `Uint8Array.prototype.isPrototypeOf` and `instanceof Uint8Array` do not work across realms (frames, Node.js VM, etc). Feel free to copy-paste the function to your own code base if you don't want the dependency: ``` const objectToString = Object.prototype.toString; export function isUint8Array(value) { return value && objectToString.call(value) === '[object Uint8Array]'; } ```
  - spicykraken 3y ago
    What do you mean by "across realms"? Is that just another way of saying `Uint8Array.prototype.isPrototypeOf` and `instanceof Uint8Array` are not available in all JS environments? I guess what I'm asking is the definition of a "Javascript Realm" in case I'm thinking it's something different.
    - mofle 3y ago
      https://weizmangal.com/2022/10/28/what-is-a-realm-in-js https://weizmangal.com/2022/10/28/what-is-a-realm-in-js Examples of this are frames in the browser and the `vm` module in Node.js.
      - spicykraken 3y ago
        Ah thanks, I thought "realms" sounded familiar and that helps clear things up a bit. Also Lavamoat and SES look really interesting thanks for the link.
  - ravenstine 3y ago
    I understand what you're saying, but that's actually in support of my point. This is still extremely trivial code to implement and, from what I can tell, doesn't warrant downloading an NPM package. Have we already forgotten the left-pad fiasco? This isn't meant as a personal attack on anyone, but we really need to frown upon needless dependencies, especially given the growing number of malicious NPM packages.
jongjong 3y ago
TBH I prefer Buffer as an abstraction. Too bad it didn't make it into native JS standard. I can't think of many use cases in JS land were Uint8Array, Uint16Array, Uint32Array, Int8Array would be absolutely necessary. Seems more useful for perf optimizations. For WebAssembly? Surely a plain array of numbers can be used in most cases. The main use case for Buffer IMO is to convert between different formats like Base64, hex or for sending raw binary data over a transport; ultimately we just need an object to represent binary. It doesn't seem appropriate for a high level language like JS to concern itself with the details of CPU architecture which warrant thinking of binary in terms of 8-bits, 16-bits or 32-bits in the first place. As an abstraction, it's rather arbitrary to treat these numbers as special. It really seems to come down to a marginal performance optimization.
- mofle 3y ago
  > I can't think of many use cases in JS land were Uint8Array, Uint16Array, Uint32Array, Int8Array would be absolutely necessary. Buffer is a subclass of Uint8Array.
- tomjakubowski 3y ago
  Sized numerics are useful even in high level, loosely and dynamically typed languages. Two examples are binary (de)serialization and OpenGL. In Python, libraries for those generally use sized numerics. High level languages will still want to do low level things
- SuboptimalEng 3y ago
  Typed arrays are essential for web apps that use WebGL and WebGPU. Being able to send this type of data to run computations on the GPU can give you 1000x speed up. You can see it in action on this WebGL fluid simulator[0] by PavelDoGreat. [0] https://github.com/PavelDoGreat/WebGL-Fluid-Simulation https://github.com/PavelDoGreat/WebGL-Fluid-Simulation
  - mateuspires 3y ago
    This simulator os awesome.
josephcsible 3y ago
> buffers expose private information through global variables, a potential security risk. Does JavaScript's security model let you effectively sandbox scripts running in the same context from each other? If not, then why does this matter?
- strken 3y ago
  From a cursory read, it matters because all Buffer.from(...) calls use a shared buffer, and buffer over-reads are a much more common vulnerability than easy access to arbitrary memory. Not a security expert etc.
- brundolf 3y ago
  Consider a web server serving multiple ongoing requests from different users (with separate permissions). Each of them uses a buffer at some point (either a Buffer or a Uint8Array) If the buffer doesn't have any cross-contamination with global state, there's no way one user could access another's data (because it's behind an object reference that never comes into the scope of the request logic for the other user). But if it did, and a malicious user found some other kind of vulnerability, they could potentially access data across-scopes
  - kklisura 3y ago
    Replace buffer in your example with plain object or just array - that rest of it still stands, right? There's something other at play: > buffers expose private information through global variables, a potential security risk. This links to following piece of code [1]: > // Somewhere in your code > const privateBuf = Buffer.from(privateKey, 'hex'); > // Rogue package can access > Buffer.from('1').buffer I've just run it in node, and my god am I shocked! > const privateBuf = Buffer.from('DEADBEEF', 'hex'); > Buffer.from('1').buffer > ArrayBuffer { [Uint8Contents]: <2f 00 00 00 00 00 00 00 de ad be ef 00 > .... [1] https://github.com/nodejs/node/issues/41588#issuecomment-1016269584 https://github.com/nodejs/node/issues/41588#issuecomment-101...
nshm 3y ago
Ok, first we screwed buffers by making them globally tracked instead of just a piece of memory. Now its time to break all binary modules again.
vendiddy 3y ago
I had to deal with binary data in a project recently and all the options made my head spin. File, Blob, Buffer, ArrayBuffer, Uint8Array, and so on. Was very confused on what to use!
- mofle 3y ago
  - Blob: Immutable raw data container with a size and MIME type, not directly readable. - File: Like a Blob, but with additional file-specific properties (e.g., filename). - ArrayBuffer: Fixed-length raw binary data in JavaScript, not directly accessible. - Uint8Array: Interface for reading/writing binary data in ArrayBuffer, showing them as 8-bit unsigned integers. - Buffer: Readable/writable raw binary data container in Node.js (subclass of Uint8Array)
  - bakkoting 3y ago
    Nit: "fixed-length" is no longer true as of very recently [1]. [1] https://github.com/tc39/proposal-resizablearraybuffer https://github.com/tc39/proposal-resizablearraybuffer
    - pwdisswordfishc 3y ago
      Now it's bounded-length.
  - explaininjs 3y ago
    A File is a Blob, every file is `instanceof Blob`.
shiomiru 3y ago
I wonder why you would introduce an extra dependency for the base64 example. It seems more trivial than left pad. https://github.com/sindresorhus/uint8array-extras/blob/cbf24e6e3c1884655452612b2cc5f266915a8df5/index.js#L107 https://github.com/sindresorhus/uint8array-extras/blob/cbf24...
- betenoire 3y ago
  > // Required as `btoa` and `atob` don't properly support Unicode: https://developer.mozilla.org/en-US/docs/Glossary/Base64#the_unicode_problem https://developer.mozilla.org/en-US/docs/Glossary/Base64#the...
keithwhor 3y ago
Personally, I won't make the switch until the utility methods are supported natively. I don't think it's worth the extra package overhead to execute `buf.toString('base64')`. The second that lands, I'm all-in on abandoning buffers.
loudgas 3y ago
>Buffer also comes with additional caveats. For instance, Buffer#slice() creates a mutable segment linked to the original Buffer, while Uint8Array#slice() creates an immutable copy, resulting in possible unpredictable behavior. The problem is not the behavior of the Buffer#slice() method, but the fact that Buffer is a subclass of Uint8Array, but changes the behavior of an inherited method. But this is an implementation detail, not specified behavior. Changing method behavior in subclasses is a key aspect of inheritance.
- mofle 3y ago
  Completely different behavior is not an implementation detail. Changing method behavior in subclasses is part of inheritance, but it shouldn't confuse or mislead. In the case of Buffer and Uint8Array, the altered `.slice()` functionality isn't a mere implementation detail; it's a significant deviation. This inconsistency can lead to unexpected bugs, especially for those who assume similar behavior based on the inheritance hierarchy. It's crucial for reliability that such fundamental behaviors remain predictable across subclasses.
  - loudgas 3y ago
    Any behavior that is not defined in the spec[0] is, by definition, an implementation detail. Relying on undefined behavior is a recipe for bugs. If you need an immutable array, and the spec doesn't require the returned array to be immutable, you should create one yourself. [0] https://tc39.es/ecma262/multipage/indexed-collections.html#sec-array.prototype.slice https://tc39.es/ecma262/multipage/indexed-collections.html#s...
    - mofle 3y ago
      You are reading the wrong spec. That is `Array#slice`, not `TypedArray#slice`. Correct spec: https://tc39.es/ecma262/multipage/indexed-collections.html#sec-%typedarray%.prototype.slice https://tc39.es/ecma262/multipage/indexed-collections.html#s... Steps 14.g.i to 14.g.ix detail the transfer of data from the original TypedArray (O) to the new TypedArray (A). It involves reading values from the original and writing them to the new array's buffer, effectively duplicating the data segment. The process ensures both arrays are distinct with separate memory spaces.
paddy_m 3y ago
Somewhat related, who has played with performant JS dataframe libraries? I mostly want to serialize from pandas/polars to JS in a fast way, with some very minor selection and indexing operations JS side. Ideally I could find a library that does dataframe like operations from reading base64 encoded Buffers/Arrays into their JS objects, then doing the right things. I'm investigating https://arrow.apache.org/docs/js/ https://arrow.apache.org/docs/js/ https://github.com/vega/falcon https://github.com/vega/falcon https://github.com/pola-rs/nodejs-polars https://github.com/pola-rs/nodejs-polars https://github.com/uwdata/arquero https://github.com/uwdata/arquero
- paddy_m 3y ago
  I added a github issue to my project where this matters literally this morning. I'd love to talk to someone dealing with the same stuff. I have baked dataframe -> JS serialization utils so many times over the past decade. I never want to do much in JS, just simple selection and filtering. But you end up needing both sides of the serialization to do it right.
- kylebarron 3y ago
  nodejs-polars is node-specific and uses native FFI. polars can be compiled to Wasm but doesn't yet have a js API out of the box. As for the fastest way to serialize data to Pandas data to the browser, you should use Parquet; it's the fastest to write on the Python side and read on the JS side, while also being compressed. See https://github.com/kylebarron/parquet-wasm https://github.com/kylebarron/parquet-wasm (full disclosure, I wrote this)
pictur 3y ago
I guess this is our new nonsense after the ESM nonsense. Come on man start fucking all the npm packages as soon as possible. Don't forget to throw away backward compatibility.
wruza 3y ago
So the only important difference is that Buffer creates a mutable view at slice(). It could use a better/distinct name for sure, e.g. mutableView(). But do you actually want to always copy a slice that is big enough? This ideological pedantic purism which eliminates practical use cases is in my book just that - impractical pedantic purism. It’s easy to advocate for when your job is shaping landscapes without having to walk these. And yes, I’m writing this still having flashbacks from ESM-only “transition”, which was more like throwing everyone into freezing waters with the promise it will warm up eventually.
- rattlesnakedave 3y ago
  > So the only important difference is that Buffer creates a mutable view at slice(). No, it also (from the article): > introduces numerous methods that are not available in other JavaScript environments. Consequently, code leveraging Buffer-specific methods needs polyfilling, preventing many valuable packages from being browser-compatible.
  - wruza 3y ago
    The polyfill argument is good, but it addresses a world where you just push html, css and js files. We live in a different one. It is valid, but more on a wishful thinking side rather than practical. And when you avoid calling it “polyfill”, it simply turns into a useful compatibility/extension library. I find it strange to argue about one dependency but then importing few others, with the only difference in taxonomy.
- moring 3y ago
  > So the only important difference is that Buffer creates a mutable view at slice(). It could use a better/distinct name for sure, e.g. mutableView(). But do you actually want to always copy a slice that is big enough? I think you are missing the point. Mutable and copied slices both have their use, and maybe a method that sometimes copies and sometimes shares does too. The problem with Buffer is that it overrides a method with well-defined semantics, then violates those. Buffer is a subclass of UInt8Array, but not a subtype nor Liskov-substitutable. In practical terms, when you get a UInt8Array and even check that it in fact is a UInt8Array, you no longer know that .slice() does even though UInt8Array's documentation explains how every UInt8Array behaves, because that documentation is now wrong. If the UInt8Array you get is a Buffer, then it is both a buffer and a UInt8Array but its .slice() behaves differently than expected.
bakkoting 3y ago
The proposal for native base64 support for Uint8Arrays is mine. I'm glad to see people are interested in using it. (So am I!) For a status update, for the last year or two the main blocker has been a conflict between a desire to have streaming support and a desire to keep the API small and simple. That's now resolved [1] by dropping streaming support, assuming I can demonstrate a reasonably efficient streaming implementation on top of the one-shot implementation, which won't be hard unless "reasonably efficient" means "with zero copies", in which case we'll need to keep arguing about it. I've also been working on documenting [2] the differences between various base64 implementations in other languages and in JS libraries to ensure we have a decent picture of the landscape when designing this. With luck, I hope to advance the proposal to stage 3 ("ready for implementations") within the next two meetings of TC39 - so either next month or January. Realistically it will probably take a little longer than that, and of course implementations take a while. But it's moving along. [1] https://github.com/tc39/proposal-arraybuffer-base64/issues/13#issuecomment-1738345569 https://github.com/tc39/proposal-arraybuffer-base64/issues/1... [2] https://gist.github.com/bakkot/16cae276209da91b652c2cb3f612af59 https://gist.github.com/bakkot/16cae276209da91b652c2cb3f612a...
- bsimpson 3y ago
  Thanks for doing this! I had to convert a ReadableSteam to base64 recently, and I was shocked at how much boilerplate it required in 2023: if (isReadableStream<Uint8Array>(value)) { const chunks = []; for await (const chunk of value) { chunks.push(chunk); } if (chunks[0].byteLength) { const length = chunks.reduce( (agg, next) => agg + next.length, 0 ); // Make the same species TypedArray that ReadableStream gave us. value = new (chunks[0].constructor as Uint8ArrayConstructor)(length); for (let i = 0, offset = 0; i < chunks.length; i++) { value.set(chunks[i], offset); offset += chunks[i].length; } } else { throw new Error(`Unrecognized readable stream type: ${ chunks[0].constructor.name }`); } } return `data:application/octet-stream;base64,${ btoa( (value as Uint8Array).reduce( (result, charCode) => result + String.fromCharCode(charCode), '' ) ) }`; I had to learn/implement a lot of little details to simply change the encoding from binary to its most common text representation. That's a day I would have rather spent doing product work.
cs2818 3y ago
Probably less common, but I write a lot of JS code interfacing with devices over serial connections. This makes Buffer's convenience methods for encoding and decoding data types in both big and little endian ordering really useful to me. It looks like I could accomplish the same thing using a DataView of an ArrayBuffer, but I don't see enough of a benefit to justify converting everything to this approach.
lexicality 3y ago
Having just spent the past few months dealing with the utter carnage caused by updating an old Node project to use modern dependencies, I predict doing this will cause a bloodbath as some libraries update but don't correctly specify semver for the breaking changes, or other libraries that depend on the Buffer usage not bothering to limit their dependencies to a major version unexpectedly breaking when `npm update` is run. I wonder what unmaintaned libraries will have to be dropped this time as the ecosystem grinds on and causes them to break? (How did we get to a place where everything is this bad?)
- cxr 3y ago
  > (How did we get to a place where everything is this bad?) Not vetting the bedrock you chose to build upon; deciding to rely on the unreliable. ("Everything" is an overstatement.)
  - Exoristos 3y ago
    What's better?
    - cxr 3y ago
      Vetting it.
    - deleted 3y ago
      [deleted]
mnkypete 3y ago
My absolute go to issue these days with Node is the persistence on doing the same type/interfaces as the browser. The absolute sh* show trying to make a FormData request in Node to work with native Node streams is mind boggling, just because fetch/FormData needs to be EXACTLY like in the browser. Sure, but how about I do not want to put a 100mb file into memory just because I want to use native fetch ....
brianzelip 3y ago
Great post, like good docs; clear, informative, and with examples.
dinkleberg 3y ago
A total tangent, but it is always fun when two totally separate things you're doing connect in a weird way. I just installed some node library and then I see this unrelated blog post that is written by the creator of that library.
- MrJohz 3y ago
  The author has written a lot (1000+) of libraries, and earned a certain amount of controversy for how they manage them, and whether all those packages are necessary. I think the most recent controversy was that they switched all of their packages over to the new module system before the ecosystem was ready. It's one of those names where I tend to raise an eyebrow when I hear it. This looks like something similar to the module/require changes, where the overall change is good, but a slower transition may help people more.
  - dinkleberg 3y ago
    Geez you aren’t kidding, this guy is prolific. I’ve seen some of his other projects on here but they all seemed to be Mac apps. Surprised to see someone churning out so many of those also has time to build and maintain 1000+ npm projects. Also this guy seems to have started the awesome list trend. I do have to wonder how many of those projects are essentially a wrapper around some native functionality just with their preferred API. Regardless, that level of scale is impressive. Thanks for the heads up though, I’ll have approach with caution.
paulddraper 3y ago
Can someone tell me when to use Uint8Array vs ArrayBuffer in my method signatures? Like, I would think that replaces a lot of Buffer usage....but the articles mentions it exactly zero times.
Aeolun 3y ago
The problem is that Buffer is just too damn convenient. I can do everything with a Uint8Array, but everything takes about 4-5x more code for no discernible benefit.
WirelessGigabit 3y ago
Personally, I believe it's better to keep on using the proven buffer vs some person's package. We don't need another left-pad. Also another reason to use a strongly typed language. You call a function that returns a buffer. The interfaces changes and you now get a UInt8Array. You will never know.
- dcgudeman 3y ago
  Is that what he is suggesting? I thought the article just wanted people to use UInt8Array instead of buffer.

217 points by ingve 3y ago

I recently ran into an issue where I needed to store an Uint8Array as a BYTEA in postgres. When i stored it directly, the Uint8Array would be modified and made bigger. When i converted it to a buffer and then stored it, it would work. Still not sure why that was the case.

TSiege 3y ago
I wonder if that's a lack of good support for Uint8Array in the client you're using with postgres?
- hersko 3y ago
  Not sure. I was using Sequalize for the ORM, but don't know if it was a Sequalize or postgres issue.

wayvey 3y ago

It's good to see that people care about native JS standards. I'm a bit concerned how Bun seems to be adding their own proprietary APIs to their JS runtime, and can't help feeling that it could be intentional vendor lock-in. Since they are venture backed I'm sure they want some ROI from the project eventually and if they start charging money it will be harder to switch away due to the proprietary APIs without refactoring your project.

thomasreggi 3y ago
It's really because we have some wonderful collaboration between vendors now, and they don't want to have to come up with their own APIs. https://wintercg.org/ https://wintercg.org/
- pjmlp 3y ago
  Cough. Vercel.
jokethrowaway 3y ago
It's still OSS, not proprietary APIs. Feel free to fork it in the future, once they need to make their money back. If it's faster than node and the API to do what I need is better (Bun.FileSystemRouter comes to mind), I don't see why not to switch over. Heck, I would accept even less retro-compatibility to get rid of transpiling, that completely oscene political mess they did with modules and treating TS as a 2nd class citizen. The amount of time I'm still hit by ESM vs CJS is insane. Truly a Python 2vs3 moment for the node.js community.
- chlorion 3y ago
  The term "proprietary" does not always refer to software licenses, but that's one of the common causes of something to be proprietary.
  - SahAssar 3y ago
    How would you define it? The wiki article for it in the context of software (https://en.wikipedia.org/wiki/Proprietary_software https://en.wikipedia.org/wiki/Proprietary_software) seems to pretty much stand it in opposition to open licenses, but I also have a sort of gut feeling that it could also refer to interfaces in OSS that are not designed to re-implementable. Not sure about that though.
    - cxr 3y ago
      "Nonstandard and controlled by one particular organization." <https://en.wiktionary.org/wiki/proprietary https://en.wiktionary.org/wiki/proprietary> XBL, a precursor to Web Components in use in Firefox for 15+ years before any ordinary webdev was talking about shadow DOM, was open source in every sense, for example. It was also proprietary. See also: "Problems with XUL". > XUL is a proprietary technology developed by Mozilla and only used by Mozilla. <https://mozilla.github.io/firefox-browser-architecture/text/0003-problems-with-xul.html#availability-of-example-code-and-help https://mozilla.github.io/firefox-browser-architecture/text/...>
      - 3y ago
brundolf 3y ago
The accusation of malice goes a bit far, but I agree it's a problem whether it's intentional or not. Bun takes a slapdash approach to shipping features and APIs, which means things get out fast but also rubs me the wrong way re: long-term implications and vision
- eyelidlessness 3y ago
  I’ll vouch for non-malice. Bun’s creator frequently tweets about hypothetical APIs he may want to introduce, often seeking feedback explicitly. The ideas are always earnest “would you find this helpful?” sorts of things. (For what it’s worth, I frequently reply that these hypotheticals are a bad idea when they break expectations etc, at least if I can fit my reasoning in tweet length. I think this has been at least marginally successful in pushing back in some cases.) The problem, with Bun but really with the ecosystem at large, is that shipping stuff is (still) the de facto way that standards kind of congeal into something resembling actual standards.
  - brundolf 3y ago
    Yeah. I follow him on twitter too, and it's both impressive and distressingly chaotic the way he spitballs and then settles on APIs in the form of twitter surveys
    - tomjakubowski 3y ago
      seems to work for twitter's CEO - though he does sometimes ignore what the survey says
    - Aeolun 3y ago
      That works just fine if you are the main person working on something.
lemper 3y ago
> can't help feeling that it could be intentional vendor lock-in. it's always vendor lock-in. can never trust companies, mate!

kev946 3y ago

I find buffer useful for its conversion functions to/from different encodings, e.g., `Buffer.from(data, 'hex')` or `Buffer.toString('base64')`. Is there a good way to do this with `Uint8Array`?

davidmurdoch 3y ago
Polyfill Buffer.
- norman784 3y ago
  I would prefer not install any dependency if possible, I don't see a necessity of installing a polyfill just to not use the built in one because it's node specific.
ash_gti 3y ago
You can use ‘atob’ and ‘btoa’ functions for some of that.
- denzquix 3y ago
  Although those functions operate on "binary strings", not Uint8Arrays, and there is no especially clean way that vanilla JS exposes to convert between the two that I am aware of.
- deleted 3y ago
  [deleted]
- esprehn 3y ago
  Those functions are fundamentally broken: https://developer.mozilla.org/en-US/docs/Glossary/Base64#javascript_support https://developer.mozilla.org/en-US/docs/Glossary/Base64#jav... See the whole section on converting arbitrary binary data and the complex ways to do it.
mofle 3y ago
People are working on bringing Base64/Hex conversion to JavaScript: https://github.com/tc39/proposal-arraybuffer-base64 https://github.com/tc39/proposal-arraybuffer-base64 I also provide a package to make the transition easier: https://github.com/sindresorhus/uint8array-extras https://github.com/sindresorhus/uint8array-extras (Feel free to copy-paste the code if you don't want another dependency)

davidmurdoch 3y ago

Buffer's `slice` is nice to have in many cases though. The pool makes it generally faster. And `allocUnsafe` is a great feature! I like Buffer.

mofle 3y ago
`Buffer.slice` is deprecated: https://nodejs.org/api/buffer.html#bufslicestart-end https://nodejs.org/api/buffer.html#bufslicestart-end
Etheryte 3y ago
As discussed in another thread, `subarray()`[0] fills the same purpose. [0] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/TypedArray/subarray https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

supermatt 3y ago

> Buffer also comes with additional caveats. For instance, Buffer#slice() creates a mutable segment linked to the original Buffer, while Uint8Array#slice() creates an immutable copy, resulting in possible unpredictable behavior. I can see how the Buffer behavior here would be preferable in cases where performance is important or memory constrained environments.

mofle 3y ago
Uint8Array has this too, but it's called `.subarray()`. The problem is that Buffer is a subclass of Uint8Array, but changes the behavior of the `.slice()` method.
- deleted 3y ago
  [deleted]

thomasreggi 3y ago

I've been using the native Uint8Array that Deno provides and it's been great overall. From time to time I may need to covert away from a Buffer type and it's a little bit of a headache, but I think the ecosystem is supporting Uint8Array, at least on the server very well.

deleted 3y ago
[deleted]

ravenstine 3y ago

I agree with using Uint8Array (although it's really ArrayBuffer that's the key difference), but what is up with the author using an NPM package to do something as trivial as checking if an object is a Uint8Array? `Uint8Array.prototype.isPrototypeOf` should be perfectly adequate and doesn't involve adding an attack vector to your application.

mofle 3y ago
`Uint8Array.prototype.isPrototypeOf` and `instanceof Uint8Array` do not work across realms (frames, Node.js VM, etc). Feel free to copy-paste the function to your own code base if you don't want the dependency: ``` const objectToString = Object.prototype.toString; export function isUint8Array(value) { return value && objectToString.call(value) === '[object Uint8Array]'; } ```
- spicykraken 3y ago
  What do you mean by "across realms"? Is that just another way of saying `Uint8Array.prototype.isPrototypeOf` and `instanceof Uint8Array` are not available in all JS environments? I guess what I'm asking is the definition of a "Javascript Realm" in case I'm thinking it's something different.
  - mofle 3y ago
    https://weizmangal.com/2022/10/28/what-is-a-realm-in-js https://weizmangal.com/2022/10/28/what-is-a-realm-in-js Examples of this are frames in the browser and the `vm` module in Node.js.
    - spicykraken 3y ago
      Ah thanks, I thought "realms" sounded familiar and that helps clear things up a bit. Also Lavamoat and SES look really interesting thanks for the link.
- ravenstine 3y ago
  I understand what you're saying, but that's actually in support of my point. This is still extremely trivial code to implement and, from what I can tell, doesn't warrant downloading an NPM package. Have we already forgotten the left-pad fiasco? This isn't meant as a personal attack on anyone, but we really need to frown upon needless dependencies, especially given the growing number of malicious NPM packages.

jongjong 3y ago

TBH I prefer Buffer as an abstraction. Too bad it didn't make it into native JS standard. I can't think of many use cases in JS land were Uint8Array, Uint16Array, Uint32Array, Int8Array would be absolutely necessary. Seems more useful for perf optimizations. For WebAssembly? Surely a plain array of numbers can be used in most cases. The main use case for Buffer IMO is to convert between different formats like Base64, hex or for sending raw binary data over a transport; ultimately we just need an object to represent binary. It doesn't seem appropriate for a high level language like JS to concern itself with the details of CPU architecture which warrant thinking of binary in terms of 8-bits, 16-bits or 32-bits in the first place. As an abstraction, it's rather arbitrary to treat these numbers as special. It really seems to come down to a marginal performance optimization.

mofle 3y ago
> I can't think of many use cases in JS land were Uint8Array, Uint16Array, Uint32Array, Int8Array would be absolutely necessary. Buffer is a subclass of Uint8Array.
tomjakubowski 3y ago
Sized numerics are useful even in high level, loosely and dynamically typed languages. Two examples are binary (de)serialization and OpenGL. In Python, libraries for those generally use sized numerics. High level languages will still want to do low level things
SuboptimalEng 3y ago
Typed arrays are essential for web apps that use WebGL and WebGPU. Being able to send this type of data to run computations on the GPU can give you 1000x speed up. You can see it in action on this WebGL fluid simulator[0] by PavelDoGreat. [0] https://github.com/PavelDoGreat/WebGL-Fluid-Simulation https://github.com/PavelDoGreat/WebGL-Fluid-Simulation
- mateuspires 3y ago
  This simulator os awesome.

josephcsible 3y ago

> buffers expose private information through global variables, a potential security risk. Does JavaScript's security model let you effectively sandbox scripts running in the same context from each other? If not, then why does this matter?

strken 3y ago
From a cursory read, it matters because all Buffer.from(...) calls use a shared buffer, and buffer over-reads are a much more common vulnerability than easy access to arbitrary memory. Not a security expert etc.
brundolf 3y ago
Consider a web server serving multiple ongoing requests from different users (with separate permissions). Each of them uses a buffer at some point (either a Buffer or a Uint8Array) If the buffer doesn't have any cross-contamination with global state, there's no way one user could access another's data (because it's behind an object reference that never comes into the scope of the request logic for the other user). But if it did, and a malicious user found some other kind of vulnerability, they could potentially access data across-scopes
- kklisura 3y ago
  Replace buffer in your example with plain object or just array - that rest of it still stands, right? There's something other at play: > buffers expose private information through global variables, a potential security risk. This links to following piece of code [1]: > // Somewhere in your code > const privateBuf = Buffer.from(privateKey, 'hex'); > // Rogue package can access > Buffer.from('1').buffer I've just run it in node, and my god am I shocked! > const privateBuf = Buffer.from('DEADBEEF', 'hex'); > Buffer.from('1').buffer > ArrayBuffer { [Uint8Contents]: <2f 00 00 00 00 00 00 00 de ad be ef 00 > .... [1] https://github.com/nodejs/node/issues/41588#issuecomment-1016269584 https://github.com/nodejs/node/issues/41588#issuecomment-101...

nshm 3y ago

Ok, first we screwed buffers by making them globally tracked instead of just a piece of memory. Now its time to break all binary modules again.

vendiddy 3y ago

I had to deal with binary data in a project recently and all the options made my head spin. File, Blob, Buffer, ArrayBuffer, Uint8Array, and so on. Was very confused on what to use!

mofle 3y ago
- Blob: Immutable raw data container with a size and MIME type, not directly readable. - File: Like a Blob, but with additional file-specific properties (e.g., filename). - ArrayBuffer: Fixed-length raw binary data in JavaScript, not directly accessible. - Uint8Array: Interface for reading/writing binary data in ArrayBuffer, showing them as 8-bit unsigned integers. - Buffer: Readable/writable raw binary data container in Node.js (subclass of Uint8Array)
- bakkoting 3y ago
  Nit: "fixed-length" is no longer true as of very recently [1]. [1] https://github.com/tc39/proposal-resizablearraybuffer https://github.com/tc39/proposal-resizablearraybuffer
  - pwdisswordfishc 3y ago
    Now it's bounded-length.
- explaininjs 3y ago
  A File is a Blob, every file is `instanceof Blob`.

shiomiru 3y ago

I wonder why you would introduce an extra dependency for the base64 example. It seems more trivial than left pad. https://github.com/sindresorhus/uint8array-extras/blob/cbf24e6e3c1884655452612b2cc5f266915a8df5/index.js#L107 https://github.com/sindresorhus/uint8array-extras/blob/cbf24...

betenoire 3y ago
> // Required as `btoa` and `atob` don't properly support Unicode: https://developer.mozilla.org/en-US/docs/Glossary/Base64#the_unicode_problem https://developer.mozilla.org/en-US/docs/Glossary/Base64#the...

keithwhor 3y ago

Personally, I won't make the switch until the utility methods are supported natively. I don't think it's worth the extra package overhead to execute `buf.toString('base64')`. The second that lands, I'm all-in on abandoning buffers.

loudgas 3y ago

>Buffer also comes with additional caveats. For instance, Buffer#slice() creates a mutable segment linked to the original Buffer, while Uint8Array#slice() creates an immutable copy, resulting in possible unpredictable behavior. The problem is not the behavior of the Buffer#slice() method, but the fact that Buffer is a subclass of Uint8Array, but changes the behavior of an inherited method. But this is an implementation detail, not specified behavior. Changing method behavior in subclasses is a key aspect of inheritance.

mofle 3y ago
Completely different behavior is not an implementation detail. Changing method behavior in subclasses is part of inheritance, but it shouldn't confuse or mislead. In the case of Buffer and Uint8Array, the altered `.slice()` functionality isn't a mere implementation detail; it's a significant deviation. This inconsistency can lead to unexpected bugs, especially for those who assume similar behavior based on the inheritance hierarchy. It's crucial for reliability that such fundamental behaviors remain predictable across subclasses.
- loudgas 3y ago
  Any behavior that is not defined in the spec[0] is, by definition, an implementation detail. Relying on undefined behavior is a recipe for bugs. If you need an immutable array, and the spec doesn't require the returned array to be immutable, you should create one yourself. [0] https://tc39.es/ecma262/multipage/indexed-collections.html#sec-array.prototype.slice https://tc39.es/ecma262/multipage/indexed-collections.html#s...
  - mofle 3y ago
    You are reading the wrong spec. That is `Array#slice`, not `TypedArray#slice`. Correct spec: https://tc39.es/ecma262/multipage/indexed-collections.html#sec-%typedarray%.prototype.slice https://tc39.es/ecma262/multipage/indexed-collections.html#s... Steps 14.g.i to 14.g.ix detail the transfer of data from the original TypedArray (O) to the new TypedArray (A). It involves reading values from the original and writing them to the new array's buffer, effectively duplicating the data segment. The process ensures both arrays are distinct with separate memory spaces.

paddy_m 3y ago

Somewhat related, who has played with performant JS dataframe libraries? I mostly want to serialize from pandas/polars to JS in a fast way, with some very minor selection and indexing operations JS side. Ideally I could find a library that does dataframe like operations from reading base64 encoded Buffers/Arrays into their JS objects, then doing the right things. I'm investigating https://arrow.apache.org/docs/js/ https://arrow.apache.org/docs/js/ https://github.com/vega/falcon https://github.com/vega/falcon https://github.com/pola-rs/nodejs-polars https://github.com/pola-rs/nodejs-polars https://github.com/uwdata/arquero https://github.com/uwdata/arquero

paddy_m 3y ago
I added a github issue to my project where this matters literally this morning. I'd love to talk to someone dealing with the same stuff. I have baked dataframe -> JS serialization utils so many times over the past decade. I never want to do much in JS, just simple selection and filtering. But you end up needing both sides of the serialization to do it right.
kylebarron 3y ago
nodejs-polars is node-specific and uses native FFI. polars can be compiled to Wasm but doesn't yet have a js API out of the box. As for the fastest way to serialize data to Pandas data to the browser, you should use Parquet; it's the fastest to write on the Python side and read on the JS side, while also being compressed. See https://github.com/kylebarron/parquet-wasm https://github.com/kylebarron/parquet-wasm (full disclosure, I wrote this)

pictur 3y ago

I guess this is our new nonsense after the ESM nonsense. Come on man start fucking all the npm packages as soon as possible. Don't forget to throw away backward compatibility.

wruza 3y ago

So the only important difference is that Buffer creates a mutable view at slice(). It could use a better/distinct name for sure, e.g. mutableView(). But do you actually want to always copy a slice that is big enough? This ideological pedantic purism which eliminates practical use cases is in my book just that - impractical pedantic purism. It’s easy to advocate for when your job is shaping landscapes without having to walk these. And yes, I’m writing this still having flashbacks from ESM-only “transition”, which was more like throwing everyone into freezing waters with the promise it will warm up eventually.

rattlesnakedave 3y ago
> So the only important difference is that Buffer creates a mutable view at slice(). No, it also (from the article): > introduces numerous methods that are not available in other JavaScript environments. Consequently, code leveraging Buffer-specific methods needs polyfilling, preventing many valuable packages from being browser-compatible.
- wruza 3y ago
  The polyfill argument is good, but it addresses a world where you just push html, css and js files. We live in a different one. It is valid, but more on a wishful thinking side rather than practical. And when you avoid calling it “polyfill”, it simply turns into a useful compatibility/extension library. I find it strange to argue about one dependency but then importing few others, with the only difference in taxonomy.
moring 3y ago
> So the only important difference is that Buffer creates a mutable view at slice(). It could use a better/distinct name for sure, e.g. mutableView(). But do you actually want to always copy a slice that is big enough? I think you are missing the point. Mutable and copied slices both have their use, and maybe a method that sometimes copies and sometimes shares does too. The problem with Buffer is that it overrides a method with well-defined semantics, then violates those. Buffer is a subclass of UInt8Array, but not a subtype nor Liskov-substitutable. In practical terms, when you get a UInt8Array and even check that it in fact is a UInt8Array, you no longer know that .slice() does even though UInt8Array's documentation explains how every UInt8Array behaves, because that documentation is now wrong. If the UInt8Array you get is a Buffer, then it is both a buffer and a UInt8Array but its .slice() behaves differently than expected.

bakkoting 3y ago

The proposal for native base64 support for Uint8Arrays is mine. I'm glad to see people are interested in using it. (So am I!) For a status update, for the last year or two the main blocker has been a conflict between a desire to have streaming support and a desire to keep the API small and simple. That's now resolved [1] by dropping streaming support, assuming I can demonstrate a reasonably efficient streaming implementation on top of the one-shot implementation, which won't be hard unless "reasonably efficient" means "with zero copies", in which case we'll need to keep arguing about it. I've also been working on documenting [2] the differences between various base64 implementations in other languages and in JS libraries to ensure we have a decent picture of the landscape when designing this. With luck, I hope to advance the proposal to stage 3 ("ready for implementations") within the next two meetings of TC39 - so either next month or January. Realistically it will probably take a little longer than that, and of course implementations take a while. But it's moving along. [1] https://github.com/tc39/proposal-arraybuffer-base64/issues/13#issuecomment-1738345569 https://github.com/tc39/proposal-arraybuffer-base64/issues/1... [2] https://gist.github.com/bakkot/16cae276209da91b652c2cb3f612af59 https://gist.github.com/bakkot/16cae276209da91b652c2cb3f612a...

bsimpson 3y ago
Thanks for doing this! I had to convert a ReadableSteam to base64 recently, and I was shocked at how much boilerplate it required in 2023: if (isReadableStream<Uint8Array>(value)) { const chunks = []; for await (const chunk of value) { chunks.push(chunk); } if (chunks[0].byteLength) { const length = chunks.reduce( (agg, next) => agg + next.length, 0 ); // Make the same species TypedArray that ReadableStream gave us. value = new (chunks[0].constructor as Uint8ArrayConstructor)(length); for (let i = 0, offset = 0; i < chunks.length; i++) { value.set(chunks[i], offset); offset += chunks[i].length; } } else { throw new Error(`Unrecognized readable stream type: ${ chunks[0].constructor.name }`); } } return `data:application/octet-stream;base64,${ btoa( (value as Uint8Array).reduce( (result, charCode) => result + String.fromCharCode(charCode), '' ) ) }`; I had to learn/implement a lot of little details to simply change the encoding from binary to its most common text representation. That's a day I would have rather spent doing product work.

cs2818 3y ago

Probably less common, but I write a lot of JS code interfacing with devices over serial connections. This makes Buffer's convenience methods for encoding and decoding data types in both big and little endian ordering really useful to me. It looks like I could accomplish the same thing using a DataView of an ArrayBuffer, but I don't see enough of a benefit to justify converting everything to this approach.

lexicality 3y ago

Having just spent the past few months dealing with the utter carnage caused by updating an old Node project to use modern dependencies, I predict doing this will cause a bloodbath as some libraries update but don't correctly specify semver for the breaking changes, or other libraries that depend on the Buffer usage not bothering to limit their dependencies to a major version unexpectedly breaking when `npm update` is run. I wonder what unmaintaned libraries will have to be dropped this time as the ecosystem grinds on and causes them to break? (How did we get to a place where everything is this bad?)

cxr 3y ago
> (How did we get to a place where everything is this bad?) Not vetting the bedrock you chose to build upon; deciding to rely on the unreliable. ("Everything" is an overstatement.)
- Exoristos 3y ago
  What's better?
  - cxr 3y ago
    Vetting it.
  - deleted 3y ago
    [deleted]

mnkypete 3y ago

My absolute go to issue these days with Node is the persistence on doing the same type/interfaces as the browser. The absolute sh* show trying to make a FormData request in Node to work with native Node streams is mind boggling, just because fetch/FormData needs to be EXACTLY like in the browser. Sure, but how about I do not want to put a 100mb file into memory just because I want to use native fetch ....

brianzelip 3y ago

Great post, like good docs; clear, informative, and with examples.

dinkleberg 3y ago

A total tangent, but it is always fun when two totally separate things you're doing connect in a weird way. I just installed some node library and then I see this unrelated blog post that is written by the creator of that library.

MrJohz 3y ago
The author has written a lot (1000+) of libraries, and earned a certain amount of controversy for how they manage them, and whether all those packages are necessary. I think the most recent controversy was that they switched all of their packages over to the new module system before the ecosystem was ready. It's one of those names where I tend to raise an eyebrow when I hear it. This looks like something similar to the module/require changes, where the overall change is good, but a slower transition may help people more.
- dinkleberg 3y ago
  Geez you aren’t kidding, this guy is prolific. I’ve seen some of his other projects on here but they all seemed to be Mac apps. Surprised to see someone churning out so many of those also has time to build and maintain 1000+ npm projects. Also this guy seems to have started the awesome list trend. I do have to wonder how many of those projects are essentially a wrapper around some native functionality just with their preferred API. Regardless, that level of scale is impressive. Thanks for the heads up though, I’ll have approach with caution.

paulddraper 3y ago

Can someone tell me when to use Uint8Array vs ArrayBuffer in my method signatures? Like, I would think that replaces a lot of Buffer usage....but the articles mentions it exactly zero times.

Aeolun 3y ago

The problem is that Buffer is just too damn convenient. I can do everything with a Uint8Array, but everything takes about 4-5x more code for no discernible benefit.

WirelessGigabit 3y ago

Personally, I believe it's better to keep on using the proven buffer vs some person's package. We don't need another left-pad. Also another reason to use a strongly typed language. You call a function that returns a buffer. The interfaces changes and you now get a UInt8Array. You will never know.

dcgudeman 3y ago
Is that what he is suggesting? I thought the article just wanted people to use UInt8Array instead of buffer.