5 ms·

TC39: Add Object.groupBy and Map.groupBy

80 points by moritzwarhier 3y ago

carterschonwald 3y ago
This seems like a nice addition. Is it soemthing that a lot of tools roll their own, or is something that can’t be done in user space? I’m relatively ignorant of the limits of js
- tibbar 3y ago
  It's fairly easy to roll your own. An example implementation of groupBy in lodash, for example: https://github.com/lodash/lodash/blob/4.17.15/lodash.js#L9393 https://github.com/lodash/lodash/blob/4.17.15/lodash.js#L939.... More broadly, this is an example of a utility function for organizing data in a data structure, which JavaScript and all other major programming languages are well-equipped to implement in arbitrary ways, so this is just a convenience function. It's not like a hook into underlying environment APIs that can't be independently shimmed.
  - jauntywundrkind 3y ago
    Trying to read lodash can (to put it mildly) be difficult. Lots of code reuse. This is probably missing checking for a bunch of corner cases, but this is the general idea: // quick & dirty Object.groupBy function groupBy(iterable, callbackFn) { const out = {} for(const [key, value] in Object.entries(iterable)) { const name = callbackFn(value, key) (out[name] || (out[name] = [])).push(value) } return out } I literally wrote something very like this yesterday (after checking & finding it only got added to Node 21, and deciding not to pull in a dependency). Edit: had keyBy impl in here, fixed! Thanks commenters!
    - tibbar 3y ago
      I think `out` should be typed as a map to an array in this case; each iteration should be appending to the corresponding array instead of overwriting the entire entry.
    - quonn 3y ago
      That‘s closer to keyBy than groupBy.
    - moritzwarhier 3y ago
      Oh yes, I agree. I remember me running into the problem of internal dependencies when extracting parts of it into a non-ESM/non-build project some years ago... Like, the internal consistency of the library probably benefits, and with tree-shaking it works reasonably well but it is still a poster example of DRY vs KISS... It has its benefits, but its API surface can be annoying, too. When I encounter it, I often have to think for a second and look at the docs, e.g. some cryptic xor or map function with three parameters, one of which being an optional config object. OTOH, it can be great to have a well-tested implementation at hand for all kinds of higher-order functions, like throttle, debounce etc
    - explaininjs 3y ago
      Aaannd… now you have a prototype pollution vulnerability :) Object.from(null), always.
  - ricardobeat 3y ago
    It's very funny to point to lodash as an example of simplicity. Here is what happens when you start unraveling the code you linked to: https://gist.github.com/ricardobeat/040af80971273f5abd71c2bfcfdfdd50 https://gist.github.com/ricardobeat/040af80971273f5abd71c2bf... I got tired, but wouldn't be surprised if the whole thing goes beyond 1000 lines. It's basically a black hole. Better hope you really, really trust their test suite, it's impossible to debug and there must only be a handful of people familiar with the whole codebase.
- erulabs 3y ago
  It can be trivially rolled by hand, but it’s also a pretty common function in other languages. I’d guess most node programmers reach for lodash for this.
  - paulddraper 3y ago
    > a pretty common function in other languages Except oddities like C, C++, Perl, PHP, and Go :)
    - erulabs 3y ago
      Oh PHP programmers would just rely on MySQL's groupBy, we both know it! :P
- paavohtl 3y ago
  It can absolutely be written in JavaScript (it is a Turing complete language after all), and it is a common utility function found in most general utility libraries like Lodash and Ramda.
- moritzwarhier 3y ago
  It is indeed something that you often roll your own, write an utility function for, or include libraries like lodash for. JS standard library is notoriously deficient. This is a good addition in my view too, especially including Map. When dealing with DB/API results, one can of course always say that such grouping in the client is an anti-pattern. But in reality it can be reasonable and useful. If you write one-off algorithms in JS to transform small amounts of data on the client, you probably habe done this. And even for data-heavy applications, it could prove to be a performance benefit for functions that use Map instances during computation. Though that's just wishful thinking until the runtime developers choose to optimize these functions, if possible.
  - pmontra 3y ago
    > When dealing with DB/API results, one can of course always say that such grouping in the client is an anti-pattern. It depends on how many data you're working on. This groupBy is useful for filters in data tables. If you have to download a zillion of records and filter them, client side filtering is a bad idea. If you have a few thousands of them, it could possibly be the faster solution overall.
    - moritzwarhier 3y ago
      Exactly.
- thenbe 3y ago
  You can roll your own or use a utility library. A simple zero-dependency library would be something like just-* [1]. Although I now prefer remeda [2] as it seems to have the best typescript support, especially the strict variants such as `groupBy.strict`. [1] https://github.com/angus-c/just#just-group-by https://github.com/angus-c/just#just-group-by [2] https://remedajs.com/docs#groupBy https://remedajs.com/docs#groupBy
jackconsidine 3y ago
Anyone know if `keyBy` will also be supported? I suppose it'd be pretty trivial to take a `groupBy` and make it `keyBy`, but then again it's pretty trivial to implement `groupBy` from scratch. I no longer install lodash that often anymore
- derefr 3y ago
  I'm a big fan of the design of Elixir's Enum.group_by, which has one-parameter and two-parameter forms. The one-parameter form is like any other language's groupBy — but the two-parameter form takes two mapping closures; passes each element into both of them; and uses the output of the first closure as the grouping key, and the output of the second closure as the value to be registered under that grouping key. This flexible primitive enables you to do basically any grouping transform you want (incl. the Lodash-style keyBy) in a single short line of code. It's a lot like a sortBy operation, in that to emulate it, you would have to do a map to extract the key, producing pairs; sort (or in this case group) the pairs; and then deep-transform the pairs inside the data structure by unwrapping the keys off them. In other words, it's something that's a bit too high-friction to reach for if the language doesn't just give it to you (you'd probably do what you're planning to do some other way); but if the language does give it to you, you'll use it quite often.
- edflsafoiewq 3y ago
  For reference, keyBy returns an object with the same keys as groupBy, but the value of a key is the last element to produce that key, instead of an array of all of the elements that did.
- bakkoting 3y ago
  No current plans for `keyBy`, and I don't know that it's really that well-motivated. (I am on TC39.)
- moritzwarhier 3y ago
  You can do that by nesting Array.prototype.map in the parameter for Object.fromEntries in an easy way (probably not as optimized as a built-in, but that might be irrelevant for most cases, since its just a duplicated iteration, not quadratic)
Waterluvian 3y ago
When it comes to helper functions on existing types, I feel like you really can’t have too many… within reason. If we added like four new kinds of object/map, that complicates things… but just formalizing more common access and iteration patterns for existing structures seems low risk. What I can’t wait for are all the set operations that would make ‘Set’ truly powerful.
mbStavola 3y ago
This is one of those functions I end up implementing in pretty much every single non-trivial JS project I work on. Glad to see some movement here.
koito17 3y ago
Nice, the JS standard library is gradually getting functions I take for granted in ClojureScript. :) I noticed this part in the proposal groupBy calls callbackfn once for each element in items, in ascending order, and constructs a new Object of arrays. Each value returned by callbackfn is coerced to a property key Does JS allow arbitrary objects as keys? I am asking because `group-by` in ClojureScript is quite flexible. e.g. you can find anagrams like so (def words ["meat" "mat" "team" "mate" "eat" "tea"]) (group-by set words) ; set is a function that creates sets from collections ;; => {#{\a \e \m \t} ["meat" "team" "mate"] ;; #{\a \m \t} ["mat"] ;; #{\a \e \t} ["eat" "tea"]} I am wondering how one could translate this using Object.groupBy as specified in the proposal.
- tibbar 3y ago
  JS does indeed allow mapping arbitrary objects to keys. However, it is the memory address of the object that is hashed, not the semantic value. So, for example, if you do: objects = {} items1 = ["a", "b"] items2 = ["a", "b"] objects[items1] = 1 objects[items2] = 2 then objects[items1] will return 1, and objects[items2] will return 2, but objects[items1] !== objects[items2]. EDIT: Sorry, this only works for dictionaries that are Maps, not Objects! See the responses. My fuzzing about in the console led me astray; you should start with `objects = new Map()` instead of `objects = {}`.
  - thegeomaster 3y ago
    This is not true. I think you're confusing JS objects with Maps. This will just coalesce the key to string and overwrite one element of the object with the other. With a map it works like you described: objects = new Map() items1 = ["a", "b"] items2 = ["a", "b"] objects.set(items1, 1) objects.set(items2, 2)
  - mediumdeviation 3y ago
    Actually that only works with Map. For plain objects the key is always the stringified cast of the key. > o = { [{}]: 1 } { '[object Object]': 1 } > k = { toString() { return 'a' } } { toString: [Function: toString] } > o = { [k]: 1 } { a: 1 }
pqdbr 3y ago
On a side note, I really wish I could learn to tell exactly if a new feature like this is already supported in a project I'm working. babel.config.js is just a mistery to me. @babel/present-env, targets: { node: 'current' }, forceAllTransforms, useBuiltIns, 'corejs', modules, @babel/plugin-proposal-object-rest-spread. I mean, I generally dump it into Chat GPT and ask for an explainer, but... how can I know for sure I can use `array.reverse()` and it will be correctly handled in older browsers? And how does the babel version in yarn.lock relate to all this? What about the .browserslistrc which contains 'defaults' ? My god.
- n2d4 3y ago
  https://compat-table.github.io/compat-table/es2016plus/ https://compat-table.github.io/compat-table/es2016plus/ Enable the "show obsolete platforms" checkmark to see the data for older versions of Babel/core-js. But probably you should just upgrade that instead.
Vt71fcAqt7 3y ago
I hope they stick to these useful helper functions instead of adding more complexity like Type Annotations that provide no type soundness or runtime checks of any kind[0] [0] https://github.com/tc39/proposal-type-annotations https://github.com/tc39/proposal-type-annotations
- wffurr 3y ago
  That’s an interesting take when the stated purpose of that proposal is “to enable developers to run programs written in TypeScript, Flow, and other static typing supersets of JavaScript without any need for transpilation”. That seems like a fine goal. Allow runtimes to execute JS with type annotations as-is.
  - Vt71fcAqt7 3y ago
    The issue for me is four-fold: 1. Adds complexity to the language 2. Adds complexity to engines 3. Adds complexity to developers, especially new developers ("wait is it typed or not") 4. Most importantly, all but guaranties we will never have true types in JavaScript for things that could benefit from it like node or electron where instant compile time isn't necessary. All this for a feature only helping some developers some of the time so they can run code that will be stripped out in production to reduce size anyway on their local browser a bit more easily.
javajosh 3y ago
I don't think they should do this especially since they don't even have an Array.prototype.peek()
aragonite 3y ago
What's the thinking behind attaching Object.groupBy to the Object constructor? With the other Object.X() functions I can easily see why it makes sense they exist on the Object constructor, because they generally follow the pattern of taking an object as argument, and say, freezes that object, returns that object's [[Prototype]], returns that object's keys, check if that object is sealed, etc. By contrast, Object.groupBy is a utility function that takes an iterable and a callback, and doesn't seem to have anything distinctively to do with "Object" per se. (I guess one exception is Object.is, which takes two values as arguments. But even Object.is makes a lot of sense existing on the Object constructor because it exposes an important abstract operation (SameValue). Object.groupBy is only a utility function)
- danvk 3y ago
  What about Object.fromEntries? See https://github.com/tc39/proposal-array-grouping https://github.com/tc39/proposal-array-grouping for why this isn’t a method on Array.prototype.
  - flqn 3y ago
    Constructs an object from kv pairs, definitely assoctiated with the "Object" prototype. Groupby is more dubious
- deleted 3y ago
  [deleted]
- LegionMammal978 3y ago
  Map.groupBy() returns a Map with the keys mapped to array values. In parallel, Object.groupBy() returns an Object with the keys cast into property names. Thus, I'd say it fits the pattern of Object-related static methods, since it's using the Object mechanism as a map.
  - zeroimpl 3y ago
    The part I’m confused on is the documentation saying the returned object from Object.groupBy doesn’t extend from Object.prototype. Is it still an object? The return value of groupBy is an object that does not inherit from %Object.prototype%.
dagurp 3y ago
I would like to have .partition as well but I guess I can get by with groupBy. Array.product would be sweet as well (i.e. like product in python's itertools).
mattlondon 3y ago
In case you need a example, there is a good one here: https://github.com/tc39/proposal-array-grouping/ https://github.com/tc39/proposal-array-grouping/
deleted 3y ago
[deleted]
gabrieledarrigo 3y ago
Holy moly, the static method is pretty horrible. Quite sad.
deleted 3y ago
[deleted]

80 points by moritzwarhier 3y ago

This seems like a nice addition. Is it soemthing that a lot of tools roll their own, or is something that can’t be done in user space? I’m relatively ignorant of the limits of js

tibbar 3y ago
It's fairly easy to roll your own. An example implementation of groupBy in lodash, for example: https://github.com/lodash/lodash/blob/4.17.15/lodash.js#L9393 https://github.com/lodash/lodash/blob/4.17.15/lodash.js#L939.... More broadly, this is an example of a utility function for organizing data in a data structure, which JavaScript and all other major programming languages are well-equipped to implement in arbitrary ways, so this is just a convenience function. It's not like a hook into underlying environment APIs that can't be independently shimmed.
- jauntywundrkind 3y ago
  Trying to read lodash can (to put it mildly) be difficult. Lots of code reuse. This is probably missing checking for a bunch of corner cases, but this is the general idea: // quick & dirty Object.groupBy function groupBy(iterable, callbackFn) { const out = {} for(const [key, value] in Object.entries(iterable)) { const name = callbackFn(value, key) (out[name] || (out[name] = [])).push(value) } return out } I literally wrote something very like this yesterday (after checking & finding it only got added to Node 21, and deciding not to pull in a dependency). Edit: had keyBy impl in here, fixed! Thanks commenters!
  - tibbar 3y ago
    I think `out` should be typed as a map to an array in this case; each iteration should be appending to the corresponding array instead of overwriting the entire entry.
  - quonn 3y ago
    That‘s closer to keyBy than groupBy.
  - moritzwarhier 3y ago
    Oh yes, I agree. I remember me running into the problem of internal dependencies when extracting parts of it into a non-ESM/non-build project some years ago... Like, the internal consistency of the library probably benefits, and with tree-shaking it works reasonably well but it is still a poster example of DRY vs KISS... It has its benefits, but its API surface can be annoying, too. When I encounter it, I often have to think for a second and look at the docs, e.g. some cryptic xor or map function with three parameters, one of which being an optional config object. OTOH, it can be great to have a well-tested implementation at hand for all kinds of higher-order functions, like throttle, debounce etc
  - explaininjs 3y ago
    Aaannd… now you have a prototype pollution vulnerability :) Object.from(null), always.
- ricardobeat 3y ago
  It's very funny to point to lodash as an example of simplicity. Here is what happens when you start unraveling the code you linked to: https://gist.github.com/ricardobeat/040af80971273f5abd71c2bfcfdfdd50 https://gist.github.com/ricardobeat/040af80971273f5abd71c2bf... I got tired, but wouldn't be surprised if the whole thing goes beyond 1000 lines. It's basically a black hole. Better hope you really, really trust their test suite, it's impossible to debug and there must only be a handful of people familiar with the whole codebase.
erulabs 3y ago
It can be trivially rolled by hand, but it’s also a pretty common function in other languages. I’d guess most node programmers reach for lodash for this.
- paulddraper 3y ago
  > a pretty common function in other languages Except oddities like C, C++, Perl, PHP, and Go :)
  - erulabs 3y ago
    Oh PHP programmers would just rely on MySQL's groupBy, we both know it! :P
paavohtl 3y ago
It can absolutely be written in JavaScript (it is a Turing complete language after all), and it is a common utility function found in most general utility libraries like Lodash and Ramda.
moritzwarhier 3y ago
It is indeed something that you often roll your own, write an utility function for, or include libraries like lodash for. JS standard library is notoriously deficient. This is a good addition in my view too, especially including Map. When dealing with DB/API results, one can of course always say that such grouping in the client is an anti-pattern. But in reality it can be reasonable and useful. If you write one-off algorithms in JS to transform small amounts of data on the client, you probably habe done this. And even for data-heavy applications, it could prove to be a performance benefit for functions that use Map instances during computation. Though that's just wishful thinking until the runtime developers choose to optimize these functions, if possible.
- pmontra 3y ago
  > When dealing with DB/API results, one can of course always say that such grouping in the client is an anti-pattern. It depends on how many data you're working on. This groupBy is useful for filters in data tables. If you have to download a zillion of records and filter them, client side filtering is a bad idea. If you have a few thousands of them, it could possibly be the faster solution overall.
  - moritzwarhier 3y ago
    Exactly.
thenbe 3y ago
You can roll your own or use a utility library. A simple zero-dependency library would be something like just-* [1]. Although I now prefer remeda [2] as it seems to have the best typescript support, especially the strict variants such as `groupBy.strict`. [1] https://github.com/angus-c/just#just-group-by https://github.com/angus-c/just#just-group-by [2] https://remedajs.com/docs#groupBy https://remedajs.com/docs#groupBy

jackconsidine 3y ago

Anyone know if `keyBy` will also be supported? I suppose it'd be pretty trivial to take a `groupBy` and make it `keyBy`, but then again it's pretty trivial to implement `groupBy` from scratch. I no longer install lodash that often anymore

derefr 3y ago
I'm a big fan of the design of Elixir's Enum.group_by, which has one-parameter and two-parameter forms. The one-parameter form is like any other language's groupBy — but the two-parameter form takes two mapping closures; passes each element into both of them; and uses the output of the first closure as the grouping key, and the output of the second closure as the value to be registered under that grouping key. This flexible primitive enables you to do basically any grouping transform you want (incl. the Lodash-style keyBy) in a single short line of code. It's a lot like a sortBy operation, in that to emulate it, you would have to do a map to extract the key, producing pairs; sort (or in this case group) the pairs; and then deep-transform the pairs inside the data structure by unwrapping the keys off them. In other words, it's something that's a bit too high-friction to reach for if the language doesn't just give it to you (you'd probably do what you're planning to do some other way); but if the language does give it to you, you'll use it quite often.
edflsafoiewq 3y ago
For reference, keyBy returns an object with the same keys as groupBy, but the value of a key is the last element to produce that key, instead of an array of all of the elements that did.
bakkoting 3y ago
No current plans for `keyBy`, and I don't know that it's really that well-motivated. (I am on TC39.)
moritzwarhier 3y ago
You can do that by nesting Array.prototype.map in the parameter for Object.fromEntries in an easy way (probably not as optimized as a built-in, but that might be irrelevant for most cases, since its just a duplicated iteration, not quadratic)

Waterluvian 3y ago

When it comes to helper functions on existing types, I feel like you really can’t have too many… within reason. If we added like four new kinds of object/map, that complicates things… but just formalizing more common access and iteration patterns for existing structures seems low risk. What I can’t wait for are all the set operations that would make ‘Set’ truly powerful.

mbStavola 3y ago

This is one of those functions I end up implementing in pretty much every single non-trivial JS project I work on. Glad to see some movement here.

koito17 3y ago

Nice, the JS standard library is gradually getting functions I take for granted in ClojureScript. :) I noticed this part in the proposal groupBy calls callbackfn once for each element in items, in ascending order, and constructs a new Object of arrays. Each value returned by callbackfn is coerced to a property key Does JS allow arbitrary objects as keys? I am asking because `group-by` in ClojureScript is quite flexible. e.g. you can find anagrams like so (def words ["meat" "mat" "team" "mate" "eat" "tea"]) (group-by set words) ; set is a function that creates sets from collections ;; => {#{\a \e \m \t} ["meat" "team" "mate"] ;; #{\a \m \t} ["mat"] ;; #{\a \e \t} ["eat" "tea"]} I am wondering how one could translate this using Object.groupBy as specified in the proposal.

tibbar 3y ago
JS does indeed allow mapping arbitrary objects to keys. However, it is the memory address of the object that is hashed, not the semantic value. So, for example, if you do: objects = {} items1 = ["a", "b"] items2 = ["a", "b"] objects[items1] = 1 objects[items2] = 2 then objects[items1] will return 1, and objects[items2] will return 2, but objects[items1] !== objects[items2]. EDIT: Sorry, this only works for dictionaries that are Maps, not Objects! See the responses. My fuzzing about in the console led me astray; you should start with `objects = new Map()` instead of `objects = {}`.
- thegeomaster 3y ago
  This is not true. I think you're confusing JS objects with Maps. This will just coalesce the key to string and overwrite one element of the object with the other. With a map it works like you described: objects = new Map() items1 = ["a", "b"] items2 = ["a", "b"] objects.set(items1, 1) objects.set(items2, 2)
- mediumdeviation 3y ago
  Actually that only works with Map. For plain objects the key is always the stringified cast of the key. > o = { [{}]: 1 } { '[object Object]': 1 } > k = { toString() { return 'a' } } { toString: [Function: toString] } > o = { [k]: 1 } { a: 1 }

pqdbr 3y ago

On a side note, I really wish I could learn to tell exactly if a new feature like this is already supported in a project I'm working. babel.config.js is just a mistery to me. @babel/present-env, targets: { node: 'current' }, forceAllTransforms, useBuiltIns, 'corejs', modules, @babel/plugin-proposal-object-rest-spread. I mean, I generally dump it into Chat GPT and ask for an explainer, but... how can I know for sure I can use `array.reverse()` and it will be correctly handled in older browsers? And how does the babel version in yarn.lock relate to all this? What about the .browserslistrc which contains 'defaults' ? My god.

n2d4 3y ago
https://compat-table.github.io/compat-table/es2016plus/ https://compat-table.github.io/compat-table/es2016plus/ Enable the "show obsolete platforms" checkmark to see the data for older versions of Babel/core-js. But probably you should just upgrade that instead.

Vt71fcAqt7 3y ago

I hope they stick to these useful helper functions instead of adding more complexity like Type Annotations that provide no type soundness or runtime checks of any kind[0] [0] https://github.com/tc39/proposal-type-annotations https://github.com/tc39/proposal-type-annotations

wffurr 3y ago
That’s an interesting take when the stated purpose of that proposal is “to enable developers to run programs written in TypeScript, Flow, and other static typing supersets of JavaScript without any need for transpilation”. That seems like a fine goal. Allow runtimes to execute JS with type annotations as-is.
- Vt71fcAqt7 3y ago
  The issue for me is four-fold: 1. Adds complexity to the language 2. Adds complexity to engines 3. Adds complexity to developers, especially new developers ("wait is it typed or not") 4. Most importantly, all but guaranties we will never have true types in JavaScript for things that could benefit from it like node or electron where instant compile time isn't necessary. All this for a feature only helping some developers some of the time so they can run code that will be stripped out in production to reduce size anyway on their local browser a bit more easily.

javajosh 3y ago

I don't think they should do this especially since they don't even have an Array.prototype.peek()

aragonite 3y ago

What's the thinking behind attaching Object.groupBy to the Object constructor? With the other Object.X() functions I can easily see why it makes sense they exist on the Object constructor, because they generally follow the pattern of taking an object as argument, and say, freezes that object, returns that object's [[Prototype]], returns that object's keys, check if that object is sealed, etc. By contrast, Object.groupBy is a utility function that takes an iterable and a callback, and doesn't seem to have anything distinctively to do with "Object" per se. (I guess one exception is Object.is, which takes two values as arguments. But even Object.is makes a lot of sense existing on the Object constructor because it exposes an important abstract operation (SameValue). Object.groupBy is only a utility function)

danvk 3y ago
What about Object.fromEntries? See https://github.com/tc39/proposal-array-grouping https://github.com/tc39/proposal-array-grouping for why this isn’t a method on Array.prototype.
- flqn 3y ago
  Constructs an object from kv pairs, definitely assoctiated with the "Object" prototype. Groupby is more dubious
deleted 3y ago
[deleted]
LegionMammal978 3y ago
Map.groupBy() returns a Map with the keys mapped to array values. In parallel, Object.groupBy() returns an Object with the keys cast into property names. Thus, I'd say it fits the pattern of Object-related static methods, since it's using the Object mechanism as a map.
- zeroimpl 3y ago
  The part I’m confused on is the documentation saying the returned object from Object.groupBy doesn’t extend from Object.prototype. Is it still an object? The return value of groupBy is an object that does not inherit from %Object.prototype%.

dagurp 3y ago

I would like to have .partition as well but I guess I can get by with groupBy. Array.product would be sweet as well (i.e. like product in python's itertools).

mattlondon 3y ago

In case you need a example, there is a good one here: https://github.com/tc39/proposal-array-grouping/ https://github.com/tc39/proposal-array-grouping/

deleted 3y ago

[deleted]

gabrieledarrigo 3y ago

Holy moly, the static method is pretty horrible. Quite sad.

deleted 3y ago

[deleted]