18 ms·

> An alternative to the + operator is the pipe | operator, which is used for set union. This suggestion did not receive much support on Python-Ideas. That's di

by kbd 8y ago

> An alternative to the + operator is the pipe | operator, which is used for set union. This suggestion did not receive much support on Python-Ideas. That's disappointing. It's always been on my Python wish list that dicts would subclass sets, as dicts are essentially sets with values attached. Pretty much everywhere you can use a set you can use a dict and it acts like the set of its keys. For example: >>> s = {'a','b','c'} >>> d = {i: i.upper() for i in s} >>> list(d) == list(s) True Dictionaries have been moving in this more ergonomic direction for a while. Originally, to union two dictionaries you had to say: >>> d2 = {'d': 'D'} >>> >>> d3 = d.copy() >>> d3.update(d2) >>> d3 {'a': 'A', 'b': 'B', 'c': 'C', 'd': 'D'} Nowadays, as the PEP points out, you can just say: >>> {**d, **d2} {'a': 'A', 'b': 'B', 'c': 'C', 'd': 'D'} There's no reason you shouldn't have always been able to say d | d2, same as sets. Now I finally get my wish that dictionaries will behave more similarly to sets and they use the wrong set of operators.

deleted 8y ago
[deleted]
guitarbill 8y ago
dict.keys() pretty much does just that: >>> a = {"foo": 1} >>> b = {"bar"} >>> a.keys() | b {'bar', 'foo'} As an aside, I like the plus operator. Begin able to merge two dictionaries in one line and have the result be a new dict is something I've needed often enough. {**d, **d2} works, but is pretty recent and still feels weird to me (not coming from a language that makes use of destructuring a lot, like Javascript).
- kbd 8y ago
  > dict.keys() pretty much does just that... a.keys() | b You're illustrating my point: >>> set(a) | b {'bar', 'foo'}
  - guitarbill 8y ago
    Yeah, I largely agree. Is that unusual these days? :D And I'd be sort of okay if dicts implemented set operators, although I don't think using `set()` or `.keys()` is a big ask. But using the pipe operator for the operation in this PEP would be a bad idea IMO: For sets, a | b == b | a. For dicts, not necessarily. So if they used the pipe operator, that could lead to surprising or unintuitive results. On the plus side, not using pipe still leaves it open for future usage, so you might get your wish yet.
    - kbd 8y ago
      Your exact point that for sets, union operator is commutative while for dictionaries it wouldn't be is one of the main objections I foresaw if I ever wrote this up as a PEP. Counterpoint: for numbers, + is commutative but for lists it's not, so it's normal for the same operator to have different commutativity depending on the type of the operands. IMO it's worth using the set operators because of the subclass relation of sets and dicts despite some small (but really, predictable) changes in behavior, but I can see how someone could have a different opinion.
    - makapuf 8y ago
      The extension operator still resembles more a set union than an addition. The fact that it is not commutative is not imo an argument since sum is even more frequently used with a commutative semantics.
103e 8y ago
If you treat dictionaries as sets of tuples union doesn’t work as expected: {(‘a’,1)} | {(‘a’, 2)} = {(‘a’,1), (‘a’, 2)} Same key maps two values.
- deleted 8y ago
  [deleted]
- coldtea 8y ago
  That's not what the parent says though. He says that sets are like dictionaries without values, e.g. a set is akin to the dictionary keys. So, in your example the ('a', 1) and ('a', 2) are the keys (in how the parent argues about) -- it's not 'a' that's the key. Same like you can do today: d = {} d[('a', 1)] = 6 d[('a', 2)] = 89 We can still express that dictionary as a set of tuples, it's just: {(('a', 1), 6), (('a', 2), 89)}
  - 103e 8y ago
    Meant to respond to top level parent who said: > dicts would subclass sets Was trying to make the case as to why treating dictionaries as a child of <generic collection> with an extend/merge operation using ‘+’ rather than a child of set with a union operation ‘|’ makes more sense (to me). Changing the behaviour of a well defined operation like union seems bad - although my case is somewhat undermined by python’s overloading of ‘+’ to mean extend.
Sean1708 8y ago
> as dicts are essentially sets with values attached. Interestingly enough some languages actually do the opposite. In Rust for example a set is literally just a dictionary with unit as the value[0] and unit is essentially a way of expressing the absence of a value (it takes up no space in memory, and you just can't do anything with it). [0]: https://doc.rust-lang.org/stable/src/std/collections/hash/set.rs.html#111-113 https://doc.rust-lang.org/stable/src/std/collections/hash/se... For posterity, the above link shows: pub struct HashSet<T, S = RandomState> { map: HashMap<T, (), S>, }
- cube2222 8y ago
  Same as Go, where everybody just uses map[Key]struct{} as sets.
  - bpicolo 8y ago
    To be fair, this is because the language does not support a type safe set type. I would use one frequently if it did
    - majewsky 8y ago
      map[T]struct{} is a type-safe set type. It's just not an ergonomic one. FWIW, I'm usually using map[T]bool and only ever inserting `true` values. It uses a bit more space, but membership checks read like if set[key] { instead of if _, ok := set[key]; ok {
- kbd 8y ago
  > Interestingly enough some languages actually do the opposite. Of course you can represent sets as dictionaries with empty values (ask anyone who programmed Perl). You're supporting my point that dicts logically subclass sets, because they can represent sets where the values can be other things as well. You're also getting at what the behavior should be if you union a dictionary and a set. Hypothetical Python: >>> s = {'a','b','c'} >>> d = {'d': 'D'} >>> d | s >>> {'d': 'D', 'a': None, 'b': None, 'c': None}
benj111 8y ago
Yes, its not so bad for the + case, but the - case seems non obvious at best.
eesmith 8y ago
"dicts would subclass sets, as dicts are essentially sets with values attached" Such a derivation would violate the Liskov substitution principle. Consider the following with set: x = {"one", "two"} y = set() y.update(x) y It result in y being {'two', 'one'} . Now, do the same with dict: y = dict() y.update(x) This gives the exception: "ValueError: dictionary update sequence element #0 has length 3; 2 is required" This means that dict cannot be used anywhere that a set can be used, which means it violates the Liskov substitution principle (see https://en.wikipedia.org/wiki/Liskov_substitution_principle https://en.wikipedia.org/wiki/Liskov_substitution_principle ) which means that if covariant methods are needed for good design then dict cannot be a subclass of set.
- kbd 8y ago
  If dicts did subclass sets, then sets would be dicts whose values are all None. In other words, your last example would be defined as: >>> s = {'one', 'two'} >>> d = {} >>> d.update(s) >>> d {'two': None, 'one': None}
  - deleted 8y ago
    [deleted]
  - pdpi 8y ago
    If sets are dicts with values of None, then they're dicts, not a superclass of dict.
  - eesmith 8y ago
    Would s["one"] = 1 raise an exception? Or convert the set into a dict? Or change the sentinel value for all the set elements? None seem like a good design since it means either the instance change its class on the fly (which Python does support) or that a dict does not act like its parent set object, breaking the is-a relationship most people expect from an OO design. It seems like the circle/ellipse problem, and the current implementation is the "drop all inheritance relationships" solution to that problem. https://en.wikipedia.org/wiki/Circle-ellipse_problem#Drop_all_inheritance_relationships https://en.wikipedia.org/wiki/Circle-ellipse_problem#Drop_al...
dtech 8y ago
> It's always been on my Python wish list that dicts would subclass sets, as dicts are essentially sets with values attached. > There's no reason you shouldn't have always been able to say d | d2, same as sets. I don't agree with this view, mainly because merging a dict is not associative, while unionizing a set is. The actual operation for "a + b" is "add everything from b set to a", and + more closely resembles that than |.
dan-robertson 8y ago
The most compelling reason to not do this is that (I claim) it’s not super obvious what to do when the keys are equal. In: { 'a' : 1 } | { 'a' : 2 } Should the result be: { 'a' : 1 } (prioritise the left hand side), or { 'a' : 2 } (prioritise the right hand side), or should it raise an error? Maybe a fourth option would be do downgrade to sets of keys and give: { 'a' } A fifth option is to magically merge values: { 'a' : 3 } or { 'a' : (1,2) } For the first two choices one loses commutativity which means that code then suddenly has to have previously cared about it (or it will do the wrong thing), even though it didn’t previously matter, and one is always potentially losing data. The third choice is safe but could cause unforeseen problems later if shared keys only happen rarely. The fourth choice also forgets a bunch of information held in the dict. In a language like Haskell, one can use traits to specify how to merge values (Monoid) but without traits (and a way to choose which trait to use) I think some kind of magic merge is not great. I claim the operations one should really want with dicts are not set operations but rather more relational ones, ie {inner,outer,left,right} joins on the keys followed by some mapping to decide how to merge values.
- zimablue 8y ago
  Great post, sets have nice properties that dictionaries don't have. Making them act similarly seems like a trap
- endgame 8y ago
  Unfortunately, even in Haskell Data.Map.Map's monoid instance is left-biased. There is the monoidal-containers package which newtype-wraps Data.Map.Map to have instance Monoid m => Monoid (MonoidalMap k m), which I think is much more sensible.
  - sametmax 8y ago
    Besides, anytime somebody compare Python to Haskell, the battle is over. They have completly different use cases and philosophy. If you want something in Haskell, you probably want the opposite in Python.
    - 8y ago
sametmax 8y ago
Practically, what your wish would accomplish ? Will it make most people more productive ? Produce less bug ? Learn faster ? Most Python coders don't even use sets more than once a year. Hell, I use collections.deque more than sets. But dicts ? We use it all the time. In fact, failing a {} + {} is a recurring disapointment in all my classrooms. Plus, in PHP and JS, arrays/objects are the "do-it-all" data structure. And it's horrible. You see the same data structure everywhere. You have to read in details what something is, and what it's for. It very nice that dict and set are very distincts, and that they have a distint set of operators. This way it's super easy to scan the code and know what the data structure is, and what it's used for. That's why I always teach set([1, 2]) and not {1, 2} first. It helps people to make a clear distinction in their mind.
- zimablue 8y ago
  I use sets a lot, sample size of one. I do data science/engineering stuff, sets of columns/sets of keys. I'm very sceptical of your claim that most developers don't use sets.
  - sametmax 8y ago
    Actually, a lot of Python users don't even know sets exist. Or they forgot, and use them randomly after they google "remove duplicates". Even among the sets users, a lot of them don't even know you can use ^, | and ~ with them. Size sample: a few hundred students and colleagues over 15 years. It make sense: for columns, it's quite commonto use dicts, or pandas dataframe. The sets of key is just the dict keys memory view. A web dev rarely needs sets. A GUI coders, a sysadmin or a geograph neither. It's not that sets are not useful, it's just that in the huge numbers of things you need to do in programming, accross all fields that Python attends to, they are pretty niche.
- rocqua 8y ago
  > It very nice that dict and set are very distincts, and that they have a distint set of operators. This way it's super easy to scan the code and know what the data structure is, and what it's used for. This is exactly why I want explicit typing in Python.
Someone 8y ago
”that dicts would subclass sets, as dicts are essentially sets with values attached.” I think a variant of https://en.wikipedia.org/wiki/Composition_over_inheritance https://en.wikipedia.org/wiki/Composition_over_inheritance applies here. Inheritance, in general, only is a good idea if there is a strong isa relation. A dictionary isn’t a set of keys, it has a set of keys. If you want to see a dictionary as a set, I think the better view would be to see it as a set of (key,value) pairs where equality of pairs is defined as equality of the key parts, ignoring the ‘value’ parts. I think it makes sense to require that such a set should behave identical to a dictionary, and that providing a ‘real’ dictionary is just an optimization, plus the addition of convenience functions, e.g. to get the set of keys. If one sees things that way, one could even define the dictionary interface as taking an equality operation on the keys and a ‘value combiner’ function that combines values, and will be used in the cases you outline (that function could add integers, concatenate strings, keep the larger value, or whatever the programmer specifies)

by kbd 8y ago

deleted 8y ago

[deleted]

guitarbill 8y ago

dict.keys() pretty much does just that: >>> a = {"foo": 1} >>> b = {"bar"} >>> a.keys() | b {'bar', 'foo'} As an aside, I like the plus operator. Begin able to merge two dictionaries in one line and have the result be a new dict is something I've needed often enough. {**d, **d2} works, but is pretty recent and still feels weird to me (not coming from a language that makes use of destructuring a lot, like Javascript).

kbd 8y ago
> dict.keys() pretty much does just that... a.keys() | b You're illustrating my point: >>> set(a) | b {'bar', 'foo'}
- guitarbill 8y ago
  Yeah, I largely agree. Is that unusual these days? :D And I'd be sort of okay if dicts implemented set operators, although I don't think using `set()` or `.keys()` is a big ask. But using the pipe operator for the operation in this PEP would be a bad idea IMO: For sets, a | b == b | a. For dicts, not necessarily. So if they used the pipe operator, that could lead to surprising or unintuitive results. On the plus side, not using pipe still leaves it open for future usage, so you might get your wish yet.
  - kbd 8y ago
    Your exact point that for sets, union operator is commutative while for dictionaries it wouldn't be is one of the main objections I foresaw if I ever wrote this up as a PEP. Counterpoint: for numbers, + is commutative but for lists it's not, so it's normal for the same operator to have different commutativity depending on the type of the operands. IMO it's worth using the set operators because of the subclass relation of sets and dicts despite some small (but really, predictable) changes in behavior, but I can see how someone could have a different opinion.
  - makapuf 8y ago
    The extension operator still resembles more a set union than an addition. The fact that it is not commutative is not imo an argument since sum is even more frequently used with a commutative semantics.

103e 8y ago

If you treat dictionaries as sets of tuples union doesn’t work as expected: {(‘a’,1)} | {(‘a’, 2)} = {(‘a’,1), (‘a’, 2)} Same key maps two values.

deleted 8y ago
[deleted]
coldtea 8y ago
That's not what the parent says though. He says that sets are like dictionaries without values, e.g. a set is akin to the dictionary keys. So, in your example the ('a', 1) and ('a', 2) are the keys (in how the parent argues about) -- it's not 'a' that's the key. Same like you can do today: d = {} d[('a', 1)] = 6 d[('a', 2)] = 89 We can still express that dictionary as a set of tuples, it's just: {(('a', 1), 6), (('a', 2), 89)}
- 103e 8y ago
  Meant to respond to top level parent who said: > dicts would subclass sets Was trying to make the case as to why treating dictionaries as a child of <generic collection> with an extend/merge operation using ‘+’ rather than a child of set with a union operation ‘|’ makes more sense (to me). Changing the behaviour of a well defined operation like union seems bad - although my case is somewhat undermined by python’s overloading of ‘+’ to mean extend.

Sean1708 8y ago

> as dicts are essentially sets with values attached. Interestingly enough some languages actually do the opposite. In Rust for example a set is literally just a dictionary with unit as the value[0] and unit is essentially a way of expressing the absence of a value (it takes up no space in memory, and you just can't do anything with it). [0]: https://doc.rust-lang.org/stable/src/std/collections/hash/set.rs.html#111-113 https://doc.rust-lang.org/stable/src/std/collections/hash/se... For posterity, the above link shows: pub struct HashSet<T, S = RandomState> { map: HashMap<T, (), S>, }

cube2222 8y ago
Same as Go, where everybody just uses map[Key]struct{} as sets.
- bpicolo 8y ago
  To be fair, this is because the language does not support a type safe set type. I would use one frequently if it did
  - majewsky 8y ago
    map[T]struct{} is a type-safe set type. It's just not an ergonomic one. FWIW, I'm usually using map[T]bool and only ever inserting `true` values. It uses a bit more space, but membership checks read like if set[key] { instead of if _, ok := set[key]; ok {
kbd 8y ago
> Interestingly enough some languages actually do the opposite. Of course you can represent sets as dictionaries with empty values (ask anyone who programmed Perl). You're supporting my point that dicts logically subclass sets, because they can represent sets where the values can be other things as well. You're also getting at what the behavior should be if you union a dictionary and a set. Hypothetical Python: >>> s = {'a','b','c'} >>> d = {'d': 'D'} >>> d | s >>> {'d': 'D', 'a': None, 'b': None, 'c': None}

benj111 8y ago

Yes, its not so bad for the + case, but the - case seems non obvious at best.

eesmith 8y ago

"dicts would subclass sets, as dicts are essentially sets with values attached" Such a derivation would violate the Liskov substitution principle. Consider the following with set: x = {"one", "two"} y = set() y.update(x) y It result in y being {'two', 'one'} . Now, do the same with dict: y = dict() y.update(x) This gives the exception: "ValueError: dictionary update sequence element #0 has length 3; 2 is required" This means that dict cannot be used anywhere that a set can be used, which means it violates the Liskov substitution principle (see https://en.wikipedia.org/wiki/Liskov_substitution_principle https://en.wikipedia.org/wiki/Liskov_substitution_principle ) which means that if covariant methods are needed for good design then dict cannot be a subclass of set.

kbd 8y ago
If dicts did subclass sets, then sets would be dicts whose values are all None. In other words, your last example would be defined as: >>> s = {'one', 'two'} >>> d = {} >>> d.update(s) >>> d {'two': None, 'one': None}
- deleted 8y ago
  [deleted]
- pdpi 8y ago
  If sets are dicts with values of None, then they're dicts, not a superclass of dict.
- eesmith 8y ago
  Would s["one"] = 1 raise an exception? Or convert the set into a dict? Or change the sentinel value for all the set elements? None seem like a good design since it means either the instance change its class on the fly (which Python does support) or that a dict does not act like its parent set object, breaking the is-a relationship most people expect from an OO design. It seems like the circle/ellipse problem, and the current implementation is the "drop all inheritance relationships" solution to that problem. https://en.wikipedia.org/wiki/Circle-ellipse_problem#Drop_all_inheritance_relationships https://en.wikipedia.org/wiki/Circle-ellipse_problem#Drop_al...

dtech 8y ago

> It's always been on my Python wish list that dicts would subclass sets, as dicts are essentially sets with values attached. > There's no reason you shouldn't have always been able to say d | d2, same as sets. I don't agree with this view, mainly because merging a dict is not associative, while unionizing a set is. The actual operation for "a + b" is "add everything from b set to a", and + more closely resembles that than |.

dan-robertson 8y ago

The most compelling reason to not do this is that (I claim) it’s not super obvious what to do when the keys are equal. In: { 'a' : 1 } | { 'a' : 2 } Should the result be: { 'a' : 1 } (prioritise the left hand side), or { 'a' : 2 } (prioritise the right hand side), or should it raise an error? Maybe a fourth option would be do downgrade to sets of keys and give: { 'a' } A fifth option is to magically merge values: { 'a' : 3 } or { 'a' : (1,2) } For the first two choices one loses commutativity which means that code then suddenly has to have previously cared about it (or it will do the wrong thing), even though it didn’t previously matter, and one is always potentially losing data. The third choice is safe but could cause unforeseen problems later if shared keys only happen rarely. The fourth choice also forgets a bunch of information held in the dict. In a language like Haskell, one can use traits to specify how to merge values (Monoid) but without traits (and a way to choose which trait to use) I think some kind of magic merge is not great. I claim the operations one should really want with dicts are not set operations but rather more relational ones, ie {inner,outer,left,right} joins on the keys followed by some mapping to decide how to merge values.

zimablue 8y ago
Great post, sets have nice properties that dictionaries don't have. Making them act similarly seems like a trap
endgame 8y ago
Unfortunately, even in Haskell Data.Map.Map's monoid instance is left-biased. There is the monoidal-containers package which newtype-wraps Data.Map.Map to have instance Monoid m => Monoid (MonoidalMap k m), which I think is much more sensible.
- sametmax 8y ago
  Besides, anytime somebody compare Python to Haskell, the battle is over. They have completly different use cases and philosophy. If you want something in Haskell, you probably want the opposite in Python.
  - 8y ago

sametmax 8y ago

Practically, what your wish would accomplish ? Will it make most people more productive ? Produce less bug ? Learn faster ? Most Python coders don't even use sets more than once a year. Hell, I use collections.deque more than sets. But dicts ? We use it all the time. In fact, failing a {} + {} is a recurring disapointment in all my classrooms. Plus, in PHP and JS, arrays/objects are the "do-it-all" data structure. And it's horrible. You see the same data structure everywhere. You have to read in details what something is, and what it's for. It very nice that dict and set are very distincts, and that they have a distint set of operators. This way it's super easy to scan the code and know what the data structure is, and what it's used for. That's why I always teach set([1, 2]) and not {1, 2} first. It helps people to make a clear distinction in their mind.

zimablue 8y ago
I use sets a lot, sample size of one. I do data science/engineering stuff, sets of columns/sets of keys. I'm very sceptical of your claim that most developers don't use sets.
- sametmax 8y ago
  Actually, a lot of Python users don't even know sets exist. Or they forgot, and use them randomly after they google "remove duplicates". Even among the sets users, a lot of them don't even know you can use ^, | and ~ with them. Size sample: a few hundred students and colleagues over 15 years. It make sense: for columns, it's quite commonto use dicts, or pandas dataframe. The sets of key is just the dict keys memory view. A web dev rarely needs sets. A GUI coders, a sysadmin or a geograph neither. It's not that sets are not useful, it's just that in the huge numbers of things you need to do in programming, accross all fields that Python attends to, they are pretty niche.
rocqua 8y ago
> It very nice that dict and set are very distincts, and that they have a distint set of operators. This way it's super easy to scan the code and know what the data structure is, and what it's used for. This is exactly why I want explicit typing in Python.

Someone 8y ago

”that dicts would subclass sets, as dicts are essentially sets with values attached.” I think a variant of https://en.wikipedia.org/wiki/Composition_over_inheritance https://en.wikipedia.org/wiki/Composition_over_inheritance applies here. Inheritance, in general, only is a good idea if there is a strong isa relation. A dictionary isn’t a set of keys, it has a set of keys. If you want to see a dictionary as a set, I think the better view would be to see it as a set of (key,value) pairs where equality of pairs is defined as equality of the key parts, ignoring the ‘value’ parts. I think it makes sense to require that such a set should behave identical to a dictionary, and that providing a ‘real’ dictionary is just an optimization, plus the addition of convenience functions, e.g. to get the set of keys. If one sees things that way, one could even define the dictionary interface as taking an equality operation on the keys and a ‘value combiner’ function that combines values, and will be used in the cases you outline (that function could add integers, concatenate strings, keep the larger value, or whatever the programmer specifies)