7 ms·

Special cases like this make generic code very hard to write. If I'm writing a library that needs to handle user-provided data agnostic of type and I want to do

by tines 6y ago

Special cases like this make generic code very hard to write. If I'm writing a library that needs to handle user-provided data agnostic of type and I want to do a sum operation, now I have to add checks all over the place for whatever special cases python has.

tgv 6y ago
There's no language in which you would be able to write a function for arbitrary data structures using sum(), and get a reasonable outcome. The problem with Python is that it doesn't have typing that can express "I can sum".
- anentropic 6y ago
  Actually, I believe this is expressible in Python's type system: # Python 3.8 from typing import Protocol, TypeVar T = TypeVar('T') class Summable(Protocol[T]): def __add__(self, other: T) -> T: ... In Python 3.4-3.7 you'd need to install typing-extensions library: from typing_extensions import Protocol
  - still_grokking 6y ago
    Interesting. It seems that "every second language" gets "type-classes" these days… C++ added such feature, this above looks like a Python version, Rust has it, Scala had them as one of the first "OOP" languages. It's nice to see that Haskell pioneered a very useful and now spreading language feature!
    - sacado2 6y ago
      Ada had them before Haskell, and I'm not even sure it was the first language with them.
- naasking 6y ago
  > There's no language in which you would be able to write a function for arbitrary data structures using sum(), and get a reasonable outcome. Computing the number of bytes used by a data type seems perfectly reasonable use of sum in that context. This is also achievable in many languages. All 'sum' needs is a map from a value to a number, which can have many sensible meanings even for arbitrary data types.
  - dragonwriter 6y ago
    And that works in python to, even if the source type is strings: sum(map(mappingFunction, iterable)) works fine as long as mappingFunction maps the elements of an iterable to numbers.
deleted 6y ago
[deleted]
wirrbel 6y ago
I think never have I written generic code for summing up numbers AND concatenating strings. Even the Haskell people would agree I think. So the design problem is in overloading `+` for string and list concatenation in Python in the first place. But I guess in terms of strings let's take a moment and remember the one truly good design decision that Python devs made about strings (which isn't usual for an imperative language). Strings are immutable by default.
- noelwelsh 6y ago
  It's very common in data processing to sum up stuff (e.g. unique visitors, total visitors, total revenue, etc.) To handle this you typically want to write a generic library that makes summing up stuff easy. I can't think of an example where you'd want to add up strings (more likely lists of strings) but it is definitely the case for other data structures (sets, vectors, lists, numbers, etc.)
  - kragen 6y ago
    Python sum() can handle summing up other data structures, although, as with foldl, you may have to give it an initial accumulator argument: >>> import numpy as np >>> xs = np.arange(9).reshape((3,3)) >>> xs array([[0, 1, 2], [3, 4, 5], [6, 7, 8]]) >>> sum(xs) # builtin sum, not numpy sum array([ 9, 12, 15]) >>> sum([[3, 11], [5]], []) [3, 11, 5] >>> sum([(3, 11), (5,)], ()) (3, 11, 5) However, that doesn't work for strings; there's a special-case check for that.
- Avi-D-coder 6y ago
  We don't just write generics over types of strings ([Char], Text, etc..) we write over Semigroups and Monoids. Concatenation with <> is standard practice.
- lmm 6y ago
  > I think never have I written generic code for summing up numbers AND concatenating strings. Even the Haskell people would agree I think. Erm, Haskell people write code that can do this all the time - you write stuff in terms of generic monoid and then it just does. I can even think of a concrete case where I used those two examples: I had essentially a report that looped over a bunch of rows and accumulated something as a secondary effect, and in one client's case that was a string explaining what had happened, and in another client's case that was a count of how many things had been processed.
throwaway_pdp09 6y ago
I recently got bitten by trying to do exactly this, concat strings with sum, and spent a while trying to find out if I was being stupid (in this case, no). It damn well should work. This works from functools import reduce nums = [1,2,3] def add(p, q): return p + q print(reduce(add, nums)) strs = ["how", "are", "you"] print(reduce(add, strs)) so why not that?
- rozab 6y ago
  Python is a language that seems to make a lot of sense at first but gets a bit weird when you dig into things. The + operator particularly is a bit funny; I recently came across a case where x+=y was not equivalent to x=x+y... Though I can't figure out what it was now.
  - tsegratis 6y ago
    Agreed, but seems haskell also fails to keep commutativity here tho? Concatenation is not addition in my opinion I've often wanted something like bits++bits concatenation to work in the same way for numbers as for strings
    - throwaway_pdp09 6y ago
      > Concatenation is not addition in my opinion For python, syntactically it is. In a general sense, an operation is what you define it to be. Just my opinion.
  - theelous3 6y ago
    You were likely doing something with threads and failed to lock `x`, which I would describe as an issue that should be expected. It doesn't really have to do with `+=` so much as it has to do with being aware of race conditions on variable unprotected by a mutex.
  - G4E 6y ago
    Python's list sum are funny like that. x+=y is equivalent to x.append(y) x = x + y creates a new list, then assign it to x (highly inefficient in a loop)
Veedrac 6y ago
You needed those checks anyway. Python is just making sure you don't forget.
dragonwriter 6y ago
> If I'm writing a library that needs to handle user-provided data agnostic of type and I want to do a sum operation, now I have to add checks all over the place for whatever special cases python has. No, you don't, because: (1) if your idea of a sum is just reducing with “+”, you use functools.reduce; and (2) if your idea of sum is a mathematical sum, you use the very slight shortcut of the sum() built-in function, which is (barring things like the runtime type checking) equivalent in result to: def sum(iterable, start=0): return functools.reduce(lambda x,y: x+y, iterable, start)
sacado2 6y ago
What's a sum operation though? Is the sum of a, b and c the same thing as the sum of b, c and a?

by tines 6y ago

tgv 6y ago

There's no language in which you would be able to write a function for arbitrary data structures using sum(), and get a reasonable outcome. The problem with Python is that it doesn't have typing that can express "I can sum".

anentropic 6y ago
Actually, I believe this is expressible in Python's type system: # Python 3.8 from typing import Protocol, TypeVar T = TypeVar('T') class Summable(Protocol[T]): def __add__(self, other: T) -> T: ... In Python 3.4-3.7 you'd need to install typing-extensions library: from typing_extensions import Protocol
- still_grokking 6y ago
  Interesting. It seems that "every second language" gets "type-classes" these days… C++ added such feature, this above looks like a Python version, Rust has it, Scala had them as one of the first "OOP" languages. It's nice to see that Haskell pioneered a very useful and now spreading language feature!
  - sacado2 6y ago
    Ada had them before Haskell, and I'm not even sure it was the first language with them.
naasking 6y ago
> There's no language in which you would be able to write a function for arbitrary data structures using sum(), and get a reasonable outcome. Computing the number of bytes used by a data type seems perfectly reasonable use of sum in that context. This is also achievable in many languages. All 'sum' needs is a map from a value to a number, which can have many sensible meanings even for arbitrary data types.
- dragonwriter 6y ago
  And that works in python to, even if the source type is strings: sum(map(mappingFunction, iterable)) works fine as long as mappingFunction maps the elements of an iterable to numbers.

deleted 6y ago

[deleted]

wirrbel 6y ago

I think never have I written generic code for summing up numbers AND concatenating strings. Even the Haskell people would agree I think. So the design problem is in overloading `+` for string and list concatenation in Python in the first place. But I guess in terms of strings let's take a moment and remember the one truly good design decision that Python devs made about strings (which isn't usual for an imperative language). Strings are immutable by default.

noelwelsh 6y ago
It's very common in data processing to sum up stuff (e.g. unique visitors, total visitors, total revenue, etc.) To handle this you typically want to write a generic library that makes summing up stuff easy. I can't think of an example where you'd want to add up strings (more likely lists of strings) but it is definitely the case for other data structures (sets, vectors, lists, numbers, etc.)
- kragen 6y ago
  Python sum() can handle summing up other data structures, although, as with foldl, you may have to give it an initial accumulator argument: >>> import numpy as np >>> xs = np.arange(9).reshape((3,3)) >>> xs array([[0, 1, 2], [3, 4, 5], [6, 7, 8]]) >>> sum(xs) # builtin sum, not numpy sum array([ 9, 12, 15]) >>> sum([[3, 11], [5]], []) [3, 11, 5] >>> sum([(3, 11), (5,)], ()) (3, 11, 5) However, that doesn't work for strings; there's a special-case check for that.
Avi-D-coder 6y ago
We don't just write generics over types of strings ([Char], Text, etc..) we write over Semigroups and Monoids. Concatenation with <> is standard practice.
lmm 6y ago
> I think never have I written generic code for summing up numbers AND concatenating strings. Even the Haskell people would agree I think. Erm, Haskell people write code that can do this all the time - you write stuff in terms of generic monoid and then it just does. I can even think of a concrete case where I used those two examples: I had essentially a report that looped over a bunch of rows and accumulated something as a secondary effect, and in one client's case that was a string explaining what had happened, and in another client's case that was a count of how many things had been processed.

throwaway_pdp09 6y ago

I recently got bitten by trying to do exactly this, concat strings with sum, and spent a while trying to find out if I was being stupid (in this case, no). It damn well should work. This works from functools import reduce nums = [1,2,3] def add(p, q): return p + q print(reduce(add, nums)) strs = ["how", "are", "you"] print(reduce(add, strs)) so why not that?

rozab 6y ago
Python is a language that seems to make a lot of sense at first but gets a bit weird when you dig into things. The + operator particularly is a bit funny; I recently came across a case where x+=y was not equivalent to x=x+y... Though I can't figure out what it was now.
- tsegratis 6y ago
  Agreed, but seems haskell also fails to keep commutativity here tho? Concatenation is not addition in my opinion I've often wanted something like bits++bits concatenation to work in the same way for numbers as for strings
  - throwaway_pdp09 6y ago
    > Concatenation is not addition in my opinion For python, syntactically it is. In a general sense, an operation is what you define it to be. Just my opinion.
- theelous3 6y ago
  You were likely doing something with threads and failed to lock `x`, which I would describe as an issue that should be expected. It doesn't really have to do with `+=` so much as it has to do with being aware of race conditions on variable unprotected by a mutex.
- G4E 6y ago
  Python's list sum are funny like that. x+=y is equivalent to x.append(y) x = x + y creates a new list, then assign it to x (highly inefficient in a loop)

Veedrac 6y ago

You needed those checks anyway. Python is just making sure you don't forget.

dragonwriter 6y ago

> If I'm writing a library that needs to handle user-provided data agnostic of type and I want to do a sum operation, now I have to add checks all over the place for whatever special cases python has. No, you don't, because: (1) if your idea of a sum is just reducing with “+”, you use functools.reduce; and (2) if your idea of sum is a mathematical sum, you use the very slight shortcut of the sum() built-in function, which is (barring things like the runtime type checking) equivalent in result to: def sum(iterable, start=0): return functools.reduce(lambda x,y: x+y, iterable, start)

sacado2 6y ago

What's a sum operation though? Is the sum of a, b and c the same thing as the sum of b, c and a?