5 ms·

One thing that I don't like with Python's scientific libraries is how they change the overall Python syntax. There are so many ways to slice an array or a data

One thing that I don't like with Python's scientific libraries is how they change the overall Python syntax. There are so many ways to slice an array or a dataframe, and only a few of them are valid Python code. Keeping the language API should have been a priority, but that is a consequence of operator overloading features.

kzrdude 5y ago
Can you explain what you mean more in detail? Libraries can't change the syntax of the Python language, not in the formal sense. Is this about things you want to be able to express in syntax but can't? Or the other way around - stuff that uses syntax/operators but should really be methods?
- ohyoutravel 5y ago
  Numpy syntax comes to mind. The extra commas often aren’t valid pure Python but are required for some operations on numpy arrays. I don’t know how this works under the hood, but expect it’s a state machine under the numpy ndarray looking for the extra commas and such. i.e. some_array[0:5,0] which isn’t valid pure Python notation.
  - kzrdude 5y ago
    Extra commas are "valid in pure python" in the following sense that I can demonstrate. Open ipython3 In [3]: class Test: ...: def __getitem__(self, index): ...: print(index) ...: In [4]: Test()[1, 2, 1:3, ..., :] (1, 2, slice(1, 3, None), Ellipsis, slice(None, None, None)) It's valid and we get the complicated tuple of integers, slices, ellipsis etc as printed. Numpy has existed for a long time. Its needs have been taken care of in upstream Python, to a big extent, and other libraries can use the same features.
    - ohyoutravel 5y ago
      Interesting! Neither myself nor my coworkers could get the snippet I posted working outside the context of an ndarray, so I had speculated at that time that it there was something else going on under the hood. You seem to have a much better grasp of Python than us, would you mind posting an example where the snipped I posted successfully accesses data from an array in pure Python? That way I can not only take the L, but correct the record and learn something in the process.
      - kzrdude 5y ago
        This program is quick & lazy but it uses a 1D python list and pretends it's a 2D list. It implements 2D slicing, giving you a square subset just like ndarray. It doesn't intend to be all correct or nice or useful. https://www.pythonmorsels.com/p/2rk5t/ https://www.pythonmorsels.com/p/2rk5t/ Is laziness a virtue? I reused the slicing implementation in `range(foo)[index]` so I didn't need to do that logic myself.
        ohyoutravel 5y ago
        Very nice!
canjobear 5y ago
I'm not sure what you're referring to. Nothing you import into Python changes its syntax. Maybe you're thinking of things like x[:, np.newaxis] where x is a numpy array? This is valid Python code outside of numpy as well, although the built-in data structures like lists and dicts won't know what to do with the :.
- wheelerof4te 5y ago
  To be precise, you could model the same behavior on your custom types by using the dunder method magic. In that case, everything is "valid" Python code. Numpy and Pandas libraries have some non-standard ways to slice arrays, get the subarrays and the data from them.
lvass 5y ago
What language wouldn't suffer from this, besides APL? Even very recent and well designed libraries like Elixir's Nx look like another APL-like language bolted on. Pipe syntax helps but not much.

kzrdude 5y ago

Can you explain what you mean more in detail? Libraries can't change the syntax of the Python language, not in the formal sense. Is this about things you want to be able to express in syntax but can't? Or the other way around - stuff that uses syntax/operators but should really be methods?

ohyoutravel 5y ago
Numpy syntax comes to mind. The extra commas often aren’t valid pure Python but are required for some operations on numpy arrays. I don’t know how this works under the hood, but expect it’s a state machine under the numpy ndarray looking for the extra commas and such. i.e. some_array[0:5,0] which isn’t valid pure Python notation.
- kzrdude 5y ago
  Extra commas are "valid in pure python" in the following sense that I can demonstrate. Open ipython3 In [3]: class Test: ...: def __getitem__(self, index): ...: print(index) ...: In [4]: Test()[1, 2, 1:3, ..., :] (1, 2, slice(1, 3, None), Ellipsis, slice(None, None, None)) It's valid and we get the complicated tuple of integers, slices, ellipsis etc as printed. Numpy has existed for a long time. Its needs have been taken care of in upstream Python, to a big extent, and other libraries can use the same features.
  - ohyoutravel 5y ago
    Interesting! Neither myself nor my coworkers could get the snippet I posted working outside the context of an ndarray, so I had speculated at that time that it there was something else going on under the hood. You seem to have a much better grasp of Python than us, would you mind posting an example where the snipped I posted successfully accesses data from an array in pure Python? That way I can not only take the L, but correct the record and learn something in the process.
    - kzrdude 5y ago
      This program is quick & lazy but it uses a 1D python list and pretends it's a 2D list. It implements 2D slicing, giving you a square subset just like ndarray. It doesn't intend to be all correct or nice or useful. https://www.pythonmorsels.com/p/2rk5t/ https://www.pythonmorsels.com/p/2rk5t/ Is laziness a virtue? I reused the slicing implementation in `range(foo)[index]` so I didn't need to do that logic myself.
      - ohyoutravel 5y ago
        Very nice!

canjobear 5y ago

I'm not sure what you're referring to. Nothing you import into Python changes its syntax. Maybe you're thinking of things like x[:, np.newaxis] where x is a numpy array? This is valid Python code outside of numpy as well, although the built-in data structures like lists and dicts won't know what to do with the :.

wheelerof4te 5y ago
To be precise, you could model the same behavior on your custom types by using the dunder method magic. In that case, everything is "valid" Python code. Numpy and Pandas libraries have some non-standard ways to slice arrays, get the subarrays and the data from them.

lvass 5y ago

What language wouldn't suffer from this, besides APL? Even very recent and well designed libraries like Elixir's Nx look like another APL-like language bolted on. Pipe syntax helps but not much.