8 ms·

Show HN: Array – A Better Python List

87 points by lauriat 6y ago

RocketSyntax 6y ago
Thank you for improving things and sharing. I use numpy & pandas, lists & dicts every day. I read your docs/github page, but can you help me see the value? However, I do think there are lots of common tasks that need to be done with lists that should be methods rather than fancy footwork =) For example: https://stackoverflow.com/questions/3462143/get-difference-between-two-lists https://stackoverflow.com/questions/3462143/get-difference-b... As you allude to w your zip loop: https://stackoverflow.com/questions/1919044/is-there-a-better-way-to-iterate-over-two-lists-getting-one-element-from-each-l https://stackoverflow.com/questions/1919044/is-there-a-bette...
- lauriat 6y ago
  Thank you for taking the time to check it out! Naturally if you're dealing with big arrays/tensors, numpy is the best choice for operating on sequences. However, ndarrays have downsides for certain use cases - as ndarrays are fixed size, adding elements is very slow, also they don't support functional methods (or rather you have to create a new array every time you apply e.g. a map), and ndarrays of any other type than numbers doesn't really make sense. Many of the methods are wrappers for built-ins, but I find the syntax of Arrays cleaner than the weirdness of the builtins. For example, while applying an async "starmap" to an Array is just a method call, with built-in lists you would have go through the whole hassle of importing both ThreadPoolExecutor and starmap, creating an executor, scheduling the function, and finally converting the result back to a list.
  - RocketSyntax 6y ago
    ndarrays "create a new array every time you apply" That resonates with me now that you explain that I can't do it. I do like chaining things in pandas like `df.select_types("float").head(100).plot.hist()`
    - nerdponx 6y ago
      Be careful though. Numpy and Pandas go through some trouble to make sure that the data inside the array is not actually copied. For instance, reshaping and slicing just return memory views. Pandas emits a somewhat-infamous warning about it that often confuses newbies.
  - lunixbochs 6y ago
    asyncmap using a thread pool with more than one worker by default is a little silly. Unless you map to a C function, you're just spawning a bunch of threads to contend for the GIL anyway.
jamespwilliams 6y ago
Looks cool. bool (__bool__) Returns whether all elements evaluate to True. I’d be worried that this will trip people up who use the if l: print l[0] # or whatever pattern
- lauriat 6y ago
  Thanks! Good point. However setting def __bool__(self): return self.nonEmpty would mess up certain methods e.g. .index for nested Arrays as __eq__ is computed elementwise and bool(Array(False, False)) would evaluate to True. Maybe a warning would be appropriate? (as is the case with ndarrays)
  - pansa2 6y ago
    > bool(Array(False, False)) would evaluate to True Isn't that consistent with the built-in `list`, though, because `bool([False, False])` is True?
    - lauriat 6y ago
      My explanation was pretty poor, let me rephrase For example, when calling Array((x, y), (z, w)).index((z, w)) the following piece of code is executed bool(Array((x, y)).__eq__((z, w))) = bool(Array(False, False)) If __bool__ returned whether the Array is nonempty, bool(Array(False, False)) would evaluate to True and the method would wrongly return 0. You're right that it would be more clear if __bool__ would behave similarly, but since Array computes operations element-wise, it isn't possible.
      - wodenokoto 6y ago
        I think the complaint is that you are using `bool` as a `all()` call on your Arrays. If you used `all()` in your implementation instead, you could be compatible with the idiomatic use of `bool(my_list)` and the _very_ common `if my_list:` structure could be used with Arrays too (like most people probably would expect from a "better list type") Regardless, Pandas struggles with the same problem, so you are at least in good company :)
nerdponx 6y ago
I feel obligated to point out the existence of the "array" package in the Python standard library: https://docs.python.org/3/library/array.html https://docs.python.org/3/library/array.html I'm sure the author is aware of it, but readers might not be.
- lauriat 6y ago
  That's why the A is capitalised ;)
Immortal333 6y ago
Chaining has its own benefits. But I think this doesn't fit the definition of "Pythonic". Again, "Pythonic" is highly debatable. But, You can always break down big chain of operations, into smaller chain using good variable naming in-between. Many operations are implemented as iterator in python on list, like filter, groupby. Looking at your code, its looks like you're not doing lazy computation. (Correct me if I wrong). This could be huge performance impact, depending upon use case of list.
- feanaro 6y ago
  > But, You can always break down big chain of operations, into smaller chain using good variable naming in-between. I don't think so. Very frequently the intermediate values represent nothing in particular and naming them simply results in visual noise. I think this is comparable to SQL or LINQ statements. Consider what those would look like if you had to name every intermediate values instead of being able to filter and group on-the-fly. Of course you can make a mess out of those too, by building huge unreadable expressions, but that's also an extreme, similar to naming every intermediate step.
- nerdponx 6y ago
  I am with you on this. Personally, I would rather continue using Toolz (https://github.com/pytoolz/toolz https://github.com/pytoolz/toolz), and contribute additional helper/utility methods to that library. The whole point of some things being functions versus methods is that they are generic rather than specialized. The generic iterator protocol is probably the best feature about the Python language, and it's both a damn shame and bad design to not use it. If you really wanted to make an improvement over built in lists, the thing to do would be to implement some kind of fully lazy "query planning" engine, like what Apache Spark has. Every method call registers a new method to be applied with the query planner, but does not execute it. Execution only occurs when you explicitly request it. That way you can effectively compile in efficient but readable code that takes multiple passes over the data into efficient operations internally that only make one pass, or at least fewer passes. This also naturally lends itself to parallelization/concurrency.
  - 6y ago
orf 6y ago
> all(map(func3, filter(func2, map(func1, zip(a, b))))) You definitely wouldn’t do this in “traditional Python”. You’d use a comprehension of some kind, or even the walrus operator, which is quite possibly faster and more readable than several chained lambdas.
- deleted 6y ago
  [deleted]
- lauriat 6y ago
  Fair enough, the example is a bit exaggerated. You could implement it with comprehensions all(func3(y) for y in (func1(x) for x in zip(a, b)) if func2(y)) It most likely is a bit faster, but I wouldn't say it's more readable.
brian_herman 6y ago
Can you do the same thing with dicts and make it so d['non_existant_key'] does not create an exception?
- basdftrewq 6y ago
  from collections import defaultdict d = defaultdict(int) d['non_existant_key']
  - st0le 6y ago
    Not quite the same what OP asked for. This will create the key and assign value 0 to it.
- lauriat 6y ago
  You can already do that with d.get("non_existant_key", default)
notretarded 6y ago
Why would I use this over numpy?
- lauriat 6y ago
  If you're doing matrix multiplication or other math operations on fixed size sequences, you shouldn't. If, however, you need the dynamic nature of the built-in list or functional methods with a touch of numpyness, you should give Array a spin.
pedrovhb 6y ago
I think this is neat but I'm not sure it's the best way to go about things. > all(map(func3, filter(func2, map(func1, zip(a, b))))) > a.zip(b).map(func1).filter(func2).forall(func3) The original is indeed terrible and the second version is a bit better. A lot better than either one, though, is splitting your logic into multiple lines and assigning a descriptive identifier to each step. Maybe even throw in some inline comments if you're particularly respectful of others' time. As tempting as it is to do something super clever and cram a ton of functionality into a small number of lines or characters (it does feel good), it's just better to be a bit more verbose and write simple, obvious code. I feel like code should be read like a book, not a puzzle.
- derwiki 6y ago
  Code is read more often than it’s written; optimize for reading.
- bko 6y ago
  > a.zip(b).map(func1).filter(func2).forall(func3) Lets make this a somewhat concrete example. --- heights = [1,2,3] widths = [4,5,6] # printing area greater than 10 # functional heights.zip(widths).map(to_area).filter(lambda area: area > 10).forall(lambda a: print("Area " + a) #Verbose way hw_zipped = zip(a,b) areas = hw_zipped.map(to_inches) big_areas = areas.filter(a: a > 10) for a in big_areas: print("Area " + a) --- Which do you prefer? I would argue the right level of abstraction is the functional way in this example, and its often the case in my experience, especially in python where you don't often use a namespace to store these intermediary variables and you have can't rely on typing
  - syrrim 6y ago
    for x, y in zip(a,b): area = to_area(x, y) if area > 10: print(f"Area {area}") >in python where you don't often use a namespace to store these intermediary variables Hm? Most python code is within a function, in my experience.
    - bko 6y ago
njharman 6y ago
> all(map(func3, filter(func2, map(func1, zip(a, b)))) That is super readable to me. Working left to right or inside out. There is one, clear, balanced, familiar, consistently used punctuation to guide you, parens, if you need it but adds little noise if you dont. The “bunch of functions taking and returning an iterator” is a great paradigm. So clean and flexiable, and powerfull. ESP combined with Python’s “many things are iterable” and is trivial to write your own iterator
- faitswulff 6y ago
  I come from Ruby, but it’s pretty unreadable to me. Not that I couldn’t, I just don’t want to. So I doubt that it’s objectively super easy to read. Any sort of reading inside out, right to left is a barrier to easy reading. This is why people like pipes in functional languages, right? You just read it in one direction.
- lifthrasiir 6y ago
  I have used Python for decades (not so much nowadays, but still) and it is very unreadable for me. It's clear that it is a data pipeline but the input and filters are all in a wrong order, thus backtracking is required for reading. I have the same complaint about str.join.
asimjalis 6y ago
I find this really useful and plan to use it. Thanks for writing and sharing. One use case for the chaining/FP style that I find particularly powerful is building out logic on the REPL. The chaining style allows me to incrementally grow my chain like a unix pipeline, see the results, use that to tweak the chain, until I finally have what I want. This type of instantaneous feedback loop is both highly productive and also extremely fun.
- lauriat 6y ago
  cheers!
topper-123 6y ago
I'd like to have a chaining operator in Python, like R is getting. Then the example could be: > a |> zip(b) |> map(func1) |> filter(func2) |> forall(func3) The advantages would be that this would work with all lists/iterables, so no need to make a special types.
- brundolf 6y ago
  The hard part with this is it sort of requires currying once you have >1 arguments, or something equivalent. I suppose Python could carve out an implicit behavior where the first or last argument is what gets fed into, but that feels potentially confusing as the calling syntax is now "lying" to you. In JavaScript doing a proper currying style isn't too hard because of arrow-syntax, but using python's function definition syntax to make a curried function would be hideous (not to mention, the standard library isn't done that way). Maybe you could have a "curryify" higher-order function. Or, the final option would be to have an explicit "insert previous value here" syntax as a part of the pipeline syntax, which is something the JS proposal has played with. Makes things more verbose (|> double(#) instead of |> double), but is maximally flexible and minimally confusing. In short: it's a lot more complicated than it seems, but I agree that this style makes this type of thing 1000x more readable.
  - topper-123 6y ago
    Just letting it implicitly be the first parameter would be good enough IMO, and a nice symmetry to self` in methods. That'd be very simple, which would be a plus in my book. Pandas allows the first param in a pipe to be a tuple[callable, str], where the second argument would signify the parameter location, e.g. `val |> (func, "param_name")` which gives some flexibility. But yeah, if you open up to piping, there are a lot of possible choices to be made and easy to go overboard also IMO.
  - housecarpenter 6y ago
    Python does have a "partial" function which does currying: from functools import partial a |> partial(zip, b) |> partial(map, func1) |> partial(filter, func2) |> partial(forall, func3) Obviously it's a bit more verbose than if the currying was done implicitly, but it's not too bad, I think. You could also import partial under a shorter name if you want. partial does have an advantage over implicit currying in that you can use keyword arguments to neatly curry on a parameter other than the first, although this isn't properly utilized by Python because most of the built-in functions have place-based rather than keyword arguments. In languages with implicit currying you have to use anonymous function expressions or functions like flip (flip(f, x, y) = f(y, x)) to deal with this. It might also be worth noting that |> doesn't essentially need to be an operator, it would just be syntactic sugar: def chain(x, *fs): y = x for f in fs: y = f(x) return y chain(a, partial(zip, b), partial(map, func1), partial(filter, func2), partial(forall, func3)) Obviously having it as an infix operator is nicer, and produces less parentheses.
goodside 6y ago
I know this is an early/experimental project, but the README could use more motivation before diving into basic usage. Asking someone to change their general-purpose containers is a big ask. It looks like Array mostly consolidates functional features already available in standard libraries, and the main innovation is a redesigned swiss-army-knife API. Good APIs are important, but my instinct is they aren’t this important. Using enhanced versions of built-in container types sounds nice, but do you really want to be keeping track of whether something is a normal list or an Array? Do you want to force people who read your code to learn this library to work with something as fundamental as lists? It’s not an impossible bar to clear (e.g. NumPy, Pandas, Dask, xarray) but it’s a high one.
- lauriat 6y ago
  Thanks for the feedback! Redesigned swiss-army-knife is well put. I’m sure Array’s not for everyone, but for some, including me it’s a nifty tool. I don’t expect people to memorise all the features of the library - the aim was to name and document each feature clearly such that finding the right method would be easy with the help of an IDE.
aldanor 6y ago
Why the JS-like naming, weird method naming convetions with strange underscores, and capitalized module name? I can't remember a single commonly used Python library with naming this strange. E.g. def removeByIndex(self, b): """ Removes the value at specified index or indices. """ ... def removeByIndex_(self, b): """ Removes the value at specified index or indices in-place. """ ... If you were to follow typical naming conventions, these would be either def remove_by_index(self, b): ... def remove_by_index_inplace(self, b): ... Or pandas-like: def remove_by_index(self, b, inplace=False): ... Or, one more step, use explicit typing as well (which also makes it more clear that the method returns self), and give a better name to the method argument rather than 'b': def remove_by_index( self, index: Union[int, Iterable[int]], inplace: bool = False, ) -> 'Array': ... Explicit type signatures in libraries like this make many things self-explanatory, like the one above.
lunixbochs 6y ago
As this doesn't use `__slots__`, every empty Array() will be 176 bytes vs the 56 bytes of [], and incur a dict allocation per array. This is due to classes without `__slots__` gaining a `__dict__` attribute for dynamic attribute assignment. Currently: >>> sys.getsizeof([]) 56 >>> a = Array() >>> sys.getsizeof(a) 72 >>> sys.getsizeof(a.__dict__) 104 with `__slots__ = []` in the Array class definition: >>> a = Array() >>> sys.getsizeof(a) 56 >>> sys.getsizeof(a.__dict__) AttributeError: 'Array' object has no attribute '__dict__'

87 points by lauriat 6y ago

Thank you for improving things and sharing. I use numpy & pandas, lists & dicts every day. I read your docs/github page, but can you help me see the value? However, I do think there are lots of common tasks that need to be done with lists that should be methods rather than fancy footwork =) For example: https://stackoverflow.com/questions/3462143/get-difference-between-two-lists https://stackoverflow.com/questions/3462143/get-difference-b... As you allude to w your zip loop: https://stackoverflow.com/questions/1919044/is-there-a-better-way-to-iterate-over-two-lists-getting-one-element-from-each-l https://stackoverflow.com/questions/1919044/is-there-a-bette...

lauriat 6y ago
Thank you for taking the time to check it out! Naturally if you're dealing with big arrays/tensors, numpy is the best choice for operating on sequences. However, ndarrays have downsides for certain use cases - as ndarrays are fixed size, adding elements is very slow, also they don't support functional methods (or rather you have to create a new array every time you apply e.g. a map), and ndarrays of any other type than numbers doesn't really make sense. Many of the methods are wrappers for built-ins, but I find the syntax of Arrays cleaner than the weirdness of the builtins. For example, while applying an async "starmap" to an Array is just a method call, with built-in lists you would have go through the whole hassle of importing both ThreadPoolExecutor and starmap, creating an executor, scheduling the function, and finally converting the result back to a list.
- RocketSyntax 6y ago
  ndarrays "create a new array every time you apply" That resonates with me now that you explain that I can't do it. I do like chaining things in pandas like `df.select_types("float").head(100).plot.hist()`
  - nerdponx 6y ago
    Be careful though. Numpy and Pandas go through some trouble to make sure that the data inside the array is not actually copied. For instance, reshaping and slicing just return memory views. Pandas emits a somewhat-infamous warning about it that often confuses newbies.
- lunixbochs 6y ago
  asyncmap using a thread pool with more than one worker by default is a little silly. Unless you map to a C function, you're just spawning a bunch of threads to contend for the GIL anyway.

jamespwilliams 6y ago

Looks cool. bool (__bool__) Returns whether all elements evaluate to True. I’d be worried that this will trip people up who use the if l: print l[0] # or whatever pattern

lauriat 6y ago
Thanks! Good point. However setting def __bool__(self): return self.nonEmpty would mess up certain methods e.g. .index for nested Arrays as __eq__ is computed elementwise and bool(Array(False, False)) would evaluate to True. Maybe a warning would be appropriate? (as is the case with ndarrays)
- pansa2 6y ago
  > bool(Array(False, False)) would evaluate to True Isn't that consistent with the built-in `list`, though, because `bool([False, False])` is True?
  - lauriat 6y ago
    My explanation was pretty poor, let me rephrase For example, when calling Array((x, y), (z, w)).index((z, w)) the following piece of code is executed bool(Array((x, y)).__eq__((z, w))) = bool(Array(False, False)) If __bool__ returned whether the Array is nonempty, bool(Array(False, False)) would evaluate to True and the method would wrongly return 0. You're right that it would be more clear if __bool__ would behave similarly, but since Array computes operations element-wise, it isn't possible.
    - wodenokoto 6y ago
      I think the complaint is that you are using `bool` as a `all()` call on your Arrays. If you used `all()` in your implementation instead, you could be compatible with the idiomatic use of `bool(my_list)` and the _very_ common `if my_list:` structure could be used with Arrays too (like most people probably would expect from a "better list type") Regardless, Pandas struggles with the same problem, so you are at least in good company :)

nerdponx 6y ago

I feel obligated to point out the existence of the "array" package in the Python standard library: https://docs.python.org/3/library/array.html https://docs.python.org/3/library/array.html I'm sure the author is aware of it, but readers might not be.

lauriat 6y ago
That's why the A is capitalised ;)

Immortal333 6y ago

Chaining has its own benefits. But I think this doesn't fit the definition of "Pythonic". Again, "Pythonic" is highly debatable. But, You can always break down big chain of operations, into smaller chain using good variable naming in-between. Many operations are implemented as iterator in python on list, like filter, groupby. Looking at your code, its looks like you're not doing lazy computation. (Correct me if I wrong). This could be huge performance impact, depending upon use case of list.

feanaro 6y ago
> But, You can always break down big chain of operations, into smaller chain using good variable naming in-between. I don't think so. Very frequently the intermediate values represent nothing in particular and naming them simply results in visual noise. I think this is comparable to SQL or LINQ statements. Consider what those would look like if you had to name every intermediate values instead of being able to filter and group on-the-fly. Of course you can make a mess out of those too, by building huge unreadable expressions, but that's also an extreme, similar to naming every intermediate step.
nerdponx 6y ago
I am with you on this. Personally, I would rather continue using Toolz (https://github.com/pytoolz/toolz https://github.com/pytoolz/toolz), and contribute additional helper/utility methods to that library. The whole point of some things being functions versus methods is that they are generic rather than specialized. The generic iterator protocol is probably the best feature about the Python language, and it's both a damn shame and bad design to not use it. If you really wanted to make an improvement over built in lists, the thing to do would be to implement some kind of fully lazy "query planning" engine, like what Apache Spark has. Every method call registers a new method to be applied with the query planner, but does not execute it. Execution only occurs when you explicitly request it. That way you can effectively compile in efficient but readable code that takes multiple passes over the data into efficient operations internally that only make one pass, or at least fewer passes. This also naturally lends itself to parallelization/concurrency.
- 6y ago

orf 6y ago

> all(map(func3, filter(func2, map(func1, zip(a, b))))) You definitely wouldn’t do this in “traditional Python”. You’d use a comprehension of some kind, or even the walrus operator, which is quite possibly faster and more readable than several chained lambdas.

deleted 6y ago
[deleted]
lauriat 6y ago
Fair enough, the example is a bit exaggerated. You could implement it with comprehensions all(func3(y) for y in (func1(x) for x in zip(a, b)) if func2(y)) It most likely is a bit faster, but I wouldn't say it's more readable.

brian_herman 6y ago

Can you do the same thing with dicts and make it so d['non_existant_key'] does not create an exception?

basdftrewq 6y ago
from collections import defaultdict d = defaultdict(int) d['non_existant_key']
- st0le 6y ago
  Not quite the same what OP asked for. This will create the key and assign value 0 to it.
lauriat 6y ago
You can already do that with d.get("non_existant_key", default)

notretarded 6y ago

Why would I use this over numpy?

lauriat 6y ago
If you're doing matrix multiplication or other math operations on fixed size sequences, you shouldn't. If, however, you need the dynamic nature of the built-in list or functional methods with a touch of numpyness, you should give Array a spin.

pedrovhb 6y ago

I think this is neat but I'm not sure it's the best way to go about things. > all(map(func3, filter(func2, map(func1, zip(a, b))))) > a.zip(b).map(func1).filter(func2).forall(func3) The original is indeed terrible and the second version is a bit better. A lot better than either one, though, is splitting your logic into multiple lines and assigning a descriptive identifier to each step. Maybe even throw in some inline comments if you're particularly respectful of others' time. As tempting as it is to do something super clever and cram a ton of functionality into a small number of lines or characters (it does feel good), it's just better to be a bit more verbose and write simple, obvious code. I feel like code should be read like a book, not a puzzle.

derwiki 6y ago
Code is read more often than it’s written; optimize for reading.
bko 6y ago
> a.zip(b).map(func1).filter(func2).forall(func3) Lets make this a somewhat concrete example. --- heights = [1,2,3] widths = [4,5,6] # printing area greater than 10 # functional heights.zip(widths).map(to_area).filter(lambda area: area > 10).forall(lambda a: print("Area " + a) #Verbose way hw_zipped = zip(a,b) areas = hw_zipped.map(to_inches) big_areas = areas.filter(a: a > 10) for a in big_areas: print("Area " + a) --- Which do you prefer? I would argue the right level of abstraction is the functional way in this example, and its often the case in my experience, especially in python where you don't often use a namespace to store these intermediary variables and you have can't rely on typing
- syrrim 6y ago
  for x, y in zip(a,b): area = to_area(x, y) if area > 10: print(f"Area {area}") >in python where you don't often use a namespace to store these intermediary variables Hm? Most python code is within a function, in my experience.
  - bko 6y ago

njharman 6y ago

> all(map(func3, filter(func2, map(func1, zip(a, b)))) That is super readable to me. Working left to right or inside out. There is one, clear, balanced, familiar, consistently used punctuation to guide you, parens, if you need it but adds little noise if you dont. The “bunch of functions taking and returning an iterator” is a great paradigm. So clean and flexiable, and powerfull. ESP combined with Python’s “many things are iterable” and is trivial to write your own iterator

faitswulff 6y ago
I come from Ruby, but it’s pretty unreadable to me. Not that I couldn’t, I just don’t want to. So I doubt that it’s objectively super easy to read. Any sort of reading inside out, right to left is a barrier to easy reading. This is why people like pipes in functional languages, right? You just read it in one direction.
lifthrasiir 6y ago
I have used Python for decades (not so much nowadays, but still) and it is very unreadable for me. It's clear that it is a data pipeline but the input and filters are all in a wrong order, thus backtracking is required for reading. I have the same complaint about str.join.

asimjalis 6y ago

I find this really useful and plan to use it. Thanks for writing and sharing. One use case for the chaining/FP style that I find particularly powerful is building out logic on the REPL. The chaining style allows me to incrementally grow my chain like a unix pipeline, see the results, use that to tweak the chain, until I finally have what I want. This type of instantaneous feedback loop is both highly productive and also extremely fun.

lauriat 6y ago
cheers!

topper-123 6y ago

I'd like to have a chaining operator in Python, like R is getting. Then the example could be: > a |> zip(b) |> map(func1) |> filter(func2) |> forall(func3) The advantages would be that this would work with all lists/iterables, so no need to make a special types.

brundolf 6y ago
The hard part with this is it sort of requires currying once you have >1 arguments, or something equivalent. I suppose Python could carve out an implicit behavior where the first or last argument is what gets fed into, but that feels potentially confusing as the calling syntax is now "lying" to you. In JavaScript doing a proper currying style isn't too hard because of arrow-syntax, but using python's function definition syntax to make a curried function would be hideous (not to mention, the standard library isn't done that way). Maybe you could have a "curryify" higher-order function. Or, the final option would be to have an explicit "insert previous value here" syntax as a part of the pipeline syntax, which is something the JS proposal has played with. Makes things more verbose (|> double(#) instead of |> double), but is maximally flexible and minimally confusing. In short: it's a lot more complicated than it seems, but I agree that this style makes this type of thing 1000x more readable.
- topper-123 6y ago
  Just letting it implicitly be the first parameter would be good enough IMO, and a nice symmetry to self` in methods. That'd be very simple, which would be a plus in my book. Pandas allows the first param in a pipe to be a tuple[callable, str], where the second argument would signify the parameter location, e.g. `val |> (func, "param_name")` which gives some flexibility. But yeah, if you open up to piping, there are a lot of possible choices to be made and easy to go overboard also IMO.
- housecarpenter 6y ago
  Python does have a "partial" function which does currying: from functools import partial a |> partial(zip, b) |> partial(map, func1) |> partial(filter, func2) |> partial(forall, func3) Obviously it's a bit more verbose than if the currying was done implicitly, but it's not too bad, I think. You could also import partial under a shorter name if you want. partial does have an advantage over implicit currying in that you can use keyword arguments to neatly curry on a parameter other than the first, although this isn't properly utilized by Python because most of the built-in functions have place-based rather than keyword arguments. In languages with implicit currying you have to use anonymous function expressions or functions like flip (flip(f, x, y) = f(y, x)) to deal with this. It might also be worth noting that |> doesn't essentially need to be an operator, it would just be syntactic sugar: def chain(x, *fs): y = x for f in fs: y = f(x) return y chain(a, partial(zip, b), partial(map, func1), partial(filter, func2), partial(forall, func3)) Obviously having it as an infix operator is nicer, and produces less parentheses.

goodside 6y ago

I know this is an early/experimental project, but the README could use more motivation before diving into basic usage. Asking someone to change their general-purpose containers is a big ask. It looks like Array mostly consolidates functional features already available in standard libraries, and the main innovation is a redesigned swiss-army-knife API. Good APIs are important, but my instinct is they aren’t this important. Using enhanced versions of built-in container types sounds nice, but do you really want to be keeping track of whether something is a normal list or an Array? Do you want to force people who read your code to learn this library to work with something as fundamental as lists? It’s not an impossible bar to clear (e.g. NumPy, Pandas, Dask, xarray) but it’s a high one.

lauriat 6y ago
Thanks for the feedback! Redesigned swiss-army-knife is well put. I’m sure Array’s not for everyone, but for some, including me it’s a nifty tool. I don’t expect people to memorise all the features of the library - the aim was to name and document each feature clearly such that finding the right method would be easy with the help of an IDE.

aldanor 6y ago

Why the JS-like naming, weird method naming convetions with strange underscores, and capitalized module name? I can't remember a single commonly used Python library with naming this strange. E.g. def removeByIndex(self, b): """ Removes the value at specified index or indices. """ ... def removeByIndex_(self, b): """ Removes the value at specified index or indices in-place. """ ... If you were to follow typical naming conventions, these would be either def remove_by_index(self, b): ... def remove_by_index_inplace(self, b): ... Or pandas-like: def remove_by_index(self, b, inplace=False): ... Or, one more step, use explicit typing as well (which also makes it more clear that the method returns self), and give a better name to the method argument rather than 'b': def remove_by_index( self, index: Union[int, Iterable[int]], inplace: bool = False, ) -> 'Array': ... Explicit type signatures in libraries like this make many things self-explanatory, like the one above.

lunixbochs 6y ago

As this doesn't use `__slots__`, every empty Array() will be 176 bytes vs the 56 bytes of [], and incur a dict allocation per array. This is due to classes without `__slots__` gaining a `__dict__` attribute for dynamic attribute assignment. Currently: >>> sys.getsizeof([]) 56 >>> a = Array() >>> sys.getsizeof(a) 72 >>> sys.getsizeof(a.__dict__) 104 with `__slots__ = []` in the Array class definition: >>> a = Array() >>> sys.getsizeof(a) 56 >>> sys.getsizeof(a.__dict__) AttributeError: 'Array' object has no attribute '__dict__'