5 ms·

Query Your Python Lists

87 points by mkalioby 2y ago

deleted 2y ago
[deleted]
sevensor 2y ago
Having seen a lot of work come to grief because of the decision to use pandas, anything that’s not pandas has my vote. Pandas: if you’re not using it interactively, don’t use it at all. This advice goes double if your use case is “read a csv.” Standard library in Python has you covered there.
- ttyprintk 2y ago
  Since DuckDB can read and write Pandas from memory, a team with varying Pandas fluency can benefit from learning DuckDB.
  - adolph 2y ago
    Since Pandas 2, Apache Arrow replaced NumPy as the backend for Pandas. Arrow is also used by Polars, DuckDB, Ibis, the list goes on. https://arrow.apache.org/overview/ https://arrow.apache.org/overview/ Apache Arrow solves most discussed problems, such as improving speed, interoperability, and data types, especially for strings. For example, the new string[pyarrow] column type is around 3.5 times more efficient. [...] The significant achievement here is zero-copy data access, mapping complex tables to memory to make accessing one terabyte of data on disk as fast and easy as one megabyte. https://airbyte.com/blog/pandas-2-0-ecosystem-arrow-polars-duckdb https://airbyte.com/blog/pandas-2-0-ecosystem-arrow-polars-d...
- c0balt 2y ago
  Both duckdb and especially polars should also be mentioned here. Polars in particular is quite good Ime if you want a pandas-alike interface (it additionally also has a more sane interface).
abdullahkhalids 2y ago
I don't understand why numeric filters are included. The library is written in python, so shouldn't a lambda function based filter be roughly as fast but much easier/clearer to write.
- MathMonkeyMan 2y ago
  I'm not the author, but this implementation has the benefit of being a JSON compatible DSL that you can serialize. Maybe that's intentional, maybe not. It does look like Python's comprehensions would be a better choice if you're writing them by hand anyway.
  - wis 2y ago
    Yea, In my opinion using Python's list comprehension is more readable and code checkable. Here's the usage example from the README: from leopards import Q l = [{"name":"John","age":"16"}, {"name":"Mike","age":"19"},{"name":"Sarah","age":"21"}] filtered= Q(l,{'name__contains':"k", "age__lt":20}) print(list(filtered)) Versus: [x for x in l if ('k' in x['name'] and int(x['age']) < 20)] Outputs: [{'name': 'Mike', 'age': '19'}] Also from the readme: > Even though, age was str in the dict, as the value of in the query dict was int, Leopards converted the value in dict automatically to match the query data type. This behaviour can be stopped by passing False to convert_types parameter. I don't like this default behavior.
    - WesleyJohnson 2y ago
      That's a bit biased, no? The actual comparison should be: filtered = Q(l,{'name__contains':"k", "age__lt":20}) Verus: filtered = [x for x in l if ('k' in x['name'] and int(x['age']) < 20)]
      - MathMonkeyMan 2y ago
        Something like Clojure would be perfect for this. Write a macro that converts {"name" (includes? "k"), "age" (< 20)} into {"name" #(includes? % "k"), "age" #(< % 20)} which is the same as {"name" (fn [name] (includes? name "k")), "age" (fn [age] (< age 20))} Then have another macro that converts that into the pattern matching code, or maybe there's already something in the standard library. You could serialize the patterns using EDN as a substitute for JSON. Fun stuff. I wrote something [similar][1] in javascript. With that it would be: const is_k_kid = tisch.compileFunction((etc) => ({ 'name': name => name.includes('k'), 'age': age => age < 20, ...etc })); const result = input.filter(is_k_kid); Yes, "...etc" is part of the DSL. [1]: https://github.com/dgoffredo/tisch https://github.com/dgoffredo/tisch
tempcommenttt 2y ago
It’s nice it’s fast at 10k dictionary entries, but how does it scale?
- deleted 2y ago
  [deleted]
- fatih-erikli-cg 2y ago
  I think 10000 is a lot enough for a queryable dataset. More of them is like computer generated things like logs etc.
- HumblyTossed 2y ago
  > but how does it scale Usually sideways, but if you stack them, you might get some vertical.
glial 2y ago
Interesting work. I'd be curious to know the timing relative to list comprehensions for similar queries, since that's the common standard library alternative for many of these examples.
- mkalioby 2y ago
  Good point, but the libray allows you to generate custom queries based on user input which will be tough by list comprehension
dsp_person 2y ago
Interesting... I've been playing with the idea of embedding more python in my C, no cython or anything just using <Python.h> and <numpy/arrayobject.h>. From one perspective it's just "free" C-bindings to a lot of optimized packages. Trying some different C-libraries, the python code is often faster. Python almost becomes C's package manager E.g. sorting 2^23 random 64-bit integers: qsort: 850ms, custom radix sort: 250ms, ksort.h: 582ms, np.sort: 107ms (including PyArray_SimpleNewFromData, PyArray_Sort). Where numpy uses intel's x86-simd-sort I believe. E.g. inserting 8M entries into a hash table (random 64-bit keys and values): MSI-style hash table: ~100ns avg insert/lookup, cc_map: ~95ns avg insert/lookup, Python.h: 91ns insert, 60ns lookup I'm curious if OPs tool might fit in similarly. I've found lmdb to be quite slow even in tmpfs with no sync, etc.
- sitkack 2y ago
  You should look at embedding Wasmtime into your C. https://github.com/bytecodealliance/wasmtime/tree/main/examples https://github.com/bytecodealliance/wasmtime/tree/main/examp...
anentropic 2y ago
Django ORM for plain lists is interesting I guess... but being faster than pandas at that is quite a surprise, bravo!
- mkalioby 2y ago
  Thanks alot.
maweki 2y ago
Embedding functionality into strings prevents any kind of static analysis. The same issue as embedding plain SQL, plain regexes, etc.. I am always in favor of declarative approaches where applicable. But whenever they are embedded in this way, you get this static analysis barrier and a possible mismatch between the imperative and declarative code, where you change a return type or field declaratively and it doesn't come up as an error in the surrounding code. A positive example is VerbalExpressions in Java, which only allow expressing valid regular expressions and every invalid regular expression is inexpressible in valid java code. Jooq is another example, which makes incorrect (even incorrectly typed) SQL code inexpressible in Java. I know python is a bit different, as there is no extensive static analysis in the compiler, but we do indeed have a lot of static analysis tools for python that could be valuable. A statically type-safe query is a wonderful thing for safety and maintainability and we do have good type-checkers for python.
- gpderetta 2y ago
  If your schema is dynamic, in most languages there isn't much you can do, but at least in python Q(name=contains('k')) it is not particularly more complex to write and certainly more composable, extensible and checkable. Alternatively go full eval and do Q("'k' in name")
- notpushkin 2y ago
  I love how PonyORM does this for SQL: it’s just Puthon `x for x in ... if ...`. Of course, if you use the same syntax for Python lists of dicts, you don’t need any library at all.
- eddd-ddde 2y ago
  I disagree. You'll be surprised to hear this, but source code... is just a very big string... If you can run static analysis on that you can run static analysis on string literals. Much like how C will give you warnings for mismatched printf arguments.
  - maweki 2y ago
    You might be surprised to hear that most compilers and static analysis tools in general do not inspect (string and other) literals, while they do indeed inspect all the other parts and structure of the abstract syntax tree.
graup 2y ago
{'name__contains':"k", "age__lt":20} Kind of tangential to this package, but I've always loved this filter query syntax. Does it have a name? I first encountered it in Django ORM, and then in DRF, which has them as URL query params. I have recently built a parser for this in Javascript to use it on the frontend. Does anyone know any JS libraries that make working with this easy? I'm thinking parsing and offering some kind of database-agnostic marshaling API. (If not, I might have to open-source my own code!)
- Daishiman 2y ago
  I think its first usage really was in Django. I've always referred to it as the Django ORM query notation.
  - mkalioby 2y ago
    As also a Django Developer for 10+ years. That is why I felt that Python needs something like this.
badmintonbaseba 2y ago
I would prefer something like `{"name": contains("k")}`, where contains("k") returns an object with a custom __eq__ that compares equal to any string (or iterable) that contains "k". Then you can just filter by equality. I recently started using this pattern for pytest equality assertions, as pytest helpfully produces a detailed diff on mismatch. It's not perfect, as pytest doesn't always produce a correct diff with this pattern, but it's better than some alternatives.
- gpderetta 2y ago
  Instead of returning an __eq__ object, I have had 'contains' just return a predicate function (that you can pass to filter for example. I guess in practice it doesn't really change much, except that calling it __eq__ is a bit misleading. A significant advantage is that you can just pass an inline lambda.
mark_l_watson 2y ago
Apologies for being off topic, but after reading the implementation code, I was amazed at how short it is! I have never been a huge fan of Python (Lisp person) but I really appreciate how concise Python can be, and the dynamic nature of Python allows the nice query syntax.
- gabrielsroka 2y ago
  > Python can be seen as a dialect of Lisp - Peter Norvig https://www.norvig.com/python-lisp.html https://www.norvig.com/python-lisp.html
  - mark_l_watson 2y ago
    Well, Peter has moved on to Python. I had lunch with him in 2001, expecting to talk about Common Lisp, but he already seemed more interested in Python. It has been a while since I studied his Lisp code, but I watch for new Python studies he releases.
- pama 2y ago
  Agreed on conciseness of the implementation. It is short and clear despite having a Max and Min that share all except one character in 30 lines of code.
James_K 2y ago
Maybe this is just me, but embedding the language in strings like this seems like it's just asking for trouble.
- HumblyTossed 2y ago
  Yes, it looks very fragile.
- mkalioby 2y ago
  Django one of most popular Web Frameworks in Python does the same trick.
pphysch 2y ago
I feel like the scale where a library like this is meaningful for performance, and therefore worth the dependency+DSL complexity, is also the scale where you should use a proper database (even just SQLite).
bityard 2y ago
Title should be prefixed with Show HN and the name of the project in order to not mislead readers about the content of the link.
dmezzetti 2y ago
Interesting project and approach, thanks for sharing! If you're interested in a simple solution to query a list with SQL including vector similarity, check this out: https://gist.github.com/davidmezzetti/f0a0b92f5281924597c9d1a7bb89562e https://gist.github.com/davidmezzetti/f0a0b92f5281924597c9d1...
Kalanos 2y ago
You can create a dataframe from a list of dictionaries in pandas `df = pd.DataFrame([{},{},{}])`
- ciupicri 2y ago
  This is supposedly a bit faster (https://github.com/mkalioby/leopards?tab=readme-ov-file#comparison-with-pandas https://github.com/mkalioby/leopards?tab=readme-ov-file#comp...).

87 points by mkalioby 2y ago

deleted 2y ago

[deleted]

Having seen a lot of work come to grief because of the decision to use pandas, anything that’s not pandas has my vote. Pandas: if you’re not using it interactively, don’t use it at all. This advice goes double if your use case is “read a csv.” Standard library in Python has you covered there.

ttyprintk 2y ago
Since DuckDB can read and write Pandas from memory, a team with varying Pandas fluency can benefit from learning DuckDB.
- adolph 2y ago
  Since Pandas 2, Apache Arrow replaced NumPy as the backend for Pandas. Arrow is also used by Polars, DuckDB, Ibis, the list goes on. https://arrow.apache.org/overview/ https://arrow.apache.org/overview/ Apache Arrow solves most discussed problems, such as improving speed, interoperability, and data types, especially for strings. For example, the new string[pyarrow] column type is around 3.5 times more efficient. [...] The significant achievement here is zero-copy data access, mapping complex tables to memory to make accessing one terabyte of data on disk as fast and easy as one megabyte. https://airbyte.com/blog/pandas-2-0-ecosystem-arrow-polars-duckdb https://airbyte.com/blog/pandas-2-0-ecosystem-arrow-polars-d...
c0balt 2y ago
Both duckdb and especially polars should also be mentioned here. Polars in particular is quite good Ime if you want a pandas-alike interface (it additionally also has a more sane interface).

abdullahkhalids 2y ago

I don't understand why numeric filters are included. The library is written in python, so shouldn't a lambda function based filter be roughly as fast but much easier/clearer to write.

MathMonkeyMan 2y ago
I'm not the author, but this implementation has the benefit of being a JSON compatible DSL that you can serialize. Maybe that's intentional, maybe not. It does look like Python's comprehensions would be a better choice if you're writing them by hand anyway.
- wis 2y ago
  Yea, In my opinion using Python's list comprehension is more readable and code checkable. Here's the usage example from the README: from leopards import Q l = [{"name":"John","age":"16"}, {"name":"Mike","age":"19"},{"name":"Sarah","age":"21"}] filtered= Q(l,{'name__contains':"k", "age__lt":20}) print(list(filtered)) Versus: [x for x in l if ('k' in x['name'] and int(x['age']) < 20)] Outputs: [{'name': 'Mike', 'age': '19'}] Also from the readme: > Even though, age was str in the dict, as the value of in the query dict was int, Leopards converted the value in dict automatically to match the query data type. This behaviour can be stopped by passing False to convert_types parameter. I don't like this default behavior.
  - WesleyJohnson 2y ago
    That's a bit biased, no? The actual comparison should be: filtered = Q(l,{'name__contains':"k", "age__lt":20}) Verus: filtered = [x for x in l if ('k' in x['name'] and int(x['age']) < 20)]
    - MathMonkeyMan 2y ago
      Something like Clojure would be perfect for this. Write a macro that converts {"name" (includes? "k"), "age" (< 20)} into {"name" #(includes? % "k"), "age" #(< % 20)} which is the same as {"name" (fn [name] (includes? name "k")), "age" (fn [age] (< age 20))} Then have another macro that converts that into the pattern matching code, or maybe there's already something in the standard library. You could serialize the patterns using EDN as a substitute for JSON. Fun stuff. I wrote something [similar][1] in javascript. With that it would be: const is_k_kid = tisch.compileFunction((etc) => ({ 'name': name => name.includes('k'), 'age': age => age < 20, ...etc })); const result = input.filter(is_k_kid); Yes, "...etc" is part of the DSL. [1]: https://github.com/dgoffredo/tisch https://github.com/dgoffredo/tisch

tempcommenttt 2y ago

It’s nice it’s fast at 10k dictionary entries, but how does it scale?

deleted 2y ago
[deleted]
fatih-erikli-cg 2y ago
I think 10000 is a lot enough for a queryable dataset. More of them is like computer generated things like logs etc.
HumblyTossed 2y ago
> but how does it scale Usually sideways, but if you stack them, you might get some vertical.

glial 2y ago

Interesting work. I'd be curious to know the timing relative to list comprehensions for similar queries, since that's the common standard library alternative for many of these examples.

mkalioby 2y ago
Good point, but the libray allows you to generate custom queries based on user input which will be tough by list comprehension

dsp_person 2y ago

Interesting... I've been playing with the idea of embedding more python in my C, no cython or anything just using <Python.h> and <numpy/arrayobject.h>. From one perspective it's just "free" C-bindings to a lot of optimized packages. Trying some different C-libraries, the python code is often faster. Python almost becomes C's package manager E.g. sorting 2^23 random 64-bit integers: qsort: 850ms, custom radix sort: 250ms, ksort.h: 582ms, np.sort: 107ms (including PyArray_SimpleNewFromData, PyArray_Sort). Where numpy uses intel's x86-simd-sort I believe. E.g. inserting 8M entries into a hash table (random 64-bit keys and values): MSI-style hash table: ~100ns avg insert/lookup, cc_map: ~95ns avg insert/lookup, Python.h: 91ns insert, 60ns lookup I'm curious if OPs tool might fit in similarly. I've found lmdb to be quite slow even in tmpfs with no sync, etc.

sitkack 2y ago
You should look at embedding Wasmtime into your C. https://github.com/bytecodealliance/wasmtime/tree/main/examples https://github.com/bytecodealliance/wasmtime/tree/main/examp...

anentropic 2y ago

Django ORM for plain lists is interesting I guess... but being faster than pandas at that is quite a surprise, bravo!

mkalioby 2y ago
Thanks alot.

maweki 2y ago

Embedding functionality into strings prevents any kind of static analysis. The same issue as embedding plain SQL, plain regexes, etc.. I am always in favor of declarative approaches where applicable. But whenever they are embedded in this way, you get this static analysis barrier and a possible mismatch between the imperative and declarative code, where you change a return type or field declaratively and it doesn't come up as an error in the surrounding code. A positive example is VerbalExpressions in Java, which only allow expressing valid regular expressions and every invalid regular expression is inexpressible in valid java code. Jooq is another example, which makes incorrect (even incorrectly typed) SQL code inexpressible in Java. I know python is a bit different, as there is no extensive static analysis in the compiler, but we do indeed have a lot of static analysis tools for python that could be valuable. A statically type-safe query is a wonderful thing for safety and maintainability and we do have good type-checkers for python.

gpderetta 2y ago
If your schema is dynamic, in most languages there isn't much you can do, but at least in python Q(name=contains('k')) it is not particularly more complex to write and certainly more composable, extensible and checkable. Alternatively go full eval and do Q("'k' in name")
notpushkin 2y ago
I love how PonyORM does this for SQL: it’s just Puthon `x for x in ... if ...`. Of course, if you use the same syntax for Python lists of dicts, you don’t need any library at all.
eddd-ddde 2y ago
I disagree. You'll be surprised to hear this, but source code... is just a very big string... If you can run static analysis on that you can run static analysis on string literals. Much like how C will give you warnings for mismatched printf arguments.
- maweki 2y ago
  You might be surprised to hear that most compilers and static analysis tools in general do not inspect (string and other) literals, while they do indeed inspect all the other parts and structure of the abstract syntax tree.

graup 2y ago

{'name__contains':"k", "age__lt":20} Kind of tangential to this package, but I've always loved this filter query syntax. Does it have a name? I first encountered it in Django ORM, and then in DRF, which has them as URL query params. I have recently built a parser for this in Javascript to use it on the frontend. Does anyone know any JS libraries that make working with this easy? I'm thinking parsing and offering some kind of database-agnostic marshaling API. (If not, I might have to open-source my own code!)

Daishiman 2y ago
I think its first usage really was in Django. I've always referred to it as the Django ORM query notation.
- mkalioby 2y ago
  As also a Django Developer for 10+ years. That is why I felt that Python needs something like this.

badmintonbaseba 2y ago

I would prefer something like `{"name": contains("k")}`, where contains("k") returns an object with a custom __eq__ that compares equal to any string (or iterable) that contains "k". Then you can just filter by equality. I recently started using this pattern for pytest equality assertions, as pytest helpfully produces a detailed diff on mismatch. It's not perfect, as pytest doesn't always produce a correct diff with this pattern, but it's better than some alternatives.

gpderetta 2y ago
Instead of returning an __eq__ object, I have had 'contains' just return a predicate function (that you can pass to filter for example. I guess in practice it doesn't really change much, except that calling it __eq__ is a bit misleading. A significant advantage is that you can just pass an inline lambda.

mark_l_watson 2y ago

Apologies for being off topic, but after reading the implementation code, I was amazed at how short it is! I have never been a huge fan of Python (Lisp person) but I really appreciate how concise Python can be, and the dynamic nature of Python allows the nice query syntax.

gabrielsroka 2y ago
> Python can be seen as a dialect of Lisp - Peter Norvig https://www.norvig.com/python-lisp.html https://www.norvig.com/python-lisp.html
- mark_l_watson 2y ago
  Well, Peter has moved on to Python. I had lunch with him in 2001, expecting to talk about Common Lisp, but he already seemed more interested in Python. It has been a while since I studied his Lisp code, but I watch for new Python studies he releases.
pama 2y ago
Agreed on conciseness of the implementation. It is short and clear despite having a Max and Min that share all except one character in 30 lines of code.

James_K 2y ago

Maybe this is just me, but embedding the language in strings like this seems like it's just asking for trouble.

HumblyTossed 2y ago
Yes, it looks very fragile.
mkalioby 2y ago
Django one of most popular Web Frameworks in Python does the same trick.

pphysch 2y ago

I feel like the scale where a library like this is meaningful for performance, and therefore worth the dependency+DSL complexity, is also the scale where you should use a proper database (even just SQLite).

bityard 2y ago

Title should be prefixed with Show HN and the name of the project in order to not mislead readers about the content of the link.

dmezzetti 2y ago

Interesting project and approach, thanks for sharing! If you're interested in a simple solution to query a list with SQL including vector similarity, check this out: https://gist.github.com/davidmezzetti/f0a0b92f5281924597c9d1a7bb89562e https://gist.github.com/davidmezzetti/f0a0b92f5281924597c9d1...

Kalanos 2y ago

You can create a dataframe from a list of dictionaries in pandas `df = pd.DataFrame([{},{},{}])`

ciupicri 2y ago
This is supposedly a bit faster (https://github.com/mkalioby/leopards?tab=readme-ov-file#comparison-with-pandas https://github.com/mkalioby/leopards?tab=readme-ov-file#comp...).