5 ms·
What category theory teaches us about dataframes
- jmount 6mo agoI like this sort of study- but it really misses the point to not give more credit for some of the observations and designs to Codd and others.
- rich_sasha 6mo agoThe article starts well, on trying to condense pandas' gaziliion of inconsistent and continuously-deprecated functions with tens of keyword arguments into a small, condensed set of composable operations - but it lost me then. The more interesting nugget for me is about this project they mention: https://modin.readthedocs.io/en/latest/index.html https://modin.readthedocs.io/en/latest/index.html called Modin, which apparently went to the effort of analysing common pandas uses and compressed the API into a mere handful of operations. Which sounds great! Sadly for me the purpose seems to have been rather to then recreate the full pandas API, only running much faster, backed by things like Ray and Dask. So it's the same API, just much faster. To me it's a shame. Pandas is clearly quite ergonomic for various exploratory interactive analyses, but the API is, imo, awful. The speed is usually not a concern for me - slow operations often seem to be avoidable, and my data tends to fit in (a lot of) RAM. I can't see that their more condensed API is public facing and usable.
- bbkane 6mo agoCheck out polars- I find it much more intuitive than pandas as it looks closer to SQL (and I learned SQL first). Maybe you'll feel the same way!
- Lyngbakr 6mo agoAgreed — I much prefer polars, too. IIRC the latest major version of pandas even introduced some polars-style syntax.
- rich_sasha 6mo agoI've looked at Polars. My sense is that Pandas is an interactive data analysis library poorly suited to production uses, and Polars is the other way around. Seemed quite verbose for example. Sometimes doing `series["2026"]` is exactly the right thing to type.
- entropicdrifter 6mo agoYou can do that in Polars, too
- mwexler 6mo agoWith some of the newest 3.x changes like copy-on-write, I find pandas getting quite verbose now as well. In a world where AI is writing the code, I guess I shouldn't complain, but when I am discovering something the ai of choice yet again missed, both pandas and polars still feel verbose and lacking sugar.
- sweezyjeezy 6mo agoThe pandas API is awful, but it's kind of interesting why. It was started as a financial time series manipulation library ('panels') in a hedge fund and a lot of the quirks come from that. For example the unique obsession with the 'index' - functions seemingly randomly returning dataframes with column data as the index, or having to write index=False every single time you write to disk, or it appending the index to the Series numpy data leading to incredibly confusing bugs. That comes from the assumption that there is almost always a meaningful index (timestamps).
- gwerbin 6mo ago> The pandas API is awful I hate to be the "you're holding it wrong" guy but 90% of "Pandas bad!" posts I find are either outright misinformed or mischaracterizing one person's particular opinion as some kind of common truth. This one is both! > That comes from the assumption that there is almost always a meaningful index (timestamps) The index can be literally any unique row label or ID. It's idiosyncratic among "data frames" (SQL has no equivalent concept, and the R community has disowned theirs), but it's really not such a crazy thing to have row labels built into your data table. Excel supports this in several different ways (frozen columns, VLOOKUP) and users expect it in just about any table-oriented GUI tool. > having to write index=False every single time you write to disk If you're actually using the index as it's meant to be used, you'd see why this isn't the default setting. > functions seemingly randomly returning dataframes with column data as the index I assume you're talking about the behavior of .groupby() and .rolling()? It's never been random. Under-documented and hard to reason about group_keys= and related options, yes. But not random. > appending the index to the Series numpy data leading to incredibly confusing bugs I've been using Pandas professionally almost daily since 2015 and I have no idea what this means.
- _diyar 6mo agoI think the commenter you are replying to might well understand these nuances. The point is not that Pandas is inscrutable, but instead that it‘s annoying to use in many common use-cases.
- sweezyjeezy 6mo ago
- rdevilla 6mo ago> Pandas is clearly quite ergonomic for various exploratory interactive analyses, but the API is, imo, awful. Having previously inherited (and now dispossessed) an un-disentangleable pile of Python, pandas, and SQL hacks reminiscent of a spreadsheet rammed with inscrutable Excel formulae, I have no idea how data scientists collaborate on anything with this technology. It's like when bioinformatics was full of write-only Perl code that was maybe executed successfully once for the purposes of a study or paper, and was kept around for future archaeologists to hopefully one day resuscitate when the need may arise again. If programmers are expected to just throw garbage like this at the next asshole with the misfortune to have to maintain code that was never designed to be maintained, it's not a surprise that the industry is once again moving towards write-only code, this time produced at scale by LLMs. It's like we're back to Visual Studio Ultimate slopping out 10k lines of XAML in response to your dragging and dropping in the WYSIWYG. There is a reason nobody does this any more.
- few 6mo agoI felt like one or two decades ago, all the rage was about rewriting programs into just two primitives: map and reduce. For example filter can be expressed as: is_even = lambda x: x % 2 == 0 mapped = map(lambda x: [x] if is_even(x) else [], data) filtered = reduce(lambda x, y: x + y, mapped, []) But then the world moved on from it because it was too rigid
- mememememememo 6mo agoPerformance aside it seems you could do most maybe a the ops with those three. I say three because your sneaky plus is a union operation. So map, reduce and union. But you are also allowing arbitrary code expressions. So it is less lego-like.
- mrlongroots 6mo agoMapReduce is nice but it doesn't, by itself, help you reason about pushdowns for one. Parquet, for example, can pushdown select/project/filter, and that's lost if you have MapReduce. And a reduce is just a shuffle + map, not very different from a distributed join. MapReduce as an escape hatch over what is fundamentally still relational algebra may be a good intuition.
- bjourne 6mo agoReductions are painful because they specify a sequence of ordered operations. Runtime is O(N), where N is the sequence length, regardless of amount of hardware. So you want to work at a higher level where you can exploit commutativity and independence of some (or even most) operations.
- ux266478 6mo agoYou're right it's primarily a runtime + compiler + language issue. I really don't understand why people tried to force functional programming in environments without decent algebraic reasoning mechanisms. Modern graph reducers have inherent confluence and aren't reliant on explicit commutation. They can do everything parallel and out of order (until they have to talk to some extrinsic thing like getting input or spitting out output), including arbitrary side-effectual mutation. We really live in the future.
- jiehong 6mo agoDups of a few days ago: - https://news.ycombinator.com/item?id=47567087 https://news.ycombinator.com/item?id=47567087
- getnormality 6mo agoHmm. Folks trying to discover the elegant core of data frame manipulation by studying... pandas usage patterns. When R's dplyr solved this over a decade ago, mostly by respecting SQL and following its lead. The pandas API feels like someone desperately needed a wheel and had never heard of a wheel, so they made a heptagon, and now millions of people are riding on heptagon wheels. Because it's locked in now, everyone uses heptagon wheels, what can you do? And now a category theorist comes along, studies the heptagon, and says hey look, you could get by on a hexagon. Maybe even a square or a triangle. That would be simpler! No. Stop. Data frames are not fundamentally different from database tables [1]. There's no reason to invent a completely new API for them. You'll get within 10% of optimal just by porting SQL to your language. Which dplyr does, and then closes most of the remaining optimality gap by going beyond SQL's limitations. You found a small core of operations that generates everything? Great. Also, did you know Brainfuck is Turing-complete? Nobody cares. Not all "complete" systems are created equal. A great DSL is not just about getting down to a small number of operations. It's about getting down to meaningful operations that are grammatically composable. The relational algebra that inspired SQL already nailed this. Build on SQL. Don't make up your own thing. Like, what is "drop duplicates"? What are duplicates? Why would anyone need to drop them? That's a pandas-brained operation. You want the distinct keys defined by a select set of key columns, like SQL and dplyr provide. Who needs a separate select and rename? Select is already using names, so why not do your name management there? One flexible select function can do it all. Again, like both SQL and dplyr. Who needs a separate difference operation? There's already a type of join, the anti-join, that gets that done more concisely and flexibly, and without adding a new primitive, just a variation on the concept of a join. Again, like both SQL and dplyr. Props to pandas for helping so many people who have no choice but to do tabular data analysis in Python, but the pandas API is not the right foundation for anything, not even a better version of pandas. [1] No, row labels and transposition are not a good enough reason to regard them as different. They are both just structures that support pivoting, which is vastly more useful, and again, implemented by both R and many popular dialects of SQL.
- fn-mote 6mo agoAmen. The author takes the 4 operations below and discusses some 3-operation thing from category theory. Not worth it, and not as clear as dplyr. > But I kept looking at the relational operators in that table (PROJECTION, RENAME, GROUPBY, JOIN) and thinking: these feel related. They all change the schema of the dataframe. Is there a deeper relationship?
- pavodive 6mo agoWhen I started reading about pandas complexity and the smaller set of operations needed, couldn't help but think of R's data.table simplicity. Granted, it's got more than 15 functions, but its simplicity seems to me very similar to what the author presented in the end.
- Lyngbakr 6mo agoBack when I used to use Stackoverflow, someone would always come along with a data.table solution when I asked a question about dplyr. The terse syntax seemed so foreign compared to the obvious verb syntax of dplyr. But then I learned data.table and I've never looked back. It's a superb tool!
- gwerbin 6mo agodata.table "simplicity" is actually a huge set of features, they just have a clever and compact way to express those features in code. At the same time, there is effectively no standard-eval programmatic interface for it, which makes it a headache for building programs rather than scripting with. data.table is amazing, but it is anything but simple IMO.
- jeremyscanvic 6mo agoIt's very insightful how they explain the difference between dataframes and SQL tables / standard relational structures!
- hermitcrab 6mo ago>a dataframe is a tuple (A, R, C, D): an array of data A, row labels R, column labels C, and a vector of column domains D. What is 'a vector of column domains D'? A description of how the data A maps to columns?
- throw_await 6mo agoI think "domain" here is like the datatype
- hermitcrab 6mo agoI guess this article is an interesting exercise from a pure maths point of view. But, as someone developing a drag and drop data wrangling tool the important thing is creating a set of composable operations/primitive that are meaningful and useful to your end user. We have ended up 73 distinct transforms in Easy Data Transform. Sure they overlap to an extent, but feel they are at the right semantic level for our users, who are not category theorists.
- mrlongroots 6mo agoAlgebras are also nice for implementations. If you can decompose a domain into a few algebraic primitives you can write nice SIMD/CUDA kernels for those primitives. To your point, I wonder if the 73 distinct transforms were just different defaults/usability wrappers over these. And you may also get into situations where kernels can be fused together or other batching constraints enable optimizations that nice algebraic primitives don't capture. But that's just systems---theory is useful in helping rethink API bloats and keeping us all honest.
- hermitcrab 6mo agoThey are effectively highly level wrappers over the most primitive operations. High enough level that they can be used from a GUI, rather than code. It is a balance. Too few transforms and they become to low level for my users. Too many and you struggle to find the transform you want.
- jimbokun 6mo agoYou don’t have to limit the transforms you offer users to just the core ones. But for your own sanity you can implement the none core ones in terms of the core ones.
- whattheheckheck 6mo agoHave you heard of the book Mathematics for Big data https://github.com/Accla/d4m https://github.com/Accla/d4m He says himself the ideas are more important than the software package
- kokhanserhii 6mo ago[dead]
- kiviuq 6mo agothere is also ZIO Prelude and ZIO schema...
- Whyachi 6mo ago[dead]
- toxik 6mo agoPandas and so on exist for the same reason Django's ORM and SqlAlchemy do: people do not want to string interpolate to talk to their database. SQL is great for DBA's, and absolutely sucks for programmers. Microsoft was really onto something with LINQ, in my opinion.
- caseyross 6mo agoInteresting idea. I feel like it could be productive to categorize operations by their result shape as well: - Row select: From N rows, produce 0-N rows. - Column select: From N columns, produce 0-N columns. - Table add: From MxN and OxP tables, produce max M+OxN+P table. - Table subtract: From MxN and OxP tables, produce min 0x0 table. This line of thinking reveals some normally hard-to-see similarities, such as `groupby` and `dedupe` sharing the same underlying mechanism. (i.e., both are "collapsing" row selects.)
- voxleone 6mo agoIt’s almost suspiciously elegant: focus on transformations and their composition, and the structure takes care of itself.