devin-petersohn

5 ms·

1.
Add Support for PyCapsule to Pyspark(github.com)
3 points by devin-petersohn 8mo ago|0 comments
2.
by devin-petersohn 8mo ago
Funny enough, I actually just (2 weeks ago) added support for streaming from Pyspark to Polars/DuckDB/etc through Arrow PyCapsule. By streaming, I mean actually streaming, not collecting all data at once. It won't be release
3.
Why Are There So Many Python Dataframes?(ponder.io)
4 points by devin-petersohn 3y ago|0 comments
4.
by devin-petersohn 4y ago
The core abstraction of Modin is the dataframe algebra, published here: https://arxiv.org/pdf/2001.00888.pdf
5.
by devin-petersohn 4y ago
I always appreciate blog posts like this, there are obviously cases where SQL shines, and in part I think the dataframe abstraction helps with filling a lot of the missing pieces that SQL doesn't handle so well (composability, debuggab
6.
Pandas is More Concise than SQL(ponder.io)
6 points by devin-petersohn 4y ago|0 comments
7.
by devin-petersohn 4y ago
There are loads of things that are not possible or are very cumbersome to write in SQL, but that pandas and many other dataframe systems allow. Examples are dropping null values based on some threshold, one-hot encoding, covariance, and cer
8.
by devin-petersohn 5y ago
This blog post is a great way to learn about the theory underlying dataframes without reading a bunch of technical jargon from our academic paper[1]. AMA! [1] https://arxiv.org/pdf/2001.00888
9.
We parallelized 600 pandas functions with Modin(ponder.io)
1 points by devin-petersohn 5y ago|1 comments
10.
by devin-petersohn 5y ago
This was my PhD focus. We identified a core "dataframe algebra"[1] that encompasses all of pandas (and R/S data.frames): a total of 16 operators that cover all 600+ operators of pandas. What you describe was exactly our aim.
11.
by devin-petersohn 6y ago
I'm one the maintainers of Modin, so I can chime in here. Dataframes are the focus of my PhD thesis, and Modin started as my PhD project. Most of the differences come down to functionality and support. Truthfully, the goals of the proj
12.
How to ensure a data scientist is never productive(medium.com)
2 points by devin-petersohn 6y ago|0 comments
13.
Modin's View of Scaling Pandas: Comparing Modin vs. Dask, Ray, Vaex, and Rapids(towardsdatascience.com)
5 points by devin-petersohn 6y ago|0 comments
14.
Dataframes are losing their statistical computing and machine learning roots(towardsdatascience.com)
6 points by devin-petersohn 7y ago|0 comments