3 ms·
Dataframes in Python are a wrapper around 2D numpy arrays, that have labels and various accessors. Operations on them are OOM slower than using the underlying a
by the__alchemist 3y ago
Dataframes in Python are a wrapper around 2D numpy arrays, that have labels and various accessors. Operations on them are OOM slower than using the underlying arrays.
- jessekv 3y agoThis is true specifically for Pandas DataFrames. Numpy arrays are themselves just wrappers around c arrays, which are contiguous chunks of memory. Polars supports operations on files larger than memory.
- YetAnotherNick 3y agoI don't know where this myth originated from but I have seen this in multiple places. Even if you think about it 2d numpy arrays can't have different type for different columns.
- theLiminator 3y agoActually numpy has something called a structured array that is pretty much what what you described.
- billyjmc 3y agoWell, if you use structured arrays or record arrays, you can do this (more or less). https://numpy.org/doc/stable/user/basics.rec.html https://numpy.org/doc/stable/user/basics.rec.html
- deleted 3y ago[deleted]
- FrustratedMonky 3y ago"myth originated" It's in the documentation. I just learned pandas recently, and I would have said this same thing. Not because I read through the numpy code, but because I read the documentation. Is it wrong? Can't a user pick up a new tool and trust some documentation without reading through 100's of libraries built on libraries. When was last time someone traced out every dependency, so they can confidently say something is "Myth".
- YetAnotherNick 3y agoWhere is it written in pandas documentation? Pandas dataframe is stored in list of 1d numpy arrays, not a single 2d array.
- ayhanfuat 3y agoThe columns with common dtypes are grouped together in something called "blocks" and inside those blocks are 2D numpy arrays. It is probably not in the documentation because it is seen as implementation detail but you can see the block manager's structure in this article (https://dkharazi.github.io/blog/blockmanager/ https://dkharazi.github.io/blog/blockmanager/) or in this talk (https://thomasjpfan.github.io/scipy-2020-lightning-talk-pandas-blockmanager https://thomasjpfan.github.io/scipy-2020-lightning-talk-pand...).
- twelvechairs 3y agoThere's a very good point here but I don't think its made clear. If your data fits into numpy arrays or structured arrays (mainly if it is in numeric types), numpy is designed for this and will likely be much faster than pandas/polars (though I've also heard pandas can be faster on very large tables). Pandas and Polars are designed for ease of use on heterogeneous data. They also include a python 'Object' data type which numpy very much does not. They are also designed more like database (e.g. 'join' operations). This allows you to work directly with imported data that numpy won't accept - after which Pandas uses numpy for underlying operations. So I think the point is if you are running into speed issues in Pandas/Polars, you may find that the time-critical operations could be things that are more efficiently done in numpy (and this would be a much bigger gain than moving from Pandas to Polars)