2 ms·
To be fair, it's clearly stated in the readme: "The performance claims only hold for small datasets, 1,000-100,000 numpy rows. Pandas perform better with large
by cerved 7y ago
To be fair, it's clearly stated in the readme:
"The performance claims only hold for small datasets, 1,000-100,000 numpy rows. Pandas perform better with larger data sets, the only functions that improve with a 1000x increase in size is rename, column drop, fillna mean, correlation matrix, value reads, and np calculations even out (np.log, np.exp as well as etc)"
- lmeyerov 7y agoThis is a real problem, akin to how modern JITs will have diff run modes for the same code. We're overdue for pandas-without-huge-overhead. See this thread for more on why pandas overhead adds up for real settings, and is crazy huge once you go to dask/ray/modin, and even worse, say spark: https://twitter.com/lmeyerov/status/1218093436286296069 https://twitter.com/lmeyerov/status/1218093436286296069 + https://twitter.com/lmeyerov/status/1220847229537157121 https://twitter.com/lmeyerov/status/1220847229537157121 . For interactive analytics apps, we want scale for bigger workloads, yet also less overhead to achieve < 100ms: ideally moving a slider will trigger lots of calcs over many widgets and their subcomponents, and the underlying stats code looks normal (e.g., df > sql > some imperative/functional lang.) Internally we write C-like buffer manips & hand-written GPU kernels & weird streaming code to achieve < 100ms wall-clock, and are trying to get tech like these up to snuff for interactive data software. As per the JIT comment, feels like a matter of time, so happy to see this (and the pressure it creates!)
- shoyer 7y agoMost of those disclaimers were added tonight, after seeing my comment :) https://github.com/firmai/pandapy/commit/692c968771bb19d4f12fac3d1eb76f8e611db165 https://github.com/firmai/pandapy/commit/692c968771bb19d4f12...