4 ms·
As a very partial; almost unrelated question: Is there any python module that you use day-to-day that you'd like to have a significant speedup with? I'm thinki
by dijit 5y ago
As a very partial; almost unrelated question: Is there any python module that you use day-to-day that you'd like to have a significant speedup with?
I'm thinking of reimplementing some python modules in rust, as that seems like the kind of weird thing I'm in to. I've done it with some success (using the excellent work of the pyo3 project) professionally, but I'd be interested in doing more.
- mritchie712 5y agopandas
- mynameis_jeff 5y agowould give https://github.com/modin-project/modin https://github.com/modin-project/modin a shot
- SnooSux 5y agoIt's been done: https://github.com/pola-rs/polars https://github.com/pola-rs/polars But I'm sure there's always room for improvement
- rytill 5y agoIt’s not like polars is a drop-in replacement, it has a totally different API.
- mrtranscendence 5y agoYou wrote "it has a totally different API", did you mean "it has an actually sane API?" Because that's what I think of when I compare pandas to polars.
- ies7 5y agoNot if you've legacy apps using pandas everywhere. Same API means I can import polars as pd and be done with it.
- fgh 5y agoThe answer would then be to have a look at polars.
- w-m 5y agoThis is a curious reply for me. I would think that there are very few parts in pandas that could be sped-up by reimplementing them with a compiled language. Pandas is plenty fast for the built-in methods, it only gets slow when you start interfacing with Python, e.g. by doing an `.apply` with your custom Python method. Obviously this interfacing part is impossible to speed up by reimplementing parts of pandas (you'd need a different API instead).
- curiousgal 5y agoI remember, when trying to squeeze some performance out of it, that a lot of the overhead came from it trying to infer types.
- zmgsabst 5y agoYou’d be awesome if you wrote a library for large image processing. You can make large Numpy arrays fine — eg, 20k x 20k or 500k x 500k, but trying to render that to anything but SVG or manual tilings pukes badly. That’s my main blocker on rendering high dimensional shapes: you can do the math, but visualizations immediately fall over (unless you do tiling yourself). There’s probably someone with a more useful idea than “gigapixel rendering” though.
- yedpodtrzitko 5y agoPydantic is quite popular library. Its author is doing exactly this - rewriting its core [0] in Rust. It's still WIP, but readme mentions that "Pydantic-core is currently around 17x faster than Pydantic Standard." [0] https://github.com/samuelcolvin/pydantic-core https://github.com/samuelcolvin/pydantic-core
- skrtskrt 5y agoPydantic is not just popular (and awesome) on its own but serves as the underpinnings of a lot of the FastAPI functionality - faster Pydantic would make a LOT of apps faster
- tclancy 5y agoNot working in Python right now, but I have 15 years of Python + Django on the web and while there are any number of attempts at this (I keep a list at https://pinboard.in/u:tclancy/t:json/t:python/ https://pinboard.in/u:tclancy/t:json/t:python/), any improvement in JSON serialization and unserialization speeds is a huge boon to projects. I am trying to think of similar bottlenecks where a drop-in replacement can be a huge performance improvement.
- JackC 5y agoThe missing thing last time I looked was a fast python json library that's byte-compatible with stdlib -- same inputs, same outputs. There are good fast options but they tend to add some (perfectly reasonable) limitation like fixed indentation size, for the sake of speed, that blocks them from being dropped into an existing public API.
- dotnet00 5y agoDefinitely matplotlib. Navigating image plots in interactive mode with even just 10000x10000 pixels is painfully slow. While I've picked up some alternatives, they don't feel as clean as matplotlib.
- wcunning 5y ago10000% -- matplotlib for visualization of a lot of different data I've looked at, but esp things like high res images in machine learning contexts is incredibly slow, even on good computers. It does fine for small vector stuff and render once and save graphs, but it's bad for what a lot of people use it for.