4 ms·
> I’ve found python optimization to be nearly intractable. Move your performance critical kernels outside Python. Numpy, Pytorch, external databases, Golang mi
by pphysch 1y ago
> I’ve found python optimization to be nearly intractable.
Move your performance critical kernels outside Python. Numpy, Pytorch, external databases, Golang microservice, etc.
It's in fact an extremely successful paradigm. Python doesn't need to be fast to be used in every domain (though async/nogil is still worth advancing to avoid idle CPUs).
- tialaramex 1y agoHowever note that (and this isn't just true for Python) if your "performance critical kernel" is "the whole program" then this does have the implication you think it does, you should not write Python. If your Python software is spending 99% of its time doing X, you can rewrite X in Rust and now the Python software using that is faster, but if your Python software is spending 7% of its time doing X, and 5% doing Y, and 3% doing Z then even if you rewrite X, and Y and Z you've shaved off no more than 15% and so it's probably time to stop writing Python at all. This is part of why Google moved to Go as I understand it.
- pphysch 1y agoSure, but there's also an argument for wrapping your pure Rust or Go executable in the thinnest layer of Python so that your users can optionally `pip install` it.
- 1718627440 1y agoAt that point you can use the OS package manager, which is more robust and installs programs in standard locations and incorporates things like process isolation.
- pphysch 11mo agoThen you are packaging for N different distros instead of just PyPI. Have fun...
- 1718627440 11mo agoA Distro is an OS. You don't package your program for Windows and expect it to work on MacOS. Don't expect this to work in the Free Software world.
- pphysch 11mo agoA distro is not strictly an OS. Ubuntu and RHEL can both run a Go binary built for Linux amd64 but they have completely different package managers.
- 1718627440 11mo ago> they have completely different package managers. Exactly, that's one case where the OSes completely differ. Just because the OSes share the same kernel, which prioritizes userspace compatibility, doesn't mean they are not a different OS. A kernel doesn't make an OS.
- pphysch 11mo agoI think you are needlessly conflating terms. My original comment was correct that you need to explicitly package your builds for many different distro package managers, if you want to go that route. That's a lot of work to maintain. Or you can just put it on PyPI which is everywhere nowadays.
- 1718627440 11mo agoWhich I as a user won't use. I trust my OS out of necessity, but I don't like to download and run random code from the internet and I don't have the time to review it. My OS instead often has patches, that strip out features I do not want and integrate it with the OS.