6 ms·

It probably depends on the type of apps one is writing. I am not advocating writing your own libs (though that is completely feasible too) but just using the ex

It probably depends on the type of apps one is writing. I am not advocating writing your own libs (though that is completely feasible too) but just using the existing ones. When one needs to convert from/to Python structures within the hot path, this of course doesn't perform well... However in most cases I worked on this was not needed - convert input to numpy / xarray /..., perform all calculations, get the result. So with about the same amount of experience, I heartily recommend python. :)

throwaway894345 5y ago
The problem is that you can't predict from the outset that all of your performance problems will be a good fit for Numpy or whatever, and if they aren't the options you have are very expensive in terms of engineering time, maintainability, etc. And it's not just performance--dependency management is still a significant problem, as is deployment in many cases. Honestly, I switched to Go. It's like a fast, statically typed, statically compiled Python. And by "fast", I mean 100-1000x faster than CPython. The package manager resolves dependencies in an instant (compared with pipenv that would take 30 minutes just to update a lockfile for a small project). Everything compiles to a static binary so you can build internal tools without requiring end users to set up a virtualenv/etc. I've rewritten real world Python programs that would be distributed as a 250MB zip file or a multi-GB docker image and converted them into Go programs which would distribute as a 2.5MB binary or 2.5MB docker image. Docker image builds went from 45 minutes (after weeks of painstakingly optimizing Dockerfiles) to ~2 minutes with a straightforward Dockerfile. The ramifications of a fast iteration loop are also hard to overstate--this was a major source of pain that virtually disappeared when the application was ported to Go. On AWS ECS/Fargate and GKE, container cold start times went from 30-50 minutes to ~30s (not sure why pulling from ECR/GCR and then starting the container took so long on these platforms--I certainly wouldn't expect a few GB to take half an hour to pull over a datacenter network). The performance also improved tremendously--in one case we traded so much maintainability in order to get a Python/Pandas system to complete requests within 60s and a naive Go rewrite would complete the same requests in 2 seconds (CPU intense parallel-friendly workload). We also looked at Dask, Polars, Spark, etc. With Python, we ran into significant Docker For Mac performance problems (the Docker VM would use all available CPU to marshal filesystem events to/from the VM, grinding the app to a halt and chewing through your battery) with Go there's no need to mount a source code filesystem to mount in the first place--you just rebuild on change (either the whole image or just rebuild the binary and `docker cp` it onto the running container). Of course, I'm sure there are other language environments that also have benefits over Python. I've just found Go to be the best I've tried by a pretty wide margin. Frankly, Python is just littered with pitfalls--you just sort of find yourself painted into these corners unexpectedly and you waste a bunch of time trying out all of these tools, frameworks, and libraries that purport to solve your problem but which introduce their own novel failure modes for nontrivial applications (e.g., Pipenv promises to solve dependency management but every lockfile operation takes half an hour, Cython promises to improve performance but it adds considerable complexity to your build tooling, Mypy promises to fix typing but it can never find the type annotations for dependencies, Pypy promises to improve performance but a ton of important ecosystem packages are unsupported, etc). Python development just feels like wandering from one tarpit to another, while in Go I just get things done.
- trulyme 5y ago
  I like Go and the benefits are real (though a GB Docker image has nothing to do with Python), it is however very verbose. I use both (for different use cases) and coming back from Go to Python is such a relief... As with everything, this is a tradeoff too. I don't feel the pain of python packages though, for me pipenv solves this nicely. To each their own I guess... :)

throwaway894345 5y ago

The problem is that you can't predict from the outset that all of your performance problems will be a good fit for Numpy or whatever, and if they aren't the options you have are very expensive in terms of engineering time, maintainability, etc. And it's not just performance--dependency management is still a significant problem, as is deployment in many cases. Honestly, I switched to Go. It's like a fast, statically typed, statically compiled Python. And by "fast", I mean 100-1000x faster than CPython. The package manager resolves dependencies in an instant (compared with pipenv that would take 30 minutes just to update a lockfile for a small project). Everything compiles to a static binary so you can build internal tools without requiring end users to set up a virtualenv/etc. I've rewritten real world Python programs that would be distributed as a 250MB zip file or a multi-GB docker image and converted them into Go programs which would distribute as a 2.5MB binary or 2.5MB docker image. Docker image builds went from 45 minutes (after weeks of painstakingly optimizing Dockerfiles) to ~2 minutes with a straightforward Dockerfile. The ramifications of a fast iteration loop are also hard to overstate--this was a major source of pain that virtually disappeared when the application was ported to Go. On AWS ECS/Fargate and GKE, container cold start times went from 30-50 minutes to ~30s (not sure why pulling from ECR/GCR and then starting the container took so long on these platforms--I certainly wouldn't expect a few GB to take half an hour to pull over a datacenter network). The performance also improved tremendously--in one case we traded so much maintainability in order to get a Python/Pandas system to complete requests within 60s and a naive Go rewrite would complete the same requests in 2 seconds (CPU intense parallel-friendly workload). We also looked at Dask, Polars, Spark, etc. With Python, we ran into significant Docker For Mac performance problems (the Docker VM would use all available CPU to marshal filesystem events to/from the VM, grinding the app to a halt and chewing through your battery) with Go there's no need to mount a source code filesystem to mount in the first place--you just rebuild on change (either the whole image or just rebuild the binary and `docker cp` it onto the running container). Of course, I'm sure there are other language environments that also have benefits over Python. I've just found Go to be the best I've tried by a pretty wide margin. Frankly, Python is just littered with pitfalls--you just sort of find yourself painted into these corners unexpectedly and you waste a bunch of time trying out all of these tools, frameworks, and libraries that purport to solve your problem but which introduce their own novel failure modes for nontrivial applications (e.g., Pipenv promises to solve dependency management but every lockfile operation takes half an hour, Cython promises to improve performance but it adds considerable complexity to your build tooling, Mypy promises to fix typing but it can never find the type annotations for dependencies, Pypy promises to improve performance but a ton of important ecosystem packages are unsupported, etc). Python development just feels like wandering from one tarpit to another, while in Go I just get things done.

trulyme 5y ago
I like Go and the benefits are real (though a GB Docker image has nothing to do with Python), it is however very verbose. I use both (for different use cases) and coming back from Go to Python is such a relief... As with everything, this is a tradeoff too. I don't feel the pain of python packages though, for me pipenv solves this nicely. To each their own I guess... :)