3 ms·
Some people have performance requirements to meet, and we’re pretty locked into Python (because some of our engineers foolishly bought into the promise that we
by throwaway894345 6y ago
Some people have performance requirements to meet, and we’re pretty locked into Python (because some of our engineers foolishly bought into the promise that we could just rewrite the slow parts in C/Pandas/multiprocessing/etc).
If this is “living the dream” then Go and other languages will blow your mind. You can have iteration that is at least as fast (especially when collaborating with other engineers) and performance that is 2-3 orders of magnitude better than Python. Not to mention a better packaging and deployment story.
The sad thing is Python could be much better, but it has to back out of some bad design decisions first.
- ris 6y ago> Not to mention a better packaging and deployment story. Go's "packaging and deployment" is enough to rule it out for new projects for me.
- ska 6y agoThis was known at the dawn of python and before, though. For example lisps in various flavors were doing this high-and-low very well decades earlier; arguably nothing has done it radically better since. So it's worth asking why that never took over the world as a counter to why python exploded. It's tempting as a technology person to wish a "better python" had been the one to gain that sort of market impact. However, we have to at least ask the question if some of its technical flaws actually contributed to its success....
- throwaway894345 6y ago> So it's worth asking why that never took over the world as a counter to why python exploded. JavaScript would like a word.
- cheez 6y agodo you mind sharing exactly what problems you've had with Python?
- throwaway894345 6y agoPerformance and dependency management are the big ones. Huge binaries are another pain point as well. We would like to run code in lambda functions, but the binaries (which aren’t that complicated) are too big 250MB so we have to run them in ECS tasks which take something like 30s to start up and building the images takes a lot longer as well (we could probably spend time to optimize the docker image builds).
- cheez 6y agoI run lambda functions in Python but I just upload the code. I suppose you are using lambda functions in a way that isn't really promoted for Python.
- throwaway894345 6y agoWe also “just upload the code”; however, the issue is that the code size is too large. Lambdas can only be 250mb, and if your dependency tree includes Numpy that’s 70mb alone. Go binaries are about 2 orders of magnitude smaller than Python binaries by my crude estimation. Go compiles source code to native machine code and the linked trims dead code. Neither of these are possible in Python, so you get enormous binaries.
- cheez 6y agoI can see why that's a problem. No tree shaking either in Python. Still it seems like aws can solve the problem by lifting your limits or something.
- dralley 6y agoThey said they work on machine learning and data engineering, which means they are very likely using very optimized native code libraries with Python bindings. Go's FFI story has a lot of overhead, so it's actually not inconceivable that Python could be faster for their specific needs.
- throwaway894345 6y agoHis first paragraph suggests he was making a general comment, but I agree that if you’re doing very specific things like calling into optimized libraries then its not much of a problem; however, if you have to call into those libraries O(n) or worse, which is often the case, then you’ll very likely lose all or most of the performance you gained due to marshaling costs (not sure how these marshaling costs stack up to Go FFI, but in ago you would just write optimized Go in this situation—e.g., move allocations out of the hot path, make sure critical paths are inlined, etc—which isn’t an option available to Python developers). But yes, if your use case is calling into an optimized C library O(1) (and you have a high degree of confidence that it will never be worse than that) then Python will work fine (or rather, you won’t be fighting with performance, but you’ll still have to deal with package management, poor documentation, huge binaries, etc).