4 ms·

I think that what's even more interesting to me is that people perceive this as such a preoccupying problem. When I was a young little programmer, the dream wa

I think that what's even more interesting to me is that people perceive this as such a preoccupying problem. When I was a young little programmer, the dream was to have a base layer of highly performant native code for handling the heavy lifting, tightly coupled to a very high level scripting language that wasn't really even trying to be performant. The thought being that the scripting language should be more focused on flexibility than performance, because, for the bits of the software that it handled, developer productivity was more important than raw performance. Fast forward a year or two, and we are living the dream. It's Python, and, while I could nitpick implementation details and language features all day, the big picture is that it's pretty awesome. So awesome that, for the machine learning and data engineering work that I do, the most performant and most productive language I have to work in is Python. (Also R, which is a pretty similar technical story, but I realize that R lost the popularity contest, and I understand why.) I guess it starts to feel to me like we're making the perfect the enemy of the good, here. We've got a well-established, productive, flexible, and easy-to-learn language where top-tier performance can generally be implemented as a library solution. Despite its many imperfections, that's pretty darned good. In my career to date, I haven't seen much cause to believe that I could realistically expect much better than that. I think that, when I do need better performance, I'd rather write a library than rewrite the language.

throwaway894345 6y ago
Some people have performance requirements to meet, and we’re pretty locked into Python (because some of our engineers foolishly bought into the promise that we could just rewrite the slow parts in C/Pandas/multiprocessing/etc). If this is “living the dream” then Go and other languages will blow your mind. You can have iteration that is at least as fast (especially when collaborating with other engineers) and performance that is 2-3 orders of magnitude better than Python. Not to mention a better packaging and deployment story. The sad thing is Python could be much better, but it has to back out of some bad design decisions first.
- ris 6y ago
  > Not to mention a better packaging and deployment story. Go's "packaging and deployment" is enough to rule it out for new projects for me.
- ska 6y ago
  This was known at the dawn of python and before, though. For example lisps in various flavors were doing this high-and-low very well decades earlier; arguably nothing has done it radically better since. So it's worth asking why that never took over the world as a counter to why python exploded. It's tempting as a technology person to wish a "better python" had been the one to gain that sort of market impact. However, we have to at least ask the question if some of its technical flaws actually contributed to its success....
  - throwaway894345 6y ago
    > So it's worth asking why that never took over the world as a counter to why python exploded. JavaScript would like a word.
- cheez 6y ago
  do you mind sharing exactly what problems you've had with Python?
  - throwaway894345 6y ago
    Performance and dependency management are the big ones. Huge binaries are another pain point as well. We would like to run code in lambda functions, but the binaries (which aren’t that complicated) are too big 250MB so we have to run them in ECS tasks which take something like 30s to start up and building the images takes a lot longer as well (we could probably spend time to optimize the docker image builds).
    - cheez 6y ago
      I run lambda functions in Python but I just upload the code. I suppose you are using lambda functions in a way that isn't really promoted for Python.
marmaduke 6y ago
I think though the perceived problem is that as soon as you have to shift between high and low levels, you have a boring friction which gets people dreaming of a different way, hence Go, Julia et al.
- throwaway894345 6y ago
  It’s that the “shift between high and low levels” costs performance, and if you have to do it O(n) times or worse (as is often the case), then that marshaling cost is a significant portion of (or even more than) the savings that you get from using a lower level language. This isn’t an issue in Go because you can just write faster Go without a marshaling overhead (or you can pay a similar overhead and call into optimized C libraries, but this is rarely necessary or worth the trouble since Go is far more maintainable and generally fast enough.

throwaway894345 6y ago

Some people have performance requirements to meet, and we’re pretty locked into Python (because some of our engineers foolishly bought into the promise that we could just rewrite the slow parts in C/Pandas/multiprocessing/etc). If this is “living the dream” then Go and other languages will blow your mind. You can have iteration that is at least as fast (especially when collaborating with other engineers) and performance that is 2-3 orders of magnitude better than Python. Not to mention a better packaging and deployment story. The sad thing is Python could be much better, but it has to back out of some bad design decisions first.

ris 6y ago
> Not to mention a better packaging and deployment story. Go's "packaging and deployment" is enough to rule it out for new projects for me.
ska 6y ago
This was known at the dawn of python and before, though. For example lisps in various flavors were doing this high-and-low very well decades earlier; arguably nothing has done it radically better since. So it's worth asking why that never took over the world as a counter to why python exploded. It's tempting as a technology person to wish a "better python" had been the one to gain that sort of market impact. However, we have to at least ask the question if some of its technical flaws actually contributed to its success....
- throwaway894345 6y ago
  > So it's worth asking why that never took over the world as a counter to why python exploded. JavaScript would like a word.
cheez 6y ago
do you mind sharing exactly what problems you've had with Python?
- throwaway894345 6y ago
  Performance and dependency management are the big ones. Huge binaries are another pain point as well. We would like to run code in lambda functions, but the binaries (which aren’t that complicated) are too big 250MB so we have to run them in ECS tasks which take something like 30s to start up and building the images takes a lot longer as well (we could probably spend time to optimize the docker image builds).
  - cheez 6y ago
    I run lambda functions in Python but I just upload the code. I suppose you are using lambda functions in a way that isn't really promoted for Python.

marmaduke 6y ago

I think though the perceived problem is that as soon as you have to shift between high and low levels, you have a boring friction which gets people dreaming of a different way, hence Go, Julia et al.

throwaway894345 6y ago
It’s that the “shift between high and low levels” costs performance, and if you have to do it O(n) times or worse (as is often the case), then that marshaling cost is a significant portion of (or even more than) the savings that you get from using a lower level language. This isn’t an issue in Go because you can just write faster Go without a marshaling overhead (or you can pay a similar overhead and call into optimized C libraries, but this is rarely necessary or worth the trouble since Go is far more maintainable and generally fast enough.