4 ms·

I invested time in a similar way with Python and it has definitely been the most lucrative skill investment for me so far. I can think of a few specific projec

by mlthoughts2018 7y ago

I invested time in a similar way with Python and it has definitely been the most lucrative skill investment for me so far. I can think of a few specific projects in my early career where I wrote things in Python while other teams in my company were trying to write similar things in C++. The Python implementations (which used Cython for a few tiny performance critical sections) ended up running much, much faster than the C++ projects (which were written by very veteran C++ engineers at the time). More than that, they were easier to test, easier to modify, and easier to teach to random new programmers joining the team. We benefited from much more user-friendly third party tools and specialized domain tools and could easily wrap any of the in-house & domain tools only accessible in C/C++.

saagarjha 7y ago
> The Python implementations (which used Cython for a few tiny performance critical sections) ended up running much, much faster than the C++ projects (which were written by very veteran C++ engineers at the time). More than that, they were easier to test, easier to modify I’m honestly really surprised. Can you provide some more detail as to how Python was beneficial for these?
- mlthoughts2018 7y ago
  It allowed me to write Cython for only small pieces of programs for which there is profiling evidence of a performance bottleneck. Cython is a slight superset of Python that compiles to heavily optimized C or C++. So it is extremely easy to write & autogenerate Python bindings to your own C extension modules. The difference is that it frees you to write Python (much easier to write & maintain) for everything, and only care about compiling down for speed in a few isolated places. Contrasted with writing everything in C/C++, this is a big advantage.
rectangletangle 7y ago
>much faster than C++ I've been writing Python for over a decade and I find this a bit surprising (not saying you're wrong). Python's strong suit was always speed of productivity over runtime speed IMO.
- xmprt 7y ago
  Could have been the C++ engineers using loads of abstractions and anti-patterns to write memory safe code at the cost of performance. Still surprising that it would be slower.
- mlthoughts2018 7y ago
  The idea that Python’s strong suit is “glue code” or flexibility is a bit of a myth. Python (CPython anyway) is nothing but a special DSL for writing C programs. Any part of Python is absolutely as fast as C if you want it to be. In the projects I’m referring to, we just used Cython for this. Instead of wasting time writing in C for the whole project, managing memory allocation, building up abstractions over structs with macros, or using these things from inscrutable C++, we just wrote simple Python, used kernprof to measure places where running time actually mattered, and targeted those few small places with a Cython extension module. Since the small part of performance critical code was much simpler in Python (using pretty much zero abstraction / functions only in Cython), the compiled Cython code was faster than corresponding components of C++ implementations (that took much longer to write and understand). Most of the rest of e.g. Python standard library calls in the surrounding glue code is already C-level extension functions in Python with no measurable running time difference than C itself, and the rest of the custom Python code was measured in the profiler to be fast enough so as not to matter. The “slow” parts of Python come from the object data model and a variety of protocols like iteration, properties and MRO. But it isn’t really fair to call it “slow” because it’s a trade-off to get those features. If you need an object that supports being iterated, having its string representation overrided, and inheriting methods from some mix-in, then you are agreeing to pay a running time cost for this. If you don’t need that stuff, you could just write what you need in e.g. Cython. With more heavily automated tools now like numba, this argument for using Python as a first choice for writing very fast code is only getting more mainstream. Most of the time, you do need a bunch of “batteries included” features of Python and running time is not the resource constraint you’re worried about. And in the occasional exceptional cases when you need raw speed, you can use extension module strategies (e.g. how numpy was made) to be as fast as C.
  - netfl0 7y ago

saagarjha 7y ago

> The Python implementations (which used Cython for a few tiny performance critical sections) ended up running much, much faster than the C++ projects (which were written by very veteran C++ engineers at the time). More than that, they were easier to test, easier to modify I’m honestly really surprised. Can you provide some more detail as to how Python was beneficial for these?

mlthoughts2018 7y ago
It allowed me to write Cython for only small pieces of programs for which there is profiling evidence of a performance bottleneck. Cython is a slight superset of Python that compiles to heavily optimized C or C++. So it is extremely easy to write & autogenerate Python bindings to your own C extension modules. The difference is that it frees you to write Python (much easier to write & maintain) for everything, and only care about compiling down for speed in a few isolated places. Contrasted with writing everything in C/C++, this is a big advantage.

rectangletangle 7y ago

>much faster than C++ I've been writing Python for over a decade and I find this a bit surprising (not saying you're wrong). Python's strong suit was always speed of productivity over runtime speed IMO.

xmprt 7y ago
Could have been the C++ engineers using loads of abstractions and anti-patterns to write memory safe code at the cost of performance. Still surprising that it would be slower.
mlthoughts2018 7y ago
The idea that Python’s strong suit is “glue code” or flexibility is a bit of a myth. Python (CPython anyway) is nothing but a special DSL for writing C programs. Any part of Python is absolutely as fast as C if you want it to be. In the projects I’m referring to, we just used Cython for this. Instead of wasting time writing in C for the whole project, managing memory allocation, building up abstractions over structs with macros, or using these things from inscrutable C++, we just wrote simple Python, used kernprof to measure places where running time actually mattered, and targeted those few small places with a Cython extension module. Since the small part of performance critical code was much simpler in Python (using pretty much zero abstraction / functions only in Cython), the compiled Cython code was faster than corresponding components of C++ implementations (that took much longer to write and understand). Most of the rest of e.g. Python standard library calls in the surrounding glue code is already C-level extension functions in Python with no measurable running time difference than C itself, and the rest of the custom Python code was measured in the profiler to be fast enough so as not to matter. The “slow” parts of Python come from the object data model and a variety of protocols like iteration, properties and MRO. But it isn’t really fair to call it “slow” because it’s a trade-off to get those features. If you need an object that supports being iterated, having its string representation overrided, and inheriting methods from some mix-in, then you are agreeing to pay a running time cost for this. If you don’t need that stuff, you could just write what you need in e.g. Cython. With more heavily automated tools now like numba, this argument for using Python as a first choice for writing very fast code is only getting more mainstream. Most of the time, you do need a bunch of “batteries included” features of Python and running time is not the resource constraint you’re worried about. And in the occasional exceptional cases when you need raw speed, you can use extension module strategies (e.g. how numpy was made) to be as fast as C.
- netfl0 7y ago