4 ms·

Oh absolutely, for some tasks Python is amazing. I use Jupyter notebooks a lot, for example, and the flexibility is an incredible feature. It just worries me w

by t8sr 3y ago

Oh absolutely, for some tasks Python is amazing. I use Jupyter notebooks a lot, for example, and the flexibility is an incredible feature. It just worries me when I sometimes see those same Jupyter notebooks running in production, crunching 100s of terabytes of data. Maybe I’m wrong, but I didn’t get the impression everyone realizes exactly how wasteful that is. I guess AWS credits are easy to come by. One thing Google did well back in the day, was making resource costs report in SWE/hours, the idea being that you see if you should go and rewrite something. If it cost 100 SWE/h to run, and it only took you a day to cut that in half, you should do it.

bb88 3y ago
Numpy is competitive with optimized C/C++. So even if it's running in a Jupyter notebook, it's still going to be insanely fast.
- GuestHNUser 3y ago
  > Numpy is competitive with optimized C/C++ Can you cite a source/example for that? I cannot imagine an optimized C program that doesn't blow python with numpy out of the water. Even a poorly written C program is likely to be 2x faster simply because it doesn't have to round trip operations from C to python and back.
  - bb88 3y ago
    I feel like this is google-able, no? I found some metrics after 30 seconds of googling.
    - LorenzoGood 3y ago
      Please post your citation for your claim.
      - bb88 3y ago
        No. Google it. I'm not your monkey.
- t8sr 3y ago
  Numpy is fine. But people write a lot of complicated code to pull JSON from somewhere, transform it in Python, and write it to parquet somewhere else, for example. JSON, the dict type and parquet are all implemented in C, but a comprehension on top of a Python iterable is just gonna be pure Python “bytecode”. It has been my experience that rewriting such things in C++, or even Go or Java is an easy way to quickly save truly incredible amounts of compute. A team I used to work with was forced to throw away a finished Python data pipeline that took them a year to build, because it cost more to run than the combined salaries of the team. And I really think if they’d had better intuition about Python’s performance under different scenarios, they could have saved a year of effort. This is why I feel it’s worth having frank discussions about trade offs when it comes to this language. It’s incredibly useful, but people in the community aren’t clearly told about its limitations. (Especially wrt performance, but also maintainability.)
  - jononor 3y ago
    If they had done performance testing from the start they could have saved a year. A pipeline that has not been performance tested was in no way "finished". Performance is not something that can be tackled on later. In any language...
    - bb88 3y ago
      I've seen bad Java/C++ code go to production before, and cost many more hours to fix it than it was just to replace the code with a working python script using the built in libraries.
  - bb88 3y ago
    > It has been my experience that rewriting such things in C++, or even Go or Java is an easy way to quickly save truly incredible amounts of compute. Sure but there's a trade off, no? Go is typically 3x the code than python. And C++ is 10x the complexity easily. There was one point back when I stopped coding C++ where one coder might not understand what another C++ coder was doing because the standard was so large. > A team I used to work with was forced to throw away a finished Python data pipeline that took them a year to build, because it cost more to run than the combined salaries of the team. You know, I have horror stories about C++ and Java as well. Usually that kind of blame goes to management for not understanding the issues up front. Pretty soon, I'll have slew of stories about go misusage as well.

by t8sr 3y ago

bb88 3y ago

Numpy is competitive with optimized C/C++. So even if it's running in a Jupyter notebook, it's still going to be insanely fast.

GuestHNUser 3y ago
> Numpy is competitive with optimized C/C++ Can you cite a source/example for that? I cannot imagine an optimized C program that doesn't blow python with numpy out of the water. Even a poorly written C program is likely to be 2x faster simply because it doesn't have to round trip operations from C to python and back.
- bb88 3y ago
  I feel like this is google-able, no? I found some metrics after 30 seconds of googling.
  - LorenzoGood 3y ago
    Please post your citation for your claim.
    - bb88 3y ago
      No. Google it. I'm not your monkey.
t8sr 3y ago
Numpy is fine. But people write a lot of complicated code to pull JSON from somewhere, transform it in Python, and write it to parquet somewhere else, for example. JSON, the dict type and parquet are all implemented in C, but a comprehension on top of a Python iterable is just gonna be pure Python “bytecode”. It has been my experience that rewriting such things in C++, or even Go or Java is an easy way to quickly save truly incredible amounts of compute. A team I used to work with was forced to throw away a finished Python data pipeline that took them a year to build, because it cost more to run than the combined salaries of the team. And I really think if they’d had better intuition about Python’s performance under different scenarios, they could have saved a year of effort. This is why I feel it’s worth having frank discussions about trade offs when it comes to this language. It’s incredibly useful, but people in the community aren’t clearly told about its limitations. (Especially wrt performance, but also maintainability.)
- jononor 3y ago
  If they had done performance testing from the start they could have saved a year. A pipeline that has not been performance tested was in no way "finished". Performance is not something that can be tackled on later. In any language...
  - bb88 3y ago
    I've seen bad Java/C++ code go to production before, and cost many more hours to fix it than it was just to replace the code with a working python script using the built in libraries.
- bb88 3y ago
  > It has been my experience that rewriting such things in C++, or even Go or Java is an easy way to quickly save truly incredible amounts of compute. Sure but there's a trade off, no? Go is typically 3x the code than python. And C++ is 10x the complexity easily. There was one point back when I stopped coding C++ where one coder might not understand what another C++ coder was doing because the standard was so large. > A team I used to work with was forced to throw away a finished Python data pipeline that took them a year to build, because it cost more to run than the combined salaries of the team. You know, I have horror stories about C++ and Java as well. Usually that kind of blame goes to management for not understanding the issues up front. Pretty soon, I'll have slew of stories about go misusage as well.