4 ms·

Python has been popular for this because it’s convenient to quickly hack on and experiment with, not because it’s the most efficient thing.

• im3w1l 3y ago
  The overhead really isn't that bad is it? Since the the python code is mostly about saying multiply matrix A with matrix B, and then that actual computation is done by optimized low level code.
  • jiggawatts 3y ago
    I suspect that this has a high chance of running afoul of Ahmdal’s Law. Even if you can parallelise the bulk of the computation, the serial parts remain single-threaded and start to dominate the total runtime.
    • taneq 3y ago
      I don’t think the serial parts of ML training are Python’s fault, are they? It’s all “operation B depends on the output of operation A”.
  • bongodongobob 3y ago
    For that stuff, yeah you're correct. What I've seen is issues with the implementation of those libraries in a project. I don't remember exactly, but I was playing with someone's wrapper for some kind of machine learning snake game and it was taking way longer than it should have on back of the napkin math. The issue was using either a dict or a list in a hot loop and changing it to the other sped it up like 1000x. So it's easy to think "yeah this library is optimized" but then you build something on top of it that is not obviously going to slow it down. But, that's the Python tradeoff.
    • 0cf8612b2e1e 3y ago
      That sounds irrelevant to Python and just a matter of slow code cropping up in libraries until someone runs a profiler.
      • littlestymaar 3y ago
        But then again if your program have places where choosing the right Python primitive is important for performance, then using python is affecting performance here since even the best algorithm in Python would be slower than the equivalent C. Most of the time it doesn't matter because there's nothing hoy on the Python side, but if there is, then Python is going to be slowing your stuff down.
    • hcarvalhoalves 3y ago
      > The issue was using either a dict or a list in a hot loop and changing it to the other sped it up like 1000x. The programmer using the wrong data structure is not a problem with the language.
      • CamperBob2 3y ago
        It really is with Python. There are simply too many containers and container-like concepts. Lists, arrays, sets, dicts...
        • 0cf8612b2e1e 3y ago
          What modern language doesn’t have those? Go kind of cheats and has maps play double duty as sets.
      • bongodongobob 3y ago
        Kinda. I guess my native tongue is C/C++ and I wouldn't expect such a huge performance difference when using an array vs a linked list or something. It's not like I had millions of items in that structure either, it was like 100. I think it contained the batch training data from each round. I tried to find the project but couldn't. I was just shocked that there was such a huge difference between primitive data structures. In that situation, I wouldn't have guessed it would make a difference.
  • llm_nerd 3y ago
    It depends on how you define overhead. Runtime overhead and memory usage is absolutely marginal, and the tightest, most perfect implementation will have trouble beating it. Instead people are trying to optimize install size of dependencies, which while maybe a fun hacking project...who really cares?