3 ms·

Conversely, freeing data science/numerical analysis tooling from the tire-fire of the Python ecosystem might actually be a net benefit for the community at larg

Conversely, freeing data science/numerical analysis tooling from the tire-fire of the Python ecosystem might actually be a net benefit for the community at large. Get a bit more variety into the options, explore some alternative approaches. Who knows, maybe some ideas from other programming languages will yield some useful advantages.

sgarland 3y ago
Python’s ecosystem is great IFF you use 3rd party tooling carefully, and understand how and when to use it. The biggest one is to use venvs liberally, and to let as few things as possible interact with system Python. If there’s a CLI tool written in Python, ideally install it with pipx. Use Poetry for every project. Unfortunately these aren’t inherently obvious until you have a raging tire fire, and even then, you have to know the source of it and that better options exist. But hey, at least it isn’t Node.
- pjmlp 3y ago
  Node has a JIT by default, while Python is still fighting to get one that is widely accepted. I still have hopes that Julia and Ruby competition makes it change.
  - sgarland 3y ago
    In theory, Node is faster than Python due to this, among other things. In practice, I have seen Node apps blocking or crashing far more often than Python apps, which drags the performance down just a bit. In the meantime, @njit decorator on Python functions, and boom, done.
    - pjmlp 3y ago
      I am yet to see such thing, to each its own I guess. @njit doesn't work in regular Python, requires installing Numba and only works in very specific use cases.
      - sgarland 3y ago
        Well yes, it’s not native, but the point was that it’s very simple to use (and install). As to specificity, I’d argue that it works on the things which stand to benefit the most from it. Parallelizing loops, vectorizing, etc.
        FridgeSeal 3y ago
        For those playing at home wondering why you can’t just do this on all Python loops: The numba jit is for numerical code. You won’t magically get better performance iterating over a bunch of things you get back from your db for example.
- FridgeSeal 3y ago
  To be fair to Node, their package management is saner and more consistent than Python’s and their build/management tooling is far, far more mature.
- IshKebab 3y ago
  > But hey, at least it isn’t Node. Why? Node's packaging story is about a million times better than Python's. You literally just listed some of the issues with Python, none of which apply to Node. > use venvs liberally Not necessary with Node (you could argue it always uses venvs but that's transparent from a user point of view). > to let as few things as possible interact with system Python. Not an issue. > If there’s a CLI tool written in Python, ideally install it with pipx. Node has one way to install things and it isn't the way that nobody has heard of (I honestly had not heard of pipx until now and I'm forced to write Python some of the time for my job). > Use Poetry for every project. You don't have to magically know which project management tool out of dozens to use with Node. Just use the overwhelmingly popular one (NPM). You missed a few more by the way. How about knowing to use `--config-settings editable_mode=compat` in order to use editable installs with static tooling? Oh and remember that that is deprecated with no alternative! And it silently doesn't work in `requirements.txt`! Yeay. Then there's `pip`'s fun "wheel error! this probably isn't a wheel error" message. Very user friendly. Oh and don't forget the whole `pip3`/`python3 -m pip` debacle. Did I say `python3`? I meant `python3` except on Windows. Node is a million miles ahead of Python in terms of packaging usability.
  - sgarland 3y ago
    > Why? Node's packaging story is about a million times better than Python's. You literally just listed some of the issues with Python, none of which apply to Node. No, instead you get to deal with absurdly slow dependency solving / installation (I gather Bun is fast, so good on that guy), the existence of leftpad, and a `node_modules` directory that is stupidly large. > Not an issue. [to let as few things as possible interact with system Python.] Obviously, because systems aren't issuing tooling in Node. > Node has one way to install things and it isn't the way that nobody has heard of (I honestly had not heard of pipx until now and I'm forced to write Python some of the time for my job). This is literally untrue – I know for a fact that both npm and yarn are extremely popular, and as mentioned, the newcomer bun exists. > How about knowing to use `--config-settings editable_mode=compat` in order to use editable installs with static tooling? Or, you know, switch to modern solutions for this: `poetry add --path /foo/package` or `pipenv install -e /foo/package`. > Oh and don't forget the whole `pip3`/`python3 -m pip` debacle. Did I say `python3`? I meant `python3` except on Windows. Don't use Windows, obvs. Seriously though, I give absolutely zero fucks about development experience on Windows with Python.
- yodelshady 3y ago
  I have literally never had a project for which Poetry, given an exact list of dependencies and flawless running on the "bad system", installed correctly on first try. Most recently it defaults to asking for a keyring, even when just downloading? I believe that's marked WONTFIX. Even with that hacked around, and I thought the point was to reduce the number of hacks required, tensorflow seems to almost universally fail in absolutely any configuration I've tried. Don't even begin to try integrating your own packages, c++ template errors are tame by comparison. I've also never seen a poetry error fit in less than 500 lines of nested exceptions. I want to like this tool but... boy, Docker doesn't give me this crap.
  - sgarland 3y ago
    Granted I haven’t tried Tensorflow with it, but I’ve literally never had any issues with it that weren’t caused by system Python changing (like Brew updating the Python minor version). And that’s fixed by installing it with pipx, which is one of their recommended ways to do so.
- dagw 3y ago
  Use Poetry for every project Poetry is tightly optimised for what they consider the 'correct' way of developing 'normal' python apps. Wander off that path and poetry becomes worse and worse.
  - sgarland 3y ago
    Maybe they have a point. It also fits with the Zen of Python (which tbf rarely seems to happen) of, “There should be one-- and preferably only one --obvious way to do it.” Node / JS in general seems to take the path of, “this existing solution doesn’t precisely fit my needs, so instead of rethinking why that might be, I’m going to reinvent the wheel.”
    - dagw 3y ago
      Maybe they have a point. Given that you where arguing that using python on the most popular OS in the world should fall outside of a 'normal' use case python should focus on, perhaps you have a too narrow view of what "correct and normal" python usage should be.

sgarland 3y ago

Python’s ecosystem is great IFF you use 3rd party tooling carefully, and understand how and when to use it. The biggest one is to use venvs liberally, and to let as few things as possible interact with system Python. If there’s a CLI tool written in Python, ideally install it with pipx. Use Poetry for every project. Unfortunately these aren’t inherently obvious until you have a raging tire fire, and even then, you have to know the source of it and that better options exist. But hey, at least it isn’t Node.

pjmlp 3y ago
Node has a JIT by default, while Python is still fighting to get one that is widely accepted. I still have hopes that Julia and Ruby competition makes it change.
- sgarland 3y ago
  In theory, Node is faster than Python due to this, among other things. In practice, I have seen Node apps blocking or crashing far more often than Python apps, which drags the performance down just a bit. In the meantime, @njit decorator on Python functions, and boom, done.
  - pjmlp 3y ago
    I am yet to see such thing, to each its own I guess. @njit doesn't work in regular Python, requires installing Numba and only works in very specific use cases.
    - sgarland 3y ago
      Well yes, it’s not native, but the point was that it’s very simple to use (and install). As to specificity, I’d argue that it works on the things which stand to benefit the most from it. Parallelizing loops, vectorizing, etc.
      - FridgeSeal 3y ago
        For those playing at home wondering why you can’t just do this on all Python loops: The numba jit is for numerical code. You won’t magically get better performance iterating over a bunch of things you get back from your db for example.
FridgeSeal 3y ago
To be fair to Node, their package management is saner and more consistent than Python’s and their build/management tooling is far, far more mature.
IshKebab 3y ago
> But hey, at least it isn’t Node. Why? Node's packaging story is about a million times better than Python's. You literally just listed some of the issues with Python, none of which apply to Node. > use venvs liberally Not necessary with Node (you could argue it always uses venvs but that's transparent from a user point of view). > to let as few things as possible interact with system Python. Not an issue. > If there’s a CLI tool written in Python, ideally install it with pipx. Node has one way to install things and it isn't the way that nobody has heard of (I honestly had not heard of pipx until now and I'm forced to write Python some of the time for my job). > Use Poetry for every project. You don't have to magically know which project management tool out of dozens to use with Node. Just use the overwhelmingly popular one (NPM). You missed a few more by the way. How about knowing to use `--config-settings editable_mode=compat` in order to use editable installs with static tooling? Oh and remember that that is deprecated with no alternative! And it silently doesn't work in `requirements.txt`! Yeay. Then there's `pip`'s fun "wheel error! this probably isn't a wheel error" message. Very user friendly. Oh and don't forget the whole `pip3`/`python3 -m pip` debacle. Did I say `python3`? I meant `python3` except on Windows. Node is a million miles ahead of Python in terms of packaging usability.
- sgarland 3y ago
  > Why? Node's packaging story is about a million times better than Python's. You literally just listed some of the issues with Python, none of which apply to Node. No, instead you get to deal with absurdly slow dependency solving / installation (I gather Bun is fast, so good on that guy), the existence of leftpad, and a `node_modules` directory that is stupidly large. > Not an issue. [to let as few things as possible interact with system Python.] Obviously, because systems aren't issuing tooling in Node. > Node has one way to install things and it isn't the way that nobody has heard of (I honestly had not heard of pipx until now and I'm forced to write Python some of the time for my job). This is literally untrue – I know for a fact that both npm and yarn are extremely popular, and as mentioned, the newcomer bun exists. > How about knowing to use `--config-settings editable_mode=compat` in order to use editable installs with static tooling? Or, you know, switch to modern solutions for this: `poetry add --path /foo/package` or `pipenv install -e /foo/package`. > Oh and don't forget the whole `pip3`/`python3 -m pip` debacle. Did I say `python3`? I meant `python3` except on Windows. Don't use Windows, obvs. Seriously though, I give absolutely zero fucks about development experience on Windows with Python.
yodelshady 3y ago
I have literally never had a project for which Poetry, given an exact list of dependencies and flawless running on the "bad system", installed correctly on first try. Most recently it defaults to asking for a keyring, even when just downloading? I believe that's marked WONTFIX. Even with that hacked around, and I thought the point was to reduce the number of hacks required, tensorflow seems to almost universally fail in absolutely any configuration I've tried. Don't even begin to try integrating your own packages, c++ template errors are tame by comparison. I've also never seen a poetry error fit in less than 500 lines of nested exceptions. I want to like this tool but... boy, Docker doesn't give me this crap.
- sgarland 3y ago
  Granted I haven’t tried Tensorflow with it, but I’ve literally never had any issues with it that weren’t caused by system Python changing (like Brew updating the Python minor version). And that’s fixed by installing it with pipx, which is one of their recommended ways to do so.
dagw 3y ago
Use Poetry for every project Poetry is tightly optimised for what they consider the 'correct' way of developing 'normal' python apps. Wander off that path and poetry becomes worse and worse.
- sgarland 3y ago
  Maybe they have a point. It also fits with the Zen of Python (which tbf rarely seems to happen) of, “There should be one-- and preferably only one --obvious way to do it.” Node / JS in general seems to take the path of, “this existing solution doesn’t precisely fit my needs, so instead of rethinking why that might be, I’m going to reinvent the wheel.”
  - dagw 3y ago
    Maybe they have a point. Given that you where arguing that using python on the most popular OS in the world should fall outside of a 'normal' use case python should focus on, perhaps you have a too narrow view of what "correct and normal" python usage should be.