11 ms·

Rye: Flask author's new project for Python bootstrapping

208 points by BiteCode_dev 3y ago

BiteCode_dev 3y ago
The project is very new and unstable. Don't expect it to solve the whole thing today. But the concept is sound.
deleted 3y ago
[deleted]
KRAKRISMOTT 3y ago
How's support for PyTorch/Jax/Tensorflow-GPU? Poetry struggles when resolving cross platform bundled driver dependencies. So far only conda/mamba is able to handle it performantly.
- qbasic_forever 3y ago
  If you can pip install those libs and they work on your platform then it looks like this should work--it's just a nice opinionated wrapper around pip, virtual env and other Python tooling. If you need conda installs because those libs have specially compiled native depencies (boatloads of fortran code, etc) then you're going to run into trouble and probably should stick with conda.
  - KRAKRISMOTT 3y ago
    Another issue is that PyTorch installation requires the dependency resolver to scrape the dependency host's HTML like a RSS reader because pypi doesn't automatically store metadata. https://github.com/python-poetry/poetry/issues/6409 https://github.com/python-poetry/poetry/issues/6409
    - qbasic_forever 3y ago
      Yeah anything that needs specially compiled or optimized binaries is an enormous pain in the python world. You really want conda for those cases.
alberth 3y ago
Off topic: it’s too bad Armin is doing so much in Rust, instead of Nim. Would be great to have a recognizable Python developer, using Nim.
- coldtea 3y ago
  The point is not to prop up Nim. I'm not sure why it would be desirable even, aside from "but I like Nim". The point is to improve Python's ecosystem. If they do it using a more established language than Nim, with great traction, that's for the better.
- atoav 3y ago
  Aside from a very subjective "I like Nim to succeed" there is literally no rational reason to pick one over the other (except, maybe the other way around, because Rust has wider adoption). So communicating that thought as if it was a generally accepted/objective statement on Armin's project isn't going to give you a lot of love.
gaganyaan 3y ago
The workspace support looks really interesting, it's something that I'm really missing in Poetry right now.
smitty1e 3y ago
My hope is that the ideas on offer inform a PEP that gets into 3.13. Python's packaging/project management story isn't among its selling points at the moment.
- coldtea 3y ago
  That's quite the understatement...
  - smitty1e 3y ago
    Mild >>> wild.
darkteflon 3y ago
This looks very nice. I’ve been quite happy with Hatch, which I adopted about 3 months ago on the strength of the PyPA’s recommendation. I sort of like where they’re coming from philosophically - sticking as close as possible to standards - and find the API pretty nice. I’ve never used Poetry so couldn’t compare. One concern I have is that it seems to largely be the work of a solo dev who also has other commitments. Absolutely nothing wrong with that in principle, of course, but I wonder how sustainable it is. I have just noticed that PyPA owns the repo - not sure if that was always the case or if it’s a recent development, but does bode well.
- thatxliner 3y ago
  It wasn’t always the case. It used to lie at ofek/hatch
- BiteCode_dev 3y ago
  It's more than those tools as it provides python itself.
fbdab103 3y ago
Has a similar look to the Posy/PyBi idea[0]. PyBi is also currently under consideration as PEP711[1]. [0] https://discuss.python.org/t/announce-pybi-and-posy/23021 https://discuss.python.org/t/announce-pybi-and-posy/23021 [1] https://peps.python.org/pep-0711/ https://peps.python.org/pep-0711/
techdragon 3y ago
Neat and it doesn’t fall into the same trap as many others trying to build similar tools in python itself by being built with rust there’s no bootstrapping issues with managing the python environment the tool to manage python environments needs. However it’s very opinionated and ignores several things I like to have control over (and have needed to use in the past) and most importantly I want to be able to compile my own python binary and pyenv makes it pretty painless. So while I’m impressed and think it looks good it’s a 100% no for me just because I have on multiple occasions had to compile ahead for testing, or apply a tiny patch python to build on an unsupported os such as a too old or new macOS… I’m not the target audience but unfortunately I suspect sue to how well known he is, this will lead to adoption and popularity and while I think this tool is good for him and other people I don’t think python needs more opinionated management tool projects getting wildly popular… I don’t want to see a repeat of the pipenv saga
- the_mitsuhiko 3y ago
  Author here. > However it’s very opinionated and ignores several things I like to have control over (and have needed to use in the past) and most importantly I want to be able to compile my own python binary and pyenv makes it pretty painless. For what it's worth I do agree with this, and in fact before I published this to GitHub, rye was capable of doing that. I somewhat intentionally stripped it down a bit to better explain and showcase what I wish that experience looked like. In an ideal world, we would not have to use custom Python installations because a tool like rye could automatically fetch it. However there are always cases where you need (eg: local python builds for development etc.) so I will definitely want to bring this back. The reason I never put this on GitHub is really that I do not want to build the next thing that just splits the ecosystem apart. But as of recent, there have been some developments that make me somewhat optimistic that there is a chance, Python packaging could be fixed once and for all, and this is supposed to be a bit of a teaser of what could be.
  - santiagobasulto 3y ago
    Armin, thanks for rye and all you've done throughout the years. I'm curious, what are these developments you've mentioned that make you optimistic?
hgatrd 3y ago
It is clever to use Rust so the tool won't break with each new Python release.
- sverhagen 3y ago
  Isn't it also declaring complete bankruptcy as far as Python's own ecosystem is concerned, that you'd have to go to Rust to fix this?
  - tsunagatta 3y ago
    Maybe if a tool like this is successful enough it could “bootstrap” the Python ecosystem into being as reliable :P
  - rtpg 3y ago
    not really? It's not like CPython is written in Python either. Almost every language that has some notion of virtual environments writes a good chunk as shell scripts for maximum environment compatibility. Sometimes you want to use the right tool for the job, and sometimes the scripting language that has a lot of environment-based configuration will not be a good fit for an "installer" that needs to handle a lot of environments cleanly.
    - sverhagen 3y ago
      I don't think that's right. Sure, the language runtime environment often uses a lower level language to speak with the underlying hardware or abstraction. But large parts of standard libraries may be written in the language itself. Build systems may have a little bit of shell scripting to get going, but larger such programs would often be written in the language itself. This certainly is the case with NPM, Yarn (JavaScript), Maven (Java), Gradle (multiple JVM languages), Rake (Ruby).
      - rtpg 3y ago
        I think Ruby's equivalent would be something like rbenv (which is just a big shell script!). Java similiarly ends up often being installed through package managers or other bespoke system installers that would at the very least need to bundle a runtime. To your point, One could write something like rye in Python, and then use PyOxidizer to package it up. I imagine a part of this is "Rust is fun", which is totally a legitimate reason to work it. And of course Rust comes with a very good distribution story (Go might have been another good answer for some people).
dpoljak 3y ago
Love the self-consciousness of it all, even having a "Should this exist" with an XKCD that jumped right to my mind as soon as I read the title. Honestly, as someone who has to periodically jump into Python to do some scraping and data wrangling I dearly hope we land on something that is sane and maintainable in the near future. I'm just so tired of (ana)conda, poetry, pyenv, and even pip itself.
andrewshadura 3y ago
I wish it supported setup.cfg
- danpalmer 3y ago
  Out of interest... why? From what I can tell the community really seems to be adopting pyproject.toml as the standard and most tools are supporting it as their primary or only configuration option now. A few are using setup.cfg, but that seems to be for legacy reasons rather than an active decision. I'm not a huge fan of TOML as a configuration format personally, but it's at least more capable than cfg files. I'm really keen to know why you feel cfg support would be useful.
  - andrewshadura 3y ago
    To be honest, I feel like setup.cfg hasn’t been given a chance to prove itself useful. Just as it was made possible to use setup.cfg only (without setup.py), it’s been almost immediately taken over by pyproject.toml. However, for those who actually migrated their projects to setup.cfg, migrating once again doesn’t bring that many benefits (but e.g. loses compat with flake8). Besides, setup.cfg’s syntax is slightly more elegant, or at least I think so :) By the way, why do you think it is more capable? As far as I know, setup.cfg has some feature toml (intentionally) doesn’t support (not that they’re useful in this context anyway).
    - danpalmer 3y ago
      I've always thought of TOML as not far off a superset of the configparser format, although with a different API in code. I realise most tools don't use the extent of configparser, but coming from JSON and YAML, I always found the more advanced configparser features counter-intuitive and awkward to use. I'm not a huge fan of TOML, but it has the necessary primitive values (strings, numbers, booleans), and some reasonable structures for managing collections and nesting.
  - snowe2010 3y ago
    Is there seriously another standard being set up in the Python community? When will it end? That’s not a rhetorical question, I’m seriously asking when the community is going to actually solve the tooling problem without making everything worse with each successive update.
r9295 3y ago
Using a highly opinionated package manager with barely any tests? Supply chain attacks exploiting package managers are actually a thing to be extremely scared of. https://news.ycombinator.com/item?id=34836275 https://news.ycombinator.com/item?id=34836275
epage 3y ago
> No Core Proprietary Stuff: Rye (with the exception of it's own tool section in the pyproject.toml) uses standardized keys. That means it uses regular requirements as you would expect. It also does not use a proprietary lock file and use pip-tools behind the scenes. The use of `pip-tools` to generate a `requirements.txt` as a lockfile is a big negative for me. Last I looked, they do not generate a cross-platform lockfile. In theory, you could still use `requirements.txt` for a cross-platform lockfile; it just might be a mess for doing incremental updates like when adding a new dependency.

208 points by BiteCode_dev 3y ago

BiteCode_dev 3y ago

The project is very new and unstable. Don't expect it to solve the whole thing today. But the concept is sound.

deleted 3y ago

[deleted]

KRAKRISMOTT 3y ago

How's support for PyTorch/Jax/Tensorflow-GPU? Poetry struggles when resolving cross platform bundled driver dependencies. So far only conda/mamba is able to handle it performantly.

qbasic_forever 3y ago
If you can pip install those libs and they work on your platform then it looks like this should work--it's just a nice opinionated wrapper around pip, virtual env and other Python tooling. If you need conda installs because those libs have specially compiled native depencies (boatloads of fortran code, etc) then you're going to run into trouble and probably should stick with conda.
- KRAKRISMOTT 3y ago
  Another issue is that PyTorch installation requires the dependency resolver to scrape the dependency host's HTML like a RSS reader because pypi doesn't automatically store metadata. https://github.com/python-poetry/poetry/issues/6409 https://github.com/python-poetry/poetry/issues/6409
  - qbasic_forever 3y ago
    Yeah anything that needs specially compiled or optimized binaries is an enormous pain in the python world. You really want conda for those cases.

alberth 3y ago

Off topic: it’s too bad Armin is doing so much in Rust, instead of Nim. Would be great to have a recognizable Python developer, using Nim.

coldtea 3y ago
The point is not to prop up Nim. I'm not sure why it would be desirable even, aside from "but I like Nim". The point is to improve Python's ecosystem. If they do it using a more established language than Nim, with great traction, that's for the better.
atoav 3y ago
Aside from a very subjective "I like Nim to succeed" there is literally no rational reason to pick one over the other (except, maybe the other way around, because Rust has wider adoption). So communicating that thought as if it was a generally accepted/objective statement on Armin's project isn't going to give you a lot of love.

gaganyaan 3y ago

The workspace support looks really interesting, it's something that I'm really missing in Poetry right now.

smitty1e 3y ago

My hope is that the ideas on offer inform a PEP that gets into 3.13. Python's packaging/project management story isn't among its selling points at the moment.

coldtea 3y ago
That's quite the understatement...
- smitty1e 3y ago
  Mild >>> wild.

darkteflon 3y ago

This looks very nice. I’ve been quite happy with Hatch, which I adopted about 3 months ago on the strength of the PyPA’s recommendation. I sort of like where they’re coming from philosophically - sticking as close as possible to standards - and find the API pretty nice. I’ve never used Poetry so couldn’t compare. One concern I have is that it seems to largely be the work of a solo dev who also has other commitments. Absolutely nothing wrong with that in principle, of course, but I wonder how sustainable it is. I have just noticed that PyPA owns the repo - not sure if that was always the case or if it’s a recent development, but does bode well.

thatxliner 3y ago
It wasn’t always the case. It used to lie at ofek/hatch
BiteCode_dev 3y ago
It's more than those tools as it provides python itself.

fbdab103 3y ago

Has a similar look to the Posy/PyBi idea[0]. PyBi is also currently under consideration as PEP711[1]. [0] https://discuss.python.org/t/announce-pybi-and-posy/23021 https://discuss.python.org/t/announce-pybi-and-posy/23021 [1] https://peps.python.org/pep-0711/ https://peps.python.org/pep-0711/

techdragon 3y ago

Neat and it doesn’t fall into the same trap as many others trying to build similar tools in python itself by being built with rust there’s no bootstrapping issues with managing the python environment the tool to manage python environments needs. However it’s very opinionated and ignores several things I like to have control over (and have needed to use in the past) and most importantly I want to be able to compile my own python binary and pyenv makes it pretty painless. So while I’m impressed and think it looks good it’s a 100% no for me just because I have on multiple occasions had to compile ahead for testing, or apply a tiny patch python to build on an unsupported os such as a too old or new macOS… I’m not the target audience but unfortunately I suspect sue to how well known he is, this will lead to adoption and popularity and while I think this tool is good for him and other people I don’t think python needs more opinionated management tool projects getting wildly popular… I don’t want to see a repeat of the pipenv saga

the_mitsuhiko 3y ago
Author here. > However it’s very opinionated and ignores several things I like to have control over (and have needed to use in the past) and most importantly I want to be able to compile my own python binary and pyenv makes it pretty painless. For what it's worth I do agree with this, and in fact before I published this to GitHub, rye was capable of doing that. I somewhat intentionally stripped it down a bit to better explain and showcase what I wish that experience looked like. In an ideal world, we would not have to use custom Python installations because a tool like rye could automatically fetch it. However there are always cases where you need (eg: local python builds for development etc.) so I will definitely want to bring this back. The reason I never put this on GitHub is really that I do not want to build the next thing that just splits the ecosystem apart. But as of recent, there have been some developments that make me somewhat optimistic that there is a chance, Python packaging could be fixed once and for all, and this is supposed to be a bit of a teaser of what could be.
- santiagobasulto 3y ago
  Armin, thanks for rye and all you've done throughout the years. I'm curious, what are these developments you've mentioned that make you optimistic?

hgatrd 3y ago

It is clever to use Rust so the tool won't break with each new Python release.

sverhagen 3y ago
Isn't it also declaring complete bankruptcy as far as Python's own ecosystem is concerned, that you'd have to go to Rust to fix this?
- tsunagatta 3y ago
  Maybe if a tool like this is successful enough it could “bootstrap” the Python ecosystem into being as reliable :P
- rtpg 3y ago
  not really? It's not like CPython is written in Python either. Almost every language that has some notion of virtual environments writes a good chunk as shell scripts for maximum environment compatibility. Sometimes you want to use the right tool for the job, and sometimes the scripting language that has a lot of environment-based configuration will not be a good fit for an "installer" that needs to handle a lot of environments cleanly.
  - sverhagen 3y ago
    I don't think that's right. Sure, the language runtime environment often uses a lower level language to speak with the underlying hardware or abstraction. But large parts of standard libraries may be written in the language itself. Build systems may have a little bit of shell scripting to get going, but larger such programs would often be written in the language itself. This certainly is the case with NPM, Yarn (JavaScript), Maven (Java), Gradle (multiple JVM languages), Rake (Ruby).
    - rtpg 3y ago
      I think Ruby's equivalent would be something like rbenv (which is just a big shell script!). Java similiarly ends up often being installed through package managers or other bespoke system installers that would at the very least need to bundle a runtime. To your point, One could write something like rye in Python, and then use PyOxidizer to package it up. I imagine a part of this is "Rust is fun", which is totally a legitimate reason to work it. And of course Rust comes with a very good distribution story (Go might have been another good answer for some people).

dpoljak 3y ago

Love the self-consciousness of it all, even having a "Should this exist" with an XKCD that jumped right to my mind as soon as I read the title. Honestly, as someone who has to periodically jump into Python to do some scraping and data wrangling I dearly hope we land on something that is sane and maintainable in the near future. I'm just so tired of (ana)conda, poetry, pyenv, and even pip itself.

andrewshadura 3y ago

I wish it supported setup.cfg

danpalmer 3y ago
Out of interest... why? From what I can tell the community really seems to be adopting pyproject.toml as the standard and most tools are supporting it as their primary or only configuration option now. A few are using setup.cfg, but that seems to be for legacy reasons rather than an active decision. I'm not a huge fan of TOML as a configuration format personally, but it's at least more capable than cfg files. I'm really keen to know why you feel cfg support would be useful.
- andrewshadura 3y ago
  To be honest, I feel like setup.cfg hasn’t been given a chance to prove itself useful. Just as it was made possible to use setup.cfg only (without setup.py), it’s been almost immediately taken over by pyproject.toml. However, for those who actually migrated their projects to setup.cfg, migrating once again doesn’t bring that many benefits (but e.g. loses compat with flake8). Besides, setup.cfg’s syntax is slightly more elegant, or at least I think so :) By the way, why do you think it is more capable? As far as I know, setup.cfg has some feature toml (intentionally) doesn’t support (not that they’re useful in this context anyway).
  - danpalmer 3y ago
    I've always thought of TOML as not far off a superset of the configparser format, although with a different API in code. I realise most tools don't use the extent of configparser, but coming from JSON and YAML, I always found the more advanced configparser features counter-intuitive and awkward to use. I'm not a huge fan of TOML, but it has the necessary primitive values (strings, numbers, booleans), and some reasonable structures for managing collections and nesting.
- snowe2010 3y ago
  Is there seriously another standard being set up in the Python community? When will it end? That’s not a rhetorical question, I’m seriously asking when the community is going to actually solve the tooling problem without making everything worse with each successive update.

r9295 3y ago

Using a highly opinionated package manager with barely any tests? Supply chain attacks exploiting package managers are actually a thing to be extremely scared of. https://news.ycombinator.com/item?id=34836275 https://news.ycombinator.com/item?id=34836275

epage 3y ago

> No Core Proprietary Stuff: Rye (with the exception of it's own tool section in the pyproject.toml) uses standardized keys. That means it uses regular requirements as you would expect. It also does not use a proprietary lock file and use pip-tools behind the scenes. The use of `pip-tools` to generate a `requirements.txt` as a lockfile is a big negative for me. Last I looked, they do not generate a cross-platform lockfile. In theory, you could still use `requirements.txt` for a cross-platform lockfile; it just might be a mess for doing incremental updates like when adding a new dependency.