4 ms·

I'm a little sad that Ruff took off as a re-implementation of a whole bunch of work that was already done, rather than as a project to improve the work that was

I'm a little sad that Ruff took off as a re-implementation of a whole bunch of work that was already done, rather than as a project to improve the work that was already done. It was nice to be able to write little extra linters or flake8 plugins in the language I was linting. Plus decades of effort had gone into making those plugins pretty great, finding the right separation of linting codes so that the right things could be linted/ignored per codebase. I understand why they did it, and they have done it remarkably well, but "rewrite it from scratch" is almost never the best answer and part of me wonders if this result could have been even better achieved another way.

Onavo 2y ago
Blame Guido. Until recently when he was bought out by Microsoft, he had been the primary blocker of higher Python performance. There were a bunch of attempts at adding a JIT to Python but Guido was more interested in splitting hairs over syntax than any real heavy lifting. Python could have been as performant as LuaJIT or V8 but their dictator ruined it. Python need more Mike Palls.
searealist 2y ago
Your argument seems ideological. There is no chance they could have improved Flake8 to be as good as Ruff is.
- burntsushi 2y ago
  Folks said the same thing to me about grep 8.5 years ago when I released ripgrep.
  - oguz-ismail 2y ago
    And you think you've done it?
  - JelteF 2y ago
    Wait... Did you misunderstand this comment? Or are you saying grep caught up with ripgrep now?
    - burntsushi 2y ago
      I'm agreeing with them. Some folks told me I should have improved grep instead of building my own thing.
  - danpalmer 2y ago
    FWIW, I think the reason I'm conflicted is probably a similar reason to why you made a separate new thing. Overcoming the inertia and ways of doing things, or proposing widespread changes, often doesn't go down well with existing maintainers – for very valid reasons. I probably wouldn't want to go into the grep project and suggest rearchitecting it, and I can see why the Ruff developers didn't want to go into projects like Flake8 and Pylint to do the same. But that doesn't stop me from feeling that there were things lost in this process and wishing for a better outcome.
  - deleted 2y ago
    [deleted]
  - tialaramex 2y ago
    A huge advantage of going your own way is that you don't need to address an audience you think are just wrong. When this happens periodically (rather than, every other week) you also get the original GNU advantage which is you can revisit an old tool but knowing what you know now, for example today "everybody" has DVCS, and so ripgrep checks for the .gitignore file.
- danpalmer 2y ago
  Yeah it's somewhat ideological, I think open source software is better for society when built as a community project than built and controlled by a VC funded company. I don't think you could get flake8 to be as fast as Ruff, but I think you could improve it to be within an order of magnitude, and honestly that's plenty good enough. There are a lot of low hanging fruit, particularly around caching. I'd push back on there being no chance of being "as good as" Ruff, because "good" is not just speed. Ruff is _not complete_, if you want all the features and linters, you still need Flake8, or you have to sacrifice some. It's also not publicly extensible, and not written in Python, both of which are fair choices but I think deviate from the ideal scenario here.
FreakLegion 2y ago
Whenever Ruff comes up, I reflexively go and check https://github.com/astral-sh/ruff/issues/970 https://github.com/astral-sh/ruff/issues/970, since Pylint is the one tool whose performance I find to be egregiously bad. Looks like Ruff is still very far from being able to replace it.
- orra 2y ago
  What about ruff + pyright? From a skim it seems like a lot of the missing features would be covered by a type checker? (Pyright isn't written in Rust, but adequate performance is still a design goal)
thrdbndndn 2y ago
I'm also sad but from a different perspective. If something can be rewritten from scratch fairly quickly and ends up being much faster, it makes you wonder what we (the general “we”) might have done “wrong” with the “decades of effort” before. To be clear: there was nothing wrong with the old toolchains (hence the quotes). And of course, Ruff/uv can do what they do because they build on all that earlier knowledge. My point is just that sometimes it's easier to start over than to fix the old, and that fact itself feels kind of sad to me.
- zahlman 2y ago
  > My point is just that sometimes it's easier to start over than to fix the old, and that fact itself feels kind of sad to me. Sad, but also liberating. There are surely more projects out there that would benefit from breaking with conventional wisdom to start over.
- lmm 2y ago
  > there was nothing wrong with the old toolchains (hence the quotes). And of course, Ruff/uv can do what they do because they build on all that earlier knowledge I don't think that's true. I think the old toolchains really were bad, partly because the Python community was uniquely resistant to building on what worked in other ecosystems, and uv in particular largely succeeds because it's the first tool in this space to finally just ignore the nonsense that had somehow become accepted wisdom in Python-land and apply a little bit of outside experience.
  - zahlman 2y ago
    > I think the old toolchains really were bad, partly because the Python community was uniquely resistant to building on what worked in other ecosystems I mostly attribute it to backwards compatibility concerns (which I in turn attribute to the traumatic 3.x migration). PyPI continued to accept egg uploads until August 2023 (https://packaging.python.org/en/latest/discussions/package-formats/#what-about-eggs https://packaging.python.org/en/latest/discussions/package-f...). The easy_install program, along the direct command-line use of `setup.py` (as opposed to having Setuptools invoke it behind the scenes), have been deprecated since October 2021 (https://setuptools.pypa.io/en/latest/history.html#v58-3-0); https://setuptools.pypa.io/en/latest/history.html#v58-3-0); but not only are they still supported, Setuptools can't even remove support for `setup.py test` (as they tried to in 72.0 and immediately reverted - https://github.com/pypa/setuptools/issues/4519 https://github.com/pypa/setuptools/issues/4519) without causing major disruption to the ecosystem. One of the packages affected was Requests, which: a) is one of the most downloaded packages on PyPI; b) is pure Python (aside from dependencies) with nothing particular complicated about its metadata (I see no reason it couldn't be done in pyproject.toml and/or setup.cfg); c) wasn't even reliant on that interface to run tests (they use Tox). (For that matter: Requests has been maintained by the PSF since mid-2019 (https://www.reddit.com/r/Python/comments/cgtp87/seems_like_requests_will_be_moved_to_a_github/ https://www.reddit.com/r/Python/comments/cgtp87/seems_like_r...) and the problem could have easily been avoided once the deprecation was announced, but nobody did anything about it. The project still defines most of its metadata in `setup.py`; `pyproject.toml` exists but is only used for pytest and isort config, while `setup.cfg` is used for flake8 config and some requirements metadata that's completely redundant with `setup.py`.) A lot of it also just has to do with lack of attention and focus. https://peps.python.org/pep-0427/ https://peps.python.org/pep-0427/ (defining the wheel format) was proposed in September 2012, and accepted in February 2013. But Setuptools itself wasn't available as a wheel until November 2013 (https://setuptools.pypa.io/en/latest/history.html#id1646 https://setuptools.pypa.io/en/latest/history.html#id1646), and in 2017 there were still reports of people having outdated versions of Pip and not being able to install Setuptools from that wheel (https://setuptools.pypa.io/en/latest/history.html#v34-0-0 https://setuptools.pypa.io/en/latest/history.html#v34-0-0). Setuptools relied on a separate package to actually make wheels until July of last year (https://setuptools.pypa.io/en/latest/history.html#v70-1-0 https://setuptools.pypa.io/en/latest/history.html#v70-1-0) - an effort which was initially proposed in June 2018 (https://github.com/pypa/setuptools/issues/1386 https://github.com/pypa/setuptools/issues/1386). It also took years to notice that Pip and Setuptools had separate implementations for the code that understands the "tags" in wheel filenames and factor it out into `packaging` (https://github.com/pypa/packaging/pull/156 https://github.com/pypa/packaging/pull/156).
n8henrie 2y ago
I am quite grateful that they rewrote so many tools that I had been using. Upgrades were super painful with them split into a dozen separate packages, I routinely ran into incompatibilities and had to pin to specific versions to deal with transitive dependencies. Given the state of python packaging and tooling, I'd say that consolidating the tooling is a big win in and of itself, and with the tremendous speed wins on top...
eviks 2y ago
> this result could have been even better achieved another way. Don't you have "decades of effort" to resolve this wonder?
hobofan 2y ago
Rewrite from scratch is exactly what the Python ecosystem needs more of. The strong push to always contribute to the existing projects rather than starting new ones in the Python community is what caused the previous subpar state of tooling. I think Astral is also picking a good middle-ground where they still aim to build as much as possible on existing Python standards (and help contributing to new ones), so it's still building on what has been done in the past.
VagabundoP 2y ago
There was probably a lot of engineering that went into those designs and that took time. Having a clean process to rewrite from is what made it so fast. They knew exactly what outcome they wanted and they had access to the internal implementations on how to do it. All that effort was not wasted at all.
fydorm 2y ago
Flake8 and Black would never have matched the performance of a Rust-based tool, even if they had 10 years of dedicated focus.

Onavo 2y ago

Blame Guido. Until recently when he was bought out by Microsoft, he had been the primary blocker of higher Python performance. There were a bunch of attempts at adding a JIT to Python but Guido was more interested in splitting hairs over syntax than any real heavy lifting. Python could have been as performant as LuaJIT or V8 but their dictator ruined it. Python need more Mike Palls.

searealist 2y ago

Your argument seems ideological. There is no chance they could have improved Flake8 to be as good as Ruff is.

burntsushi 2y ago
Folks said the same thing to me about grep 8.5 years ago when I released ripgrep.
- oguz-ismail 2y ago
  And you think you've done it?
- JelteF 2y ago
  Wait... Did you misunderstand this comment? Or are you saying grep caught up with ripgrep now?
  - burntsushi 2y ago
    I'm agreeing with them. Some folks told me I should have improved grep instead of building my own thing.
- danpalmer 2y ago
  FWIW, I think the reason I'm conflicted is probably a similar reason to why you made a separate new thing. Overcoming the inertia and ways of doing things, or proposing widespread changes, often doesn't go down well with existing maintainers – for very valid reasons. I probably wouldn't want to go into the grep project and suggest rearchitecting it, and I can see why the Ruff developers didn't want to go into projects like Flake8 and Pylint to do the same. But that doesn't stop me from feeling that there were things lost in this process and wishing for a better outcome.
- deleted 2y ago
  [deleted]
- tialaramex 2y ago
  A huge advantage of going your own way is that you don't need to address an audience you think are just wrong. When this happens periodically (rather than, every other week) you also get the original GNU advantage which is you can revisit an old tool but knowing what you know now, for example today "everybody" has DVCS, and so ripgrep checks for the .gitignore file.
danpalmer 2y ago
Yeah it's somewhat ideological, I think open source software is better for society when built as a community project than built and controlled by a VC funded company. I don't think you could get flake8 to be as fast as Ruff, but I think you could improve it to be within an order of magnitude, and honestly that's plenty good enough. There are a lot of low hanging fruit, particularly around caching. I'd push back on there being no chance of being "as good as" Ruff, because "good" is not just speed. Ruff is _not complete_, if you want all the features and linters, you still need Flake8, or you have to sacrifice some. It's also not publicly extensible, and not written in Python, both of which are fair choices but I think deviate from the ideal scenario here.

FreakLegion 2y ago

Whenever Ruff comes up, I reflexively go and check https://github.com/astral-sh/ruff/issues/970 https://github.com/astral-sh/ruff/issues/970, since Pylint is the one tool whose performance I find to be egregiously bad. Looks like Ruff is still very far from being able to replace it.

orra 2y ago
What about ruff + pyright? From a skim it seems like a lot of the missing features would be covered by a type checker? (Pyright isn't written in Rust, but adequate performance is still a design goal)

thrdbndndn 2y ago

I'm also sad but from a different perspective. If something can be rewritten from scratch fairly quickly and ends up being much faster, it makes you wonder what we (the general “we”) might have done “wrong” with the “decades of effort” before. To be clear: there was nothing wrong with the old toolchains (hence the quotes). And of course, Ruff/uv can do what they do because they build on all that earlier knowledge. My point is just that sometimes it's easier to start over than to fix the old, and that fact itself feels kind of sad to me.

zahlman 2y ago
> My point is just that sometimes it's easier to start over than to fix the old, and that fact itself feels kind of sad to me. Sad, but also liberating. There are surely more projects out there that would benefit from breaking with conventional wisdom to start over.
lmm 2y ago
> there was nothing wrong with the old toolchains (hence the quotes). And of course, Ruff/uv can do what they do because they build on all that earlier knowledge I don't think that's true. I think the old toolchains really were bad, partly because the Python community was uniquely resistant to building on what worked in other ecosystems, and uv in particular largely succeeds because it's the first tool in this space to finally just ignore the nonsense that had somehow become accepted wisdom in Python-land and apply a little bit of outside experience.
- zahlman 2y ago
  > I think the old toolchains really were bad, partly because the Python community was uniquely resistant to building on what worked in other ecosystems I mostly attribute it to backwards compatibility concerns (which I in turn attribute to the traumatic 3.x migration). PyPI continued to accept egg uploads until August 2023 (https://packaging.python.org/en/latest/discussions/package-formats/#what-about-eggs https://packaging.python.org/en/latest/discussions/package-f...). The easy_install program, along the direct command-line use of `setup.py` (as opposed to having Setuptools invoke it behind the scenes), have been deprecated since October 2021 (https://setuptools.pypa.io/en/latest/history.html#v58-3-0); https://setuptools.pypa.io/en/latest/history.html#v58-3-0); but not only are they still supported, Setuptools can't even remove support for `setup.py test` (as they tried to in 72.0 and immediately reverted - https://github.com/pypa/setuptools/issues/4519 https://github.com/pypa/setuptools/issues/4519) without causing major disruption to the ecosystem. One of the packages affected was Requests, which: a) is one of the most downloaded packages on PyPI; b) is pure Python (aside from dependencies) with nothing particular complicated about its metadata (I see no reason it couldn't be done in pyproject.toml and/or setup.cfg); c) wasn't even reliant on that interface to run tests (they use Tox). (For that matter: Requests has been maintained by the PSF since mid-2019 (https://www.reddit.com/r/Python/comments/cgtp87/seems_like_requests_will_be_moved_to_a_github/ https://www.reddit.com/r/Python/comments/cgtp87/seems_like_r...) and the problem could have easily been avoided once the deprecation was announced, but nobody did anything about it. The project still defines most of its metadata in `setup.py`; `pyproject.toml` exists but is only used for pytest and isort config, while `setup.cfg` is used for flake8 config and some requirements metadata that's completely redundant with `setup.py`.) A lot of it also just has to do with lack of attention and focus. https://peps.python.org/pep-0427/ https://peps.python.org/pep-0427/ (defining the wheel format) was proposed in September 2012, and accepted in February 2013. But Setuptools itself wasn't available as a wheel until November 2013 (https://setuptools.pypa.io/en/latest/history.html#id1646 https://setuptools.pypa.io/en/latest/history.html#id1646), and in 2017 there were still reports of people having outdated versions of Pip and not being able to install Setuptools from that wheel (https://setuptools.pypa.io/en/latest/history.html#v34-0-0 https://setuptools.pypa.io/en/latest/history.html#v34-0-0). Setuptools relied on a separate package to actually make wheels until July of last year (https://setuptools.pypa.io/en/latest/history.html#v70-1-0 https://setuptools.pypa.io/en/latest/history.html#v70-1-0) - an effort which was initially proposed in June 2018 (https://github.com/pypa/setuptools/issues/1386 https://github.com/pypa/setuptools/issues/1386). It also took years to notice that Pip and Setuptools had separate implementations for the code that understands the "tags" in wheel filenames and factor it out into `packaging` (https://github.com/pypa/packaging/pull/156 https://github.com/pypa/packaging/pull/156).

n8henrie 2y ago

I am quite grateful that they rewrote so many tools that I had been using. Upgrades were super painful with them split into a dozen separate packages, I routinely ran into incompatibilities and had to pin to specific versions to deal with transitive dependencies. Given the state of python packaging and tooling, I'd say that consolidating the tooling is a big win in and of itself, and with the tremendous speed wins on top...

eviks 2y ago

> this result could have been even better achieved another way. Don't you have "decades of effort" to resolve this wonder?

hobofan 2y ago

Rewrite from scratch is exactly what the Python ecosystem needs more of. The strong push to always contribute to the existing projects rather than starting new ones in the Python community is what caused the previous subpar state of tooling. I think Astral is also picking a good middle-ground where they still aim to build as much as possible on existing Python standards (and help contributing to new ones), so it's still building on what has been done in the past.

VagabundoP 2y ago

There was probably a lot of engineering that went into those designs and that took time. Having a clean process to rewrite from is what made it so fast. They knew exactly what outcome they wanted and they had access to the internal implementations on how to do it. All that effort was not wasted at all.

fydorm 2y ago

Flake8 and Black would never have matched the performance of a Rust-based tool, even if they had 10 years of dedicated focus.