4 ms·

It’s a big deal because normally linters have been written in pure python and they are prohibitively slow for large code bases. This tool provides an important

It’s a big deal because normally linters have been written in pure python and they are prohibitively slow for large code bases. This tool provides an important speedup that will save my team hour’s of waiting for CI.

fbdab103 4y ago
I am curious how big your codebase is that it saves hours. I never found the existing Python based tooling to be slow enough that I felt the need to consider alternatives. Though, I will gladly take the speedup.
- lost_tourist 4y ago
  Any program that took longer than several minutes to "lint" for me seemed to blow up anyway, usually by running out of memory or hitting some infinite recursion.
- scbrg 4y ago
  https://xkcd.com/1205/ https://xkcd.com/1205/ is relevant here. Our codebase is somewhere between 200k and 300k LOC. Running the flake8 suite takes minutes. We haven't integrated ruff, but during a test run it ran in less than a second, IIRC. So, with, say 20-30 commits per day, that's about an hour, every day. Sure, it's not like we're usually sitting there twiddling our thumbs during builds, but sometimes we actually do have to wait for a build to get ready. Just the waiting time during those rare occasions add up. [edit]: flake8, remarkably, dropped roughly an order of magnitude in performance between version 2 and 3. So that didn't help.
  - pydry 4y ago
    We fixed that in a previous project by running the linter only on changed files.
  - brenns10 4y ago
    > [edit]: flake8, remarkably, dropped roughly an order of magnitude in performance between version 2 and 3. So that didn't help. Sorry for the drive by, but did you try bisecting the flake8 change that slowed it down? Unless there were thousands of commits between releases, it would only take <10 runs to bisect, an automated bisect should be under an hour with a test case that takes a few minutes. Could be an interesting thing to try?
    - scbrg 4y ago
      Doubt it would be helpful. It appears that the model changed dramatically between those two versions. For instance, the whole API (which we relied on) was dropped[0] and some legacy API was glued in place. In either case, the information wouldn't be all that useful. We run our stuff off of Debian packages (yes, we're old school), and we're not particularly interested in maintaining our own fork. [0] https://flake8.pycqa.org/en/latest/user/python-api.html#legacy-api https://flake8.pycqa.org/en/latest/user/python-api.html#lega...
brigandish 4y ago
I get that bit, what I don't get (or don't see a good reason for) is why Ruff isn't the link and this is. Link to the actual project if that's the subject. If someone wants to compare performance, write a blog post or something instead of wasting everyone's time pointing to references in a different project.

fbdab103 4y ago

I am curious how big your codebase is that it saves hours. I never found the existing Python based tooling to be slow enough that I felt the need to consider alternatives. Though, I will gladly take the speedup.

lost_tourist 4y ago
Any program that took longer than several minutes to "lint" for me seemed to blow up anyway, usually by running out of memory or hitting some infinite recursion.
scbrg 4y ago
https://xkcd.com/1205/ https://xkcd.com/1205/ is relevant here. Our codebase is somewhere between 200k and 300k LOC. Running the flake8 suite takes minutes. We haven't integrated ruff, but during a test run it ran in less than a second, IIRC. So, with, say 20-30 commits per day, that's about an hour, every day. Sure, it's not like we're usually sitting there twiddling our thumbs during builds, but sometimes we actually do have to wait for a build to get ready. Just the waiting time during those rare occasions add up. [edit]: flake8, remarkably, dropped roughly an order of magnitude in performance between version 2 and 3. So that didn't help.
- pydry 4y ago
  We fixed that in a previous project by running the linter only on changed files.
- brenns10 4y ago
  > [edit]: flake8, remarkably, dropped roughly an order of magnitude in performance between version 2 and 3. So that didn't help. Sorry for the drive by, but did you try bisecting the flake8 change that slowed it down? Unless there were thousands of commits between releases, it would only take <10 runs to bisect, an automated bisect should be under an hour with a test case that takes a few minutes. Could be an interesting thing to try?
  - scbrg 4y ago
    Doubt it would be helpful. It appears that the model changed dramatically between those two versions. For instance, the whole API (which we relied on) was dropped[0] and some legacy API was glued in place. In either case, the information wouldn't be all that useful. We run our stuff off of Debian packages (yes, we're old school), and we're not particularly interested in maintaining our own fork. [0] https://flake8.pycqa.org/en/latest/user/python-api.html#legacy-api https://flake8.pycqa.org/en/latest/user/python-api.html#lega...

brigandish 4y ago

I get that bit, what I don't get (or don't see a good reason for) is why Ruff isn't the link and this is. Link to the actual project if that's the subject. If someone wants to compare performance, write a blog post or something instead of wasting everyone's time pointing to references in a different project.