7 ms·

I love Ruff and I'm glad that Charlie and the rest of the team are able to work on such tools full-time. I'm also happy to see that the author of Maturin (Rust+

I love Ruff and I'm glad that Charlie and the rest of the team are able to work on such tools full-time. I'm also happy to see that the author of Maturin (Rust+Python interop) is involved, as Maturin is a fantastic project with great ease-of-use. For those who aren't familiar, Ruff is a very fast Python linter that supersedes a variety of tools like isort, flake8, and perhaps eventually Black [1]. My understanding is that since Ruff is an all-in-one tool, it can parse the underlying Python files just once -- and since this is the most expensive phase of these kinds of tools, adding any extra functionality on top (such as linting, formatting, ...) is very cheap. And that's to say nothing of the fact that Ruff is built on Rust, with the usual performance wins that Rust projects tend to have. I'm a little apprehensive of how development will look given a VC raise with the expectation of VC returns. But for now, I'm delighted and can't wait to see what the team comes up with. [1]: https://github.com/charliermarsh/ruff/issues/1904 https://github.com/charliermarsh/ruff/issues/1904

swyx 3y ago
if it's any consolation, Accel is in my mind one of the "good ones". they have a record of backing "good" devtool companies that manage open source and commercial concerns well - Vercel, Sentry, etc. (i'm sure theres more, those are just the two that i'm closest to)
- echelon 3y ago
  What are the good VC companies? Not just for dev tools, but in general. Where do you find this information?
  - swyx 3y ago
    just being "in the scene" for a few years :) tbh i care more about individual investors than the fund brand, but its a fallback whenever meeting someone new edit: https://overcast.fm/+t_0aYbn-o/12:00 https://overcast.fm/+t_0aYbn-o/12:00 listen to steve krouse talk about how accel encouraged him for val.town even tho he didnt initially believe in it himself. thats not normal investor behavior. look out for people like that, by definition they are rare
Night_Thastus 3y ago
I thought Black was an auto-formatter, similar to clang-format, but for Python. Ruff seems to be only doing linting, not formatting. Am I missing something?
- senko 3y ago
  Ruff will be getting more autoformat capabilities: see the link in parent poster and also read the first comment there, from one of Black maintainers.
  - Night_Thastus 3y ago
    Pretty bold to try to replace Flake8 and Black with one tool. I hope they succeed. Would be great to have a tool that does both and is substantially faster.
    - senko 3y ago
      Indeed. I do hope they'll expose Ruff as a Python module / API in the future. I'm currently using Black to format Python code that's in-memory (never gets written out to disk). With Black (as an imported Py module), it's just a matter of passing in a string and getting one back. With Ruff as it is now, I'd have to write that out to disk, spawn Ruff process, then read the formatted file back in. Do that for many files and the speed advantages disappear, and actually it's slower.
      - BiteCode_dev 3y ago
        You can mount the file system in memory for that, but I see your point.
      - deleted 3y ago
        [deleted]
      - zamnos 3y ago
        Why doesn't ruff reading/writing from stdin/out, combined with passing the data in with python subprocess.run and os.pipe work here? cat bad.py | ruff check --fix --stdin-filename stdin.py - > good.py Substituting bad.py and good.py for in-memory os.pipe objects for your data. This still involves spawning a subprocess, which can add up, but you're not having to read/write to disk at least.
pid-1 3y ago
Even if no VC was involved, how do you make money developing an open source linter?
- klibertp 3y ago
  I think we're going to see a shift in developers expectations towards the cost of their tools, as a side-effect of Copilot & co. If you already pay $100/mo in subscriptions for IDE, its plugins, and services it integrates with, then forking additional $5 for a linter doesn't look as absurd as it used to...
- darkstar999 3y ago
  They are probably looking for an ecosystem around the tool, like how github built on git.
- qbasic_forever 3y ago
  How did Hashi corp make money (and an eventual billion dollar plus IPO!) developing open source tools like vagrant, consul and terraform... Clearly the creator of ruff is going to expand from an open source tool into a company focusing on Python tooling and developer experience, services, etc. There is a market for stuff like that just based on pycharm and jet brains' success in the space alone.
  - nwellnhof 3y ago
    HashiCorp made a $270 million loss last year, more than 50% of their revenue.
  - spullara 3y ago
    A lot of the money comes from Vault which is open source but only used by larger organizations that want support.
    - andygeorge 3y ago
      Vault namespacing also requires an enterprise license, so yeah if you're doing anything useful with Vault at scale, you're paying for it.
cauthon 3y ago
just starting to dabble in Rust, and I've seen PyO3 mentioned a number of times for writing rust bindings. What's the difference between Maturin and PyO3?
- shadaj 3y ago
  PyO3 is the library that enables Python <-> Rust bindings, Maturin is a build tool for packaging PyO3 Rust libraries (which export Python APIs) as Python packages!
  - cauthon 3y ago
    thanks!
lozenge 3y ago
The main reason pylint is slow is it tries to infer the types of the variables, and it predates type annotations so it can go through multiple functions/control paths across multiple files to figure these out. This approach is obviated by type annotations and type checkers. Ruff is built for speed from the start and doesn't look at type annotations because you should run a type checker alongside.
- zx8080 3y ago
  Just to be sure. Is Ruff faster by lacking type checking? Also, does Ruff work well on codebase without type hints?
  - npage97 3y ago
    Ruff is definitely faster by lacking type-checking. That requires way more analysis. Ruff will work fine without type hints. I believe it rightfully leaves it to mypy for those who want those features. Mypy transpiles itself to c using mypyc and that can still take a while to complete when caches get invalidated.
  - rattray 3y ago
    Ruff is faster than many tools which don't do type checking. Ruff works well regardless of whether you use type hints.
    - fsckboy 3y ago
      that's dodging the question a little bit? let's say Ruff is faster because it's so good, how much faster than type checking alternatives would this great tool be if it did type checking?
      - viraptor 3y ago
        Without building the tool to compare, it's really hard to tell. In my experience of rewriting some Python and Ruby code into compiled versions, you can expect 10-20x improvements. But it's both really dependant on the code itself and on how you're going to reimplement it - most of the time it doesn't make sense to do perfect 1:1 rewrites.
toyg 3y ago
I actually cursed Maturin a few years ago, as it refused to compile on OpenBSD and hence broke some lib I planned to use. Python is becoming a bit too reliant on Rust. Rust is good but, in term of portability, is just not as mature as C. If your lib relies on Rust, please please please test it on something beyond Linux and Mac.
- deleted 3y ago
  [deleted]
- criddell 3y ago
  > please test it on something beyond Linux and Mac That’s asking a lot. If your platform is poorly supported by Rust, then maybe that’s where your efforts should be directed. If you fix that, lots of interesting stuff beyond one library is unlocked.
  - senko 3y ago
    Theo de Raadt engaging the Rust community would be a sight to behold :)
    - steveklabnik 3y ago
      I am not aware of him saying anything about Rust beyond this: https://marc.info/?l=openbsd-misc&m=151233345723889&w=2 https://marc.info/?l=openbsd-misc&m=151233345723889&w=2 Which doesn't mean he hasn't, of course!
      - robocat 3y ago
        That email (from Dec 2017) ends with “Such ecosystems come with incredible costs. For instance, rust cannot even compile itself on i386 at present time because it exhausts the address space.”. I presume Theo is actually complaining about using more than 3 GB (?) of memory, but still it really shows the different cost-versus-benefit decisions that we all make.
Cort3z 3y ago
Ruff is great, but it lacks a lot of the rules[1]. As far as I understand it reads every file in parallel, if that is still the case there are categories of issues it cannot detect. In my case ruff cannot fully replace my existing tools, but makes for a great companion. 1: https://github.com/charliermarsh/ruff/issues/970 https://github.com/charliermarsh/ruff/issues/970

swyx 3y ago

if it's any consolation, Accel is in my mind one of the "good ones". they have a record of backing "good" devtool companies that manage open source and commercial concerns well - Vercel, Sentry, etc. (i'm sure theres more, those are just the two that i'm closest to)

echelon 3y ago
What are the good VC companies? Not just for dev tools, but in general. Where do you find this information?
- swyx 3y ago
  just being "in the scene" for a few years :) tbh i care more about individual investors than the fund brand, but its a fallback whenever meeting someone new edit: https://overcast.fm/+t_0aYbn-o/12:00 https://overcast.fm/+t_0aYbn-o/12:00 listen to steve krouse talk about how accel encouraged him for val.town even tho he didnt initially believe in it himself. thats not normal investor behavior. look out for people like that, by definition they are rare

Night_Thastus 3y ago

I thought Black was an auto-formatter, similar to clang-format, but for Python. Ruff seems to be only doing linting, not formatting. Am I missing something?

senko 3y ago
Ruff will be getting more autoformat capabilities: see the link in parent poster and also read the first comment there, from one of Black maintainers.
- Night_Thastus 3y ago
  Pretty bold to try to replace Flake8 and Black with one tool. I hope they succeed. Would be great to have a tool that does both and is substantially faster.
  - senko 3y ago
    Indeed. I do hope they'll expose Ruff as a Python module / API in the future. I'm currently using Black to format Python code that's in-memory (never gets written out to disk). With Black (as an imported Py module), it's just a matter of passing in a string and getting one back. With Ruff as it is now, I'd have to write that out to disk, spawn Ruff process, then read the formatted file back in. Do that for many files and the speed advantages disappear, and actually it's slower.
    - BiteCode_dev 3y ago
      You can mount the file system in memory for that, but I see your point.
    - deleted 3y ago
      [deleted]
    - zamnos 3y ago
      Why doesn't ruff reading/writing from stdin/out, combined with passing the data in with python subprocess.run and os.pipe work here? cat bad.py | ruff check --fix --stdin-filename stdin.py - > good.py Substituting bad.py and good.py for in-memory os.pipe objects for your data. This still involves spawning a subprocess, which can add up, but you're not having to read/write to disk at least.

pid-1 3y ago

Even if no VC was involved, how do you make money developing an open source linter?

klibertp 3y ago
I think we're going to see a shift in developers expectations towards the cost of their tools, as a side-effect of Copilot & co. If you already pay $100/mo in subscriptions for IDE, its plugins, and services it integrates with, then forking additional $5 for a linter doesn't look as absurd as it used to...
darkstar999 3y ago
They are probably looking for an ecosystem around the tool, like how github built on git.
qbasic_forever 3y ago
How did Hashi corp make money (and an eventual billion dollar plus IPO!) developing open source tools like vagrant, consul and terraform... Clearly the creator of ruff is going to expand from an open source tool into a company focusing on Python tooling and developer experience, services, etc. There is a market for stuff like that just based on pycharm and jet brains' success in the space alone.
- nwellnhof 3y ago
  HashiCorp made a $270 million loss last year, more than 50% of their revenue.
- spullara 3y ago
  A lot of the money comes from Vault which is open source but only used by larger organizations that want support.
  - andygeorge 3y ago
    Vault namespacing also requires an enterprise license, so yeah if you're doing anything useful with Vault at scale, you're paying for it.

cauthon 3y ago

just starting to dabble in Rust, and I've seen PyO3 mentioned a number of times for writing rust bindings. What's the difference between Maturin and PyO3?

shadaj 3y ago
PyO3 is the library that enables Python <-> Rust bindings, Maturin is a build tool for packaging PyO3 Rust libraries (which export Python APIs) as Python packages!
- cauthon 3y ago
  thanks!

lozenge 3y ago

The main reason pylint is slow is it tries to infer the types of the variables, and it predates type annotations so it can go through multiple functions/control paths across multiple files to figure these out. This approach is obviated by type annotations and type checkers. Ruff is built for speed from the start and doesn't look at type annotations because you should run a type checker alongside.

zx8080 3y ago
Just to be sure. Is Ruff faster by lacking type checking? Also, does Ruff work well on codebase without type hints?
- npage97 3y ago
  Ruff is definitely faster by lacking type-checking. That requires way more analysis. Ruff will work fine without type hints. I believe it rightfully leaves it to mypy for those who want those features. Mypy transpiles itself to c using mypyc and that can still take a while to complete when caches get invalidated.
- rattray 3y ago
  Ruff is faster than many tools which don't do type checking. Ruff works well regardless of whether you use type hints.
  - fsckboy 3y ago
    that's dodging the question a little bit? let's say Ruff is faster because it's so good, how much faster than type checking alternatives would this great tool be if it did type checking?
    - viraptor 3y ago
      Without building the tool to compare, it's really hard to tell. In my experience of rewriting some Python and Ruby code into compiled versions, you can expect 10-20x improvements. But it's both really dependant on the code itself and on how you're going to reimplement it - most of the time it doesn't make sense to do perfect 1:1 rewrites.

toyg 3y ago

I actually cursed Maturin a few years ago, as it refused to compile on OpenBSD and hence broke some lib I planned to use. Python is becoming a bit too reliant on Rust. Rust is good but, in term of portability, is just not as mature as C. If your lib relies on Rust, please please please test it on something beyond Linux and Mac.

deleted 3y ago
[deleted]
criddell 3y ago
> please test it on something beyond Linux and Mac That’s asking a lot. If your platform is poorly supported by Rust, then maybe that’s where your efforts should be directed. If you fix that, lots of interesting stuff beyond one library is unlocked.
- senko 3y ago
  Theo de Raadt engaging the Rust community would be a sight to behold :)
  - steveklabnik 3y ago
    I am not aware of him saying anything about Rust beyond this: https://marc.info/?l=openbsd-misc&m=151233345723889&w=2 https://marc.info/?l=openbsd-misc&m=151233345723889&w=2 Which doesn't mean he hasn't, of course!
    - robocat 3y ago
      That email (from Dec 2017) ends with “Such ecosystems come with incredible costs. For instance, rust cannot even compile itself on i386 at present time because it exhausts the address space.”. I presume Theo is actually complaining about using more than 3 GB (?) of memory, but still it really shows the different cost-versus-benefit decisions that we all make.

Cort3z 3y ago

Ruff is great, but it lacks a lot of the rules[1]. As far as I understand it reads every file in parallel, if that is still the case there are categories of issues it cannot detect. In my case ruff cannot fully replace my existing tools, but makes for a great companion. 1: https://github.com/charliermarsh/ruff/issues/970 https://github.com/charliermarsh/ruff/issues/970