10 ms·

Writing and linting Python at scale

119 points by el_duderino 3y ago

ayhanfuat 3y ago
(It's a podcast, not an article)
- gostsamo 3y ago
  The description links to the tech article. https://engineering.fb.com/2023/08/07/developer-tools/fixit-2-linter-meta/ https://engineering.fb.com/2023/08/07/developer-tools/fixit-...
  - jpc0 3y ago
    https://engineering.fb.com/2023/08/07/developer-tools/fixit-2-linter-meta/ https://engineering.fb.com/2023/08/07/developer-tools/fixit-... Just a test, for some reason your link isn't clickable for me.
    - chefandy 3y ago
      Looks like they unintentionally (or intentionally for some reason I can't parse ATM?) triggered code formatting. If you have a line preceeded by two or more blank lines that is indented with two or more spaces then it'll <pre> formatting. Like This
      - gostsamo 3y ago
        unintentionally, sorry for that one.
  - dang 3y ago
    Changed to that from https://engineering.fb.com/2023/11/21/production-engineering/writing-linting-python-at-scale-meta/ https://engineering.fb.com/2023/11/21/production-engineering.... Thanks! (I fixed the link in your comment btw. More at https://news.ycombinator.com/formatdoc https://news.ycombinator.com/formatdoc)
infocollector 3y ago
They are talking about this project : https://pypi.org/project/fixit/2.0.0a1/#description https://pypi.org/project/fixit/2.0.0a1/#description https://libcst.readthedocs.io/en/latest/ https://libcst.readthedocs.io/en/latest/ powers Fixit 2. I would love to hear if there are any production users here for this new linter, and how is that working out for them?
lagt_t 3y ago
But but but HN told me that python is only good for small scale and prototypes, the engineers at meta are wrong
- DarkNova6 3y ago
  The engineers at meta worte a language + runtime on top of PHP to make it work "at scale". If Facebook wants to make something work they have enough resources to throw at the problem to solve it. Regardless of whether it makes sense or not.
  - jerf 3y ago
    I mean, as far as I'm concerned, everything Facebook has done has done nothing but reconfirm how unwise it is to build large-scale infrastructure on dynamic scripting languages. They have the resources to move heaven and earth to do what amounts to turning their dynamic scripting language back into static languages in everything but name, and they have the hole they've dug themselves into that justifies it. I have neither. I don't think that Facebook necessarily made a mistake at the time, though. Static languages have come a long way. I can prototype in them very nearly as quickly as I can in dynamic languages now, with the crossover point for where the static language is simply straight-up an advantage being roughly one to two weeks. Circa 2004 I would not even remotely make that claim. By the time static languages reached that state they were already in deep. But I would consider it a mistake for most cases to start right now, in 2023, with dynamic scripting languages as a base layer.
    - crabbone 3y ago
      > dynamic scripting languages. Why keep repeating this nonsense? "Dynamic" or "scripting" aren't features of languages. When anyone says something like this, it's like talking about square chicken... (i.e. a category error). Obviously, you had some idea in your mind, and you wanted to communicate it somehow, but your readers will not know what it was unless you make an effort to analyze what you want to say and make sure it doesn't have internal contradictions and that readers can within reason understand what you are trying to say. You seem to be complaining about something. My guess is that your problem isn't even with the language. Your problem is that some tools for working with your kind of programs are missing or aren't effective.
wdroz 3y ago
I'm happy with Ruff[0], it's very fast. [0] -- https://github.com/astral-sh/ruff https://github.com/astral-sh/ruff
- ltbarcly3 3y ago
  Unfortunately ruff is very inconsistent and has lots of differences from the flake8 plugins it tries to emulate. Lots of rules are confused by irrelevant context so that it can miss lots of things it should find when the equivalent flake8 plugin still find them. It's automatic fixing of issues will happily introduce other issues that it doesn't find until the next run. I've tried pretty hard to use it and gave up, it's just not remotely ready.
  - charliermarsh 3y ago
    If you notice differences vs. the originating plugins, would definitely appreciate it if you could file an issue! We tend to be very responsive especially when it comes to matters of correctness. Candidly, for Flake8 plugins, my experience is that Ruff tends to be more consistent, more robust in its inference and, at this point, more extensively tested than the original implementations -- both via our own testing and via the significant number of projects and companies that now use Ruff in production. (As compared to Pylint, though, we catch fewer issues, since Pylint does some type inference across files, which Ruff doesn't support yet.) Ruff is also designed such that we will iteratively lint-and-fix until there are no more fixable issues, so if you've seen the linter introduce _new_ fixable diagnostics, that would be a bug too. Regardless, thanks for giving it a try :)
  - jamwil 3y ago
    This isn’t really accurate in my experience. I’ve been using it on several projects large and small for some time now and can only recall one time autofix introduced an issue of any sort and that was many versions ago. It will also handle cascading fixes just fine.
  - drcongo 3y ago
    We've gone all in on Ruff for several months now, across many projects, and not noticed any of these issues.
_whiteCaps_ 3y ago
There's a lot of neat Python research coming from Facebook. libcst (which Fixit uses) is super cool - I use it in https://gitlab.com/harford/logzy https://gitlab.com/harford/logzy
- kstrauser 3y ago
  Why do you use that instead of flake8, which also finds that? (Not meant to be a pointed question; genuinely curious.)
  - _whiteCaps_ 3y ago
    AFAIK flake8 only lints and doesn't apply the fixes. fixit probably does a better job than my tool but it was fun to write. :-)
driggs 3y ago
It looks like the interesting feature of their tool Fixit 2 is that its lint rules know how to auto-apply themselves. I'm fine with an auto code formatter, an auto import organizer, but not sure how much I trust a linter to auto-apply "fixes".
- n00j 3y ago
  In the C#/.NET world this is the standard and it works very well. It definitely increases productivity to not have to double check each and every instance of violating a lint error, as you can just have the fix applied to an entire project without having to worry to much.
  - crabbone 3y ago
    > increases productivity to not have to double check each and every instance of violating a lint error, At what cosmic speed should you be pumping out code for this to be a concern? Also, in C#/.NET, where programmers predominantly use MSVS, which is an atrocious editor with MSBuild, which is an atrocious build system, both hampering productivity... Also, plenty of linter errors are actual errors that need non-trivial fixing. So, I struggle to understand how that may be automatically fixed (at least in the context of Python). A trivial example: misspelled variable name -- how would the linter know if that's a typo, or that the programmer intended to declare this variable, but forgot to do that?
    - insanitybit 3y ago
      > Also, plenty of linter errors are actual errors that need non-trivial fixing. These sound like they're not lints at all.
      - crabbone 3y ago
        No, not really. It is perfectly possible to write unintentionally valid code, which would be caught by linter due to the author not following some convention. Trivial and popular example of such code is assignment instead of comparison in the context of condition. Some languages allow for this to happen, but it's known as a possible mistake and so the conventions in such languages would discourage the use of assignment in the context of condition, even though it technically produces valid code.
Andrew018 3y ago
[dead]
dang 3y ago
Related and a bit recent: Fixit 2: Meta’s auto-fixing linter for Python - https://news.ycombinator.com/item?id=37036262 https://news.ycombinator.com/item?id=37036262 - Aug 2023 (11 comments)
drchaim 3y ago
I dream with a unique tool that makes all formatting and linting and sorting and whatever. I don't care too much of the final style. I just prefer not to think too much about it. Ruff is close, let's see.
- appplication 3y ago
  The fact that ruff is expanding with VC funding has me cautiously optimistic. They will hopefully be funded and incentivized to capture the existing fragmented ecosystem of Python dev support tooling. I’m sure their monetization will be more of the same cloud hosted perks etc we see from similar companies. Which is just fine with me, as it keeps their core offering OSS orthogonal to their value prop, which is usually convenience related rather than feature related. I normally hate one-tool-to-rule-them-all ambitions, but when it comes to linting, type hints, auto formatting, import ordering, etc (basically anything you’d want to put in a pre-commit hook), all of that could have a much more coherent story if brought under one hood. Subverting pre-commit itself would also be great. The story around the virtual envs these hooks run in is incohesive with development best practices (define your dependencies in one and only one place), and author has made it aggressively clear on multiple occasions they are uninterested in fixing that.

119 points by el_duderino 3y ago

ayhanfuat 3y ago

(It's a podcast, not an article)

gostsamo 3y ago
The description links to the tech article. https://engineering.fb.com/2023/08/07/developer-tools/fixit-2-linter-meta/ https://engineering.fb.com/2023/08/07/developer-tools/fixit-...
- jpc0 3y ago
  https://engineering.fb.com/2023/08/07/developer-tools/fixit-2-linter-meta/ https://engineering.fb.com/2023/08/07/developer-tools/fixit-... Just a test, for some reason your link isn't clickable for me.
  - chefandy 3y ago
    Looks like they unintentionally (or intentionally for some reason I can't parse ATM?) triggered code formatting. If you have a line preceeded by two or more blank lines that is indented with two or more spaces then it'll <pre> formatting. Like This
    - gostsamo 3y ago
      unintentionally, sorry for that one.
- dang 3y ago
  Changed to that from https://engineering.fb.com/2023/11/21/production-engineering/writing-linting-python-at-scale-meta/ https://engineering.fb.com/2023/11/21/production-engineering.... Thanks! (I fixed the link in your comment btw. More at https://news.ycombinator.com/formatdoc https://news.ycombinator.com/formatdoc)

infocollector 3y ago

They are talking about this project : https://pypi.org/project/fixit/2.0.0a1/#description https://pypi.org/project/fixit/2.0.0a1/#description https://libcst.readthedocs.io/en/latest/ https://libcst.readthedocs.io/en/latest/ powers Fixit 2. I would love to hear if there are any production users here for this new linter, and how is that working out for them?

lagt_t 3y ago

But but but HN told me that python is only good for small scale and prototypes, the engineers at meta are wrong

DarkNova6 3y ago
The engineers at meta worte a language + runtime on top of PHP to make it work "at scale". If Facebook wants to make something work they have enough resources to throw at the problem to solve it. Regardless of whether it makes sense or not.
- jerf 3y ago
  I mean, as far as I'm concerned, everything Facebook has done has done nothing but reconfirm how unwise it is to build large-scale infrastructure on dynamic scripting languages. They have the resources to move heaven and earth to do what amounts to turning their dynamic scripting language back into static languages in everything but name, and they have the hole they've dug themselves into that justifies it. I have neither. I don't think that Facebook necessarily made a mistake at the time, though. Static languages have come a long way. I can prototype in them very nearly as quickly as I can in dynamic languages now, with the crossover point for where the static language is simply straight-up an advantage being roughly one to two weeks. Circa 2004 I would not even remotely make that claim. By the time static languages reached that state they were already in deep. But I would consider it a mistake for most cases to start right now, in 2023, with dynamic scripting languages as a base layer.
  - crabbone 3y ago
    > dynamic scripting languages. Why keep repeating this nonsense? "Dynamic" or "scripting" aren't features of languages. When anyone says something like this, it's like talking about square chicken... (i.e. a category error). Obviously, you had some idea in your mind, and you wanted to communicate it somehow, but your readers will not know what it was unless you make an effort to analyze what you want to say and make sure it doesn't have internal contradictions and that readers can within reason understand what you are trying to say. You seem to be complaining about something. My guess is that your problem isn't even with the language. Your problem is that some tools for working with your kind of programs are missing or aren't effective.

wdroz 3y ago

I'm happy with Ruff[0], it's very fast. [0] -- https://github.com/astral-sh/ruff https://github.com/astral-sh/ruff

ltbarcly3 3y ago
Unfortunately ruff is very inconsistent and has lots of differences from the flake8 plugins it tries to emulate. Lots of rules are confused by irrelevant context so that it can miss lots of things it should find when the equivalent flake8 plugin still find them. It's automatic fixing of issues will happily introduce other issues that it doesn't find until the next run. I've tried pretty hard to use it and gave up, it's just not remotely ready.
- charliermarsh 3y ago
  If you notice differences vs. the originating plugins, would definitely appreciate it if you could file an issue! We tend to be very responsive especially when it comes to matters of correctness. Candidly, for Flake8 plugins, my experience is that Ruff tends to be more consistent, more robust in its inference and, at this point, more extensively tested than the original implementations -- both via our own testing and via the significant number of projects and companies that now use Ruff in production. (As compared to Pylint, though, we catch fewer issues, since Pylint does some type inference across files, which Ruff doesn't support yet.) Ruff is also designed such that we will iteratively lint-and-fix until there are no more fixable issues, so if you've seen the linter introduce _new_ fixable diagnostics, that would be a bug too. Regardless, thanks for giving it a try :)
- jamwil 3y ago
  This isn’t really accurate in my experience. I’ve been using it on several projects large and small for some time now and can only recall one time autofix introduced an issue of any sort and that was many versions ago. It will also handle cascading fixes just fine.
- drcongo 3y ago
  We've gone all in on Ruff for several months now, across many projects, and not noticed any of these issues.

_whiteCaps_ 3y ago

There's a lot of neat Python research coming from Facebook. libcst (which Fixit uses) is super cool - I use it in https://gitlab.com/harford/logzy https://gitlab.com/harford/logzy

kstrauser 3y ago
Why do you use that instead of flake8, which also finds that? (Not meant to be a pointed question; genuinely curious.)
- _whiteCaps_ 3y ago
  AFAIK flake8 only lints and doesn't apply the fixes. fixit probably does a better job than my tool but it was fun to write. :-)

driggs 3y ago

It looks like the interesting feature of their tool Fixit 2 is that its lint rules know how to auto-apply themselves. I'm fine with an auto code formatter, an auto import organizer, but not sure how much I trust a linter to auto-apply "fixes".

n00j 3y ago
In the C#/.NET world this is the standard and it works very well. It definitely increases productivity to not have to double check each and every instance of violating a lint error, as you can just have the fix applied to an entire project without having to worry to much.
- crabbone 3y ago
  > increases productivity to not have to double check each and every instance of violating a lint error, At what cosmic speed should you be pumping out code for this to be a concern? Also, in C#/.NET, where programmers predominantly use MSVS, which is an atrocious editor with MSBuild, which is an atrocious build system, both hampering productivity... Also, plenty of linter errors are actual errors that need non-trivial fixing. So, I struggle to understand how that may be automatically fixed (at least in the context of Python). A trivial example: misspelled variable name -- how would the linter know if that's a typo, or that the programmer intended to declare this variable, but forgot to do that?
  - insanitybit 3y ago
    > Also, plenty of linter errors are actual errors that need non-trivial fixing. These sound like they're not lints at all.
    - crabbone 3y ago
      No, not really. It is perfectly possible to write unintentionally valid code, which would be caught by linter due to the author not following some convention. Trivial and popular example of such code is assignment instead of comparison in the context of condition. Some languages allow for this to happen, but it's known as a possible mistake and so the conventions in such languages would discourage the use of assignment in the context of condition, even though it technically produces valid code.

Andrew018 3y ago

[dead]

dang 3y ago

Related and a bit recent: Fixit 2: Meta’s auto-fixing linter for Python - https://news.ycombinator.com/item?id=37036262 https://news.ycombinator.com/item?id=37036262 - Aug 2023 (11 comments)

drchaim 3y ago

I dream with a unique tool that makes all formatting and linting and sorting and whatever. I don't care too much of the final style. I just prefer not to think too much about it. Ruff is close, let's see.

appplication 3y ago
The fact that ruff is expanding with VC funding has me cautiously optimistic. They will hopefully be funded and incentivized to capture the existing fragmented ecosystem of Python dev support tooling. I’m sure their monetization will be more of the same cloud hosted perks etc we see from similar companies. Which is just fine with me, as it keeps their core offering OSS orthogonal to their value prop, which is usually convenience related rather than feature related. I normally hate one-tool-to-rule-them-all ambitions, but when it comes to linting, type hints, auto formatting, import ordering, etc (basically anything you’d want to put in a pre-commit hook), all of that could have a much more coherent story if brought under one hood. Subverting pre-commit itself would also be great. The story around the virtual envs these hooks run in is incohesive with development best practices (define your dependencies in one and only one place), and author has made it aggressively clear on multiple occasions they are uninterested in fixing that.