3 ms·

Huge fan of Mypy but you lost me at LOC worship. Why do we do this? The more LOC is somehow attributed to more features? That simply isn’t true. More LOC means

Huge fan of Mypy but you lost me at LOC worship. Why do we do this? The more LOC is somehow attributed to more features? That simply isn’t true. More LOC means two things. One, what you are trying to do with the language is pushing its limits. Or two, you don’t understand the domain. Most large monorepo’s I have seen fall into the latter category while few reside in the former. Game engines, mature enterprise software, and a few others are large (probably OP’s codebase too), but we seem to revel in the fact that we have so much code to grok. Sometimes conciseness is better than cleverness. Back to mypy. I’d throw in black as well. Combining black, flake8, mypy, on precommit has loads of advantages. It’s completely opinionated but I find it helps me write better Python code.

charliermarsh 4y ago
I actually agree with this. And I like your second point around LOC as a yellow-flag around poor domain understanding. I tried to acknowledge the deficiencies in LOC in the second paragraph... but it felt like a useful shorthand for conveying a sense of scale and complexity (at least to an order-of-magnitude) in a domain-agnostic way. Outside of Mypy, we use: black, isort, flake8, docformatter, and autoflake (to remove unused imports) -- all on pre-commit + enforced on CI. I'd like to see Black cannibalize more of that toolchain :)
- gabereiser 4y ago
  Yeah, understandable. It's Python so things can easily get verbose. I've known too many engineers that see LOC as a measurement of complexity when those things aren't directly related. Complexity is usually unwrapped into a simplified model of things (human nature) but often there's 2x more plumbing code to cobble it back into complex mode to work with it. Not saying it's the case in your codebase but it's been what I've seen in large Python codebases. A complex domain is broken down into simple domain models then wrapped in services that add that complexity back because the original domain model was simplified from a business perspective already. Complexity to me comes from obtuse concepts. The business trying to cast too wide a net or something along those lines. Microservices help to concise that vision into reusable bits but its gets a bad wrap because of the context switching (which means your domains aren't well defined.) Keep using those tools though, they will save you so much time down the road when you enforce test coverage and fully lean in on that autoflake.
kortex 4y ago
I don't see it as "LOC-worship". It's about the most objective measure of codebase complexity/scope as could exist. It doesn't equate to features, but it certainly correlates to it, more business logic means more code. > Sometimes conciseness is better than cleverness. Isn't concise and clever on the same side of the balance? Cleverness and abstraction is how you cut down on LOC. Abstraction has its own cost, so while you can trade LOC for cleverness/abstraction, there's limits/costs to that as well. Also also, python/mypy still lacks support for higher-kinded types, which really puts a soft-ceiling on the amount of abstraction you can type-safely utilize. Aaaanyways, I think the whole original point is, "we have a large codebase, and mypy helps keep the wheels on at speed". +1 on black and precommit. That has done wonders for codebase consistency. I'd run flake8 too but the maintainer is weirdly opposed to supporting pyproject.toml for config.
- gabereiser 4y ago
  Concise and clever aren't necessarily related. Concise means conveying your objective as clearly as you can, as small as you can. Clever is John Carmack's q_sqrt. Code that is small, that works, but no one understands why. LOC should be looked at as a negative at a certain point. Software Engineers spend more time reading code than writing code. The more LOC you have, the more an engineer needs to read to get work done. LOC is not a good measurement of quality software nor is it a measurement of complexity of the problem, only the complexity of the code.
- benji-york 4y ago
  > I'd run flake8 too but the maintainer is weirdly opposed to supporting pyproject.toml for config. I haven't used it in anger yet, but FlakeHeaven (https://github.com/flakeheaven/flakeheaven https://github.com/flakeheaven/flakeheaven) looks like a nice fork that includes pyproject.toml support.

charliermarsh 4y ago

I actually agree with this. And I like your second point around LOC as a yellow-flag around poor domain understanding. I tried to acknowledge the deficiencies in LOC in the second paragraph... but it felt like a useful shorthand for conveying a sense of scale and complexity (at least to an order-of-magnitude) in a domain-agnostic way. Outside of Mypy, we use: black, isort, flake8, docformatter, and autoflake (to remove unused imports) -- all on pre-commit + enforced on CI. I'd like to see Black cannibalize more of that toolchain :)

gabereiser 4y ago
Yeah, understandable. It's Python so things can easily get verbose. I've known too many engineers that see LOC as a measurement of complexity when those things aren't directly related. Complexity is usually unwrapped into a simplified model of things (human nature) but often there's 2x more plumbing code to cobble it back into complex mode to work with it. Not saying it's the case in your codebase but it's been what I've seen in large Python codebases. A complex domain is broken down into simple domain models then wrapped in services that add that complexity back because the original domain model was simplified from a business perspective already. Complexity to me comes from obtuse concepts. The business trying to cast too wide a net or something along those lines. Microservices help to concise that vision into reusable bits but its gets a bad wrap because of the context switching (which means your domains aren't well defined.) Keep using those tools though, they will save you so much time down the road when you enforce test coverage and fully lean in on that autoflake.

kortex 4y ago

I don't see it as "LOC-worship". It's about the most objective measure of codebase complexity/scope as could exist. It doesn't equate to features, but it certainly correlates to it, more business logic means more code. > Sometimes conciseness is better than cleverness. Isn't concise and clever on the same side of the balance? Cleverness and abstraction is how you cut down on LOC. Abstraction has its own cost, so while you can trade LOC for cleverness/abstraction, there's limits/costs to that as well. Also also, python/mypy still lacks support for higher-kinded types, which really puts a soft-ceiling on the amount of abstraction you can type-safely utilize. Aaaanyways, I think the whole original point is, "we have a large codebase, and mypy helps keep the wheels on at speed". +1 on black and precommit. That has done wonders for codebase consistency. I'd run flake8 too but the maintainer is weirdly opposed to supporting pyproject.toml for config.

gabereiser 4y ago
Concise and clever aren't necessarily related. Concise means conveying your objective as clearly as you can, as small as you can. Clever is John Carmack's q_sqrt. Code that is small, that works, but no one understands why. LOC should be looked at as a negative at a certain point. Software Engineers spend more time reading code than writing code. The more LOC you have, the more an engineer needs to read to get work done. LOC is not a good measurement of quality software nor is it a measurement of complexity of the problem, only the complexity of the code.
benji-york 4y ago
> I'd run flake8 too but the maintainer is weirdly opposed to supporting pyproject.toml for config. I haven't used it in anger yet, but FlakeHeaven (https://github.com/flakeheaven/flakeheaven https://github.com/flakeheaven/flakeheaven) looks like a nice fork that includes pyproject.toml support.