4 ms·

I wish people would just forget about pre-commit, this thing is especially useless in a setting where a CI/CD pipeline exists. It's not that hard to write a sim

I wish people would just forget about pre-commit, this thing is especially useless in a setting where a CI/CD pipeline exists. It's not that hard to write a simple Makefile or shellscript to run linters on push. Pre-commit is one of the most annoying tools that have come into existence in recent years that everyone seems to be cargo-culting. It doesn't play well with editors since in order to find the actual binary path, you'd have to open up a sqlite database to fish out the virtualenv pre-commit created. Pre-commit also increases maintenance burden, since its configuration is completely separate from your usual requirements-dev.txt/pyproject.toml/setup.cfg etc. If you have dev dependencies in one of these files because making your editors to find the pre-commit created binaries are hard, now you have to keep both versions in sync. I really don't see the point of any pre-commit hooks unless you are the one guy that doesn't use a modern CI/CD platform.

globular-toast 4y ago
Our CI pipelines invoke pre-commit. That way it's trivial to run the exact same tools locally as would be run in CI. Running the tools locally is basically about tightening the development loop. Many of the commonly used tools (e.g. black, isort etc.) actually make the changes to the files so you'll never even commit failing versions. Do you really want to push changes to some remote CI system only to be told it's failed some boring QA check? There's nothing at all stopping you from doing that. Pre-commit is completely optional for each developer. I would just recommend it for sanity reasons.
- wyuenho 4y ago
  > Do you really want to push changes to some remote CI system only to be told it's failed some boring QA check? Yes, that's what CI/CD is for. > Pre-commit is completely optional for each developer. No it is not, it's installed at your git pre-commit hook, and it gets run every fucking time I commit, even if I've set up my editor correctly to auto-format and lint everything at I develop, meaning 99.9999% of the time sure the code I commit will pass all these linter checks. I can use git commit --no-verify to bypass pre-commit, but then again, what's the point of using pre-commit in the first place if you need to bypass it? There is absolutely no point of linting twice locally and thrice in total just to hit the first stage of your deployment.
  - chucksmash 4y ago
    rm ./.git/hooks/pre-commit Or just do not install the hook in the first place. Choice is still the developer's unless your team is doing something else here that limits your ability to delete/rename files locally.
    - wyuenho 4y ago
      That does not alleviate the problem with difficulty having your editor to use the correct version and settings for the linters and formatters. This approach also does not address the issue of having 2 separate sets of configuration when you are trying address problem number 1 above.
  - doliveira 4y ago
    How is it mandatory? You can just not install the hooks
    - wyuenho 4y ago
      1. Because it defeats the purpose of pre-commit. 2. pre-commit install installs the hook by default. 3. If you happen to fail linting a few times a year, there's always an anal coworker telling your boss in a 1-on-1 you are not following some BS ways of working.
      - doliveira 4y ago
        > 1. Because it defeats the purpose of pre-commit. No, it doesn't. Pre-commits are just a DX bonus so you won't have to wait for the CI/CD server. 2. pre-commit install installs the hook by default. Am I missing something? If you run "$ pre-commit install" of course it will install the hooks. Do you mean that poetry install, setup.py themselves are making the magic of installing the hooks themselves? If so, it's indeed a bad practice. 3. If you happen to fail linting a few times a year, there's always an anal coworker telling your boss in a 1-on-1 you are not following some BS ways of working. I don't get why anyone would care, unless you're polluting the CI/CD history with tons of tiny commits all day along just to run linting.
doliveira 4y ago
Pre-commit is optional, you can just not install them into .git/ ... Although I'd indeed prefer just before push, or just make display warnings One thing that's really annoying these days are CI/CD that can't be replicated locally, generating quite annoying delays in the development. Jenkins seems particularly problematic in this regard: the steps get encoded in some cryptic pet Jenkins server, and then you have to wait minutes until an agent picks it up and reaches the step you actually care about. Other tools are a little quicker, but still... So, I think at the very least pre-commit hooks help with this "over-reliance" on the CI/CD server. It's so much better DX when you can run parts of the pipeline instantaneously.
- wyuenho 4y ago
  > One thing that's really annoying these days are CI/CD that can't be replicated locally pre-commit just runs your linter, formatter and tests. Surely you can fully replicate this step locally. Just run make lint both locally and on CI. Anything else that's hard to replicate has to do with the distributed systems that you are probably working on because these systems are all probably proprietary stuff that live on the cloud. > Although I'd indeed prefer just before push, or just make display warnings Wholehearted agree. This is a compromise I can live with.
  - doliveira 4y ago
    My point is that using these tools at least makes people put these scripts within the repo instead of just in the CI server.
    - wyuenho 4y ago
      There's no "instead of" to speak of. Either you run the same linting step both locally and on CI, or just on CI, you can't skip linting on CI. Every properly set up CI pipeline has some combination of Makefile/shell script/Dockerfile that runs exactly the same way locally and on CI, which the script checked into the repo itself. If your CI scripts don't exist in your repo, you are doing it wrong.
carso 4y ago
Pre-commit running tools in it's own virtual environment is a feature, not a bug, in my book -- it means that the dependencies for my linter tools aren't mixed in with the dependencies for the code I'm writing. And, keeping things separate from setup.cfg or pyproject.toml is optional: The tools still look for configuration in their usual places, so it's still possible have your black options in pyproject.toml and just a bare-bones entry to call black in your .pre-commit file if you prefer.
- wyuenho 4y ago
  Except your usual configuration don't necessarily work for your pre-commit hooks. A prime example is mypy.
  - js2 4y ago
    Sure it can. The pre-commit tool is just a framework for running multiple hooks at commit time against just the files which are modified in that commit. You can configure those hooks however you want. You also don't have to run the hooks at pre-commit time. Just don't hook pre-commit into your checkout. The pre-commit tool can also be configured to run its checks at a different stages than, well, the pre-commit stage: https://pre-commit.com/#top_level-default_stages https://pre-commit.com/#top_level-default_stages
fluidcruft 4y ago
Do you have any tutorials for setting up CI/CD? My impression was that's all stuff that runs in the cloud but if it's something I can use on my own personal projects I'd play with it. Frankly a lot of these things become unintelligible. I've used pre-commit for things like black and autopep8 and that's all pretty understandable to me. The CI/CD things I've read all seem like everyone already understands some giant Rube Goldberg contraption that they're strapping on things for some reason that probably matters to giant dev teams.
- natpalmer1776 4y ago
  Good example of CI/CD that's not a rube goldberg machine per se: https://github.blog/2022-02-02-build-ci-cd-pipeline-github-actions-four-steps/ https://github.blog/2022-02-02-build-ci-cd-pipeline-github-a...
  - fluidcruft 4y ago
    Thank you. So does CI/CD require becoming dependent on a cloud service such as github?
    - natpalmer1776 4y ago
      Not necessarily, it just depends on how invested you are in the CI/CD pipeline for any given project, your preferences regarding self-hosting vs. cloud, and the amount of time you have to dedicate to the subject. Strictly speaking, any tool or set of tools that allow you to trigger building & deploying/publishing artifacts in response to source control commits can be used to build a CI/CD pipeline. One could write bash scripts linked to a cron job that pulls a remote repository every n minutes and then performs some scripted actions to integrate changes between branches before building & publishing the artifact to a local SFTP server. If you prefer a more mature solution with better documentation however, there is a (non-exhaustive) list of CI/CD tools on this awesome-devops list: https://github.com/wmariuss/awesome-devops#continuous-integration-delivery https://github.com/wmariuss/awesome-devops#continuous-integr... --- edit: I saw gitlab on the list and realized it is probably the closest self-hosted equivalent to the github option I mentioned previously https://about.gitlab.com/features/continuous-integration/ https://about.gitlab.com/features/continuous-integration/
js2 4y ago
Pre-commit can also be configured to use the project's dependencies, which is what I do with my repos when there is overlap. You don't have to use its built-in integrations. Indeed, I find its integrations most useful for tools that aren't specific to my project: things like the white-space checks, yaml checks, etc. You can also set things up the other way around, having the CI/CD system install and and run pre-commit's checks as a build step. Pre-commit provides a nice framework, I find, for running these checks. The advantage of pre-commit is that it catches mistakes before they are committed. Most CI/CD systems are setup to only validate the tip of the branch (i.e. the last commit in the PR), not all the commits along the way. Yes, you can configure a CI/CD system to test each commit, but it's usually swimming upstream to do so. And yes, you can squash all of a PR's commits into a single commit, but there are good reasons NOT to do that. So assuming you have multiple commits in a change, it's nice to know they have likely all been validated in a project using pre-commmit. I'll make an appeal to authority here: I've been a professional developer for decades. I've worked with a variety of VCS's and build systems. I've written plenty of Makefiles. Nonetheless, I still find pre-commit useful. I use it even in combination with a Makefile sometimes. You'd be horrified, I guess, to know some of my Makefiles have a rule which runs `pre-commit run --all`. As an example, earlier this week I setup a new repo which build packages an AWS Lambda written in javascript, and deploys it using the AWS "sam" CLI via a CI/CD system. So the deployed code is Javascript. The repo contains shell-scripts to assist with deployment. And there are multiple yaml files. There's a yaml file to configure the CI/CD system, and there's the CloudFormation template files. Here's what I configured pre-commit to do: 1. Check for whitespace nits. 2. Check for syntax errors in the yaml files. 3. Validate the CloudFormation templates. https://aws.amazon.com/blogs/infrastructure-and-automation/use-git-pre-commit-hooks-avoid-aws-cloudformation-errors/ https://aws.amazon.com/blogs/infrastructure-and-automation/u... 4. Run shellcheck against the shell scripts. 5. Run eslint and prettifier against the javascript. 6. Run "npm test" as a local step. Normally I'd leave (6) out because in most projects its too time-consuming, but in this project the tests run quickly enough that I just made it a pre-commit check. The "npm test" step runs jest, which is installed as a dev dependency in the project's package.json. The other tools are all installed by pre-commit itself. > I really don't see the point of any pre-commit hooks unless you are the one guy that doesn't use a modern CI/CD platform. I've tried to make an argument for why to use pre-commit above. Nevertheless, you don't have to install pre-commit's hooks in your checkout. They are ideally there to save you time having to correct mistakes after the fact. It sounds like there's an impedance mismatch between your personal workflow and how you've seen pre-commit set up. Perhaps by resolving that mismatch by adjusting the pre-commit configuration, you can enjoy pre-commits benefits w/o experiencing the issues you've run into.
areeh 4y ago
It's convenient to run the lint step faster/sooner than at CI/CD time. Depending on your setup, the separate linter deps handled by pre-commit can be more convenient than hassle both locally and in your CI/CD pipeline (re the makefile script you mention). Having done it both ways several times I lean pre-commit for now
- wyuenho 4y ago
  You've already run them at edit time.

globular-toast 4y ago

Our CI pipelines invoke pre-commit. That way it's trivial to run the exact same tools locally as would be run in CI. Running the tools locally is basically about tightening the development loop. Many of the commonly used tools (e.g. black, isort etc.) actually make the changes to the files so you'll never even commit failing versions. Do you really want to push changes to some remote CI system only to be told it's failed some boring QA check? There's nothing at all stopping you from doing that. Pre-commit is completely optional for each developer. I would just recommend it for sanity reasons.

wyuenho 4y ago
> Do you really want to push changes to some remote CI system only to be told it's failed some boring QA check? Yes, that's what CI/CD is for. > Pre-commit is completely optional for each developer. No it is not, it's installed at your git pre-commit hook, and it gets run every fucking time I commit, even if I've set up my editor correctly to auto-format and lint everything at I develop, meaning 99.9999% of the time sure the code I commit will pass all these linter checks. I can use git commit --no-verify to bypass pre-commit, but then again, what's the point of using pre-commit in the first place if you need to bypass it? There is absolutely no point of linting twice locally and thrice in total just to hit the first stage of your deployment.
- chucksmash 4y ago
  rm ./.git/hooks/pre-commit Or just do not install the hook in the first place. Choice is still the developer's unless your team is doing something else here that limits your ability to delete/rename files locally.
  - wyuenho 4y ago
    That does not alleviate the problem with difficulty having your editor to use the correct version and settings for the linters and formatters. This approach also does not address the issue of having 2 separate sets of configuration when you are trying address problem number 1 above.
- doliveira 4y ago
  How is it mandatory? You can just not install the hooks
  - wyuenho 4y ago
    1. Because it defeats the purpose of pre-commit. 2. pre-commit install installs the hook by default. 3. If you happen to fail linting a few times a year, there's always an anal coworker telling your boss in a 1-on-1 you are not following some BS ways of working.
    - doliveira 4y ago
      > 1. Because it defeats the purpose of pre-commit. No, it doesn't. Pre-commits are just a DX bonus so you won't have to wait for the CI/CD server. 2. pre-commit install installs the hook by default. Am I missing something? If you run "$ pre-commit install" of course it will install the hooks. Do you mean that poetry install, setup.py themselves are making the magic of installing the hooks themselves? If so, it's indeed a bad practice. 3. If you happen to fail linting a few times a year, there's always an anal coworker telling your boss in a 1-on-1 you are not following some BS ways of working. I don't get why anyone would care, unless you're polluting the CI/CD history with tons of tiny commits all day along just to run linting.

doliveira 4y ago

Pre-commit is optional, you can just not install them into .git/ ... Although I'd indeed prefer just before push, or just make display warnings One thing that's really annoying these days are CI/CD that can't be replicated locally, generating quite annoying delays in the development. Jenkins seems particularly problematic in this regard: the steps get encoded in some cryptic pet Jenkins server, and then you have to wait minutes until an agent picks it up and reaches the step you actually care about. Other tools are a little quicker, but still... So, I think at the very least pre-commit hooks help with this "over-reliance" on the CI/CD server. It's so much better DX when you can run parts of the pipeline instantaneously.

wyuenho 4y ago
> One thing that's really annoying these days are CI/CD that can't be replicated locally pre-commit just runs your linter, formatter and tests. Surely you can fully replicate this step locally. Just run make lint both locally and on CI. Anything else that's hard to replicate has to do with the distributed systems that you are probably working on because these systems are all probably proprietary stuff that live on the cloud. > Although I'd indeed prefer just before push, or just make display warnings Wholehearted agree. This is a compromise I can live with.
- doliveira 4y ago
  My point is that using these tools at least makes people put these scripts within the repo instead of just in the CI server.
  - wyuenho 4y ago
    There's no "instead of" to speak of. Either you run the same linting step both locally and on CI, or just on CI, you can't skip linting on CI. Every properly set up CI pipeline has some combination of Makefile/shell script/Dockerfile that runs exactly the same way locally and on CI, which the script checked into the repo itself. If your CI scripts don't exist in your repo, you are doing it wrong.

carso 4y ago

Pre-commit running tools in it's own virtual environment is a feature, not a bug, in my book -- it means that the dependencies for my linter tools aren't mixed in with the dependencies for the code I'm writing. And, keeping things separate from setup.cfg or pyproject.toml is optional: The tools still look for configuration in their usual places, so it's still possible have your black options in pyproject.toml and just a bare-bones entry to call black in your .pre-commit file if you prefer.

wyuenho 4y ago
Except your usual configuration don't necessarily work for your pre-commit hooks. A prime example is mypy.
- js2 4y ago
  Sure it can. The pre-commit tool is just a framework for running multiple hooks at commit time against just the files which are modified in that commit. You can configure those hooks however you want. You also don't have to run the hooks at pre-commit time. Just don't hook pre-commit into your checkout. The pre-commit tool can also be configured to run its checks at a different stages than, well, the pre-commit stage: https://pre-commit.com/#top_level-default_stages https://pre-commit.com/#top_level-default_stages

fluidcruft 4y ago

Do you have any tutorials for setting up CI/CD? My impression was that's all stuff that runs in the cloud but if it's something I can use on my own personal projects I'd play with it. Frankly a lot of these things become unintelligible. I've used pre-commit for things like black and autopep8 and that's all pretty understandable to me. The CI/CD things I've read all seem like everyone already understands some giant Rube Goldberg contraption that they're strapping on things for some reason that probably matters to giant dev teams.

natpalmer1776 4y ago
Good example of CI/CD that's not a rube goldberg machine per se: https://github.blog/2022-02-02-build-ci-cd-pipeline-github-actions-four-steps/ https://github.blog/2022-02-02-build-ci-cd-pipeline-github-a...
- fluidcruft 4y ago
  Thank you. So does CI/CD require becoming dependent on a cloud service such as github?
  - natpalmer1776 4y ago
    Not necessarily, it just depends on how invested you are in the CI/CD pipeline for any given project, your preferences regarding self-hosting vs. cloud, and the amount of time you have to dedicate to the subject. Strictly speaking, any tool or set of tools that allow you to trigger building & deploying/publishing artifacts in response to source control commits can be used to build a CI/CD pipeline. One could write bash scripts linked to a cron job that pulls a remote repository every n minutes and then performs some scripted actions to integrate changes between branches before building & publishing the artifact to a local SFTP server. If you prefer a more mature solution with better documentation however, there is a (non-exhaustive) list of CI/CD tools on this awesome-devops list: https://github.com/wmariuss/awesome-devops#continuous-integration-delivery https://github.com/wmariuss/awesome-devops#continuous-integr... --- edit: I saw gitlab on the list and realized it is probably the closest self-hosted equivalent to the github option I mentioned previously https://about.gitlab.com/features/continuous-integration/ https://about.gitlab.com/features/continuous-integration/

js2 4y ago

Pre-commit can also be configured to use the project's dependencies, which is what I do with my repos when there is overlap. You don't have to use its built-in integrations. Indeed, I find its integrations most useful for tools that aren't specific to my project: things like the white-space checks, yaml checks, etc. You can also set things up the other way around, having the CI/CD system install and and run pre-commit's checks as a build step. Pre-commit provides a nice framework, I find, for running these checks. The advantage of pre-commit is that it catches mistakes before they are committed. Most CI/CD systems are setup to only validate the tip of the branch (i.e. the last commit in the PR), not all the commits along the way. Yes, you can configure a CI/CD system to test each commit, but it's usually swimming upstream to do so. And yes, you can squash all of a PR's commits into a single commit, but there are good reasons NOT to do that. So assuming you have multiple commits in a change, it's nice to know they have likely all been validated in a project using pre-commmit. I'll make an appeal to authority here: I've been a professional developer for decades. I've worked with a variety of VCS's and build systems. I've written plenty of Makefiles. Nonetheless, I still find pre-commit useful. I use it even in combination with a Makefile sometimes. You'd be horrified, I guess, to know some of my Makefiles have a rule which runs `pre-commit run --all`. As an example, earlier this week I setup a new repo which build packages an AWS Lambda written in javascript, and deploys it using the AWS "sam" CLI via a CI/CD system. So the deployed code is Javascript. The repo contains shell-scripts to assist with deployment. And there are multiple yaml files. There's a yaml file to configure the CI/CD system, and there's the CloudFormation template files. Here's what I configured pre-commit to do: 1. Check for whitespace nits. 2. Check for syntax errors in the yaml files. 3. Validate the CloudFormation templates. https://aws.amazon.com/blogs/infrastructure-and-automation/use-git-pre-commit-hooks-avoid-aws-cloudformation-errors/ https://aws.amazon.com/blogs/infrastructure-and-automation/u... 4. Run shellcheck against the shell scripts. 5. Run eslint and prettifier against the javascript. 6. Run "npm test" as a local step. Normally I'd leave (6) out because in most projects its too time-consuming, but in this project the tests run quickly enough that I just made it a pre-commit check. The "npm test" step runs jest, which is installed as a dev dependency in the project's package.json. The other tools are all installed by pre-commit itself. > I really don't see the point of any pre-commit hooks unless you are the one guy that doesn't use a modern CI/CD platform. I've tried to make an argument for why to use pre-commit above. Nevertheless, you don't have to install pre-commit's hooks in your checkout. They are ideally there to save you time having to correct mistakes after the fact. It sounds like there's an impedance mismatch between your personal workflow and how you've seen pre-commit set up. Perhaps by resolving that mismatch by adjusting the pre-commit configuration, you can enjoy pre-commits benefits w/o experiencing the issues you've run into.

areeh 4y ago

It's convenient to run the lint step faster/sooner than at CI/CD time. Depending on your setup, the separate linter deps handled by pre-commit can be more convenient than hassle both locally and in your CI/CD pipeline (re the makefile script you mention). Having done it both ways several times I lean pre-commit for now

wyuenho 4y ago
You've already run them at edit time.