3 ms·

Are you really using a checkout on a production server to install and configure your software? This feels like a "Doctor, doctor - it hurts when I stick a spoo

Are you really using a checkout on a production server to install and configure your software? This feels like a "Doctor, doctor - it hurts when I stick a spoon in my eye" moment - "Don't do that!" Build tools and build servers exist for a reason. Subversion has "svn export" if you insist on working that way. git can git clone repos --depth 1; rm -rf repos/.git if you really, really must. And what you are calling polluted history, they are calling efficient collaboration.

shoo 11y ago
If there exists appropriate tooling (e.g. for filtering commit logs based on what touched the code belonging to a single project / what touched the code belonging to a single project and its transitive dependencies) then I dare say a huge monorepo history over many projects and libraries isn't polluted history, it is remarkably rich and useful history. In big places with solid engineering teams and a long-term investment in infrastructure it seems achievable to build the tools to do this properly. (famously: google, which isn't a very fair example). I think many people's default reaction to monorepos -- fear, loathing and disbelief, say -- is based upon working with open source tools / related cloud services / distributed workflows - that scale poorly when dealing with large repositories, at least with default configuration and without a lot of mucking about. That said, this default reaction is arguably a healthy reaction if the reality is that most teams / organisations do not have the resources to invest in setting up tooling / infrastructure beyond what they can get for free / rent for a few hundred dollars a month.
StavrosK 11y ago
> Build tools and build servers exist for a reason. What's the build server going to do? Copy the Python code from the repo to the other server? It doesn't seem like setting up a build server just to do "git checkout; rm .git" is worth the effort.
- shoo 11y ago
  Suppose you've invested in a build server anyway, for CI. Then you can just add/extend a job to create a release package, and deploy it. Doesn't take that much effort, and lets you avoid infecting your production boxes with dev infrastructure/scaffolding, as `NickNameNick` suggests.
  - StavrosK 11y ago
    What sort of def scaffolding do you mean? The benefits of just pulling the repo are that you can use ansible-pull to automatically update the servers whenever new code is merged to master, which would be more complicated if you had the CI server doing the pushing.
    - shoo 11y ago
      > What sort of def scaffolding do you mean? version control software, to start with. perhaps more. i guess my viewpoint is coloured as the last thing i was working on was deployed to client desktop machines. there was absolutely no way we were going to be pushing to those with version control. if you are deploying to a server you own, i guess in principle you can do whatever you want. one of my colleagues rigged a build to create a debian package for the server-side version of the code we worked on. that took a bit of mucking about to get going but was very nice when it was set up.
      - StavrosK 11y ago
        Building a deb is also an option, I generally don't prefer it because just pulling is easier to set up, and debs don't have that many advantages when you're dealing with interpreted server code.
- NickNameNick 11y ago
  I mostly work with java, so the build server is doing a clean compilation, to ensure the code word outside my dev environment. Pulling in known versions of dependencies, some packaging, unit and integration tests etc. Mostly using Maven. Whilst some of that is somewhat specific to java, I'd still want a build server to ensure I hadn't managed to pollute my dev environment with and undeclared dependency before I pushed my code to production. And I'd still want to run at least some integration tests, even for the simplest project. And I really REALLY don't want my internet facing servers to be able to reach into my internal network, which means a checkout simply isn't going to work anyway. Lastly I don't want my code on those servers either. You can set up .htaccess or equivalent to prevent access to .svn or .git (or dotfiles in general..) but those files really shouldn't have been there in the first place.
  - StavrosK 11y ago
    > I'd still want a build server to ensure I hadn't managed to pollute my dev environment with and undeclared dependency before I pushed my code to production. In my case, the production server is the build server. You can't have dev dependencies there, because you don't develop there. > And I'd still want to run at least some integration tests, even for the simplest project. Integration tests run on the dev or CI server, in my case. > And I really REALLY don't want my internet facing servers to be able to reach into my internal network, which means a checkout simply isn't going to work anyway. The code servers aren't in the internal network, in my case. They're already internet-facing, since I need to pull from them. > Lastly I don't want my code on those servers either. You can set up .htaccess or equivalent to prevent access to .svn or .git (or dotfiles in general..) but those files really shouldn't have been there in the first place. In my case, if my code isn't on the server, the server doesn't run. There's no need to set up .htaccess, since the HTTP server doesn't serve anything off the filesystem.
    - NickNameNick 11y ago
      >> I'd still want a build server to ensure I hadn't managed to pollute my dev environment with and undeclared dependency before I pushed my code to production. >In my case, the production server is the build server. You can't have dev dependencies there, because you don't develop there. The problem is not that dev dependencies exist, the problem is that they might be undeclared or unrecorded. I have imagemagic on my dev machine, but if it isn't installed as part of the server setup script, then it won't be on the production server. The tests might run fine on a dev server that happens to have imagemagic on it, but fail in production. >Integration tests run on the dev If you run your integration tests in your dev environment, you might not catch some on the fly configuration changes or utility installations - undeclared dependencies.
- perlgeek 11y ago
  Set up a virtualenv with defined versions of all dependencies, on install the python code there. And tar up the virtualenv. Or create a .deb or .rpm package out of the python code. In a non-trivial setup, just doing a 'git checkout' isn't enough for a deployment.
  - StavrosK 11y ago
    I'll just point out that your assumption that I'm only doing a "git checkout" is completely unwarranted.
    - twic 11y ago
      Given that in the comment to which that was a reply, you opined that 'It doesn't seem like setting up a build server just to do "git checkout; rm .git" is worth the effort.', it's not completely unwarranted.
      - StavrosK 11y ago
        The part that the build server can handle is the "git checkout" part, maybe create a virtualenv with the dependencies. In that case, doing a "git checkout" is enough for deployment, because you're already assuming that the rest of the deployment happens elsewhere.
- falcolas 11y ago
  If you're working with Python, you can create builds of individual modules via zip archives. This lets you create a versionable, signable build artifact which can be easily deployed almost everywhere. They can even include static versions of you dependencies, making the deploy less reliant on what is installed on the target server.
  - murph 11y ago
    A great tool for this is pex: https://github.com/pantsbuild/pex https://github.com/pantsbuild/pex

shoo 11y ago

If there exists appropriate tooling (e.g. for filtering commit logs based on what touched the code belonging to a single project / what touched the code belonging to a single project and its transitive dependencies) then I dare say a huge monorepo history over many projects and libraries isn't polluted history, it is remarkably rich and useful history. In big places with solid engineering teams and a long-term investment in infrastructure it seems achievable to build the tools to do this properly. (famously: google, which isn't a very fair example). I think many people's default reaction to monorepos -- fear, loathing and disbelief, say -- is based upon working with open source tools / related cloud services / distributed workflows - that scale poorly when dealing with large repositories, at least with default configuration and without a lot of mucking about. That said, this default reaction is arguably a healthy reaction if the reality is that most teams / organisations do not have the resources to invest in setting up tooling / infrastructure beyond what they can get for free / rent for a few hundred dollars a month.

StavrosK 11y ago

> Build tools and build servers exist for a reason. What's the build server going to do? Copy the Python code from the repo to the other server? It doesn't seem like setting up a build server just to do "git checkout; rm .git" is worth the effort.

shoo 11y ago
Suppose you've invested in a build server anyway, for CI. Then you can just add/extend a job to create a release package, and deploy it. Doesn't take that much effort, and lets you avoid infecting your production boxes with dev infrastructure/scaffolding, as `NickNameNick` suggests.
- StavrosK 11y ago
  What sort of def scaffolding do you mean? The benefits of just pulling the repo are that you can use ansible-pull to automatically update the servers whenever new code is merged to master, which would be more complicated if you had the CI server doing the pushing.
  - shoo 11y ago
    > What sort of def scaffolding do you mean? version control software, to start with. perhaps more. i guess my viewpoint is coloured as the last thing i was working on was deployed to client desktop machines. there was absolutely no way we were going to be pushing to those with version control. if you are deploying to a server you own, i guess in principle you can do whatever you want. one of my colleagues rigged a build to create a debian package for the server-side version of the code we worked on. that took a bit of mucking about to get going but was very nice when it was set up.
    - StavrosK 11y ago
      Building a deb is also an option, I generally don't prefer it because just pulling is easier to set up, and debs don't have that many advantages when you're dealing with interpreted server code.
NickNameNick 11y ago
I mostly work with java, so the build server is doing a clean compilation, to ensure the code word outside my dev environment. Pulling in known versions of dependencies, some packaging, unit and integration tests etc. Mostly using Maven. Whilst some of that is somewhat specific to java, I'd still want a build server to ensure I hadn't managed to pollute my dev environment with and undeclared dependency before I pushed my code to production. And I'd still want to run at least some integration tests, even for the simplest project. And I really REALLY don't want my internet facing servers to be able to reach into my internal network, which means a checkout simply isn't going to work anyway. Lastly I don't want my code on those servers either. You can set up .htaccess or equivalent to prevent access to .svn or .git (or dotfiles in general..) but those files really shouldn't have been there in the first place.
- StavrosK 11y ago
  > I'd still want a build server to ensure I hadn't managed to pollute my dev environment with and undeclared dependency before I pushed my code to production. In my case, the production server is the build server. You can't have dev dependencies there, because you don't develop there. > And I'd still want to run at least some integration tests, even for the simplest project. Integration tests run on the dev or CI server, in my case. > And I really REALLY don't want my internet facing servers to be able to reach into my internal network, which means a checkout simply isn't going to work anyway. The code servers aren't in the internal network, in my case. They're already internet-facing, since I need to pull from them. > Lastly I don't want my code on those servers either. You can set up .htaccess or equivalent to prevent access to .svn or .git (or dotfiles in general..) but those files really shouldn't have been there in the first place. In my case, if my code isn't on the server, the server doesn't run. There's no need to set up .htaccess, since the HTTP server doesn't serve anything off the filesystem.
  - NickNameNick 11y ago
    >> I'd still want a build server to ensure I hadn't managed to pollute my dev environment with and undeclared dependency before I pushed my code to production. >In my case, the production server is the build server. You can't have dev dependencies there, because you don't develop there. The problem is not that dev dependencies exist, the problem is that they might be undeclared or unrecorded. I have imagemagic on my dev machine, but if it isn't installed as part of the server setup script, then it won't be on the production server. The tests might run fine on a dev server that happens to have imagemagic on it, but fail in production. >Integration tests run on the dev If you run your integration tests in your dev environment, you might not catch some on the fly configuration changes or utility installations - undeclared dependencies.
perlgeek 11y ago
Set up a virtualenv with defined versions of all dependencies, on install the python code there. And tar up the virtualenv. Or create a .deb or .rpm package out of the python code. In a non-trivial setup, just doing a 'git checkout' isn't enough for a deployment.
- StavrosK 11y ago
  I'll just point out that your assumption that I'm only doing a "git checkout" is completely unwarranted.
  - twic 11y ago
    Given that in the comment to which that was a reply, you opined that 'It doesn't seem like setting up a build server just to do "git checkout; rm .git" is worth the effort.', it's not completely unwarranted.
    - StavrosK 11y ago
      The part that the build server can handle is the "git checkout" part, maybe create a virtualenv with the dependencies. In that case, doing a "git checkout" is enough for deployment, because you're already assuming that the rest of the deployment happens elsewhere.
falcolas 11y ago
If you're working with Python, you can create builds of individual modules via zip archives. This lets you create a versionable, signable build artifact which can be easily deployed almost everywhere. They can even include static versions of you dependencies, making the deploy less reliant on what is installed on the target server.
- murph 11y ago
  A great tool for this is pex: https://github.com/pantsbuild/pex https://github.com/pantsbuild/pex