7 ms·

Vendor. Everything. Always. I appreciate the fact that Google is investing in infrastructure to make the Go ecosystem robust, I really do. Having said that I

Vendor. Everything. Always. I appreciate the fact that Google is investing in infrastructure to make the Go ecosystem robust, I really do. Having said that I encourage everyone to do a simple experiment. Check out your code into some random directory. Copy the directory to a USB drive and walk it over to an air gapped machine (no wi-fi, no ethernet, clean dev environment installed). Copy the directory to the box, make some small code change and try to build your binary. If your build fails you're doing it wrong. In real life one of the following will happen[1]: - Your network connection to the Internet will be down - Your network connection to the corporate network will be down - The services you depend on (Github, Google proxy, Docker hub) will be down - The services you depend on will throttle you - The services you depend on will serve you wrong or corrupt data - The services you depend on will be offline for maintenance - The certificates on the service you depend on were not renewed in time - The account you use on services you depend on will be inexplicably suspended Murphy's law being what it is, some or all of the above will happen when your SaaS is on fire, your customers are screaming, and you're trying to roll out a fix. Meanwhile your company is losing approximately your personal lifetime earnings in revenue every half an hour. Vendor. Everything. Always. (and have backups, lots of backups) [1] Based on highly educational career experience

ndarwincorn 7y ago
I don't see any of those as reasons to commit dependencies to the same VCS repo your source control lives in. Your first point is particularly laughable. If your network connection to the internet is down and you're a SaaS provider, you're probably going to need to fix that before you can roll out a fix to your hosted software. That said, they're all good points to consider when building out a deployment pipeline, and in general mirroring your dependencies is a great idea. But committing a vendor directory is a poor solution/mitigation for most of those risks.
- TheSwordsman 7y ago
  I'm not sure a solution that solves 100% of the problems, with minimal to no side effects, is a poor solution.
  - Groxx 7y ago
    There are definitely downsides - a gigantic git repo is a big one. Lengthy clones, far slower git operations, etc. Many projects I've seen go this route will exceed 10s or 100s of gigabytes pretty quickly, and git is very unhappy about sizes like that.
- AYBABTME 7y ago
  How is it a poor solution? It is extremely simple, pragmatic and effective.
  - sigmonsays 7y ago
    I dont think its a poor solution. I have had great success keeping all of GOPATH in git. Its just easier to deploy new packges. The build machine doesn't even need internet access at this point.
- ownagefool 7y ago
  They themselves are not, but lets look at another consideration. How will you catch a backdoor added to one of your dependencies? - Scan them and hope someone else noticed and reported before you used it? - Assume your team mates reviewed them when they were updated? - Add SIEM like capabilities and monitor network connections? It should really be all 3. There are counter points of course, but if you vendor them, they'll in a PR and reviewed just the same as the rest of your code. Of course, where do you stop? You trust your distro to do the right thing and you don't review that code, etc. So the more generic question is why do you trust the golang modules you use?
- programd 7y ago
  > Your first point is particularly laughable. If your network connection to the internet is down and you're a SaaS provider, you're probably going to need to fix that before you can roll out a fix to your hosted software. Don't assume that your SaaS runs on the same network as your build machines. In most cases this is not the case - or at least it shouldn't be. Your SaaS might be broken in some way, you're trying to create a build to fix it on your secure corp network and your connectivity to the Net disappears. And it's 3 AM in San Francisco and everyone who can fix it is asleep, and meanwhile your European customers are not happy. Let's be clear. You can mitigate the effect of all these problems one way or another. But the mitigations are easier if you have less moving parts and less dependencies on systems you don't control. Lets face it, ultimately it all comes down to limiting your business risk at least cost (because money). One simple way to do that is if your build environment travels with your code down the years.
  - ndarwincorn 7y ago
    > Don't assume that your SaaS runs on the same network as your build machines. That wasn't the assumption. It's laughable because under any condition of where your hosting and build infrastructure live, you'll need to restore your internet connection before you can deploy whatever software fix you're trying to ('deploy' in the sense that your customers have access to the deployed fix).
- dewey 7y ago
  > I don't see any of those as reasons to commit dependencies to the same VCS repo your source control lives in. Why not? What's the downside except having a "update dependency" commit from time to time. It's a great solution to always have your dependencies at hand, offline and even versioned. Makes it easy to see when a bug got introduced by a dependency or roll back an update. Especially for projects that don't have a build pipeline, cached dependencies or just smaller dependencies that aren't very well backed up from other people. I think it's a great way to make sure things are still running after a few years (personal projects) and everything still works without having to hunt down some dependencies or find alternative mirrors.
  - SamWhited 7y ago
    > What's the downside except having a "update dependency" commit from time to time. This isn't a downside, it's a chance to vet and code review your dependencies which you should be doing anyways (but nobody does).
tshannon 7y ago
I thought vendoring was being deprecated in favor of modules. Maybe I misunderstood though.
- chimeracoder 7y ago
  > I thought vendoring was being deprecated in favor of modules. Maybe I misunderstood though. The original proposal (or sketch) of modules eliminated vendoring. It was quickly updated to include support for vendoring in response to feedback. The decision not to use vendoring (by default) has been controversial. That said, vendoring isn't "deprecated" in the sense that Go modules still allow for vendoring; they just happen not to vendor by default.
- programd 7y ago
  That's kind of the problem - well, issue - that I'm worried about. Now you have to do some special things in your build environment to vendor stuff (set env variables, compiler options), and by default your build infrastructure is depending on supposedly always-on external services. I rather think it should be the other way around. By default your build environment should gather all required artifacts locally, and only if you want to depend on external services should you have to create non-default options. I would have been slightly happier of vendoring was the default happy path, and module proxies were an alternative. I wonder if Google folks are subconsciously influenced by the idea that all this infrastructure will exist forever and never decay because they have access to seemingly infalible and highly available Google services. The gradual decay of Perl CPAN (which I've always loved BTW) and the reliability/security issues with the Node ecosystem are instructive counterexamples. I have to say that I'm not dogmatic about this, there are good cases to be made for certain features which go with module proxies. But ultimately if you've got a long running business you now have to do more work to make sure your code will still compile in 5 years - e.g. create and maintain a module proxy service in perpetuity. This as opposed to just archiving a bunch of files organized as a git repo. Anyway, my experience says vendor all the things. You'll be glad you did.
kodablah 7y ago
> Vendor. Everything. Always. [...] Your network connection to the corporate network will be down I think this is a bit extreme and becomes very hard to manage dependencies department wide. I can walk over to the storage machine if corporate network is down. I know that some people want the entire internet, every git repo, every docker image, every game/binary asset, and every needed dependency available locally, but this is just not reasonable. This kind of hardline stance disregards the benefit of intranet stores, be it for code, docker images, whatever. There is a middleground between not relying on third party services and requiring the ability to build after a fresh clone in a vacuum. Some builds simply require more than vendorable pieces, and just telling them they're doing it wrong is like telling someone they are doing it wrong relying on their corporate email servers instead of walking across office and delivering a hand-written note. Maybe something like, "vendor code as much as you can" is a more reasonable approach than "always" this, "you're doing it wrong" that.
- programd 7y ago
  > I know that some people want the entire internet, every git repo, every docker image, every game/binary asset, and every needed dependency available locally, but this is just not reasonable. Sure it is. In most Enterprise outfits its even policy. Ask yourself why that is. I don't mean to be flippant about it, your point is actually quite reasonable. If you are a solo developer, or a small company, then don't worry about it. Github will be up, Google proxy will have your back, your Net connection will work. Once per year when all the stars align just wrong your builds might get delayed but it isn't worth worrying about it. However in certain industries if the delay during some incident is costing you more then a few digits per minute, even once per year, management starts to notice. Long term support contracts are also a thing, e.g. ensuring that your code still has the artifacts to build on Red Hat 5 or some such. If you think about it that's nothing but outsourcing vendoring to a third party for large amounts of money.
nikolay 7y ago
Yeah, we do - in our Docker repo. But I don't want my Dockerfiles polluted with checksums of modules I download. This allows me to build images safely and then it's all vendored in our private Docker repo. This just makes our lives easier and more intelligent.
nnutter 7y ago
Good luck deploying your vendored, air-gap-built binary.
marcrosoft 7y ago
I agree. Unfortunately you now have to go out of your way to vendor with modules enabled which is soon the default.
- Strom 7y ago
  You can create the vendor directory with "go mod vendor" and have everything use that vendor directory by having "-mod=vendor" in your GOFLAGS env variable. Is there something I'm missing?
danielparks 7y ago
> Check out your code into some random directory. Copy the directory to a USB drive and walk it over to an air gapped machine (no wi-fi, no ethernet, clean dev environment installed). Copy the directory to the box, make some small code change and try to build your binary. I don’t follow. Why is the code change important? Why not just use the go module cache? Seems like a much cleaner solution with very little overhead.
- _ph_ 7y ago
  The idea of making a change and rebuilding is about proving that the build process does not require a network connection and that you have properly resolved the depencenies on that machine.
  - danielparks 7y ago
    The change is unnecessary. You can just rebuild. If you’ve built once using the standard go module functionality then you will be able to rebuild as long as you don’t pull in more dependencies. Naturally you can move the cache around.
no_wizard 7y ago
I'm curious, do you or know if its policy, to vendor node_modules the same way as you do go modules?
- cjbprime 7y ago
  This isn't usually done, because node_modules often contain machine specific (e.g. 32-bit vs 64-bit x86) or OS specific (e.g. Linux vs Mac) compiled native code.
Groxx 7y ago
Alternatively: have a shared cache. Share / replicate that around instead. It's just a local mirror, and you can even literally run it as a local mirror, so you can transparently convert your not-vendored-anything project into an equivalent-to-vendored-everything project without immensely bloating each git repo with duplicates.
pjmlp 7y ago
While I agree with you, we in the Java and .NET communities enjoy decentralized library distributions, so what most Java/.NET shops end up doing is setting their Nexus/NuGET servers in-house, and the proxies that care that all teams actually use those servers. Just like vendoring, in a single place, and all teams profit from the speed of access. The only downside is the update process to get IT to add new libraries to it, as they tend to require license validations from legal department.
weitzj 7y ago
I would also say: vendor your tools. There is a bit of a pattern emerging, where you add a „tools“ packages and this tools package just contains blank _ imports to external Go tools you use (e.g. gometalinter) This way when you run god mod vendor, you will also be able to vendor your tools and build them locally from your repo. This works quite well with a Makefile for now. There is an open issue tracking this vendor-tools approach

ndarwincorn 7y ago

I don't see any of those as reasons to commit dependencies to the same VCS repo your source control lives in. Your first point is particularly laughable. If your network connection to the internet is down and you're a SaaS provider, you're probably going to need to fix that before you can roll out a fix to your hosted software. That said, they're all good points to consider when building out a deployment pipeline, and in general mirroring your dependencies is a great idea. But committing a vendor directory is a poor solution/mitigation for most of those risks.

TheSwordsman 7y ago
I'm not sure a solution that solves 100% of the problems, with minimal to no side effects, is a poor solution.
- Groxx 7y ago
  There are definitely downsides - a gigantic git repo is a big one. Lengthy clones, far slower git operations, etc. Many projects I've seen go this route will exceed 10s or 100s of gigabytes pretty quickly, and git is very unhappy about sizes like that.
AYBABTME 7y ago
How is it a poor solution? It is extremely simple, pragmatic and effective.
- sigmonsays 7y ago
  I dont think its a poor solution. I have had great success keeping all of GOPATH in git. Its just easier to deploy new packges. The build machine doesn't even need internet access at this point.
ownagefool 7y ago
They themselves are not, but lets look at another consideration. How will you catch a backdoor added to one of your dependencies? - Scan them and hope someone else noticed and reported before you used it? - Assume your team mates reviewed them when they were updated? - Add SIEM like capabilities and monitor network connections? It should really be all 3. There are counter points of course, but if you vendor them, they'll in a PR and reviewed just the same as the rest of your code. Of course, where do you stop? You trust your distro to do the right thing and you don't review that code, etc. So the more generic question is why do you trust the golang modules you use?
programd 7y ago
> Your first point is particularly laughable. If your network connection to the internet is down and you're a SaaS provider, you're probably going to need to fix that before you can roll out a fix to your hosted software. Don't assume that your SaaS runs on the same network as your build machines. In most cases this is not the case - or at least it shouldn't be. Your SaaS might be broken in some way, you're trying to create a build to fix it on your secure corp network and your connectivity to the Net disappears. And it's 3 AM in San Francisco and everyone who can fix it is asleep, and meanwhile your European customers are not happy. Let's be clear. You can mitigate the effect of all these problems one way or another. But the mitigations are easier if you have less moving parts and less dependencies on systems you don't control. Lets face it, ultimately it all comes down to limiting your business risk at least cost (because money). One simple way to do that is if your build environment travels with your code down the years.
- ndarwincorn 7y ago
  > Don't assume that your SaaS runs on the same network as your build machines. That wasn't the assumption. It's laughable because under any condition of where your hosting and build infrastructure live, you'll need to restore your internet connection before you can deploy whatever software fix you're trying to ('deploy' in the sense that your customers have access to the deployed fix).
dewey 7y ago
> I don't see any of those as reasons to commit dependencies to the same VCS repo your source control lives in. Why not? What's the downside except having a "update dependency" commit from time to time. It's a great solution to always have your dependencies at hand, offline and even versioned. Makes it easy to see when a bug got introduced by a dependency or roll back an update. Especially for projects that don't have a build pipeline, cached dependencies or just smaller dependencies that aren't very well backed up from other people. I think it's a great way to make sure things are still running after a few years (personal projects) and everything still works without having to hunt down some dependencies or find alternative mirrors.
- SamWhited 7y ago
  > What's the downside except having a "update dependency" commit from time to time. This isn't a downside, it's a chance to vet and code review your dependencies which you should be doing anyways (but nobody does).

tshannon 7y ago

I thought vendoring was being deprecated in favor of modules. Maybe I misunderstood though.

chimeracoder 7y ago
> I thought vendoring was being deprecated in favor of modules. Maybe I misunderstood though. The original proposal (or sketch) of modules eliminated vendoring. It was quickly updated to include support for vendoring in response to feedback. The decision not to use vendoring (by default) has been controversial. That said, vendoring isn't "deprecated" in the sense that Go modules still allow for vendoring; they just happen not to vendor by default.
programd 7y ago
That's kind of the problem - well, issue - that I'm worried about. Now you have to do some special things in your build environment to vendor stuff (set env variables, compiler options), and by default your build infrastructure is depending on supposedly always-on external services. I rather think it should be the other way around. By default your build environment should gather all required artifacts locally, and only if you want to depend on external services should you have to create non-default options. I would have been slightly happier of vendoring was the default happy path, and module proxies were an alternative. I wonder if Google folks are subconsciously influenced by the idea that all this infrastructure will exist forever and never decay because they have access to seemingly infalible and highly available Google services. The gradual decay of Perl CPAN (which I've always loved BTW) and the reliability/security issues with the Node ecosystem are instructive counterexamples. I have to say that I'm not dogmatic about this, there are good cases to be made for certain features which go with module proxies. But ultimately if you've got a long running business you now have to do more work to make sure your code will still compile in 5 years - e.g. create and maintain a module proxy service in perpetuity. This as opposed to just archiving a bunch of files organized as a git repo. Anyway, my experience says vendor all the things. You'll be glad you did.

kodablah 7y ago

> Vendor. Everything. Always. [...] Your network connection to the corporate network will be down I think this is a bit extreme and becomes very hard to manage dependencies department wide. I can walk over to the storage machine if corporate network is down. I know that some people want the entire internet, every git repo, every docker image, every game/binary asset, and every needed dependency available locally, but this is just not reasonable. This kind of hardline stance disregards the benefit of intranet stores, be it for code, docker images, whatever. There is a middleground between not relying on third party services and requiring the ability to build after a fresh clone in a vacuum. Some builds simply require more than vendorable pieces, and just telling them they're doing it wrong is like telling someone they are doing it wrong relying on their corporate email servers instead of walking across office and delivering a hand-written note. Maybe something like, "vendor code as much as you can" is a more reasonable approach than "always" this, "you're doing it wrong" that.

programd 7y ago
> I know that some people want the entire internet, every git repo, every docker image, every game/binary asset, and every needed dependency available locally, but this is just not reasonable. Sure it is. In most Enterprise outfits its even policy. Ask yourself why that is. I don't mean to be flippant about it, your point is actually quite reasonable. If you are a solo developer, or a small company, then don't worry about it. Github will be up, Google proxy will have your back, your Net connection will work. Once per year when all the stars align just wrong your builds might get delayed but it isn't worth worrying about it. However in certain industries if the delay during some incident is costing you more then a few digits per minute, even once per year, management starts to notice. Long term support contracts are also a thing, e.g. ensuring that your code still has the artifacts to build on Red Hat 5 or some such. If you think about it that's nothing but outsourcing vendoring to a third party for large amounts of money.

nikolay 7y ago

Yeah, we do - in our Docker repo. But I don't want my Dockerfiles polluted with checksums of modules I download. This allows me to build images safely and then it's all vendored in our private Docker repo. This just makes our lives easier and more intelligent.

nnutter 7y ago

Good luck deploying your vendored, air-gap-built binary.

marcrosoft 7y ago

I agree. Unfortunately you now have to go out of your way to vendor with modules enabled which is soon the default.

Strom 7y ago
You can create the vendor directory with "go mod vendor" and have everything use that vendor directory by having "-mod=vendor" in your GOFLAGS env variable. Is there something I'm missing?

danielparks 7y ago

> Check out your code into some random directory. Copy the directory to a USB drive and walk it over to an air gapped machine (no wi-fi, no ethernet, clean dev environment installed). Copy the directory to the box, make some small code change and try to build your binary. I don’t follow. Why is the code change important? Why not just use the go module cache? Seems like a much cleaner solution with very little overhead.

_ph_ 7y ago
The idea of making a change and rebuilding is about proving that the build process does not require a network connection and that you have properly resolved the depencenies on that machine.
- danielparks 7y ago
  The change is unnecessary. You can just rebuild. If you’ve built once using the standard go module functionality then you will be able to rebuild as long as you don’t pull in more dependencies. Naturally you can move the cache around.

no_wizard 7y ago

I'm curious, do you or know if its policy, to vendor node_modules the same way as you do go modules?

cjbprime 7y ago
This isn't usually done, because node_modules often contain machine specific (e.g. 32-bit vs 64-bit x86) or OS specific (e.g. Linux vs Mac) compiled native code.

Groxx 7y ago

Alternatively: have a shared cache. Share / replicate that around instead. It's just a local mirror, and you can even literally run it as a local mirror, so you can transparently convert your not-vendored-anything project into an equivalent-to-vendored-everything project without immensely bloating each git repo with duplicates.

pjmlp 7y ago

While I agree with you, we in the Java and .NET communities enjoy decentralized library distributions, so what most Java/.NET shops end up doing is setting their Nexus/NuGET servers in-house, and the proxies that care that all teams actually use those servers. Just like vendoring, in a single place, and all teams profit from the speed of access. The only downside is the update process to get IT to add new libraries to it, as they tend to require license validations from legal department.

weitzj 7y ago

I would also say: vendor your tools. There is a bit of a pattern emerging, where you add a „tools“ packages and this tools package just contains blank _ imports to external Go tools you use (e.g. gometalinter) This way when you run god mod vendor, you will also be able to vendor your tools and build them locally from your repo. This works quite well with a Makefile for now. There is an open issue tracking this vendor-tools approach