4 ms·

>If the main module imports example.com/mod1/pkg1 and a separate example.com/mod1/pkg2 imports example.com/mod2, there is no way for example.com/mod2 to affect

by Groxx 10mo ago

>If the main module imports example.com/mod1/pkg1 and a separate example.com/mod1/pkg2 imports example.com/mod2, there is no way for example.com/mod2 to affect the build or run code on the developer’s machine, so you don’t need to consider it a dependency. iirc (I do not have a test setup at the moment to verify) it does affect your dependency resolution, and therefore your build, though its code does not exist in your binary. I know this because those AWS / K8S / Google Cloud libraries cause MASSIVE problems with their constant breaking changes without major version changes, and their importing other libraries that also have frequent breaking changes without major version changes, even if the dependency those unused subpackages include (and therefore raise the minimum version) is only used by some other module that needs a lower version (iirc). It's quite a headache sometimes, and could be rather easily solved if you could set upper bounds and not just lower. Or if those giga-projects would stop doing such obviously bad things. The version-affecting behavior is kinda unavoidable afaict. If they didn't include those unused version constraints, `go build ...` or just importing a new package within existing modules could cause your build to fail, forcing you to rerun version resolution. That'd probably just lead people to feel like "go is broken UGH", and go leans awfully hard towards avoiding that kind of thing. Mostly for the better, but not quite always / not in all ways.

nasretdinov 10mo ago
These libraries are honestly so bad that whenever I have to interact with them I just split those dependencies into a separate binary :)
- supriyo-biswas 10mo ago
  I just write my own, small, focused client for these AWS services since the SDKs are generally just so unergonomic. With vibe coding becoming a thing it's become even easier to do that.
  - kbolino 10mo ago
    You may already know these things, but for others who may not: the AWS SDK for Go suffers from shockingly bad discoverability. For example, take the S3 library, github.com/aws/aws-sdk-go-v2/service/s3. If you have an s3.Client and you look at e.g. the ListObjectsV2 method, you might have no idea that there is a ListObjectsV2Paginator which makes it much easier to use, because nowhere in the method docs is it mentioned. Indeed, most operations that paginate have more ergonomic paginators, but none of them tell you this. But that isn't even the worst of it. Say you want to download or upload a file to S3. If you haven't worked with AWS for other languages, you might think that you just do GetObject and PutObject. And yes, for small files, that's generally fine. But for large files you want to use resumable downloads and multipart uploads. So you look and lo, there is no simple way to do this in the AWS SDK for Go. But actually, there is! It's in a totally unrelated and unlinked package, called github.com/aws/aws-sdk-go-v2/feature/s3/manager. Now you're getting some religion, so you ask "what are the other so-called 'feature' packages?" and you try to browse pkg.go.dev at the github.com/aws/aws-sdk-go-v2/feature level but nope, that's not a Go module so there's nothing to see there. In fact, you can't even browse the other s3 features, never mind find out what other services have features. Fortunately, you can browse their GitHub repo at least: https://github.com/aws/aws-sdk-go-v2/tree/main/feature https://github.com/aws/aws-sdk-go-v2/tree/main/feature It's quite clear that they use poorly thought-out cross-language codegen for this, which partly explains the lack of ergonomics, but also shows that they don't much care whether you use their stuff properly.
    - jamesfinlayson 10mo ago
      Yeah the AWS PHP SDK is similarly bad. The Java SDK is okay but still completely lacking in examples for anything outside of the basics. AWS seems to optimise for an SDK that be completely generated but not for an SDK that tells you what you want to know.
- onionisafruit 10mo ago
  I dabbled in Kubernetes at work a few years ago. I was shocked at the state of their go.mod files. Especially because I had heard Kubernetes used as an example of great Go software. Right off the bat I needed to copy a dozen or so `replace` statements into my go.mod. I was sure I just didn’t understand something and spent an hour or so looking for the “right way” only to discover other open source controllers that had the same replaces in their go.mods. Maybe that was a transitory stage and it’s straightened out now. I certainly hope so.
  - cpuguy83 10mo ago
    Definitely do not look at Kubernetes as a good example of go code and especially not how it handles deps (btw, it only relatively recently switched to go mods). Not to say it is a bad project. Not at all.
  - jerf 10mo ago
    I think a lot of people assume Kubernetes must be a good Go example, because it's so big and successful. But it started life as a Java project and was ported into Go, and that shows in some of the architecture, Also, in general across all languages, picking the absolute largest projects you can is often not a great idea in terms of copying design unless you too are going to be that large, e.g., I wouldn't suggest using Firefox as an example of C++ necessarily. Such projects always end up developing solutions to problems you will never have, and solving problems we don't have is one of the most common mistakes software developers make.
    - nullpoint420 10mo ago
      I like your comparison about Firefox not being a great example of "good" C/C++. I'd personally consider myself comfortable in C/C++. I've built Wayland compositors, H264 backends for live-streaming, and built Chromium occasionally for testing. Despite being a die-hard Firefox (zen) user - I still have not been able to compile Firefox! To be fair, this was pre-Firefox quantum days so I hope their SVM and build tools have improved.
    - lokar 10mo ago
nvarsj 10mo ago
godep and vendoring completely solved this problem. Go modules relies too heavily on dependencies being good citizens, which is a very naive approach to dependency management.
- quacker 10mo ago
  You can still do vendoring with Go modules, using the `go mod vendor` command.
  - nvarsj 10mo ago
    I'm a bit out of date now, but 'go mod vendor' did not in fact vendor all dependencies when I last used it. It seemed more like a CI caching mechanism then actual complete dependency vendoring.
    - majewsky 10mo ago
      This is not the experience that I have. I use vendoring heavily because it's nice to be able to review all the relevant code changes when Renovatebot updates a dependency [1], and to get a feel for how heavy or lightweight certain dependencies are. If vendoring was incomplete, I would see it trying to download missing dependencies at compile time, but the "go: downloading $MODULE $VERSION" lines only show up when I expect them too, e.g. during "go get -u" or "go mod tidy/vendor". [1] Before you ask, I'm not reading the full diff on something like x/sys. Mostly on third-party dependencies where I find it harder to judge the reliability of the maintainers.
      - Groxx 9mo ago
        I'll throw in a +1, afaik vendoring is a complete and reliable solution here. Go's `mod`-related commands have had quite a large number of breaking changes in behavior through the years, so I won't say it's a stable solution (aside from completely disabling modules, that has always worked fine from my use). I've had to fix build scripts at least once or twice a year due to that, e.g. when they started validating that the vendor folder was unmodified (a very reasonable thing to do, but still breaking). But once that is overcome it has always worked fine.
FiloSottile 10mo ago
It's tricky, to the point that I made a little playground to explore it. https://github.com/FiloSottile/mostly-harmless/tree/main/deps https://github.com/FiloSottile/mostly-harmless/tree/main/dep... The example.com/mod2 go.mod does not in fact affect version resolution, because it's not even fetched. However, it affects the example.com/mod1 go.mod, and the example.com/mod1 go.mod affects version resolution. This doesn't help with the problem you are describing, but it still has value from a security point of view, because example.com/mod2 truly doesn't matter except to the extent that was already checked into example.com/mod1, which you do need to trust. If you try to "go build" or "go test" something in example.com/mod2, you actually do get an error since Go 1.17, as if it was not in your dependency tree at all. You need to "go get" it like any new dependency.

by Groxx 10mo ago

nasretdinov 10mo ago

These libraries are honestly so bad that whenever I have to interact with them I just split those dependencies into a separate binary :)

supriyo-biswas 10mo ago
I just write my own, small, focused client for these AWS services since the SDKs are generally just so unergonomic. With vibe coding becoming a thing it's become even easier to do that.
- kbolino 10mo ago
  You may already know these things, but for others who may not: the AWS SDK for Go suffers from shockingly bad discoverability. For example, take the S3 library, github.com/aws/aws-sdk-go-v2/service/s3. If you have an s3.Client and you look at e.g. the ListObjectsV2 method, you might have no idea that there is a ListObjectsV2Paginator which makes it much easier to use, because nowhere in the method docs is it mentioned. Indeed, most operations that paginate have more ergonomic paginators, but none of them tell you this. But that isn't even the worst of it. Say you want to download or upload a file to S3. If you haven't worked with AWS for other languages, you might think that you just do GetObject and PutObject. And yes, for small files, that's generally fine. But for large files you want to use resumable downloads and multipart uploads. So you look and lo, there is no simple way to do this in the AWS SDK for Go. But actually, there is! It's in a totally unrelated and unlinked package, called github.com/aws/aws-sdk-go-v2/feature/s3/manager. Now you're getting some religion, so you ask "what are the other so-called 'feature' packages?" and you try to browse pkg.go.dev at the github.com/aws/aws-sdk-go-v2/feature level but nope, that's not a Go module so there's nothing to see there. In fact, you can't even browse the other s3 features, never mind find out what other services have features. Fortunately, you can browse their GitHub repo at least: https://github.com/aws/aws-sdk-go-v2/tree/main/feature https://github.com/aws/aws-sdk-go-v2/tree/main/feature It's quite clear that they use poorly thought-out cross-language codegen for this, which partly explains the lack of ergonomics, but also shows that they don't much care whether you use their stuff properly.
  - jamesfinlayson 10mo ago
    Yeah the AWS PHP SDK is similarly bad. The Java SDK is okay but still completely lacking in examples for anything outside of the basics. AWS seems to optimise for an SDK that be completely generated but not for an SDK that tells you what you want to know.
onionisafruit 10mo ago
I dabbled in Kubernetes at work a few years ago. I was shocked at the state of their go.mod files. Especially because I had heard Kubernetes used as an example of great Go software. Right off the bat I needed to copy a dozen or so `replace` statements into my go.mod. I was sure I just didn’t understand something and spent an hour or so looking for the “right way” only to discover other open source controllers that had the same replaces in their go.mods. Maybe that was a transitory stage and it’s straightened out now. I certainly hope so.
- cpuguy83 10mo ago
  Definitely do not look at Kubernetes as a good example of go code and especially not how it handles deps (btw, it only relatively recently switched to go mods). Not to say it is a bad project. Not at all.
- jerf 10mo ago
  I think a lot of people assume Kubernetes must be a good Go example, because it's so big and successful. But it started life as a Java project and was ported into Go, and that shows in some of the architecture, Also, in general across all languages, picking the absolute largest projects you can is often not a great idea in terms of copying design unless you too are going to be that large, e.g., I wouldn't suggest using Firefox as an example of C++ necessarily. Such projects always end up developing solutions to problems you will never have, and solving problems we don't have is one of the most common mistakes software developers make.
  - nullpoint420 10mo ago
    I like your comparison about Firefox not being a great example of "good" C/C++. I'd personally consider myself comfortable in C/C++. I've built Wayland compositors, H264 backends for live-streaming, and built Chromium occasionally for testing. Despite being a die-hard Firefox (zen) user - I still have not been able to compile Firefox! To be fair, this was pre-Firefox quantum days so I hope their SVM and build tools have improved.
  - lokar 10mo ago

nvarsj 10mo ago

godep and vendoring completely solved this problem. Go modules relies too heavily on dependencies being good citizens, which is a very naive approach to dependency management.

quacker 10mo ago
You can still do vendoring with Go modules, using the `go mod vendor` command.
- nvarsj 10mo ago
  I'm a bit out of date now, but 'go mod vendor' did not in fact vendor all dependencies when I last used it. It seemed more like a CI caching mechanism then actual complete dependency vendoring.
  - majewsky 10mo ago
    This is not the experience that I have. I use vendoring heavily because it's nice to be able to review all the relevant code changes when Renovatebot updates a dependency [1], and to get a feel for how heavy or lightweight certain dependencies are. If vendoring was incomplete, I would see it trying to download missing dependencies at compile time, but the "go: downloading $MODULE $VERSION" lines only show up when I expect them too, e.g. during "go get -u" or "go mod tidy/vendor". [1] Before you ask, I'm not reading the full diff on something like x/sys. Mostly on third-party dependencies where I find it harder to judge the reliability of the maintainers.
    - Groxx 9mo ago
      I'll throw in a +1, afaik vendoring is a complete and reliable solution here. Go's `mod`-related commands have had quite a large number of breaking changes in behavior through the years, so I won't say it's a stable solution (aside from completely disabling modules, that has always worked fine from my use). I've had to fix build scripts at least once or twice a year due to that, e.g. when they started validating that the vendor folder was unmodified (a very reasonable thing to do, but still breaking). But once that is overcome it has always worked fine.

FiloSottile 10mo ago

It's tricky, to the point that I made a little playground to explore it. https://github.com/FiloSottile/mostly-harmless/tree/main/deps https://github.com/FiloSottile/mostly-harmless/tree/main/dep... The example.com/mod2 go.mod does not in fact affect version resolution, because it's not even fetched. However, it affects the example.com/mod1 go.mod, and the example.com/mod1 go.mod affects version resolution. This doesn't help with the problem you are describing, but it still has value from a security point of view, because example.com/mod2 truly doesn't matter except to the extent that was already checked into example.com/mod1, which you do need to trust. If you try to "go build" or "go test" something in example.com/mod2, you actually do get an error since Go 1.17, as if it was not in your dependency tree at all. You need to "go get" it like any new dependency.