15 ms·

Abusing Go's Infrastructure

381 points by efge 2y ago

arccy 2y ago
it's a known issue https://github.com/golang/go/issues/31866 https://github.com/golang/go/issues/31866
- yjftsjthsd-h 2y ago
  That fix would help with accidents, but wouldn't someone intentionally hoping doing it just add a .mod and .go file to the root?
  - jerf 2y ago
    How do you "fix" that at all? In the end, there is no definition of "a source control repository that is a Go module" that is robust to this sort of "attack"... although calling it an "attack" is kind of dubious, the reasons why this is a bad thing strike me as very strained and relatively weak. Mostly it hurts Google by hosting too much stuff, but, good luck bringing them down that way.
- oooyay 2y ago
  Color me unsurprised Marwan is on this issue. He and Aaron wrote Athens, Marwan wrote (to my knowledge) the first Go download protocol implementation that Athens is based on. This issue is kind of curious because Athens already uses the go mod download -json command mentioned as a preflight check for module verification. More or less, if the repo passes the go module commands understanding of a module then Athens will serve it. In more verboten terms: - a module version, pseudo version, or +incompatible must be able to be formulated - that module (and it's dependencies) must produce a valid checksum The checksum of modules just has to do with the current .mod and all files + recursively for each dependency. So, as the author pointed out you can have lots of space for arbitrary files by design so long as you have a basic go program.
kyrra 2y ago
Googler, opinions are my own. I know nothing about this space. I would hope the Go team collaborated with GCP and Drive, as hosting malicious files is something Google has to deal with all the time. This isn't much different from other endpoints Google already allows people to put random data on.
- vineyardmike 2y ago
  > I would hope the Go team collaborated with GCP and Drive Former Googler, I know nothing about the Go Dev Tools team, but Google collaborates in this way better than almost any massive company I've worked at or heard about from close friends. Google is really good at having a central team manage infrastructure, and share it across the company. As long as it's not a messenger app. Surely (pure guessing) the go team is using the internal blob store, and I think there is some internal-infra teams that handle abuse and file scanning automatically.
8organicbits 2y ago
I know pypi has some non-python projects as well. Python needs the ability to distribute wheels, which are compiled binaries, as the user may not be able to compile library code. Lots of that code is written in C, but Golang[1] is also possible. I can't find an example, but I believe I've seen this used for distributing applications (not libraries) as well. It's kinda cool to write some app in C, upload to pypi, and then ask users to install with `pip install`. [1] https://github.com/popatam/gopy_build_wheel_example https://github.com/popatam/gopy_build_wheel_example
- bee_rider 2y ago
  Hypothetically if they did try to add some requirement to use Python, people could just comply maliciously by providing the most minimal stub of Python code, right? Linux, but ls is written in Python. So it is probably better just to not play games.
  - Maxatar 2y ago
    You could embed the binary data in a Python string and then have the installer dump that string to a file.
- rfoo 2y ago
  pip install cmake or even proprietary binaries, pip install nvidia-cudnn-cu12
  - IshKebab 2y ago
    Yeah I copied CMake's idea of using PyPI and I also use it to distribute some pure Rust CLI tools using Maturin. It works really well. Pip is... well it's about on par with most other package managers, i.e. not great, not terrible, but it has some pretty huge advantages over any other software distribution method on Linux: * Very likely to be installed already on Linux and probably Mac too. * Doesn't require root to install. You can even have isolated installs via pyenv. * I don't have to ask anyone's permission to publish a package. * I only have to make one package. If any can think of a better option I'm all ears but until then I'm fairly happy with this hack.
verdverm 2y ago
CUE's module system is finally rolling out, MVS likes Go's, but built on OCI infra. If you are interested in dependency management systems, here are some links - proposal: https://github.com/cue-lang/proposal/tree/main/designs/modules.v3 https://github.com/cue-lang/proposal/tree/main/designs/modul... - custom registry: https://cuelang.org/docs/tutorial/working-with-a-custom-module-registry/ https://cuelang.org/docs/tutorial/working-with-a-custom-modu... - road map: https://github.com/orgs/cue-lang/projects/10/views/8 https://github.com/orgs/cue-lang/projects/10/views/8 - in 0.9.0-alpha-5, modules become enabled by default: https://github.com/cue-lang/cue/releases/tag/v0.9.0-alpha.5 https://github.com/cue-lang/cue/releases/tag/v0.9.0-alpha.5 For Go Sum, the Trillian project backs the transparency log: https://github.com/google/trillian https://github.com/google/trillian CUE plans to piggyback on the OCI options with attestations and such
- dgellow 2y ago
  What does that have to do with the linked article?
  - verdverm 2y ago
    The CUE team worked with the Go team on the module system. From these interactions, and community input, they decided against using a proxy like Go has. The "exploit" in the article was one of the reasons they made this decision, and chose to use OCI registries instead. The V1 proposal actually proposed using the same Go proxy servers as a stopgap, which received significant pushback from the community (I was probably the loudest voice against the idea). The Go team was supportive at the time, but this would have been exactly what OP talks about, having non-Go projects in the proxy/sumdb. So CUE's module design can be seen as an evolution on Go's, building on the good parts while addressing some of the shortcomings. Fun fact, CUE started as a fork of Go, mainly for the internal compiler tooling and packages
    - infogulch 2y ago
      One thing that the Go module system solves that seems to be unaddressed in CUE's design based on OCI is the sum database / transparency log. I could add a "Statement that we might wish to make for a module M" to the "Module contents assurance" section: - The content of module M is the same content that everyone else sees for the same `$path@$version`. Though I guess users can utilize existing solutions like https://github.com/sigstore/cosign https://github.com/sigstore/cosign or rekor (mentioned elsewhere itt).
miki123211 2y ago
Any online service that lets users upload material that is then publicly visible will eventually be used for command-and-control, copyright infringement and hosting CSAM. This is especially true for services that have other important uses besides file hosting and hence are hard to block. This already happened to Twitter[1], Telegram[2], and even the PGP key infrastructure[3], not to mention obvious suspects like GitHub. [1] https://pentestlab.blog/2017/09/26/command-and-control-twitter/ https://pentestlab.blog/2017/09/26/command-and-control-twitt... [2] https://www.blazeinfosec.com/post/leveraging-telegram-as-a-command-control/ https://www.blazeinfosec.com/post/leveraging-telegram-as-a-c... [3] https://torrentfreak.com/openpgp-keyservers-now-store-irremovable-torrent-magnet-links-180907/ https://torrentfreak.com/openpgp-keyservers-now-store-irremo...
- deleted 2y ago
  [deleted]
- liquidgecka 2y ago
  And Gmail and Google groups, and Google drive, and Gchat, on and on. The data you store doesn't even have to be public. With Gmail they would distribute credentials to log in and read attachments that they uploaded via imap. (I am a former Google SAD-SRE [Spam, Abuse, Delivery])
  - zepolen 2y ago
    Question, how would you know without invading the user's privacy?
    - kccqzy 2y ago
      An algorithm that processes private user data is by itself not invading anyone's privacy. It's clear to me that invasion of privacy only happens when humans look at private user data directly, or look at user data that's not sufficiently processed by an algorithm. Otherwise, something as simple as a spell checker would be an invasion of privacy because it literally looks at every word in an email you write. That's absurd.
ithkuil 2y ago
I toyed with the idea of piggybacking on (i.e. abusing) the golang proxy and sumdb to have a free transparent log of checksums of arbitrary URLs https://getsum.pub/ https://getsum.pub/
- arccy 2y ago
  sounds convoluted. If you just want a public transparency log, the public rekor instance under the sigstore project is much more appropriate for that. https://www.sigstore.dev/ https://www.sigstore.dev/ https://docs.sigstore.dev/logging/overview/ https://docs.sigstore.dev/logging/overview/
  - lpapez 2y ago
    Sure, but the gosum database is a critical piece of worldwide software infrastructure, so you can count on it being accesible behind many firewalls and always up. And it's completely free and anonymus. Perfect for the purpose.
  - ithkuil 2y ago
    Yeah when I did that there was no public rekor instance ran by the sigstore project so I choose the only available public transparency log I could bend to my needs (x509 transparency logs were an alternative but it'd quickly hit rate limits by acme providers)
  - skybrian 2y ago
    Interesting! Looks like it's being used by some npm packages [1] and soon homebrew will be using it [2]. Any other interesting usage? As a user, the npm usage doesn't seem very prominent. On an npm's web page, there's a checkmark next to the version number on the right side that I hadn't paid any attention to before, with more information at the very bottom of the page. Here's an example. [3] [1] https://blog.sigstore.dev/npm-provenance-ga/ https://blog.sigstore.dev/npm-provenance-ga/ [2] https://blog.sigstore.dev/homebrew-build-provenance/ https://blog.sigstore.dev/homebrew-build-provenance/ [3] https://www.npmjs.com/package/fast-check https://www.npmjs.com/package/fast-check
9dev 2y ago
Off topic: took a look at the domain, had a foreboding on the innuendo, found mostly what I expected on put.as …
- moonlion_eth 2y ago
  I made the mistake of doing that at work
- yazzku 2y ago
  Not off-topic at all; came here just for this.
- KolmogorovComp 2y ago
  https://put.as/ https://put.as/ Mildly NSFW.
- mdtrooper 2y ago
  Yes, it is a spanish plural word. But....
IshKebab 2y ago
Maybe I'm being stupid but what exactly is the issue here? It's probably a bit wasteful of the proxy to cache non-Go repos, but even if it didn't you could make it store arbitrary data just by having it cache a Go repo surely? Sounds like a complete non-issue unless I've missed something.
- arandomusername 2y ago
  you're right.
- gnfargbl 2y ago
  I don't think you've missed anything. The news here appears to be that a unsecured public proxy is willing to proxy things and make them available to the public in an unsecured fashion. The article does make the point that some monitored networks might trust golang proxy URLs more than arbitrary web URLs and that this could be used for bypassing reputation filters etc -- but there are already several ways to do that, and this one doesn't seem particularly special.
ConchiGambero 2y ago
[dead]
palata 2y ago
That's maybe naive, but... how is that different than just pushing files to e.g. a GitHub repository? Is it just the fact that you need to create an account for GitHub? Because I can store arbitrary data there, too. Without the 500M limit...
- withinboredom 2y ago
  GitHub has some pretty stringent rate limits for anon requests.
erik_seaberg 2y ago
W3C laid the groundwork for everything on the Web to be heavily cacheable, so it's weird that there are so few general-purpose proxy caches. Are publishers sending short "Cache-Control: max-age" or "Vary: Cookie" responses when they didn't need to? Are too many ISPs paying for transit rather than peering?
- lmz 2y ago
  In general there's no way to ensure the cache hasn't tampered with the contents (e.g. ISP proxy ad injection on non HTTPS sites). For software downloads usually there are signatures and checksums. Arbitrary content, not so much.
  - arccy 2y ago
    There was HTTP SXG (signed exchanges) but it never seemed to get any traction https://web.dev/articles/signed-exchanges https://web.dev/articles/signed-exchanges
    - iainmerrick 2y ago
      Even if the content is signed, there’s still the issue that the proxy gets to see everything you read, right?
      - arccy 2y ago
        that's the point of a proxy that can share contents between clients...
  - yencabulator 2y ago
    Maybe use cache only when Subresource Integrity is present. https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity https://developer.mozilla.org/en-US/docs/Web/Security/Subres...

381 points by efge 2y ago

arccy 2y ago

it's a known issue https://github.com/golang/go/issues/31866 https://github.com/golang/go/issues/31866

yjftsjthsd-h 2y ago
That fix would help with accidents, but wouldn't someone intentionally hoping doing it just add a .mod and .go file to the root?
- jerf 2y ago
  How do you "fix" that at all? In the end, there is no definition of "a source control repository that is a Go module" that is robust to this sort of "attack"... although calling it an "attack" is kind of dubious, the reasons why this is a bad thing strike me as very strained and relatively weak. Mostly it hurts Google by hosting too much stuff, but, good luck bringing them down that way.
oooyay 2y ago
Color me unsurprised Marwan is on this issue. He and Aaron wrote Athens, Marwan wrote (to my knowledge) the first Go download protocol implementation that Athens is based on. This issue is kind of curious because Athens already uses the go mod download -json command mentioned as a preflight check for module verification. More or less, if the repo passes the go module commands understanding of a module then Athens will serve it. In more verboten terms: - a module version, pseudo version, or +incompatible must be able to be formulated - that module (and it's dependencies) must produce a valid checksum The checksum of modules just has to do with the current .mod and all files + recursively for each dependency. So, as the author pointed out you can have lots of space for arbitrary files by design so long as you have a basic go program.

kyrra 2y ago

Googler, opinions are my own. I know nothing about this space. I would hope the Go team collaborated with GCP and Drive, as hosting malicious files is something Google has to deal with all the time. This isn't much different from other endpoints Google already allows people to put random data on.

vineyardmike 2y ago
> I would hope the Go team collaborated with GCP and Drive Former Googler, I know nothing about the Go Dev Tools team, but Google collaborates in this way better than almost any massive company I've worked at or heard about from close friends. Google is really good at having a central team manage infrastructure, and share it across the company. As long as it's not a messenger app. Surely (pure guessing) the go team is using the internal blob store, and I think there is some internal-infra teams that handle abuse and file scanning automatically.

8organicbits 2y ago

I know pypi has some non-python projects as well. Python needs the ability to distribute wheels, which are compiled binaries, as the user may not be able to compile library code. Lots of that code is written in C, but Golang[1] is also possible. I can't find an example, but I believe I've seen this used for distributing applications (not libraries) as well. It's kinda cool to write some app in C, upload to pypi, and then ask users to install with `pip install`. [1] https://github.com/popatam/gopy_build_wheel_example https://github.com/popatam/gopy_build_wheel_example

bee_rider 2y ago
Hypothetically if they did try to add some requirement to use Python, people could just comply maliciously by providing the most minimal stub of Python code, right? Linux, but ls is written in Python. So it is probably better just to not play games.
- Maxatar 2y ago
  You could embed the binary data in a Python string and then have the installer dump that string to a file.
rfoo 2y ago
pip install cmake or even proprietary binaries, pip install nvidia-cudnn-cu12
- IshKebab 2y ago
  Yeah I copied CMake's idea of using PyPI and I also use it to distribute some pure Rust CLI tools using Maturin. It works really well. Pip is... well it's about on par with most other package managers, i.e. not great, not terrible, but it has some pretty huge advantages over any other software distribution method on Linux: * Very likely to be installed already on Linux and probably Mac too. * Doesn't require root to install. You can even have isolated installs via pyenv. * I don't have to ask anyone's permission to publish a package. * I only have to make one package. If any can think of a better option I'm all ears but until then I'm fairly happy with this hack.

verdverm 2y ago

CUE's module system is finally rolling out, MVS likes Go's, but built on OCI infra. If you are interested in dependency management systems, here are some links - proposal: https://github.com/cue-lang/proposal/tree/main/designs/modules.v3 https://github.com/cue-lang/proposal/tree/main/designs/modul... - custom registry: https://cuelang.org/docs/tutorial/working-with-a-custom-module-registry/ https://cuelang.org/docs/tutorial/working-with-a-custom-modu... - road map: https://github.com/orgs/cue-lang/projects/10/views/8 https://github.com/orgs/cue-lang/projects/10/views/8 - in 0.9.0-alpha-5, modules become enabled by default: https://github.com/cue-lang/cue/releases/tag/v0.9.0-alpha.5 https://github.com/cue-lang/cue/releases/tag/v0.9.0-alpha.5 For Go Sum, the Trillian project backs the transparency log: https://github.com/google/trillian https://github.com/google/trillian CUE plans to piggyback on the OCI options with attestations and such

dgellow 2y ago
What does that have to do with the linked article?
- verdverm 2y ago
  The CUE team worked with the Go team on the module system. From these interactions, and community input, they decided against using a proxy like Go has. The "exploit" in the article was one of the reasons they made this decision, and chose to use OCI registries instead. The V1 proposal actually proposed using the same Go proxy servers as a stopgap, which received significant pushback from the community (I was probably the loudest voice against the idea). The Go team was supportive at the time, but this would have been exactly what OP talks about, having non-Go projects in the proxy/sumdb. So CUE's module design can be seen as an evolution on Go's, building on the good parts while addressing some of the shortcomings. Fun fact, CUE started as a fork of Go, mainly for the internal compiler tooling and packages
  - infogulch 2y ago
    One thing that the Go module system solves that seems to be unaddressed in CUE's design based on OCI is the sum database / transparency log. I could add a "Statement that we might wish to make for a module M" to the "Module contents assurance" section: - The content of module M is the same content that everyone else sees for the same `$path@$version`. Though I guess users can utilize existing solutions like https://github.com/sigstore/cosign https://github.com/sigstore/cosign or rekor (mentioned elsewhere itt).

miki123211 2y ago

Any online service that lets users upload material that is then publicly visible will eventually be used for command-and-control, copyright infringement and hosting CSAM. This is especially true for services that have other important uses besides file hosting and hence are hard to block. This already happened to Twitter[1], Telegram[2], and even the PGP key infrastructure[3], not to mention obvious suspects like GitHub. [1] https://pentestlab.blog/2017/09/26/command-and-control-twitter/ https://pentestlab.blog/2017/09/26/command-and-control-twitt... [2] https://www.blazeinfosec.com/post/leveraging-telegram-as-a-command-control/ https://www.blazeinfosec.com/post/leveraging-telegram-as-a-c... [3] https://torrentfreak.com/openpgp-keyservers-now-store-irremovable-torrent-magnet-links-180907/ https://torrentfreak.com/openpgp-keyservers-now-store-irremo...

deleted 2y ago
[deleted]
liquidgecka 2y ago
And Gmail and Google groups, and Google drive, and Gchat, on and on. The data you store doesn't even have to be public. With Gmail they would distribute credentials to log in and read attachments that they uploaded via imap. (I am a former Google SAD-SRE [Spam, Abuse, Delivery])
- zepolen 2y ago
  Question, how would you know without invading the user's privacy?
  - kccqzy 2y ago
    An algorithm that processes private user data is by itself not invading anyone's privacy. It's clear to me that invasion of privacy only happens when humans look at private user data directly, or look at user data that's not sufficiently processed by an algorithm. Otherwise, something as simple as a spell checker would be an invasion of privacy because it literally looks at every word in an email you write. That's absurd.

ithkuil 2y ago

I toyed with the idea of piggybacking on (i.e. abusing) the golang proxy and sumdb to have a free transparent log of checksums of arbitrary URLs https://getsum.pub/ https://getsum.pub/

arccy 2y ago
sounds convoluted. If you just want a public transparency log, the public rekor instance under the sigstore project is much more appropriate for that. https://www.sigstore.dev/ https://www.sigstore.dev/ https://docs.sigstore.dev/logging/overview/ https://docs.sigstore.dev/logging/overview/
- lpapez 2y ago
  Sure, but the gosum database is a critical piece of worldwide software infrastructure, so you can count on it being accesible behind many firewalls and always up. And it's completely free and anonymus. Perfect for the purpose.
- ithkuil 2y ago
  Yeah when I did that there was no public rekor instance ran by the sigstore project so I choose the only available public transparency log I could bend to my needs (x509 transparency logs were an alternative but it'd quickly hit rate limits by acme providers)
- skybrian 2y ago
  Interesting! Looks like it's being used by some npm packages [1] and soon homebrew will be using it [2]. Any other interesting usage? As a user, the npm usage doesn't seem very prominent. On an npm's web page, there's a checkmark next to the version number on the right side that I hadn't paid any attention to before, with more information at the very bottom of the page. Here's an example. [3] [1] https://blog.sigstore.dev/npm-provenance-ga/ https://blog.sigstore.dev/npm-provenance-ga/ [2] https://blog.sigstore.dev/homebrew-build-provenance/ https://blog.sigstore.dev/homebrew-build-provenance/ [3] https://www.npmjs.com/package/fast-check https://www.npmjs.com/package/fast-check

9dev 2y ago

Off topic: took a look at the domain, had a foreboding on the innuendo, found mostly what I expected on put.as …

moonlion_eth 2y ago
I made the mistake of doing that at work
yazzku 2y ago
Not off-topic at all; came here just for this.
KolmogorovComp 2y ago
https://put.as/ https://put.as/ Mildly NSFW.
mdtrooper 2y ago
Yes, it is a spanish plural word. But....

IshKebab 2y ago

Maybe I'm being stupid but what exactly is the issue here? It's probably a bit wasteful of the proxy to cache non-Go repos, but even if it didn't you could make it store arbitrary data just by having it cache a Go repo surely? Sounds like a complete non-issue unless I've missed something.

arandomusername 2y ago
you're right.
gnfargbl 2y ago
I don't think you've missed anything. The news here appears to be that a unsecured public proxy is willing to proxy things and make them available to the public in an unsecured fashion. The article does make the point that some monitored networks might trust golang proxy URLs more than arbitrary web URLs and that this could be used for bypassing reputation filters etc -- but there are already several ways to do that, and this one doesn't seem particularly special.

ConchiGambero 2y ago

[dead]

palata 2y ago

That's maybe naive, but... how is that different than just pushing files to e.g. a GitHub repository? Is it just the fact that you need to create an account for GitHub? Because I can store arbitrary data there, too. Without the 500M limit...

withinboredom 2y ago
GitHub has some pretty stringent rate limits for anon requests.

erik_seaberg 2y ago

W3C laid the groundwork for everything on the Web to be heavily cacheable, so it's weird that there are so few general-purpose proxy caches. Are publishers sending short "Cache-Control: max-age" or "Vary: Cookie" responses when they didn't need to? Are too many ISPs paying for transit rather than peering?

lmz 2y ago
In general there's no way to ensure the cache hasn't tampered with the contents (e.g. ISP proxy ad injection on non HTTPS sites). For software downloads usually there are signatures and checksums. Arbitrary content, not so much.
- arccy 2y ago
  There was HTTP SXG (signed exchanges) but it never seemed to get any traction https://web.dev/articles/signed-exchanges https://web.dev/articles/signed-exchanges
  - iainmerrick 2y ago
    Even if the content is signed, there’s still the issue that the proxy gets to see everything you read, right?
    - arccy 2y ago
      that's the point of a proxy that can share contents between clients...
- yencabulator 2y ago
  Maybe use cache only when Subresource Integrity is present. https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity https://developer.mozilla.org/en-US/docs/Web/Security/Subres...