4 ms·

In what world do you have a machine which downloads source code to build it, but doesn't have outbound internet access so it can't download source code or build

In what world do you have a machine which downloads source code to build it, but doesn't have outbound internet access so it can't download source code or build dependencies? Like, effectively the "build machine" here is a locked down docker container that runs "git clone && npm build", right? How do you do either of those activities without outbound network access? And outbound network access is enough on its own to create a reverse shell, even without any open inbound ports. The miss here isn't that the build container had network access, it's that the build container both ran untrusted code, and had access to secrets.

arccy 2y ago
It's common, doesn't mean it's secure. A lot of linux distros in their packaging will separate download (allows outbound to fetch dependencies), from build (no outside access). Unfortunately, in some ecosystems, even downloading packages using the native package managers is unsafe because of postinstall scripts or equivalent.
- zahlman 2y ago
  >Unfortunately, in some ecosystems, even downloading packages using the native package managers is unsafe because of postinstall scripts or equivalent. Funny you should mention this because I was just psyching myself up to submit my blog piece from last night on the topic. In Python, downloading packages using the native package installer (Pip, which really doesn't itself do anything that could be called package management) is unsafe because of build scripts - unless you tell it to only accept pre-built packages, defeating the point of the systems these Linux distros are using. (I assume/hope people in this position are aware of the problem and have rigged up another solution with the API. In the post I commented that I don't know of such solutions being publicly available, but surely they exist somewhere.) You'd be justified in wondering why the build script runs when you only ask to download the package. It's mainly because of the historically atrocious approach to metadata (and all the legacy packages for which installation is still supported). But from reading the issue trackers, it seems like the code paths aren't especially easy to disentangle, either - since they've gone so long with the assumption baked in that the problem isn't really solvable. In other HN posts I've complained about people pointing out things in the Python packaging ecosystem that aren't really problems. But this really is one. https://zahlman.github.io/posts/2025/02/28/python-packaging-3/ https://zahlman.github.io/posts/2025/02/28/python-packaging-...
areyourllySorry 2y ago
you can always limit said network access to npm.
- TheDong 2y ago
  You can't since a large number of npm post-install scripts also make random arbitrary network calls. This includes things like downloading and compiling pre-compiled binaries for the native architecture hosted on random servers. npm is really cool.
  - cdmyrm 2y ago
    [flagged]
katbyte 2y ago
you use a language where you have all your deps local to the repo? ie go vendor?
ndriscoll 2y ago
Even if your builders are downloading dependencies on the fly, you can and should force that through an artifact repository (e.g. artifactory) you control. They shouldn't need arbitrary outbound Internet access. The builder needs a token injected with read-only pull permissions for a write-through cache and push permissions to the path it is currently building for. The only thing it needs to talk to is the artifactory instance.
fc417fc802 2y ago
If you don't network isolate your build tooling then how do you have any confidence that your inputs are what you believe them to be? I run my build tools in a network namespace with no connection to the outside world. The dependencies are whatever I explicitly checked into the repo or otherwise placed within the directory tree.
- TheDong 2y ago
  You don't have any confidence beyond what lockfiles give you (which is to say the npm postinstall scripts could be very impure, non-hermetic, and output random strings). But if you require users to vendor all their dependencies, fully isolate all network traffic during build, be perfectly pure and reproducible and hermetic, presumably use nix/bazel/etc... well, you won't have any users. If you want a perfectly secure system with 0 users, it's pretty easy to build that.
  - fc417fc802 2y ago
    > But if you require users I'm not suggesting that a commercial service should require this. You asked "In what world do you have ..." and I'm pointing out that it's actually a fairly common practice. Particularly in any security conscious environment. Anyone not doing it is cutting corners to save time, which to be clear isn't always a bad thing. There's nothing wrong if my small personal website doesn't have a network isolated fully reproducible build. On the other hand, any widely distributed binaries definitely should. For example, I fully expect that my bank uses network isolated builds for their website. They are an absolutely massive target after all.
  - pixl97 2y ago
    Most banks and larger enterprises do exactly this. Devs don't get to go out and pick random libraries with out a code review and then it's placed on a local repository. There are just far too many insecure and 'typo' malware to pull off the internet raw.
    - NegativeK 2y ago
mcoliver 2y ago
There are plenty of worlds that take security more seriously and practice defense in depth. Your response could use a little less hubris and a more genuinely inquisitive tone. Looks like others have already chimed in here but to respond to your (what feels like sarcasm) questions: - You can have a submission process that accepts a package or downloads dependencies, and then passes it to another machine that is on an isolated network for code execution / build which then returns the built package and logs to the network facing machine for consumption. Now sure if your build machine is still exposing everything on it to the user supplied code (instead of sandboxing the actual npm build/make/etc.. command) you could insert malicious code that zips up the whole filesystem, env vars, etc.. and exfiltrates them through your built app in this case snagging the secrets. I don't disagree that the secrets on the build machine were the big miss, but I also think designing the build system differently could have helped.
- TheDong 2y ago
  You have to meet your users where they are. Your users are not using nix and bazel, they're using npm and typescript. If your users are using bazel, it's easy to separate "download" from "build", but if you're meeting your users over here where cows aren't spherical, you can't take security that seriously. Security doesn't help if all your users leave.
  - jasonkester 2y ago
    The simple solution would be to check your node-modules folder into source control. Then your build machine wouldn’t need to download anything from anywhere except your repository.
gtirloni 2y ago
In a world with an internal proxy/mirror for dependencies and no internet access allowed by build systems.
- dboreham 2y ago
  Which is not the world we live in.
  - gtirloni 2y ago
    s/we/I/
  - cdmyrm 2y ago
    [flagged]
cdmyrm 2y ago
It's called air-gapping, and lots of adults do it.
zahlman 2y ago
> The miss here isn't that the build container had network access, it's that the build container both ran untrusted code, and had access to secrets. If you're providing a build container service then you pretty much have to run untrusted code (the customer's) in the container, yes? So then the problem is really just the bad Firebase config... ?

arccy 2y ago

It's common, doesn't mean it's secure. A lot of linux distros in their packaging will separate download (allows outbound to fetch dependencies), from build (no outside access). Unfortunately, in some ecosystems, even downloading packages using the native package managers is unsafe because of postinstall scripts or equivalent.

zahlman 2y ago
>Unfortunately, in some ecosystems, even downloading packages using the native package managers is unsafe because of postinstall scripts or equivalent. Funny you should mention this because I was just psyching myself up to submit my blog piece from last night on the topic. In Python, downloading packages using the native package installer (Pip, which really doesn't itself do anything that could be called package management) is unsafe because of build scripts - unless you tell it to only accept pre-built packages, defeating the point of the systems these Linux distros are using. (I assume/hope people in this position are aware of the problem and have rigged up another solution with the API. In the post I commented that I don't know of such solutions being publicly available, but surely they exist somewhere.) You'd be justified in wondering why the build script runs when you only ask to download the package. It's mainly because of the historically atrocious approach to metadata (and all the legacy packages for which installation is still supported). But from reading the issue trackers, it seems like the code paths aren't especially easy to disentangle, either - since they've gone so long with the assumption baked in that the problem isn't really solvable. In other HN posts I've complained about people pointing out things in the Python packaging ecosystem that aren't really problems. But this really is one. https://zahlman.github.io/posts/2025/02/28/python-packaging-3/ https://zahlman.github.io/posts/2025/02/28/python-packaging-...

areyourllySorry 2y ago

you can always limit said network access to npm.

TheDong 2y ago
You can't since a large number of npm post-install scripts also make random arbitrary network calls. This includes things like downloading and compiling pre-compiled binaries for the native architecture hosted on random servers. npm is really cool.
- cdmyrm 2y ago
  [flagged]

katbyte 2y ago

you use a language where you have all your deps local to the repo? ie go vendor?

ndriscoll 2y ago

Even if your builders are downloading dependencies on the fly, you can and should force that through an artifact repository (e.g. artifactory) you control. They shouldn't need arbitrary outbound Internet access. The builder needs a token injected with read-only pull permissions for a write-through cache and push permissions to the path it is currently building for. The only thing it needs to talk to is the artifactory instance.

fc417fc802 2y ago

If you don't network isolate your build tooling then how do you have any confidence that your inputs are what you believe them to be? I run my build tools in a network namespace with no connection to the outside world. The dependencies are whatever I explicitly checked into the repo or otherwise placed within the directory tree.

TheDong 2y ago
You don't have any confidence beyond what lockfiles give you (which is to say the npm postinstall scripts could be very impure, non-hermetic, and output random strings). But if you require users to vendor all their dependencies, fully isolate all network traffic during build, be perfectly pure and reproducible and hermetic, presumably use nix/bazel/etc... well, you won't have any users. If you want a perfectly secure system with 0 users, it's pretty easy to build that.
- fc417fc802 2y ago
  > But if you require users I'm not suggesting that a commercial service should require this. You asked "In what world do you have ..." and I'm pointing out that it's actually a fairly common practice. Particularly in any security conscious environment. Anyone not doing it is cutting corners to save time, which to be clear isn't always a bad thing. There's nothing wrong if my small personal website doesn't have a network isolated fully reproducible build. On the other hand, any widely distributed binaries definitely should. For example, I fully expect that my bank uses network isolated builds for their website. They are an absolutely massive target after all.
- pixl97 2y ago
  Most banks and larger enterprises do exactly this. Devs don't get to go out and pick random libraries with out a code review and then it's placed on a local repository. There are just far too many insecure and 'typo' malware to pull off the internet raw.
  - NegativeK 2y ago

mcoliver 2y ago

There are plenty of worlds that take security more seriously and practice defense in depth. Your response could use a little less hubris and a more genuinely inquisitive tone. Looks like others have already chimed in here but to respond to your (what feels like sarcasm) questions: - You can have a submission process that accepts a package or downloads dependencies, and then passes it to another machine that is on an isolated network for code execution / build which then returns the built package and logs to the network facing machine for consumption. Now sure if your build machine is still exposing everything on it to the user supplied code (instead of sandboxing the actual npm build/make/etc.. command) you could insert malicious code that zips up the whole filesystem, env vars, etc.. and exfiltrates them through your built app in this case snagging the secrets. I don't disagree that the secrets on the build machine were the big miss, but I also think designing the build system differently could have helped.

TheDong 2y ago
You have to meet your users where they are. Your users are not using nix and bazel, they're using npm and typescript. If your users are using bazel, it's easy to separate "download" from "build", but if you're meeting your users over here where cows aren't spherical, you can't take security that seriously. Security doesn't help if all your users leave.
- jasonkester 2y ago
  The simple solution would be to check your node-modules folder into source control. Then your build machine wouldn’t need to download anything from anywhere except your repository.

gtirloni 2y ago

In a world with an internal proxy/mirror for dependencies and no internet access allowed by build systems.

dboreham 2y ago
Which is not the world we live in.
- gtirloni 2y ago
  s/we/I/
- cdmyrm 2y ago
  [flagged]

cdmyrm 2y ago

It's called air-gapping, and lots of adults do it.

zahlman 2y ago

> The miss here isn't that the build container had network access, it's that the build container both ran untrusted code, and had access to secrets. If you're providing a build container service then you pretty much have to run untrusted code (the customer's) in the container, yes? So then the problem is really just the bad Firebase config... ?