9 ms·

Out of curiosity, is it really necessary to have the separate artifact server? Pinning dependencies by hash ought to be sufficient.

fortran77 5y ago
It's nice to have a build machine that can complete a build when it's disconnected from the Internet
- kawsper 5y ago
  How often does that happen?
  - 83457 5y ago
    Sometimes I use an air gapped test lab. Setup of certain software and projects is a real pain. Sounds like this approach could help.
  - oauea 5y ago
    https://hn.algolia.com/?q=cloudflare+down https://hn.algolia.com/?q=cloudflare+down https://hn.algolia.com/?q=akamari+down https://hn.algolia.com/?q=akamari+down https://hn.algolia.com/?q=github+down https://hn.algolia.com/?q=github+down
    - jrockway 5y ago
      Docker Hub also has rate limits and outages, so yet another thing you want to cache if you promise customers "we'll install our software in your Kubernetes cluster in 15 minutes 99.95% of the time".
    - kawsper 5y ago
      I don't see how any of those are build machines that are offline.
      - oauea 5y ago
        Build machines generally want to be 'online' so they can connect to the internet. They want to connect to the internet so they can access services hosted on the internet. Many of those services use Cloudflare, Akamari or Github. If any of those are down, those services are down. Now it's great that your build machine is online, but the packages it's trying to fetch won't be available because their CDN is offline. Now if only you had a local caching proxy.
  - ex_amazon_sde 5y ago
    It's standard practice in many large companies.
yardstick 5y ago
Seems reasonable enough depending on your use case. In our situation we store our own (private, commercial) artifacts as well as third party ones, so we already need to have a server, and we know our server is configured, maintained & monitored in a secure fashion whereas I have no guarantees with public servers. Plus our build servers don’t have access out to the internet either, for security. Supply chain attacks like SolarWinds and Kaseya are too common these days. Edit: Also, our local servers are faster at serving requests, allowing for faster builds, and ensures no issues with broken builds if a public repo went offline or was under attack.
zerkten 5y ago
It's going to depend on your circumstances. You don't share any context about your app or any of your development. Rather than looking at this from the perspective of needing an artifact server, you just look at it as a case of supply chain protection. If pinning dependencies counters all the threats in your threat model, then fine. If not, you need to be doing something to counter them. An artifact server, or vendoring your dependencies, provides provides a lot of additional control where chokepoints or additional audits can be inserted. If there was no management cost or hassle then you'd just have an artifact server to give you a free abstraction, but it's a trade-off for many people. It's also not a solution in itself, you need to be able to do audits and use the artifact server to your advantage. The problem is really with the threat models and whether someone really knows what they need to defend against. I find that many engineers are naïve to the threats since they've never personally had exposure to an environment where these are made visible and countered. At other times, engineers are aware, but it's a problem of influencing management.
jrockway 5y ago
Security and availability don't have to be mutually exclusive. I remember in the early day of Go modules our Docker builds (that did "go mod download") would be rate-limited by Github, so a local cache was necessary to get builds to succeed 100% of the time. (Yes, you can plumb through some authentication material to avoid this, but Github is slow even when they're not rate limiting you!) Honestly, that thing was faster than the official Go Module Proxy so I kept it around longer than necessary and the results were good. Even if you cache modules on your own infrastructure, you should still validate the checksums to prevent insiders from tampering with the cached modules. I'll also mention that any speed increases really depend on your CI provider. I self-hosted Jenkins on a Rather Large AWS machine, and had the proxy nearby, so it was fast. But if you use a CI provider, they tend to be severely network-limited at the node level and so even if you have a cache nearby, you are still going to download at dialup speeds. (I use CircleCI now and at one point cached our node_modules. It was as slow as just getting them from NPM. And like, really slow... often 2 minutes to download, whereas on my consumer Internet connection it's on the order of 10 seconds. Shared resources... always a disaster.)
- helsinki 5y ago
  Tangential non-sequitor: > self-hosted on AWS People forgot what self-hosting actually means.
  - jrockway 5y ago
    I disagree with that analysis. That basically means running some binary yourself, with the alternative being buying a SaaS product that is hosted by the developer. You may think that "self-hosting" means having your own physical server, but I have never heard anyone else use the expression like that.
  - paulryanrogers 5y ago
    Meanings can evolve over time. I tend to think of self hosted as installing from code and managing myself, whether on local hardware or remote.
  - jijji 5y ago
tikkabhuna 5y ago
We use an artifact server and our build servers are completely airgapped. We know exactly what dependencies are used across the organisation. We can take centralised action against malicious dependencies. I wouldn't bother having one if you're small (<25) people. If you start having a centralised Infosec group, then it starts to become necessary.
- njharman 5y ago
  Airgaped? Really? Everytime a build happens someone physically moves a Thu drive or some other media too from the build server? Airgap means not networked, even internally. Not just "blocked" from internet.
  - tikkabhuna 5y ago
    Wikipedia[1] offers a slightly relaxed definition, although I agree, I (and my colleagues) abuse the term. The artifact repository server connects to the internet via a proxy. Build servers have no access to the internet. [1] https://en.wikipedia.org/wiki/Air_gap_(networking) https://en.wikipedia.org/wiki/Air_gap_(networking)
formerly_proven 5y ago
Depending on what you're using for package management, an "artifact server" can be as simple as a directory in git or a dumb HTTP server. File names are non-colliding and you don't really need an audit log on that server, because all references are through lock-files in git with hashes (right? RIGHT?), so it basically doesn't matter what's on there.

fortran77 5y ago

It's nice to have a build machine that can complete a build when it's disconnected from the Internet

kawsper 5y ago
How often does that happen?
- 83457 5y ago
  Sometimes I use an air gapped test lab. Setup of certain software and projects is a real pain. Sounds like this approach could help.
- oauea 5y ago
  https://hn.algolia.com/?q=cloudflare+down https://hn.algolia.com/?q=cloudflare+down https://hn.algolia.com/?q=akamari+down https://hn.algolia.com/?q=akamari+down https://hn.algolia.com/?q=github+down https://hn.algolia.com/?q=github+down
  - jrockway 5y ago
    Docker Hub also has rate limits and outages, so yet another thing you want to cache if you promise customers "we'll install our software in your Kubernetes cluster in 15 minutes 99.95% of the time".
  - kawsper 5y ago
    I don't see how any of those are build machines that are offline.
    - oauea 5y ago
      Build machines generally want to be 'online' so they can connect to the internet. They want to connect to the internet so they can access services hosted on the internet. Many of those services use Cloudflare, Akamari or Github. If any of those are down, those services are down. Now it's great that your build machine is online, but the packages it's trying to fetch won't be available because their CDN is offline. Now if only you had a local caching proxy.
- ex_amazon_sde 5y ago
  It's standard practice in many large companies.

yardstick 5y ago

Seems reasonable enough depending on your use case. In our situation we store our own (private, commercial) artifacts as well as third party ones, so we already need to have a server, and we know our server is configured, maintained & monitored in a secure fashion whereas I have no guarantees with public servers. Plus our build servers don’t have access out to the internet either, for security. Supply chain attacks like SolarWinds and Kaseya are too common these days. Edit: Also, our local servers are faster at serving requests, allowing for faster builds, and ensures no issues with broken builds if a public repo went offline or was under attack.

zerkten 5y ago

It's going to depend on your circumstances. You don't share any context about your app or any of your development. Rather than looking at this from the perspective of needing an artifact server, you just look at it as a case of supply chain protection. If pinning dependencies counters all the threats in your threat model, then fine. If not, you need to be doing something to counter them. An artifact server, or vendoring your dependencies, provides provides a lot of additional control where chokepoints or additional audits can be inserted. If there was no management cost or hassle then you'd just have an artifact server to give you a free abstraction, but it's a trade-off for many people. It's also not a solution in itself, you need to be able to do audits and use the artifact server to your advantage. The problem is really with the threat models and whether someone really knows what they need to defend against. I find that many engineers are naïve to the threats since they've never personally had exposure to an environment where these are made visible and countered. At other times, engineers are aware, but it's a problem of influencing management.

jrockway 5y ago

Security and availability don't have to be mutually exclusive. I remember in the early day of Go modules our Docker builds (that did "go mod download") would be rate-limited by Github, so a local cache was necessary to get builds to succeed 100% of the time. (Yes, you can plumb through some authentication material to avoid this, but Github is slow even when they're not rate limiting you!) Honestly, that thing was faster than the official Go Module Proxy so I kept it around longer than necessary and the results were good. Even if you cache modules on your own infrastructure, you should still validate the checksums to prevent insiders from tampering with the cached modules. I'll also mention that any speed increases really depend on your CI provider. I self-hosted Jenkins on a Rather Large AWS machine, and had the proxy nearby, so it was fast. But if you use a CI provider, they tend to be severely network-limited at the node level and so even if you have a cache nearby, you are still going to download at dialup speeds. (I use CircleCI now and at one point cached our node_modules. It was as slow as just getting them from NPM. And like, really slow... often 2 minutes to download, whereas on my consumer Internet connection it's on the order of 10 seconds. Shared resources... always a disaster.)

helsinki 5y ago
Tangential non-sequitor: > self-hosted on AWS People forgot what self-hosting actually means.
- jrockway 5y ago
  I disagree with that analysis. That basically means running some binary yourself, with the alternative being buying a SaaS product that is hosted by the developer. You may think that "self-hosting" means having your own physical server, but I have never heard anyone else use the expression like that.
- paulryanrogers 5y ago
  Meanings can evolve over time. I tend to think of self hosted as installing from code and managing myself, whether on local hardware or remote.
- jijji 5y ago

tikkabhuna 5y ago

We use an artifact server and our build servers are completely airgapped. We know exactly what dependencies are used across the organisation. We can take centralised action against malicious dependencies. I wouldn't bother having one if you're small (<25) people. If you start having a centralised Infosec group, then it starts to become necessary.

njharman 5y ago
Airgaped? Really? Everytime a build happens someone physically moves a Thu drive or some other media too from the build server? Airgap means not networked, even internally. Not just "blocked" from internet.
- tikkabhuna 5y ago
  Wikipedia[1] offers a slightly relaxed definition, although I agree, I (and my colleagues) abuse the term. The artifact repository server connects to the internet via a proxy. Build servers have no access to the internet. [1] https://en.wikipedia.org/wiki/Air_gap_(networking) https://en.wikipedia.org/wiki/Air_gap_(networking)

formerly_proven 5y ago

Depending on what you're using for package management, an "artifact server" can be as simple as a directory in git or a dumb HTTP server. File names are non-colliding and you don't really need an audit log on that server, because all references are through lock-files in git with hashes (right? RIGHT?), so it basically doesn't matter what's on there.