12 ms·

This is why our build systems don’t use public repositories directly, and why we always pin to an exact version. Any third party dependencies (js/python/java/c/

• deleted 5y ago
  [deleted]
• ericpauley 5y ago
  Out of curiosity, is it really necessary to have the separate artifact server? Pinning dependencies by hash ought to be sufficient.
  • fortran77 5y ago
    It's nice to have a build machine that can complete a build when it's disconnected from the Internet
    • kawsper 5y ago
      How often does that happen?
      • 83457 5y ago
        Sometimes I use an air gapped test lab. Setup of certain software and projects is a real pain. Sounds like this approach could help.
      • oauea 5y ago
        https://hn.algolia.com/?q=cloudflare+down https://hn.algolia.com/?q=cloudflare+down https://hn.algolia.com/?q=akamari+down https://hn.algolia.com/?q=akamari+down https://hn.algolia.com/?q=github+down https://hn.algolia.com/?q=github+down
        • jrockway 5y ago
          Docker Hub also has rate limits and outages, so yet another thing you want to cache if you promise customers "we'll install our software in your Kubernetes cluster in 15 minutes 99.95% of the time".
        • kawsper 5y ago
          I don't see how any of those are build machines that are offline.
          • oauea 5y ago
            Build machines generally want to be 'online' so they can connect to the internet. They want to connect to the internet so they can access services hosted on the internet. Many of those services use Cloudflare, Akamari or Github. If any of those are down, those services are down. Now it's great that your build machine is online, but the packages it's trying to fetch won't be available because their CDN is offline. Now if only you had a local caching proxy.
      • ex_amazon_sde 5y ago
        It's standard practice in many large companies.
  • yardstick 5y ago
    Seems reasonable enough depending on your use case. In our situation we store our own (private, commercial) artifacts as well as third party ones, so we already need to have a server, and we know our server is configured, maintained & monitored in a secure fashion whereas I have no guarantees with public servers. Plus our build servers don’t have access out to the internet either, for security. Supply chain attacks like SolarWinds and Kaseya are too common these days. Edit: Also, our local servers are faster at serving requests, allowing for faster builds, and ensures no issues with broken builds if a public repo went offline or was under attack.
  • zerkten 5y ago
    It's going to depend on your circumstances. You don't share any context about your app or any of your development. Rather than looking at this from the perspective of needing an artifact server, you just look at it as a case of supply chain protection. If pinning dependencies counters all the threats in your threat model, then fine. If not, you need to be doing something to counter them. An artifact server, or vendoring your dependencies, provides provides a lot of additional control where chokepoints or additional audits can be inserted. If there was no management cost or hassle then you'd just have an artifact server to give you a free abstraction, but it's a trade-off for many people. It's also not a solution in itself, you need to be able to do audits and use the artifact server to your advantage. The problem is really with the threat models and whether someone really knows what they need to defend against. I find that many engineers are naïve to the threats since they've never personally had exposure to an environment where these are made visible and countered. At other times, engineers are aware, but it's a problem of influencing management.
  • jrockway 5y ago
    Security and availability don't have to be mutually exclusive. I remember in the early day of Go modules our Docker builds (that did "go mod download") would be rate-limited by Github, so a local cache was necessary to get builds to succeed 100% of the time. (Yes, you can plumb through some authentication material to avoid this, but Github is slow even when they're not rate limiting you!) Honestly, that thing was faster than the official Go Module Proxy so I kept it around longer than necessary and the results were good. Even if you cache modules on your own infrastructure, you should still validate the checksums to prevent insiders from tampering with the cached modules. I'll also mention that any speed increases really depend on your CI provider. I self-hosted Jenkins on a Rather Large AWS machine, and had the proxy nearby, so it was fast. But if you use a CI provider, they tend to be severely network-limited at the node level and so even if you have a cache nearby, you are still going to download at dialup speeds. (I use CircleCI now and at one point cached our node_modules. It was as slow as just getting them from NPM. And like, really slow... often 2 minutes to download, whereas on my consumer Internet connection it's on the order of 10 seconds. Shared resources... always a disaster.)
    • helsinki 5y ago
      Tangential non-sequitor: > self-hosted on AWS People forgot what self-hosting actually means.
      • jrockway 5y ago
        I disagree with that analysis. That basically means running some binary yourself, with the alternative being buying a SaaS product that is hosted by the developer. You may think that "self-hosting" means having your own physical server, but I have never heard anyone else use the expression like that.
      • paulryanrogers 5y ago
        Meanings can evolve over time. I tend to think of self hosted as installing from code and managing myself, whether on local hardware or remote.
      • jijji 5y ago
  • tikkabhuna 5y ago
    We use an artifact server and our build servers are completely airgapped. We know exactly what dependencies are used across the organisation. We can take centralised action against malicious dependencies. I wouldn't bother having one if you're small (<25) people. If you start having a centralised Infosec group, then it starts to become necessary.
    • njharman 5y ago
      Airgaped? Really? Everytime a build happens someone physically moves a Thu drive or some other media too from the build server? Airgap means not networked, even internally. Not just "blocked" from internet.
      • tikkabhuna 5y ago
        Wikipedia[1] offers a slightly relaxed definition, although I agree, I (and my colleagues) abuse the term. The artifact repository server connects to the internet via a proxy. Build servers have no access to the internet. [1] https://en.wikipedia.org/wiki/Air_gap_(networking) https://en.wikipedia.org/wiki/Air_gap_(networking)
  • formerly_proven 5y ago
    Depending on what you're using for package management, an "artifact server" can be as simple as a directory in git or a dumb HTTP server. File names are non-colliding and you don't really need an audit log on that server, because all references are through lock-files in git with hashes (right? RIGHT?), so it basically doesn't matter what's on there.
• helsinki 5y ago
  FYI - You can overwrite an existing package’s release/version via pip (at least when using Artifactory’s PyPi). Not safe to assume pinning the version ‘freezes’ anything.
  • nerdponx 5y ago
    Make a "lockfile" with the pip-compile tool [0] that includes hashes. Unless you happen to fetch the hash after the package has been compromised, this should keep you safe from an overwritten version. [0]: https://pypi.org/project/pip-tools/ https://pypi.org/project/pip-tools/
  • oefrha 5y ago
    No, you cannot overwrite a file on PyPI once uploaded, even if you delete the release first. This policy has been in place for many years.
  • globular-toast 5y ago
    Pip also supports packages hashes so you can be sure you're getting the exact same package that you got last time.
• grincho 5y ago
  Yes, this is why I implemented hash-checking in pip (https://pip.pypa.io/en/stable/topics/repeatable-installs/#hash-checking https://pip.pypa.io/en/stable/topics/repeatable-installs/#ha...). Running your own server is certainly another way to solve the problem (and lets you work offline), but keeping the pinning info in version control gives you a built-in audit trail, code reviews, and one fewer server to maintain.
  • stavros 5y ago
    Doesn't Poetry do this by default in its lockfile too?
    • c618b9b695c4 5y ago
      There is an active PEP[0] for defining lockfiles. 0: https://www.python.org/dev/peps/pep-0665/ https://www.python.org/dev/peps/pep-0665/
• specialist 5y ago
  I did a bunch of nodejs stuff at my last gig. These teams had the of practice keeping packages up to date. Drove me frikkin nuts. So much churn, chaos. Is this a JavaScript thing? Carried over from frontend development? Exasperated, I finally stopped advocating for locking everything down. Everyone treated me like I was crazy. (Reproducible builds?? Pfft!!) Happens with enterprisey Java, Sprint, Maven projects too. (I can't even comment on Python projects; I was just happy when I could get them to run.) What's going on? FOMO? Lock ~~Look~~ down dependencies. Only upgrade when absolutely necessary. Keep things simple to facilitate catching regression bugs and such. Oh well. I moved on. I don't miss it one bit.
  • oblio 5y ago
    Java/Spring/Maven should have locked down dependencies by default. They have to go out of their way to not do that. Not that some people don't, anyway. Typos: > I stopped advocating for locking everything down. started? > Look down dependencies. lock?
    • nvrspyx 5y ago
      I think the first typo is actually correct. They're saying that they stopped advocating for it because everyone treated them like they were crazy for doing so.
    • specialist 5y ago
      Edited. Thanks for proofreading my rant. :)
    • watermelon0 5y ago
      AFAIK Maven and Gradle don't have any built-in way of locking down dependencies (direct or transient), unless I missed something.
      • cesarb 5y ago
        In my experience, Maven always uses the exact version specified in your pom.xml (and the pom.xml of your dependencies, transitively), it never uses a newer version automatically. That is, the built-in way in Maven is locked down dependencies.
• ageofwant 5y ago
  You should mention this in your interviews. Keeping up to date with the state of the art is implicit for me. If I need to spend months retraining or up training because of company policy I expect to be compensated for that while employed.
  • yardstick 5y ago
    We are upfront about it in interviews. We definitely don’t want unhappy developers, but we also don’t want insecure code. We do upgrade libraries but we do so only after analysing risk and impact. None of our developers have spent months training to develop our code.
• pjmlp 5y ago
  I have been doing internal repositories and vendoring since 2000, only for personal projects do I do otherwise as they are mostly throw away stuff. Teaching good security practices and application lifecycle management seems to always be an uphill battle.
• Nursie 5y ago
  This is how it always used to be, back in the before[1] times. Libraries would be selected carefully and dependencies would be kept locally so that you could always reproduce a build. The world is different now, and just being able to select a package and integrate it like that is a massive effectiveness multiplier, but I think the industry at large lost something in the transition. ([1] before internet package management, and before even stuff like apt and yum)
  • goodpoint 5y ago
    > I think the industry at large lost something in the transition. Lost a lot of trust and security with the advent of language-specific installers that pull libraries and their dependencies from random URLs without any 3rd party vetting.
    • mikepurvis 5y ago
      I agree, though I think there are a couple pieces to the puzzle. Like, quite apart from the enormous convenience of new releases being available right away, and even being able to drop in a git URL and run your dependency off a branch while you wait for your upstream PR to merge, pip provides isolation between workspaces that apt has never been able to do natively because it's too generic— the closest thing is lightweight unshares, but those were years after virtualenv, require more ceremony to get going with, and may be isolated from the outside system in ways that impede productivity (there's also never really been a consistent tooling story for it, with having to cobble together debootstrap + whatever the chroot flavor of the week is). But the biggest issue is when npm, cargo, etc showed how to include a dependency multiple times at different versions. Operating system package managers (other than Nix & friends) have no concept for how this could or would work. Pip has no story for it, and the various pip-to-apt bridges like py2deb and stdeb if anything just exacerbate the problem, by mixing your project's dependencies with those on your system. Anyway, yes. Things were lost, 100%. But I don't think the system you want (isolation, version freedom, but with safe/vetted packages) is going to be something that's possible to evolve out of any of the current systems. It's got to be something that goes back to first principles.
• yjftsjthsd-h 5y ago
  Sorry, I don't quite see how this would protect against supply chain attacks. If an upstream dependency is back-doored, they just have to silently add their code in an otherwise reasonable sounding release, and now you will happily download that version, add it to your internal mirrors, and pin its version forever. Unless you actually read the diff on every update, which I think is impractical (although you're welcome to correct me), I don't see how this is buying you much.
  • ozzythecat 5y ago
    It’s not 100% bullet proof, but it’s safer than pulling any random repos directly from the internet. It’s also good that your business doesn’t have to rely on a third party every time you need to pull down your dependencies and build your software.
    • DSingularity 5y ago
      Protects you against a dependency being hijacked and retroactively embedding malware in prior versions . Check summing would protect against that without the trouble of manually hosting the dependency.
  • polynomial 5y ago
    My read was GP is implying they do read the diff every time, and only change the pinned version after a manual review- "only after we are happy." This does seem impractical at even a modest scale.
    • zerkten 5y ago
      What does "modest scale" mean? Some people have to have all dependency updates reviewed and have some level of independent security team monitoring dependency changes. Not everyone has this, but the context is key as some domains have this consistently. Some of this is cultural in addition to being paranoid about security, or having strict compliance requirements. For your average startup, they may not have any time to dedicate to this and other fires, but that's not a universal situation.
• pornel 5y ago
  Own hosting, pinning, and checksum checks are defenses against network-based attacks and compromise of the package repository. They do nothing against trojans like these. You will be running your own pinned checksummed version of the malicious code. If you want to stop malware published by the legitimate package author, you need to review the code you're pulling in and/or tightly sandbox it (and effective sandboxing is usually impossible for dependencies running in your own process).
  • tremon 5y ago
    Well, it does induce a timed delay between upstream release of a compromised package and when it enters the own codebase. As long as the exploit is found and published before someone manually uploads it to the local store, you're safe. But yes, a better option would be to run your own acceptance tests on each new upstream release, and that includes profiling disk/network/cpu usage across different releases.
    • jameshart 5y ago
      Also introduces that same time delay for getting security patches released by package maintainers into your build pipeline.
      • yardstick 5y ago
        Not true. We assess each update in case of high priority issues that need a quick update. Also we are cautious of using dependencies that don’t provide long term support/back fixes - where possible I pick stable, responsibly-managed dependencies. Postgres is a great example, security fixes are applied across multiple major versions, not just the latest.
        • rexelhoff 5y ago
          Sounds exactly like what we do for medical device software. Version pinning, self-managed forks and code reviews of the dependency on upgrade.
• actually_a_dog 5y ago
  Yep. You should also be hosting and deploying from wheels[0], even for stuff you create internally. If you're doing it right, you'll end up hosting your own internal PyPi server[1], which, luckily, isn't hard[2]. We did this at one of my previous companies, and, of all the things that ever went wrong with our deploy processes, our internal PyPi server was literally never the culprit. --- [0]: https://pythonwheels.com/ https://pythonwheels.com/ [1]: https://github.com/testdrivenio/private-pypi https://github.com/testdrivenio/private-pypi [2]: https://testdriven.io/blog/private-pypi/ https://testdriven.io/blog/private-pypi/