8 ms·

I recently had a similar discussion with people using npm for building a CSS framework library. I tried to explain the concept of getting a pre-downloaded tarba

• mateuszf 7y ago
  It can be done using package managers too, for example using Nix. The problem here is not related to distribution method but rather dependency hell in javascriptland.
  • weberc2 7y ago
    I would say the main issue is that Nix is super difficult and requires you to intimately know your entire dependency tree down into Linux particularities. JavaScript makes matters worse, but even packaging a nontrivial Python package pulled directly from Pypi is often difficult. I want to like Nix, but it is far too pedantic to be practical. And this isn’t even mentioning the usability issues.
    • glglwty 7y ago
      I think you only need to specify your immediate dependencies in nix?
      • weberc2 7y ago
        Someone has to write the package definitions for those dependencies, and the public package repository doesn’t have broad coverage for many popular packages in many languages. This is understandable in that this is a massive effort, but that’s also the point—the effort to manage packages is huge and the extra effort compared to other tools is not a good bargain for many projects, nor is it much of consolation for project maintainers who would otherwise like to use it.
• Gravityloss 7y ago
  It can be sold as a new invention in a few years.
• draw_down 7y ago
  Maybe it wasn’t explained well.
• jwr 7y ago
  To follow up: I ended up building my own Docker images which contained the build tools (also known as the npm burning dumpster tire fire with extras thrown in), source code and all dependencies. This allows me to get reproducible builds, the modern-day analog of "./configure; make".
• maple3142 7y ago
  I sometimes have problem doing `./configure && make` because dependencies varies across different distro, which have some subtle differences that make it not easy as expected.
  • tjoff 7y ago
    But it is consistent, which is very important.
  • regularfry 7y ago
    If you can do `./configure && make`, you can also do a `checkinstall` on debianesque systems, which gives you a package so you don't have to do it again.
    • pbhjpbhj 7y ago
      checkinstall does rpm/slack too (or did), not sure how debianesque they are.
  • jwr 7y ago
    True. But at least it doesn't auto-download half the internet (with new and improved vulnerabilities of the day) and dump it all in your "project directory" (whatever that might be, I'm still fuzzy on the exact definition).
• Tade0 7y ago
  Truth be told the whole front-end ecosystem relies on those servers being available, so in their absence any development everywhere would grind to a halt - as it almost did during the left-pad scandal. While I agree that this is not exactly the sanest approach, within the ecosystem there's no incentive to work differently. Also, like someone else mentioned - dependency hell only got worse over time - setting up a new project you're likely to have several versions of the same library in your node_modules.
  • tomcooks 7y ago
    > Truth be told the whole front-end ecosystem relies on those servers being available, so in their absence any development everywhere would grind to a halt There's a world out there of LAMP/LEMP stack web developers that wholeheartedly disagree with this perception.
  • weberc2 7y ago
    There’s a pretty straightforward solution to this problem which is to run your own NPM server even if it’s only a caching proxy. For some orgs, the uptime provided by the third party servers is sufficient.
    • doomslice 7y ago
      Having been at 3 companies that did this, I can say with certainty that our caching proxies (artifactory in all of them) were much more likely to go down than the public repos were.
      • regularfry 7y ago
        Yes, but at least in that case you have the fallback of the public repos.
      • _-o-_ 7y ago
        Yeah, I think I've lost a day in total when our local npm was finicky on several occasions or failed to fetch latest version of a package that we've developed in house, but made publicly available.
• pwagland 7y ago
  Don't turn it into an ageist thing, unless you are particularly old, there will be plenty of older developers that also don't understand this. It largely comes from understanding _why_ reproducibility is a good thing, and there are _lots_ of open source maintainers that understand this that are likely "younger than you". The vast majority of developers though focus on other things, that SQL injection attacks are still a thing indicate that we as a community have a _long_ way to go on the security front.
  • was_boring 7y ago
    It is way bigger then security, and hits upon quality of life and reliability. Without a deterministic build I don't know what I check in actually works across environment, or if the deployment artifact works.
    • rtpg 7y ago
      checksums are a thing, as well as package mirrors or outright holding onto the file and checking in a binary. There's loads of ways to know what you are using without needing to strip mtime's from zip files.
      • zaphar 7y ago
        Checksums, solve a different problem. How does a developer know his code which works on his laptop will work on production. One of, but not the only, conditions for that are that his development artifact builds with the same dependencies as it will build in production. Their are a lot of possible causes for Dev/Prod to differ wildly. One of them are poor build practices.
    • SkyPuncher 7y ago
      > Without a deterministic build I don't know what I check in actually works across environment, or if the deployment artifact works. I use to have this concern, but it's really not an issue with modern package managers. Even without a checksum in a lock file (e.g. gem bundler), I haven't actually had a deployment break because of dependencies changes. The biggest issue is being blocked for a couple hours because a package repo went down. Most of this is solved by using a self-hosted or 3rd party proxy package manager.
• radicalbyte 7y ago
  I like using Artifactory for this, you keep a local copy of all NPM / nuget / etc dependencies hosted next to your build infrastructure. What shocked me most about NPM is that it used to have absolutely 0 verification built in, yet it was being heavily promoted by very well known, educated and experienced tech celebs. All at a time when it was basically a hobby toy.
  • etimberg 7y ago
    Last time I tried to use Artifactory for it's caching it barely worked. Would randomly 404 when a package was not in it's cache yet and you'd end up retrying a CI build for a while until everything was there.
    • anon73044 7y ago
      At work I'm running ~2000 builds a day across 14 repos, mix of c# and php. We're proxying several hundred NPM, composer and nuget packages on 2 artifactory containers running in high availability. It was a rocky start but about 2 months in everything was pretty stable. If you're still apposed to artifactory you could try sonatype nexus
• danmckinney 7y ago
  I don't think it's specific to just younger devs, but there absolutely is an argument for having your own artifact repository that you can control.
• JustSomeNobody 7y ago
  I don’t think it’s age so much as it’s “well, that’s just what everyone does now”. If you want to work in this industry you have to follow the herd; for better or worse. I certainly don’t agree with it, but people have bills to pay.
  • jlgaddis 7y ago
    Your comment reminds me of the "Lemmings walking off a cliff" animated GIF that was popular once animated GIFs became a thing on the web.
  • fargle 7y ago
    What if "well, that's just what everyone does" is just wrong? I trimmed the word "now" from the quote because lazy sloppy short-term thinking is nothing new. Good configuration management takes a lot of work and pays great returns, only not in the very short-term. It's not surprising that inexperienced devs will fall into this trap. And just why do you "Have to follow the herd"? Pretty sure just about everything new and creative (good or bad) was a result of not following the herd. I recommend going where you need to go. If a herd starts following you, that's great. At least they'll be slipping and sliding on the shit you leave behind, not the other way round.
  • kbr2000 7y ago
    Managers and C-levels surely have appetite for this. In the end it just degrades mostly to trial-and-error, which I'm sure is no problem because the people have to do the real work get sick of it and get replaced (sometimes voluntarily, sometimes with force), or play the game nice out of fear losing their job (thus feeding the beast). It's not very surprising since 'products' nowadays are more like 'services' instead, and offer some kind of encapsulation: people are not interested much in how something ticks behind the scenes, and so it can drive down the behind-the-scene quality. They also are conditioned to accept lousy excuses (including none) for outages and breakdowns, because they got sold 'magic', and boy that is magical.. You'll might have to work in obscurity to keep up operations quality, which is in turn a driver for your own demise. Game over. I'd like to suggest to you the following reading material: - Bullshit Jobs (David Graeber, 2018) - The Dilbert Principle (Scott Adams, 2000) - Future Shock (Alvin Toffler, 1970)
• y0y 7y ago
  I know you're taking some flack for making it about age, but I do think there's some merit there. I'm 34. I feel like I'm in the middle in terms of software developer / sysadmin age. Lots of bright minds came before me and there are lots of bright minds out there right now in their early 20s. A key difference is I can remember a time when network connectivity was flaky. When it was hardly a given. Even when it was available, the download times alone could often constitute a large part of the overall build time. I think that we have all become complacent with regard to internet connectivity and service availability, but I think the younger you are the more complacent you are likely to be. If github.com goes down entirely, let's be honest - there are a lot of Jenkins builds that are going to be in the red.
  • IAmEveryone 7y ago
    People adapt to the situation they experience. Github (and other repositories) tend to be stable, so they are used. Network speeds have increased, so we use the network and expend less efforts on local caching etc. There's nothing "complacent" about this: previous generations also relied on infrastructure and didn't plan for prolonged power outages or had backup ham radio network links for when AOL was down. People using nom install instead of custom makefiles aren't ignorant or stupid, they have found better ways to achieve their needs. And if 10 years on the job don't create the need to learn some skill, there is no reason to invest time into it. And I have complete confidence that people would be able to come with some workable solutions rather quickly if the githubcopalypse ever happens. There is some cultural component at play among the "luddites" here as well, maybe comparable to preppers? It feels like planning for really exciting emergencies when one's skills that have been derided for so long are suddenly needed and safe the day. In this analogy, I guess Makefiles are the equivalent of very masculine hunting and zombie-defending skills.
    • tonyarkles 7y ago
      I half agree with you, and am chuckling a little bit about the "very masculine hunting and zombie-defending skills" part, but... I'd like to offer some perspective. I'm 36 and work on a pretty broad set of consulting projects: some schematic/PCB/mechanical design, firmware, some lower-level desktop/server code, and up and up to web/mobile apps. "Full full stack" if you will. I live in a "major" Canadian city (although not in the top 15 by population), and I also own two wonderful properties about an hour out of town in a quite rural area. One is a cabin on a lake, and the other is a church from the 1910s. Sometimes I head out to one of these places to do the "Deep Work" thing, distraction-free, and sometimes it's to take time off, but end up getting an emergency call from a client. In either case, my Internet connectivity is limited to tethering, and depending on a few factors, that can either work fantastically well or poorly. Going from the lowest-level to the highest-level projects, there's a very clearly declining probability of the project being able to build during a low-connectivity event. The embedded stuff pretty much always works just fine (it's a Makefile, or CMake). The C desktop/server stuff? Always works fine (any dependencies were pre-installed). Python/Ruby/Elixir web backend projects usually go OK, although I've occasionally ran into issues where the package manager wants to check for updates. Node front-end builds sometimes start to fall apart, and Android (via Android Studio) often refuse to build at all! (Some kind of weird Maven/Gradle thing that needs to go out and check something, even though the dependencies have all been pre-installed...) It's extraordinarily frustrating when you can't change a line of code and hit "Build" to test a change locally. Everything's already present on the machine! It worked just fine 5 minutes ago! To your prepper comment, and the previous comments about infrastructure, there's a significant population of the world that doesn't have 100% reliable infrastructure, even in Canada and the US. The tools we have used to work just fine in that environment, but are getting progressively worse.
• pojntfx 7y ago
  18yo JS/Ruby/Go dev here. `make` is certainly known to us; it's just that platform-specific build systems (`mage` for Go, `rake` for Ruby, `gulp` for JS etc.) are simply more sane to set up per languages than using `make` for all languages and allows for efficient code reuse.
  • celticmusic 7y ago
    You have make call the platform specific build systems.
    • Spivak 7y ago
      What's the point in a project that only uses a single build system? You're essentially getting nothing out of make except indirection. Might as well just have a build.sh at the root of the repo.
      • frenchyatwork 7y ago
        > You're essentially getting nothing out of make except indirection The thing you're getting is some familiarity, I think (because of the indirection). It's sort of like using a `package.json` for the scripts section, but you get to have comments, and there's less other random stuff in there. I definitely understand why many would think this is over-kill though. A proper readme is probably just as good/better.
      • celticmusic 7y ago
        familiarity, documenting the process for building, and last but not least, access to the entire unix toolchain.
    • pojntfx 7y ago
      I have done so in the past. This leads to a lot of duplication though.
      • asdasdasdasdwd 7y ago
• AndyMcConachie 7y ago
  s/younger/less experiened/
• mr__y 7y ago
  >building a CSS framework library I think that might have been the root cause of the problem.
• colinjfw 7y ago
  Not alien to me, likely younger.
• ekidd 7y ago
  > without relying on any third-party servers being available and without pulling in dependencies that might have changed. There are two different issues here: 1. Not pulling in changed dependencies. This is what "lock files" are for: To limit builds to known version of every dependency. npm was terrible about this for a long time. Most other language package managers are better. 2. Not relying on third party servers to be available. Personally, I've mostly worked with Ruby's bundler and Rust's cargo, and in over 10 years, I've lost maybe two days of work because of package server outages. That's less than I've lost due to S3 outages, less than I've lost due to broken backup systems, and less than I've lost due to complex RAID failures. For the clients and employers in question, this was acceptable downtime. In the rare cases where one day of noticeable build downtime every 5 years is unacceptable, then it's usually possible to just "vendor" the dependencies, usually by running a one-line command. For many small to midsize businesses (and many growing startups), a small risk of brief outages is an acceptable engineering trade-off.
  • hn_throwaway_99 7y ago
    > npm was terrible about this for a long time. That's putting it lightly. To me "terrible" would mean they just didn't support it, "batshit insane" means they supported lock files but ignored them every time you ran npm install. My favorite comment from this stackoverflow post: https://stackoverflow.com/questions/45022048/why-does-npm-install-rewrite-package-lock-json https://stackoverflow.com/questions/45022048/why-does-npm-in... "Why would you expect something called package lock to lock the packages? Package lock is analogous to how, when you put any key into a door lock, the lock reshapes itself to match whatever key was put in, then opens the door. Now if you'll excuse me, I'm late for a tea party with a rabbit."
    • hinkley 7y ago
      I think one of the scenes with Humpty Dumpty could be an appropriate allusion as well.
  • dirkf 7y ago
    At work we also cache all 3rd party dependencies locally. Not because the 3rd party servers might be down but to be sure that one/two/... years from now we can still recreate the same software we delivered at that time to a customer. Our build machines typically didn't even have internet access. If a dependency was not available offline (due to developer error or whatever) we would know quickly. In certain industry branches this is even mandatory if you want to be seriously considered as a supplier. If a build tool does not support reproducible builds in such a way (both fixing dependency version and getting it from a cache somehow) or makes it difficult then it is considered as a hobby toy that has no place in the workplace. Even for small businesses I would advocate to take this seriously from the beginning. It's not that hard and will save you headaches later on when suddenly reproducible builds become important.
• winrid 7y ago
  Easiest thing to do to solve this would be to have something like artifactory or even squid in front of npm and use exact version numbers. Front end devs are going to use NPM. Solve the other problems separately like you would with any other build system.
  • eckza 7y ago
    Sure. And then when something unexpected happens, you have two problems.
    • winrid 7y ago
      You mean clearing a cache? I don't understand. Using a proxy to protect yourself from missing dependencies/down servers is a bad thing now?
• asdfman123 7y ago
  You'll get blank stares and resistance whenever you throw a bunch of jargon at people that they don't understand, though. It makes people feel insecure. You can't expect new programmers who sound like primarily front-end devs to learn all the hot new stuff and all the old stuff, too. If you want someone with that kind of knowledge, you need to hire a senior dev with 20 years of experience.