7 ms·

It's amazing how much stuff breaks when GitHub goes down. I'm doing some Rust coding right now, the rust-s3 crate tells me that to look up what feature I need t

by xfbs 5y ago

It's amazing how much stuff breaks when GitHub goes down. I'm doing some Rust coding right now, the rust-s3 crate tells me that to look up what feature I need to enable (tokio + rustls), I need to look into the Cargo.toml. Well, the repo won't load, and nor can I clone it. Well okay, fuck that I can use the default dependencies. But no wait I can't, I can't even do a cargo build because cargo uses a github repository as the source of truth for all crates. No more Rust for me today :(

pasabagi 5y ago
Cargo has an --offline option. It's actually pretty possible to use rust totally offline - the doccumentation can be built, for instance, then locally served with (iirc) cargo doc.
- jiggawatts 5y ago
  Offline should be the default, not an option. Otherwise nobody will notice how fragile their workflow is until it is too late to fix it!
  - Aeolun 5y ago
    I recently realized I was the only person running the API locally when suddenly all our unmocked tests started failing only on my machine. All other tests, including the ones on the CI server, were connecting to the hosted integration environment for any unmocked call (and thus randomly modifying state there).
- xfbs 5y ago
  I build and host documentation on every commit anyways in the CI. And yes, that is true, I eventually figured it out (had some issues with it at first) but it seems like GitHub is back up so all's well anyhow. I do however wish that there was some public mirror that cargo could fall back to, wouldn't that make a lot of sense?
  - still_grokking 5y ago
    It would be good if such things would be hosted decentralized on something like say IPFS.
kleiba 5y ago
It's amazing how much stuff breaks when GitHub goes down. That's right. Someone should really come up with a decentralized VCS. /scnr
- catskul2 5y ago
  IMO Git is decentralization-ready but the rest of the infrastructure necessary to make it practical is not widely available/in use. The necessary peer-to-peer and networks of trust are still not a solved problem, or if they are they've for some reason are not popular enough to be in wide use.
  - PaulDavisThe1st 5y ago
    It's widely available and widely used, just not as widely used as github. Setting up your own git server is not hard, but it's not as easy as just getting github or gitlab to run it for you. Way too many people take the easier path, even though the harder path is not actually that hard. There are also multiple solutions.
    - mr_toad 5y ago
      It’s ok for small private projects, but user management is hard. You can’t just let any rando have commit privileges.
      - PaulDavisThe1st 5y ago
        that's what things like gitlab (not the website) or gitea are actually for.
    - catskul2 5y ago
      > Setting up your own git server is not hard Bullshit. Getting it set up in a secure and public way is an order of magnitude or two more difficult than it would need to be for mass use which is what we're talking about here. When you have a situation where a the vast majority of people find something too difficult, the problem (from the perspective of actually getting it happen) is not the people the problem is with the proposed solution. > There are also multiple solutions Multiple solutions does not generally make things easier/better even if it's desirable for other reasons.
revskill 5y ago
I guess same story for Go!
- krasin 5y ago
  In Go, it's customary to use 'go mod vendor' to put your dependencies into the main repository. While it's not universally recognized as a good technique, it saves the adopters of this approach from the downtime today.
  - jen20 5y ago
    I’m not sure this is customary at all - I rarely encounter a vendor directory anymore. Note you also need to build differently if you go this route: `-mod=vendor`, otherwise the directory will be ignored in modern Go.
    - krasin 5y ago
      With more outages of our amazing, but overly centralized development ecosystem, the popularity of this approach will likely surge. It helps that Go supports the vendoring workflow as it makes the choice practical. As for building with a flag: true, but very minor, as it's rare to execute 'go build' directly. In most projects I've seen, it's either Bazel, or some kind of "build.sh".
    - travisd 5y ago
      I've never had to use -mod=vendor, so I just looked it up: -mod mode module download mode to use: readonly, vendor, or mod. By default, if a vendor directory is present and the go version in go.mod is 1.14 or higher, the go command acts as if -mod=vendor were set. Otherwise, the go command acts as if -mod=readonly were set. See https://golang.org/ref/mod#build-commands for details. If there's a vendor directory, it's used by default. As for my two cents, I use it frequently when building Docker images so that I don't have to pass secrets into the image to clone private modules (but I don't check the vendor directory into Git).
    - 5y ago
birktj 5y ago
You can view the source of a crate on docs.rs (see [1] for the Cargo.toml of rust-s3). Also I am pretty sure cargo only depends on GitHub for authentication for uploading crates and not for the actual contents. Trying to build an empty crate with rust-s3 as a dependency right now seems to work fine. [1]: https://docs.rs/crate/rust-s3/0.28.0/source/Cargo.toml https://docs.rs/crate/rust-s3/0.28.0/source/Cargo.toml
- ATsch 5y ago
  As I understand it, the crates themselves are not stored on github, but the crate index is, as it uses a git repo to get "free" delta compression and auditing.
anothernewdude 5y ago
I can't stand builds that just reach out to the internet for things.
- lanstin 5y ago
  Yeah seems like basic hygiene especially given the supply chain attacks but also a lost cause. No one has the skills to do builds without the internet. Even forcing teams to use an allow list for the internet involving fighting a lot of angry people.
  - contingencies 5y ago
    One of the few decent uses for containers is to enforce proxied internet so build process artifacts can be auto-stored for subsequent builds. For the worst offender I am aware of, try building a flutter project... it silent internet gets artifacts from at least three different packaging systems (node, cocoapods, android packages), all of which have caused hard-to-debug failures.
- duped 5y ago
  How do you build software in 2021 then? Do you write everything from scratch?
  - spoils19 5y ago
    ... yes? I've often been ridiculed for wanting to build everything in-house, but events like these just validate that sometimes, to get the most reliable and best software, you'll have to reinvent the wheel a few times.
    - duped 5y ago
      To be honest that's deserving of ridicule. Reinventing wheels is how you introduce fragility and instability, not prevent it.
  - still_grokking 5y ago
etra0 5y ago
Beside the `--offline`, you can also use `--vendor` to include all the dependencies in a folder to be committed alongside your project. Useful when you don't want to rely on external fetch every time!
- jiggawatts 5y ago
  Both should be the default. Rust has a robust memory model, but everything else about it insists on copying the fragility of the NPM ecosystem. The recent hoopla around a bunch of Rust mods quitting revealed that my suspicions are precisely true — key Rust staff also sit on the NPM board!
  - pclmulqdq 5y ago
    Cargo is the one thing keeping me off the rust ecosystem. The fundamentals of the language are great, but the tight coupling of the rust language with cargo's package management really irks me - it introduces as many correctness and security problems as the memory model solves.
    - duped 5y ago
      What languages have package management systems that solve those problems? Cargo does have options to fix these, like mentioned in other comments. I'm not convinced they should be defaults.
      - pclmulqdq 5y ago
        I'm suggesting that "languages that have package managers" is generally the problem. I think the go package manager gets the closest to solving these problems thanks to its reliance on one trusted source that has STRONG incentives to make sure that packages are available and trusted, but fundamentally, a language and a package manager are very different products, and I don't want them to be bundled.
dandotway 5y ago
In the 1970s-80s we managed with email lists and tarballs on multiple mirrors. Git itself decentralizes source control, and yet we all want to use single-point-of-failure Github. Anyone remember when Github took down youtube-dl? I wonder how much Big Brother data Microsoft is gathering on Github developers. "Oh, as part of our hiring process, just like we scan your FB and Twitter, we also scan your Github activity to evaluate your performance as best as our machine learning overlords can assess it. Do you create Github projects and then abandon them when unfixed issues accumulate? Our algorithm thinks you're a bad hire."
- adeelk93 5y ago
  > Anyone remember when Github took down youtube-dl? I would blame the law (DMCA), not those forced to abide by the law (Github)
  - Teever 5y ago
    Your argument is pure sophistry. Microsoft is a member of the RIAA and works closely with the MPAA to further control over society with DRM and intellectual property restrictions like the DMCA. It is well within the power of an entity the size of Microsoft to oppose something like the DMCA and the fact that they don't indicates not that they can't as you imply, but that they don't want to.
  - skeeter2020 5y ago
    Especially because they worked to try and make sure this wouldn't happen again.
  - deleted 5y ago
    [deleted]
  - kevin_thibedeau 5y ago
    The law also permits content to be restored immediately upon receipt of a counternotice. Somehow the data silos never get around to supporting that.
dom96 5y ago
Should be fairly easy for Cargo to solve this: why doesn't it already mirror its source of truth to GitLab and other Git hosters?
- nixgeek 5y ago
  Probably because the level of effort required to configure that and then monitor/manage it continuing to operate doesn't make sense when GitHub has an outage or two a year.
fowlie 5y ago
I was working on my Nix config. I had just added a small command line utility and wanted to install it, but then got 504 errors from github.com. Annoying!
strenholme 5y ago
This is why a lot of companies use a product like jFrog artifactory to handle dependencies. Instead of trying to download files from GitHub and other random places on the internet, the artifactory is a local mirror of all of those files, with servers on the artifactory which run the relevant services such as a npm repo of packages, a npm repo of packages, and so on. This way, if GitHub falls off of the face of the earth tomorrow, the packages still build. Also: If some open source maintainer deletes an older version of a package, replacing it with a new one with a different API, the local package will still build. (The local build process will have to be set up to use the local artifactory to pull dependencies, of course.)
avjinder 5y ago
I just got an itch to try out Emacs, after years of vim. Installed Spacemacs and opened it. It tried to install a bunch of packages, mostly from Github it seems because it kept failing.

by xfbs 5y ago

pasabagi 5y ago

Cargo has an --offline option. It's actually pretty possible to use rust totally offline - the doccumentation can be built, for instance, then locally served with (iirc) cargo doc.

jiggawatts 5y ago
Offline should be the default, not an option. Otherwise nobody will notice how fragile their workflow is until it is too late to fix it!
- Aeolun 5y ago
  I recently realized I was the only person running the API locally when suddenly all our unmocked tests started failing only on my machine. All other tests, including the ones on the CI server, were connecting to the hosted integration environment for any unmocked call (and thus randomly modifying state there).
xfbs 5y ago
I build and host documentation on every commit anyways in the CI. And yes, that is true, I eventually figured it out (had some issues with it at first) but it seems like GitHub is back up so all's well anyhow. I do however wish that there was some public mirror that cargo could fall back to, wouldn't that make a lot of sense?
- still_grokking 5y ago
  It would be good if such things would be hosted decentralized on something like say IPFS.

kleiba 5y ago

It's amazing how much stuff breaks when GitHub goes down. That's right. Someone should really come up with a decentralized VCS. /scnr

catskul2 5y ago
IMO Git is decentralization-ready but the rest of the infrastructure necessary to make it practical is not widely available/in use. The necessary peer-to-peer and networks of trust are still not a solved problem, or if they are they've for some reason are not popular enough to be in wide use.
- PaulDavisThe1st 5y ago
  It's widely available and widely used, just not as widely used as github. Setting up your own git server is not hard, but it's not as easy as just getting github or gitlab to run it for you. Way too many people take the easier path, even though the harder path is not actually that hard. There are also multiple solutions.
  - mr_toad 5y ago
    It’s ok for small private projects, but user management is hard. You can’t just let any rando have commit privileges.
    - PaulDavisThe1st 5y ago
      that's what things like gitlab (not the website) or gitea are actually for.
  - catskul2 5y ago
    > Setting up your own git server is not hard Bullshit. Getting it set up in a secure and public way is an order of magnitude or two more difficult than it would need to be for mass use which is what we're talking about here. When you have a situation where a the vast majority of people find something too difficult, the problem (from the perspective of actually getting it happen) is not the people the problem is with the proposed solution. > There are also multiple solutions Multiple solutions does not generally make things easier/better even if it's desirable for other reasons.

revskill 5y ago

I guess same story for Go!

krasin 5y ago
In Go, it's customary to use 'go mod vendor' to put your dependencies into the main repository. While it's not universally recognized as a good technique, it saves the adopters of this approach from the downtime today.
- jen20 5y ago
  I’m not sure this is customary at all - I rarely encounter a vendor directory anymore. Note you also need to build differently if you go this route: `-mod=vendor`, otherwise the directory will be ignored in modern Go.
  - krasin 5y ago
    With more outages of our amazing, but overly centralized development ecosystem, the popularity of this approach will likely surge. It helps that Go supports the vendoring workflow as it makes the choice practical. As for building with a flag: true, but very minor, as it's rare to execute 'go build' directly. In most projects I've seen, it's either Bazel, or some kind of "build.sh".
  - travisd 5y ago
    I've never had to use -mod=vendor, so I just looked it up: -mod mode module download mode to use: readonly, vendor, or mod. By default, if a vendor directory is present and the go version in go.mod is 1.14 or higher, the go command acts as if -mod=vendor were set. Otherwise, the go command acts as if -mod=readonly were set. See https://golang.org/ref/mod#build-commands for details. If there's a vendor directory, it's used by default. As for my two cents, I use it frequently when building Docker images so that I don't have to pass secrets into the image to clone private modules (but I don't check the vendor directory into Git).
  - 5y ago

birktj 5y ago

You can view the source of a crate on docs.rs (see [1] for the Cargo.toml of rust-s3). Also I am pretty sure cargo only depends on GitHub for authentication for uploading crates and not for the actual contents. Trying to build an empty crate with rust-s3 as a dependency right now seems to work fine. [1]: https://docs.rs/crate/rust-s3/0.28.0/source/Cargo.toml https://docs.rs/crate/rust-s3/0.28.0/source/Cargo.toml

ATsch 5y ago
As I understand it, the crates themselves are not stored on github, but the crate index is, as it uses a git repo to get "free" delta compression and auditing.

anothernewdude 5y ago

I can't stand builds that just reach out to the internet for things.

lanstin 5y ago
Yeah seems like basic hygiene especially given the supply chain attacks but also a lost cause. No one has the skills to do builds without the internet. Even forcing teams to use an allow list for the internet involving fighting a lot of angry people.
- contingencies 5y ago
  One of the few decent uses for containers is to enforce proxied internet so build process artifacts can be auto-stored for subsequent builds. For the worst offender I am aware of, try building a flutter project... it silent internet gets artifacts from at least three different packaging systems (node, cocoapods, android packages), all of which have caused hard-to-debug failures.
duped 5y ago
How do you build software in 2021 then? Do you write everything from scratch?
- spoils19 5y ago
  ... yes? I've often been ridiculed for wanting to build everything in-house, but events like these just validate that sometimes, to get the most reliable and best software, you'll have to reinvent the wheel a few times.
  - duped 5y ago
    To be honest that's deserving of ridicule. Reinventing wheels is how you introduce fragility and instability, not prevent it.
- still_grokking 5y ago

etra0 5y ago

Beside the `--offline`, you can also use `--vendor` to include all the dependencies in a folder to be committed alongside your project. Useful when you don't want to rely on external fetch every time!

jiggawatts 5y ago
Both should be the default. Rust has a robust memory model, but everything else about it insists on copying the fragility of the NPM ecosystem. The recent hoopla around a bunch of Rust mods quitting revealed that my suspicions are precisely true — key Rust staff also sit on the NPM board!
- pclmulqdq 5y ago
  Cargo is the one thing keeping me off the rust ecosystem. The fundamentals of the language are great, but the tight coupling of the rust language with cargo's package management really irks me - it introduces as many correctness and security problems as the memory model solves.
  - duped 5y ago
    What languages have package management systems that solve those problems? Cargo does have options to fix these, like mentioned in other comments. I'm not convinced they should be defaults.
    - pclmulqdq 5y ago
      I'm suggesting that "languages that have package managers" is generally the problem. I think the go package manager gets the closest to solving these problems thanks to its reliance on one trusted source that has STRONG incentives to make sure that packages are available and trusted, but fundamentally, a language and a package manager are very different products, and I don't want them to be bundled.

dandotway 5y ago

In the 1970s-80s we managed with email lists and tarballs on multiple mirrors. Git itself decentralizes source control, and yet we all want to use single-point-of-failure Github. Anyone remember when Github took down youtube-dl? I wonder how much Big Brother data Microsoft is gathering on Github developers. "Oh, as part of our hiring process, just like we scan your FB and Twitter, we also scan your Github activity to evaluate your performance as best as our machine learning overlords can assess it. Do you create Github projects and then abandon them when unfixed issues accumulate? Our algorithm thinks you're a bad hire."

adeelk93 5y ago
> Anyone remember when Github took down youtube-dl? I would blame the law (DMCA), not those forced to abide by the law (Github)
- Teever 5y ago
  Your argument is pure sophistry. Microsoft is a member of the RIAA and works closely with the MPAA to further control over society with DRM and intellectual property restrictions like the DMCA. It is well within the power of an entity the size of Microsoft to oppose something like the DMCA and the fact that they don't indicates not that they can't as you imply, but that they don't want to.
- skeeter2020 5y ago
  Especially because they worked to try and make sure this wouldn't happen again.
- deleted 5y ago
  [deleted]
- kevin_thibedeau 5y ago
  The law also permits content to be restored immediately upon receipt of a counternotice. Somehow the data silos never get around to supporting that.

dom96 5y ago

Should be fairly easy for Cargo to solve this: why doesn't it already mirror its source of truth to GitLab and other Git hosters?

nixgeek 5y ago
Probably because the level of effort required to configure that and then monitor/manage it continuing to operate doesn't make sense when GitHub has an outage or two a year.

fowlie 5y ago

I was working on my Nix config. I had just added a small command line utility and wanted to install it, but then got 504 errors from github.com. Annoying!

strenholme 5y ago

This is why a lot of companies use a product like jFrog artifactory to handle dependencies. Instead of trying to download files from GitHub and other random places on the internet, the artifactory is a local mirror of all of those files, with servers on the artifactory which run the relevant services such as a npm repo of packages, a npm repo of packages, and so on. This way, if GitHub falls off of the face of the earth tomorrow, the packages still build. Also: If some open source maintainer deletes an older version of a package, replacing it with a new one with a different API, the local package will still build. (The local build process will have to be set up to use the local artifactory to pull dependencies, of course.)

avjinder 5y ago

I just got an itch to try out Emacs, after years of vim. Installed Spacemacs and opened it. It tried to install a bunch of packages, mostly from Github it seems because it kept failing.