16 ms·

As an SRE Manager, this is causing me a hell of a headache this morning. In 30 days a bunch of images we depend on may just disappear. We mostly depend on ima

As an SRE Manager, this is causing me a hell of a headache this morning. In 30 days a bunch of images we depend on may just disappear. We mostly depend on images from relatively large organizations (`alpine`, `node`, `golang`, etc), so one would want to believe that we'll be fine - they're all either in the open source program or will pay. But I can't hang my hat on that. If those images disappear, we lose the ability to release and that's not acceptable. There's no way for us to see which organizations have paid and which haven't. Which are members of the open source program and which aren't. I can't even tell which images are likely at risk. The best I can come up with, at the moment, is waiting for each organization to make some sort of announcement with one of "We've paid, don't worry", "We're migrating, here's where", or "We've applied to the open source program". And if organizations don't do that... I mean, 30 days isn't enough time to find alternatives and migrate. So we're just left basically hoping that nothing blows up in 30 days. And companies that do that to me give me a very strong incentive to never use their products and tools if I can avoid it.

ohgodplsno 4y ago
Or you could, you know, host a Docker registry and reupload those images to something you control. Worst case scenario, in 30 days, nothing is gone from Docker and you can just spin it down. Your job as an SRE is not to look at things and go "oh well, nothing we can do lol".
- dbingham 4y ago
  Yes, that involves ripping out Docker Hub everywhere. It's a significant chunk of work, not something easily fit into 30 days on a team that is already strapped for resources with more work than we can do.
  - mysterydip 4y ago
    I'm not familiar with how Docker works, so forgive the ignorance. I thought the point of docker images was portability? Is it not just taking the references and pointing to a new instance under your control?
    - creshal 4y ago
      I'm not too familiar with docker myself, but gitlab's selfhosted omnibus includes a container registry that Just Works™ for our small team.
    - friendzis 4y ago
      Most production workloads do not use docker directly, but rather use it as sorts of "installation format" that other services schedule (spin up, connect, spin down, upgrade). One of typical defaults is to always try and pull new image even if requested version is available in node-local cache. On one hand it prevents issues where services would fail to start on certain nodes in the event of repository downtime. On the other hand it blocks service startup altogether. With such a set up availability of registry is mission critical for continuous operation. Some people think it is a perfectly reasonable idea to set up defaults to always pull, point to latest version and not have local cache/mirror. Judging from the number of upvotes on OP, depending on third party remote without any SLA to be always available for production workloads seems to be the default.
  - Faaak 4y ago
    Setting up harbor as a docker proxy-cache is actually quite simple
    - djbusby 4y ago
      The link https://goharbor.io/docs/2.1.0/administration/configure-proxy-cache/ https://goharbor.io/docs/2.1.0/administration/configure-prox...
    - pram 4y ago
      [flagged]
- bushbaba 4y ago
  That’s unplanned work. There’s other work needing to be done as well.
  - ohgodplsno 4y ago
    And a sudden fire is also unplanned work, but that's still your work. If this is such a threat, then maybe shift priorities around.
    - mynameisvlad 4y ago
      Are we not allowed to complain about unnecessary unplanned work being foisted on us with 30 days notice? That seems like an entirely relevant complaint for this forum but from your first reply, you’re acting like somehow it’s the greatest offense in the world that someone pointed this out.
      - ohgodplsno 4y ago
        Come on, 30 days notice is a walk in the park. Additionally, OP was the one complaining that changing a few URLs and eventually spinning up a new server. It's quite literally a one day or two job, unless you're at a company the size of Amazon (in which case, luckily for you, you're not the only SRE, so it's still just a few days). > The best I can come up with, at the moment, is waiting for each organization to make some sort of announcement with one of "We've paid, don't worry", "We're migrating, here's where", or "We've applied to the open source program". And if organizations don't do that... I mean, 30 days isn't enough time to find alternatives and migrate. This is the original comment. The best they can come up with is... do nothing and wait to see if the smoke turns into a fire ? I've seen better uses of time. 30 days is enough time to find an alternative, migrate _and_ get regular coffee breaks too.
        mynameisvlad 4y ago
        30 days is nowhere near enough times for people with real jobs that have other things to do rather than drop everything to do this. Once again, completely needlessly. You’re making a mountain out of an entirely valid complaint. Quoting your own profile, stay mad.
- fieldcny 4y ago
  i didn't read it as that, they were stating the realities, assumptions were made, those assumptions are now invalid, they are working on alternatives, 30days is a short deadline for something like this and docker as an organization is behaving poorly. all of that seems pretty true, and frankly no one should support a company that does something like this. I get they need to figure out how to make money, but time has shown the worst way to do that is to screw over customers or potential customers. I like the poster will never trust docker, and will never use their tooling or formats, pod-man all the way.
  - unilynx 4y ago
    Earlier events already had us slowly switching out docker for pod man, and the tooling is more similar than I had expected. Half of the work is ensuring the images are explicitly prefixed with docker.io/ And this week it turns out that makes the now problematic spots a lot more greppable
- hoherd 4y ago
  Imagine you shipped software that included references to docker hub images. That software will no longer work if any of the referenced images are deleted from docker hub. This will be the case with any helm charts that reference images that are deleted from docker hub. Some of those charts will not have variables that let you override the docker images and tags, so some of those will not be usable without creating a new release. This is one of the primary reasons to vendor your third party docker images into a docker registry that you control.
  - aprdm 4y ago
    Yes vendor them all too.
  - account42 4y ago
    [flagged]
  - ohgodplsno 4y ago
    "Don't release software that can pull code from random services on the internet then execute it without making that configurable" has been standard since the internet was available, just about. Vendor your helm charts if they are production critical. Vendor the docker images if they are production critical. Vendor the libraries if they are critical. As an added bonus, you even help making a saner internet where you don't pull left-pad three billion times a month.
- planetafro 4y ago
  This exactly. We have pipelines designed specifically for this reason. We pull, patch, and perform minor edits to images we use. We then version lock all-the-things for consistency. Not saying this is good news, but in the Enterprise, you have to plan for shit like this.
richardwhiuk 4y ago
You should be escrowing any Docker images you depend on, I'd have thought.
- nine_k 4y ago
  Good thing is that setting up an image registry in AWS is so simple! (Ha-ha, only serious.)
  - ZiiS 4y ago
    Is https://aws.amazon.com/ecr/ https://aws.amazon.com/ecr/ no good?
    - nine_k 4y ago
      It exactly is good.
    - web3-is-a-scam 4y ago
      It's great. We migrated our images to it months ago (not because of this, bandwidth issues mainly, we also vendor our base images on it) and it has given us exactly 0 problems.
      - GabeIsko 4y ago
        This is what I don't understand about Docker's policy switch. Aren't all the companies that would potentially pay them just going to switch to one of the main CSPs Container Registry service?
        web3-is-a-scam 4y ago
        It’s baffling. Moving registries is extremely easy.
        ZiiS 4y ago
        I assume that is what they want. Without the jucy margins on running the containers hosting images is never going to be profitable.
sergiotapia 4y ago
If your business is depending on these open source projects to exist, shouldn't you be paying them so they can then pay for Docker?
- goodpoint 4y ago
  No, business love freeloading.
- dahdum 4y ago
  Not every open source project wants to deal with donations / payments that could force incorporation, tax filings, bank accounts, credit/debit cards, and other paperwork. I certainly wouldn't want to deal with that for a side project.
  - BlueTemplar 4y ago
    If you are part of an organization, you already need to deal with most of those ?
    - tetha 4y ago
      You might need to deal with this on the receiving end.
TimWolla 4y ago
The images you mention (alpine, node, golang) are all so-called “Docker Official Images”. Those are all the ones without a slash as the namespace separator in them: https://hub.docker.com/search?q=&type=image&image_filter=official https://hub.docker.com/search?q=&type=image&image_filter=off... They are versioned and reviewed here: https://github.com/docker-library/official-images https://github.com/docker-library/official-images I don't expect them to go away. Disclosure: I maintain two of them (spiped, adminer).
- karamanolev 4y ago
  Useful information, bad look for Docker - "Oh, no slash as the namespace separator. Good and easy way to tell, that's how I would've done it!".
  - cshimmin 4y ago
    I mean, it's not a terrible convention. On the website they have a badge ("docker official image"), but devs aren't usually looking at the website, they're looking at their Dockerfile in vim or whatever. This is a straightforward way to communicate that semantically through namespacing. Still, shame on docker for the rug-pull.
    - zamnos 4y ago
      It's better than none, but explicit over implicit. If it were namespaced like PULL docker.org/offical/alpine:latest that would be better, imo.
      - _cenw 4y ago
        They are also available at docker.io/library/alpine and equivalent, and I'd advise anyone to start using this format as more distros might break the default registry[1]. [1]: https://man.archlinux.org/man/containers-registries.conf.5.en#NOTE:_RISK_OF_USING_UNQUALIFIED_IMAGE_NAMES https://man.archlinux.org/man/containers-registries.conf.5.e...
- 4y ago
deleted 4y ago
[deleted]
tyler33 4y ago
the bad thing about other computers, could happen to everybody, it is harder to use your machine but better long termn
aprdm 4y ago
You can vendor images. Never have your product depend on something that is in the internet. Spin up Harbour locally and put it in the middle to cache at the very least.
- mc4ndr3 4y ago
  Imagine if everyone actually did this. Then we would have a myriad of base images hiding even more malware than we do currently. Not to mention vertically integrating the entire Docker layer set defeats the whole point of using Docker in the first place.
  - aprdm 4y ago
    No, you're wrong. Everyone who wants to stay in business and makes money actually does it. Has been my experience in all big companies, it's a business continuity problem /not to do it/. You can and should run security in the vendored images.
  - twblalock 4y ago
    I've never worked somewhere that didn't have an internal Artifactory with copies of everything. Not doing that is unusual, and actually less secure. Do you think it's sane or secure for all of your builds to depend on downloading packages from the public internet?
  - wlesieutre 4y ago
    They're internal mirrors of public images, if there's something in your infrastructure installing malware on them you've got bigger problems
  - tehbeard 4y ago
    That's.... I don't know how you even arrived at that idea of that being what happens? Are you imagining some kludged together perl script to hackily save the tarballs, written by someone who is then immediately let go? What they're suggesting is basically setting up a cache for it locally in-between them and the "main repo" and ensuring the cache doesn't delete after x days and/or keep backups of the images they depend on. If the package disappears, or the main repo falls over (cough github, cough), your devs, CI & prod aren't sat twiddling thumbs unable to work... and if the package is nuked off the planet? You've got some time then to find an alternate / see where they move to.
friendzis 4y ago
> If those images disappear, we lose the ability to release and that's not acceptable. left-pad moment once again. > I mean, 30 days isn't enough time to find alternatives and migrate. Maybe take control of mission critical dependencies and self-host?
- Woodi 4y ago
  > Maybe take control of mission critical dependencies and self-host? Last few years prove that this option is a no-go - they just don't do such things ! Independence ? Self-sufficiency ? Security ? Local, fast access ? Obviousness ? No payment required ? Avoid at all costs !
  - dividedbyzero 4y ago
    > Last few years prove that this option is a no-go - they just don't do such things ! Who are "they"?
    - ipaddr 4y ago
      Busy DevOps crews?
  - web3-is-a-scam 4y ago
    taking responsibility for our supply chain and things we depend on that use mostly for free? absolutely preposterous, the business demands more features.
KingLancelot 4y ago
[dead]
softfalcon 4y ago
First of all, want to say, that sounds deeply frustrating. Secondly, if this is a serious worry. I would recommend creating your own private docker registry. https://docs.docker.com/registry/deploying/ https://docs.docker.com/registry/deploying/ Then I would download all current versions of the images you use within your org and push them up to said registry. It’s not a perfect solution, but you’ll be able to pull the images if they disappear and considering this will take only a few minutes to set up somewhere, could be a life saver. As well, I should note that most cloud providers also have a container registry service you can use instead of this. We use the google one to back up vital images to in case Docker Hub were to have issues. Is this a massive pain in the butt? Yup! But it sure beats failed deploys! Good luck out there!
- bravetraveler 4y ago
  As someone who maintains the registries we use globally at work, +1. I know people groan at running infrastructure, but the registry software is really well documented and flexible. If you don't need to 'push', but only pull - configuring them as pull through caches is nice for availability and reliability -- while also saving from nickle/diming. They will get things from a configurable upstream, proxy.remoteurl. Contrary to what the documentation says, this can work with anything speaking the API. Not just Dockerhub. edit: My one criticism, it's not good from an HTTPS hardening perspective. It's functional, but audits find non-issues. You'll want nginx or something in front to ensure good HSTS header coverage for non-actionable requests, for example.
  - Yeroc 4y ago
    All good points but while this saves you from the docker images disappearing it does nothing to solve the issue of those images no longer receiving important security updates and bug fixes going forward.
    - bravetraveler 4y ago
      Indeed, buying time at most :) The situation just presented an opportunity for improvement, I don't intend to suggest it as a cure - but a good step! Edit: For anyone curious, our upstream is actually the same software somewhere else, utilized by CICD. That being the origin allows for pushes, with the pull-through caches being read-only by nature
nickcw 4y ago
> Which are members of the open source program and which aren't. You can tell which are members of the open source program if you go to their docker hub page and you'll see a banner "SPONSORED OSS" Here is an example: https://hub.docker.com/r/rclone/rclone https://hub.docker.com/r/rclone/rclone
mc4ndr3 4y ago
That's a fair point, and when someone with a working brain mentions the fallout throughout the Internet that would result, I expect Docker Inc. will reverse course and embark on a PR campaign pretending it was all a mere tawdry joke.
- asda_ 4y ago
  [dead]
deleted 4y ago
[deleted]
cpitman 4y ago
Many of the responses here are talking about how to vendor/cache images instead of depending on an online registry, but remember that you also need access to a supply chain for these images. Base images will continue to be patched/updated, and you need those to keep your own images up to date. Unless the suggestion is to build all images, from the bottom up, from scratch.
- sangnoir 4y ago
  It's a stop-gap measure. There are dozens of companies chomping at the bit to replace Docker as THE docker registry: I'd bet someone at Github is very busy at this very moment.
  - hosh 4y ago
    The article talked about using the Github Container Registry, which was launched in 2020.
    - dividedbyzero 4y ago
      Those very busy people at Github may well be in marketing
- web3-is-a-scam 4y ago
  Typically when you "cache" something, you're gonna expire it at some point...no? If the image is patched, it eventually gets refreshed in the mirror. If the image dissapears at least we still have it until we figure out where the heck it went.
- ParetoOptimal 4y ago
  > Base images will continue to be patched/updated, and you need those to keep your own images up to date. Unless the suggestion is to build all images, from the bottom up, from scratch. If docker pushes people to that, hopefully more reproducible solutions like nix and it's ux friendly "porcelains" such as https://devenv.sh/ https://devenv.sh/ gain market share.
- 4y ago
leshenka 4y ago
How hard is it to spin your own registry and clone those images there? I’m not heavily invested in my company’s infrastructure but as far as I can tell we have our own docker and npm registries
maxyurk 4y ago
Use JFrog Artifactory. If you're with self hosting there's a free JFrog Container Registry edition.
SergeAx 4y ago
Our organization currently caching all and every external dependencies we are using: Go, Python, npm and .NET packages, Docker images, Linux deb packages, so everything is contained inside our perimeter. We did that after one day our self-hosted Gitlab runners were throttled and then rate-limited by some package repository and all CI pipelines halted.
xyst 4y ago
if you haven't mirrored the docker images your application needs to a private registry, then you are doing it wrong.
jjav 4y ago
> If those images disappear, we lose the ability to release and that's not acceptable. This shines light on why it is so risky (from both availability and security perspectives) to be dependent on any third party for the build pipeline of a product. I have always insisted that all dependencies must be pulled from a local source even if the ultimate origin is upstream. I am continuously surprised how many groups simply rely on some third party service (or a dozen of them) to be always and perpetually available or their product build goes boom.
- darkhelmet 4y ago
  Likewise. I've always insisted on building from in-house copies of external dependencies for precisely this kind of scenario. It astonishes me the number of people who didn't get why. Having things like docker rate-limiting/shutdowns, regular supply chain attacks, etc has been helping though. Slightly related: actually knowing for sure that you've got a handle on all of the external dependencies is sometimes harder than it should be. Building in an environment with no outbound network access turns up all sorts of terrible things - far more often than it should. The kind that worry me are supposedly self-contained packages that internally do a bunch of "curl | sudo bash" type processing in their pre/post-install scripts. Those are good to know about before it is too late.
  - jjav 4y ago
    > Building in an environment with no outbound network access turns up all sorts of terrible things Yes, highly recommended to build on such a system, it'll shake out the roaches that lie hidden. In a small startup environment, the very least to do is at least keep a local repository of all external dependencies and build off that, so that if a third party goes offline or deletes what you needed you're still good. For larger enterprises with more resources, best is to build everything from source code kept in local repositories and do those builds, as you say, in machines with no network connectivity. That way you are guaranteed that the every bit of code in your product can be (re)built from source even far in the future.
- SoftTalker 4y ago
  I run a local Ubuntu mirror for the work systems I manage, for this reason.
jayp1418 4y ago
That's why better to have NetBSD + pkgsrc combo for servers.
- drdaeman 4y ago
  You misspelled Nixpkgs ;-) I'm kidding, of course, but IIRC pkgsrc (and alikes, such as APT) has a number of limitations, for example a very limited ability to have multiple versions of the same package installed, making it less than optimal replacement. (I believe a lot of people depend on ability to spin up a new version while the old is running, then do the cutover and shut down the old one after it's not is use.)
  - goodpoint 4y ago
    > APT ... has a number of limitations ...and crucial features, like having security fixes backported.
    - pxc 4y ago
      What you're alluding to is an Ubuntu decision about what goes into their repos, and has nothing to do with apt itself.
      - goodpoint 4y ago
        If anything it's usually a Debian decision. But it's more than that.
  - anthk 4y ago
    And Guix, too. Specially Guix.
  - pxc 4y ago
    Capabilities aside, if you're reproducible and source-based, you're gonna survive binary artifact repository outages a lot better than if you're not. If there were a comparable culling of the Nixpkgs binary cache, pipelines relying on Nix for their packages would be affected in a much less invasive way: they'd see Nix silently fall back to upstream sources, and reproducibly build from source, wherever the caches binary artifacts became unavailable.
BlueTemplar 4y ago
How viable is it to fork Docker Hub ?
the8472 4y ago
Why not use your own registry with a pull-through cache?
salawat 4y ago
Time for you to locally clone the dockerfiles you're reliant on, build up your own in house repository, and then do what has been done since time immemorial. Mirror the important shit. No excuses, just do. Yes, it's work. I guarantee though, you'll be less exposed to externally created drama. Making sure your org stays up to date though, that's on you.
websap 4y ago
You probably wanna move to AWS Public ECR Gallery. They have a notion of official images. AWS is in a better position to offer long term coverage.
owaislone 4y ago
Good time/opportunity to get your team/company to invest in a registry+proxy to host all images you depend on.
amouat 4y ago
Just to be clear, the official images are definitely not at risk, and I say that as a Docker captain. Official images are hugely important to Docker, now and going forward.
phpisthebest 4y ago
Any organization that has the means to pay, should pay for another service that is not openly hostile to users...
caeril 4y ago
This whole thing is so weird. Why do so many organizations depend on the internet to function? It wasn't too long ago that it was standard practice to vendor your dependencies; that is, dump your dependencies into a vendor/ directory and keep that directory updated and backed up. But now, you all think it's 100% acceptable to just throw your hands up if github is down, or a maven repository is down, or docker hub makes a policy change? Every year that goes by it becomes clear that we are actually regressing as a profession.
- phaedrus 4y ago
  There are some places that still work the old way, such as where I work - and we're finding we're increasingly out-of-touch with younger developers who grew up in a connected world. We had a recent college grad engineer (developer) who didn't work out as a hire. Some examples of the disconnect: Try as I might, I couldn't get him to understand the difference between "git" the tool and "Github" the website. He kept making me nervous because he'd slip up and use the two terms interchangeably. (We have sensitive data that shouldn't be uploaded to the cloud.) He didn't seem to completely understand files/folders and the desktop metaphor. He didn't seem to understand the difference between personal devices and work devices. The last straw for our boss to let him go was he turned in a project that used a free web service on the cloud to upload data and get back the rows sorted. (Refer back to what I said above about: sensitive data.) It didn't appear he was being obstinate, it was a tech-cultural difference. "Radical semantic disconnect" as I've seen the term used in science fiction.
  - FearNotDaniel 4y ago
    Oh dear, sounds like a tricky one, but I'm not sure that the "cultural" difference is what really mattered: the question is, did he have a willingness to understand where his worldview was falling short; to see that his limited experience of the world and college education was only a tiny subset of human experience and technical practices; to actively engage with those differences and continue learning? Unfortunately over the years I've also had some bad experiences with recent grads and the biggest problems usually boiled down to arrogance rather than ignorance... e.g. I can sort-of understand somewhere along the line that someone could have mistakenly learned that this new thing called "unicode" was invented for unix machines (after all, the names sort of sound similar) and that therefore we have absolutely no business trying to use it on a Windows system. But to then absolutely insist until you are blue in the face that you must be right about this because you learned it in college and everyone else in the team is just wrong, no matter what evidence is produced... well that is a difficult situation that I have personally encountered.
yenda 4y ago
Sounds like you could save yourself some time and budget by offering to pay for those images your are using?
ownagefool 4y ago
> I mean, 30 days isn't enough time to find alternatives and migrate. Write a script to iterate the images and push them to your own registry. This will buy you time in the event anything does dissapear.
web3-is-a-scam 4y ago
This is why our team vendors the images we depend on into AWS Elastic Container Registry.
eecc 4y ago
> If those images disappear, we lose the ability to release and that's not acceptable It’s your responsibility to ensure your own business continuity. You should review how your build pipeline depends on resources outside of your org perimeter, and deploy a private registry under your own control. btw, you could also contribute some mirroring bandwidth to the community. You must’ve heard that the cloud is just someone else’s computer.
agilob 4y ago
Very first thing you should have is a mirroring docker hub proxy. Im surprised SRE manager doesn't have it, why?
anecdotal1 4y ago
What's so hard about making your team build and host the images you rely on? Install Gitlab, clone these projects onto it, it will usually detect and build the container images. You may have to manually fire off builds for older tags/branches but it will work
TeeMassive 4y ago
And this is why in many organizations I insist on mirroring every packages and images that we depend on.

ohgodplsno 4y ago

Or you could, you know, host a Docker registry and reupload those images to something you control. Worst case scenario, in 30 days, nothing is gone from Docker and you can just spin it down. Your job as an SRE is not to look at things and go "oh well, nothing we can do lol".

dbingham 4y ago
Yes, that involves ripping out Docker Hub everywhere. It's a significant chunk of work, not something easily fit into 30 days on a team that is already strapped for resources with more work than we can do.
- mysterydip 4y ago
  I'm not familiar with how Docker works, so forgive the ignorance. I thought the point of docker images was portability? Is it not just taking the references and pointing to a new instance under your control?
  - creshal 4y ago
    I'm not too familiar with docker myself, but gitlab's selfhosted omnibus includes a container registry that Just Works™ for our small team.
  - friendzis 4y ago
    Most production workloads do not use docker directly, but rather use it as sorts of "installation format" that other services schedule (spin up, connect, spin down, upgrade). One of typical defaults is to always try and pull new image even if requested version is available in node-local cache. On one hand it prevents issues where services would fail to start on certain nodes in the event of repository downtime. On the other hand it blocks service startup altogether. With such a set up availability of registry is mission critical for continuous operation. Some people think it is a perfectly reasonable idea to set up defaults to always pull, point to latest version and not have local cache/mirror. Judging from the number of upvotes on OP, depending on third party remote without any SLA to be always available for production workloads seems to be the default.
- Faaak 4y ago
  Setting up harbor as a docker proxy-cache is actually quite simple
  - djbusby 4y ago
    The link https://goharbor.io/docs/2.1.0/administration/configure-proxy-cache/ https://goharbor.io/docs/2.1.0/administration/configure-prox...
  - pram 4y ago
    [flagged]
bushbaba 4y ago
That’s unplanned work. There’s other work needing to be done as well.
- ohgodplsno 4y ago
  And a sudden fire is also unplanned work, but that's still your work. If this is such a threat, then maybe shift priorities around.
  - mynameisvlad 4y ago
    Are we not allowed to complain about unnecessary unplanned work being foisted on us with 30 days notice? That seems like an entirely relevant complaint for this forum but from your first reply, you’re acting like somehow it’s the greatest offense in the world that someone pointed this out.
    - ohgodplsno 4y ago
      Come on, 30 days notice is a walk in the park. Additionally, OP was the one complaining that changing a few URLs and eventually spinning up a new server. It's quite literally a one day or two job, unless you're at a company the size of Amazon (in which case, luckily for you, you're not the only SRE, so it's still just a few days). > The best I can come up with, at the moment, is waiting for each organization to make some sort of announcement with one of "We've paid, don't worry", "We're migrating, here's where", or "We've applied to the open source program". And if organizations don't do that... I mean, 30 days isn't enough time to find alternatives and migrate. This is the original comment. The best they can come up with is... do nothing and wait to see if the smoke turns into a fire ? I've seen better uses of time. 30 days is enough time to find an alternative, migrate _and_ get regular coffee breaks too.
      - mynameisvlad 4y ago
        30 days is nowhere near enough times for people with real jobs that have other things to do rather than drop everything to do this. Once again, completely needlessly. You’re making a mountain out of an entirely valid complaint. Quoting your own profile, stay mad.
fieldcny 4y ago
i didn't read it as that, they were stating the realities, assumptions were made, those assumptions are now invalid, they are working on alternatives, 30days is a short deadline for something like this and docker as an organization is behaving poorly. all of that seems pretty true, and frankly no one should support a company that does something like this. I get they need to figure out how to make money, but time has shown the worst way to do that is to screw over customers or potential customers. I like the poster will never trust docker, and will never use their tooling or formats, pod-man all the way.
- unilynx 4y ago
  Earlier events already had us slowly switching out docker for pod man, and the tooling is more similar than I had expected. Half of the work is ensuring the images are explicitly prefixed with docker.io/ And this week it turns out that makes the now problematic spots a lot more greppable
hoherd 4y ago
Imagine you shipped software that included references to docker hub images. That software will no longer work if any of the referenced images are deleted from docker hub. This will be the case with any helm charts that reference images that are deleted from docker hub. Some of those charts will not have variables that let you override the docker images and tags, so some of those will not be usable without creating a new release. This is one of the primary reasons to vendor your third party docker images into a docker registry that you control.
- aprdm 4y ago
  Yes vendor them all too.
- account42 4y ago
  [flagged]
- ohgodplsno 4y ago
  "Don't release software that can pull code from random services on the internet then execute it without making that configurable" has been standard since the internet was available, just about. Vendor your helm charts if they are production critical. Vendor the docker images if they are production critical. Vendor the libraries if they are critical. As an added bonus, you even help making a saner internet where you don't pull left-pad three billion times a month.
planetafro 4y ago
This exactly. We have pipelines designed specifically for this reason. We pull, patch, and perform minor edits to images we use. We then version lock all-the-things for consistency. Not saying this is good news, but in the Enterprise, you have to plan for shit like this.

richardwhiuk 4y ago

You should be escrowing any Docker images you depend on, I'd have thought.

nine_k 4y ago
Good thing is that setting up an image registry in AWS is so simple! (Ha-ha, only serious.)
- ZiiS 4y ago
  Is https://aws.amazon.com/ecr/ https://aws.amazon.com/ecr/ no good?
  - nine_k 4y ago
    It exactly is good.
  - web3-is-a-scam 4y ago
    It's great. We migrated our images to it months ago (not because of this, bandwidth issues mainly, we also vendor our base images on it) and it has given us exactly 0 problems.
    - GabeIsko 4y ago
      This is what I don't understand about Docker's policy switch. Aren't all the companies that would potentially pay them just going to switch to one of the main CSPs Container Registry service?
      - web3-is-a-scam 4y ago
        It’s baffling. Moving registries is extremely easy.
      - ZiiS 4y ago
        I assume that is what they want. Without the jucy margins on running the containers hosting images is never going to be profitable.

sergiotapia 4y ago

If your business is depending on these open source projects to exist, shouldn't you be paying them so they can then pay for Docker?

goodpoint 4y ago
No, business love freeloading.
dahdum 4y ago
Not every open source project wants to deal with donations / payments that could force incorporation, tax filings, bank accounts, credit/debit cards, and other paperwork. I certainly wouldn't want to deal with that for a side project.
- BlueTemplar 4y ago
  If you are part of an organization, you already need to deal with most of those ?
  - tetha 4y ago
    You might need to deal with this on the receiving end.

TimWolla 4y ago

The images you mention (alpine, node, golang) are all so-called “Docker Official Images”. Those are all the ones without a slash as the namespace separator in them: https://hub.docker.com/search?q=&type=image&image_filter=official https://hub.docker.com/search?q=&type=image&image_filter=off... They are versioned and reviewed here: https://github.com/docker-library/official-images https://github.com/docker-library/official-images I don't expect them to go away. Disclosure: I maintain two of them (spiped, adminer).

karamanolev 4y ago
Useful information, bad look for Docker - "Oh, no slash as the namespace separator. Good and easy way to tell, that's how I would've done it!".
- cshimmin 4y ago
  I mean, it's not a terrible convention. On the website they have a badge ("docker official image"), but devs aren't usually looking at the website, they're looking at their Dockerfile in vim or whatever. This is a straightforward way to communicate that semantically through namespacing. Still, shame on docker for the rug-pull.
  - zamnos 4y ago
    It's better than none, but explicit over implicit. If it were namespaced like PULL docker.org/offical/alpine:latest that would be better, imo.
    - _cenw 4y ago
      They are also available at docker.io/library/alpine and equivalent, and I'd advise anyone to start using this format as more distros might break the default registry[1]. [1]: https://man.archlinux.org/man/containers-registries.conf.5.en#NOTE:_RISK_OF_USING_UNQUALIFIED_IMAGE_NAMES https://man.archlinux.org/man/containers-registries.conf.5.e...
4y ago

deleted 4y ago

[deleted]

tyler33 4y ago

the bad thing about other computers, could happen to everybody, it is harder to use your machine but better long termn

aprdm 4y ago

You can vendor images. Never have your product depend on something that is in the internet. Spin up Harbour locally and put it in the middle to cache at the very least.

mc4ndr3 4y ago
Imagine if everyone actually did this. Then we would have a myriad of base images hiding even more malware than we do currently. Not to mention vertically integrating the entire Docker layer set defeats the whole point of using Docker in the first place.
- aprdm 4y ago
  No, you're wrong. Everyone who wants to stay in business and makes money actually does it. Has been my experience in all big companies, it's a business continuity problem /not to do it/. You can and should run security in the vendored images.
- twblalock 4y ago
  I've never worked somewhere that didn't have an internal Artifactory with copies of everything. Not doing that is unusual, and actually less secure. Do you think it's sane or secure for all of your builds to depend on downloading packages from the public internet?
- wlesieutre 4y ago
  They're internal mirrors of public images, if there's something in your infrastructure installing malware on them you've got bigger problems
- tehbeard 4y ago
  That's.... I don't know how you even arrived at that idea of that being what happens? Are you imagining some kludged together perl script to hackily save the tarballs, written by someone who is then immediately let go? What they're suggesting is basically setting up a cache for it locally in-between them and the "main repo" and ensuring the cache doesn't delete after x days and/or keep backups of the images they depend on. If the package disappears, or the main repo falls over (cough github, cough), your devs, CI & prod aren't sat twiddling thumbs unable to work... and if the package is nuked off the planet? You've got some time then to find an alternate / see where they move to.

friendzis 4y ago

> If those images disappear, we lose the ability to release and that's not acceptable. left-pad moment once again. > I mean, 30 days isn't enough time to find alternatives and migrate. Maybe take control of mission critical dependencies and self-host?

Woodi 4y ago
> Maybe take control of mission critical dependencies and self-host? Last few years prove that this option is a no-go - they just don't do such things ! Independence ? Self-sufficiency ? Security ? Local, fast access ? Obviousness ? No payment required ? Avoid at all costs !
- dividedbyzero 4y ago
  > Last few years prove that this option is a no-go - they just don't do such things ! Who are "they"?
  - ipaddr 4y ago
    Busy DevOps crews?
- web3-is-a-scam 4y ago
  taking responsibility for our supply chain and things we depend on that use mostly for free? absolutely preposterous, the business demands more features.

KingLancelot 4y ago

[dead]

softfalcon 4y ago

First of all, want to say, that sounds deeply frustrating. Secondly, if this is a serious worry. I would recommend creating your own private docker registry. https://docs.docker.com/registry/deploying/ https://docs.docker.com/registry/deploying/ Then I would download all current versions of the images you use within your org and push them up to said registry. It’s not a perfect solution, but you’ll be able to pull the images if they disappear and considering this will take only a few minutes to set up somewhere, could be a life saver. As well, I should note that most cloud providers also have a container registry service you can use instead of this. We use the google one to back up vital images to in case Docker Hub were to have issues. Is this a massive pain in the butt? Yup! But it sure beats failed deploys! Good luck out there!

bravetraveler 4y ago
As someone who maintains the registries we use globally at work, +1. I know people groan at running infrastructure, but the registry software is really well documented and flexible. If you don't need to 'push', but only pull - configuring them as pull through caches is nice for availability and reliability -- while also saving from nickle/diming. They will get things from a configurable upstream, proxy.remoteurl. Contrary to what the documentation says, this can work with anything speaking the API. Not just Dockerhub. edit: My one criticism, it's not good from an HTTPS hardening perspective. It's functional, but audits find non-issues. You'll want nginx or something in front to ensure good HSTS header coverage for non-actionable requests, for example.
- Yeroc 4y ago
  All good points but while this saves you from the docker images disappearing it does nothing to solve the issue of those images no longer receiving important security updates and bug fixes going forward.
  - bravetraveler 4y ago
    Indeed, buying time at most :) The situation just presented an opportunity for improvement, I don't intend to suggest it as a cure - but a good step! Edit: For anyone curious, our upstream is actually the same software somewhere else, utilized by CICD. That being the origin allows for pushes, with the pull-through caches being read-only by nature

nickcw 4y ago

> Which are members of the open source program and which aren't. You can tell which are members of the open source program if you go to their docker hub page and you'll see a banner "SPONSORED OSS" Here is an example: https://hub.docker.com/r/rclone/rclone https://hub.docker.com/r/rclone/rclone

mc4ndr3 4y ago

That's a fair point, and when someone with a working brain mentions the fallout throughout the Internet that would result, I expect Docker Inc. will reverse course and embark on a PR campaign pretending it was all a mere tawdry joke.

asda_ 4y ago
[dead]

deleted 4y ago

[deleted]

cpitman 4y ago

Many of the responses here are talking about how to vendor/cache images instead of depending on an online registry, but remember that you also need access to a supply chain for these images. Base images will continue to be patched/updated, and you need those to keep your own images up to date. Unless the suggestion is to build all images, from the bottom up, from scratch.

sangnoir 4y ago
It's a stop-gap measure. There are dozens of companies chomping at the bit to replace Docker as THE docker registry: I'd bet someone at Github is very busy at this very moment.
- hosh 4y ago
  The article talked about using the Github Container Registry, which was launched in 2020.
  - dividedbyzero 4y ago
    Those very busy people at Github may well be in marketing
web3-is-a-scam 4y ago
Typically when you "cache" something, you're gonna expire it at some point...no? If the image is patched, it eventually gets refreshed in the mirror. If the image dissapears at least we still have it until we figure out where the heck it went.
ParetoOptimal 4y ago
> Base images will continue to be patched/updated, and you need those to keep your own images up to date. Unless the suggestion is to build all images, from the bottom up, from scratch. If docker pushes people to that, hopefully more reproducible solutions like nix and it's ux friendly "porcelains" such as https://devenv.sh/ https://devenv.sh/ gain market share.
4y ago

leshenka 4y ago

How hard is it to spin your own registry and clone those images there? I’m not heavily invested in my company’s infrastructure but as far as I can tell we have our own docker and npm registries

maxyurk 4y ago

Use JFrog Artifactory. If you're with self hosting there's a free JFrog Container Registry edition.

SergeAx 4y ago

Our organization currently caching all and every external dependencies we are using: Go, Python, npm and .NET packages, Docker images, Linux deb packages, so everything is contained inside our perimeter. We did that after one day our self-hosted Gitlab runners were throttled and then rate-limited by some package repository and all CI pipelines halted.

xyst 4y ago

if you haven't mirrored the docker images your application needs to a private registry, then you are doing it wrong.

jjav 4y ago

> If those images disappear, we lose the ability to release and that's not acceptable. This shines light on why it is so risky (from both availability and security perspectives) to be dependent on any third party for the build pipeline of a product. I have always insisted that all dependencies must be pulled from a local source even if the ultimate origin is upstream. I am continuously surprised how many groups simply rely on some third party service (or a dozen of them) to be always and perpetually available or their product build goes boom.

darkhelmet 4y ago
Likewise. I've always insisted on building from in-house copies of external dependencies for precisely this kind of scenario. It astonishes me the number of people who didn't get why. Having things like docker rate-limiting/shutdowns, regular supply chain attacks, etc has been helping though. Slightly related: actually knowing for sure that you've got a handle on all of the external dependencies is sometimes harder than it should be. Building in an environment with no outbound network access turns up all sorts of terrible things - far more often than it should. The kind that worry me are supposedly self-contained packages that internally do a bunch of "curl | sudo bash" type processing in their pre/post-install scripts. Those are good to know about before it is too late.
- jjav 4y ago
  > Building in an environment with no outbound network access turns up all sorts of terrible things Yes, highly recommended to build on such a system, it'll shake out the roaches that lie hidden. In a small startup environment, the very least to do is at least keep a local repository of all external dependencies and build off that, so that if a third party goes offline or deletes what you needed you're still good. For larger enterprises with more resources, best is to build everything from source code kept in local repositories and do those builds, as you say, in machines with no network connectivity. That way you are guaranteed that the every bit of code in your product can be (re)built from source even far in the future.
SoftTalker 4y ago
I run a local Ubuntu mirror for the work systems I manage, for this reason.

jayp1418 4y ago

That's why better to have NetBSD + pkgsrc combo for servers.

drdaeman 4y ago
You misspelled Nixpkgs ;-) I'm kidding, of course, but IIRC pkgsrc (and alikes, such as APT) has a number of limitations, for example a very limited ability to have multiple versions of the same package installed, making it less than optimal replacement. (I believe a lot of people depend on ability to spin up a new version while the old is running, then do the cutover and shut down the old one after it's not is use.)
- goodpoint 4y ago
  > APT ... has a number of limitations ...and crucial features, like having security fixes backported.
  - pxc 4y ago
    What you're alluding to is an Ubuntu decision about what goes into their repos, and has nothing to do with apt itself.
    - goodpoint 4y ago
      If anything it's usually a Debian decision. But it's more than that.
- anthk 4y ago
  And Guix, too. Specially Guix.
- pxc 4y ago
  Capabilities aside, if you're reproducible and source-based, you're gonna survive binary artifact repository outages a lot better than if you're not. If there were a comparable culling of the Nixpkgs binary cache, pipelines relying on Nix for their packages would be affected in a much less invasive way: they'd see Nix silently fall back to upstream sources, and reproducibly build from source, wherever the caches binary artifacts became unavailable.

BlueTemplar 4y ago

How viable is it to fork Docker Hub ?

the8472 4y ago

Why not use your own registry with a pull-through cache?

salawat 4y ago

Time for you to locally clone the dockerfiles you're reliant on, build up your own in house repository, and then do what has been done since time immemorial. Mirror the important shit. No excuses, just do. Yes, it's work. I guarantee though, you'll be less exposed to externally created drama. Making sure your org stays up to date though, that's on you.

websap 4y ago

You probably wanna move to AWS Public ECR Gallery. They have a notion of official images. AWS is in a better position to offer long term coverage.

owaislone 4y ago

Good time/opportunity to get your team/company to invest in a registry+proxy to host all images you depend on.

amouat 4y ago

Just to be clear, the official images are definitely not at risk, and I say that as a Docker captain. Official images are hugely important to Docker, now and going forward.

phpisthebest 4y ago

Any organization that has the means to pay, should pay for another service that is not openly hostile to users...

caeril 4y ago

This whole thing is so weird. Why do so many organizations depend on the internet to function? It wasn't too long ago that it was standard practice to vendor your dependencies; that is, dump your dependencies into a vendor/ directory and keep that directory updated and backed up. But now, you all think it's 100% acceptable to just throw your hands up if github is down, or a maven repository is down, or docker hub makes a policy change? Every year that goes by it becomes clear that we are actually regressing as a profession.

phaedrus 4y ago
There are some places that still work the old way, such as where I work - and we're finding we're increasingly out-of-touch with younger developers who grew up in a connected world. We had a recent college grad engineer (developer) who didn't work out as a hire. Some examples of the disconnect: Try as I might, I couldn't get him to understand the difference between "git" the tool and "Github" the website. He kept making me nervous because he'd slip up and use the two terms interchangeably. (We have sensitive data that shouldn't be uploaded to the cloud.) He didn't seem to completely understand files/folders and the desktop metaphor. He didn't seem to understand the difference between personal devices and work devices. The last straw for our boss to let him go was he turned in a project that used a free web service on the cloud to upload data and get back the rows sorted. (Refer back to what I said above about: sensitive data.) It didn't appear he was being obstinate, it was a tech-cultural difference. "Radical semantic disconnect" as I've seen the term used in science fiction.
- FearNotDaniel 4y ago
  Oh dear, sounds like a tricky one, but I'm not sure that the "cultural" difference is what really mattered: the question is, did he have a willingness to understand where his worldview was falling short; to see that his limited experience of the world and college education was only a tiny subset of human experience and technical practices; to actively engage with those differences and continue learning? Unfortunately over the years I've also had some bad experiences with recent grads and the biggest problems usually boiled down to arrogance rather than ignorance... e.g. I can sort-of understand somewhere along the line that someone could have mistakenly learned that this new thing called "unicode" was invented for unix machines (after all, the names sort of sound similar) and that therefore we have absolutely no business trying to use it on a Windows system. But to then absolutely insist until you are blue in the face that you must be right about this because you learned it in college and everyone else in the team is just wrong, no matter what evidence is produced... well that is a difficult situation that I have personally encountered.

yenda 4y ago

Sounds like you could save yourself some time and budget by offering to pay for those images your are using?

ownagefool 4y ago

> I mean, 30 days isn't enough time to find alternatives and migrate. Write a script to iterate the images and push them to your own registry. This will buy you time in the event anything does dissapear.

web3-is-a-scam 4y ago

This is why our team vendors the images we depend on into AWS Elastic Container Registry.

eecc 4y ago

> If those images disappear, we lose the ability to release and that's not acceptable It’s your responsibility to ensure your own business continuity. You should review how your build pipeline depends on resources outside of your org perimeter, and deploy a private registry under your own control. btw, you could also contribute some mirroring bandwidth to the community. You must’ve heard that the cloud is just someone else’s computer.

agilob 4y ago

Very first thing you should have is a mirroring docker hub proxy. Im surprised SRE manager doesn't have it, why?

anecdotal1 4y ago

What's so hard about making your team build and host the images you rely on? Install Gitlab, clone these projects onto it, it will usually detect and build the container images. You may have to manually fire off builds for older tags/branches but it will work

TeeMassive 4y ago

And this is why in many organizations I insist on mirroring every packages and images that we depend on.