16 ms·
Docker update ToS: Image retention limits imposed on free accounts
- laksjd 6y agoI just got an email notification and while I can understand that they're doing this (all those GB must add up to a significant cost), the relatively short notice seems unnecessary.
- deleted 6y ago[deleted]
- ownagefool 6y ago6 month notice doesn't seem terrible for a free service imo.
- amenod 6y ago3 month notice - they will start on Nov 1st. Still not bad.
- ownagefool 6y agoImage retention is 6 months though, so it seems slightly unclear if the timer starts counting from today or from Nov 1st. Best to assume the worst but still plenty of time to write a cron that pulls all your images, assuming for some reason you need images you don't pull for > 6 months.
- francislavoie 6y agoThat includes public images? That'll hurt OSS. That's a bummer. It wouldn't surprise me if people move to Github's registry for open source projects. https://github.com/features/packages https://github.com/features/packages
- gramakri 6y agoOnly if they never got pulled for 6 months
- jimktrains2 6y agoIf it's oss there should be a docker file available to build yourself?
- gruez 6y ago>It wouldn't surprise me if people move to Github's registry for open source projects. https://github.com/features/packages https://github.com/features/packages The egress pricing is going to be a dealbreaker. The free plan only includes 1GB out.
- toastal 6y agoGitLab also has container registry (on GitLab.com it's a part of your 10GB repository budget) https://docs.gitlab.com/ee/user/packages/container_registry/ https://docs.gitlab.com/ee/user/packages/container_registry/
- NathanKP 6y agoProbably not at 50 cents per GB of data transfer outside of Github Actions. Unfortunately the only place you can viably use Github registry right now is inside Github actions
- laurencerowe 6y agoThat pricing is for private repos. It's free for public repos.
- nickjj 6y agoThe FAQ[0] says pulling an image once every 6 months will prevent it from being purged by resetting the timer. It doesn't seem like a big deal really. It just means old public images from years ago that haven't been pulled or pushed to will get removed. [0]: https://www.docker.com/pricing/retentionfaq https://www.docker.com/pricing/retentionfaq
- klysm 6y agolooks like there will be some bots that pull images on a periodic basis cropping up
- dgellow 6y agoYep, everybody will have a small scheduled Github Action pulling their image once per month or similar ¯\_(ツ)_/¯
- jacekm 6y agoWhy not just pay this 60$/year? I mean if it's something important then it's worth paying for. If not - there is cheaper storage available when one can archive their containers.
- klysm 6y agoI agree it’s probably better to just pay, but certainly if it’s that cheap to circumvent then people will do it to save $60
- res0nat0r 6y agoEveryone should just start using Googles Cloud Build service IMO, it will cost you pennies. You can literally just do a `docker build -t gcr.io/project/image:0.0.1` and it will automatically tar up and send your build directly to their build service and create the container. It's about the cheapest and easiest build service I've seen. https://cloud.google.com/cloud-build https://cloud.google.com/cloud-build
- 6y ago
- rmoriz 6y agoSo this means that open source projects need to pay to keep older images alive?
- PaywallBuster 6y agotl;dr images hosted on free accounts without downloads for 6 months will be (scheduled) removed.
- oauea 6y agoTime for someone to create a new service that will pull your images into /dev/null once a month.
- wildpeaks 6y agoNo need for a new service, a simple Github Action with a cron trigger can do it.
- mr__y 6y agoThis could also be solved by one person running a service that would crawl all public docker images and pull those that are close to expiration automatically every 6 months. At this moment I'm just curious how much resources would be needed for that
- remram 6y agoIf you just need to send the request, not read the content in full, that can be done by one free-tier cloud VM.
- ptspts 6y agoWhich hosting provider and product provides such free-tier cloud VMs?
- judge2020 6y agoNone of the big providers offer always-free VMs, but if you could just ping the registry endpoint, Lambda might be able to do it since you get 3.2 million seconds of compute "always free"[0]. Same for Google Cloud Run, although it's only 1GB egress a month[1], and Azure Functions[2]. 0: https://aws.amazon.com/free/?all-free-tier.sort-by=item.additionalFields.SortRank&all-free-tier.sort-order=asc&awsf.Free%20Tier%20Types=tier%23always-free&all-free-tier.q=lambda&all-free-tier.q_operator=AND https://aws.amazon.com/free/?all-free-tier.sort-by=item.addi... 1: https://cloud.google.com/free#always-free-products_5 https://cloud.google.com/free#always-free-products_5 2: https://azure.microsoft.com/en-us/free/free-account-faq/ https://azure.microsoft.com/en-us/free/free-account-faq/
- wildpeaks 6y agoThis will begin November 1, 2020
- ncrmro 6y agoBeen wondering when image space would start to be a concern. Actually just set up my own private registry and pull though registry. Pretty easy stuff although no real GUI to browse as of yet. This is all sitting on my NAS running in Rancher OS
- chrisandchris 6y agoTake a look at Portus, a project maintained by SUSE, which has a pretty nice GUI for a private docker registry. https://github.com/SUSE/Portus https://github.com/SUSE/Portus
- GordonS 6y agoThis looks fantastic, thanks for posting it!
- ncrmro 6y agoActually spent some time looking at this today. It’s a bit more complex than I was hopping. As right now I’m the only user. And the only way to conecto to my registry atm is through wireguard. It is cool seeing opensuse. Same the rancher
- chrisandchris 6y agoIt‘s a bit an overkill if you‘re alone but even then, I‘m using it for my own private registry too because it‘s the nicest/easiest way IMHO for adding auth to a docker registry.
- CameronNemo 6y agoNot sure if it is production ready, but some of Cisco's containers team have been working on an OCI compliant image repository server. https://github.com/anuvu/zot https://github.com/anuvu/zot
- freedomben 6y agoIf I'm reading this correctly, a single pull every <6 months would avoid this. This seems like NBD to me. Still, I keep my images mirrored on quay.io and I would recommend that to others (disclaimer: I work for Red Hat which acquired quay.io)
- deleted 6y ago[deleted]
- brutos 6y agoThis will be quite bad for reproducible science. Publishing bioinformatics tools as containers was becoming quite popular. Many of these tools have a tiny niche audience and when a scientist wants to try to reproduce some results from a paper published years ago with a specific version of a tool they might be out of luck.
- hvs 6y agoMaybe they should switch to Github. https://github.com/features/packages https://github.com/features/packages
- toomuchtodo 6y agoOr store the containers in the Internet Archive alongside the paper. They’re just tarballs. Lots of options as long as you're comfortable with object storage.
- captn3m0 6y agoquay is another alternative.
- brutos 6y agoThis still means that tools published in the last few years until now might just be gone soon. The people who uploaded the images might have graduated or moved on and none will be there to save the work.
- icebraining 6y agoSounds like a job for the Archive Team, as long as there's some way to identify the images worth saving.
- maxfan8 6y agoYep, just mentioned it to the Archive Team IRC. We're probably going to selectively archive particular Docker images, although that's a lot of manual labor. If you have any ideas wrt to selecting important images, that'd be great.
- morpheuskafka 6y agoThis seems like a non-issue, if you need to keep a rarely-used image alive for some reason just write a cron job to pull it once every six months. If the goal is long term archival it should be entrusted to something like Internet Archive.
- bithavoc 6y agoThis is fine and completely fair, I bet is not cheap paying for storage for docker images no one cares about.
- fgribreau 6y agoDocker is the new Heroku. Cronjobs will pull images to simulate image activity
- gramakri 6y agoDoes anyone know what is docker's business model these days?
- gtirloni 6y agoAfter they sold Docker Enterprise to Mirantis, I don't know anymore. Probably hold on long enough to get acquired?
- CameronNemo 6y agoWhy would anyone want to acquire them after they sold off the majority of their client base? Did they retain a significant amount of talent?
- fapjacks 6y agoYeah.
- dawnerd 6y ago“Enterprise”
- thebouv 6y agoContinuously throw stuff at the wall to see what sticks?
- Bedon292 6y agoIf they are doing this, they should add stats on when the last time an image was pulled, so you can see what is at risk of being removed. Would be curious about a graph, like NPM has a weekly downloads one so you can see how active something is.
- manishyt 6y ago(I work for Docker). We will be updating the UI to show status of each image (active or inactive). We will be updating the FAQ shortly to clarify this.
- djsumdog 6y agoSeems like companies are relearning what they should have in the 2001 dotcom bust. Keep free stuff free and add paid stuff. If your free stuff isn't sustainable, you really should have though that through early on. This limit seems reasonable, because storage costs are expensive. But it should have been implemented day one so people have reasonable expectations on retention. Other's have mentioned open source projects and artifacts for scientific publication being two niche use cases where people still might want this data years later, but it'd be rare for it to be pulled every six months. I only have a few things on docker hub, but I'll probably move them to a self-hosted repo pretty soon. At least if it's self hosted, I know it will stay up until I die and my credit cards stop working.
- Macha 6y agoI think in Docker's case, in their original plan this free unlimited hosting was probably sustainable in a freemium model where businesses paid for Docker Enterprise and Docker.com was about marketing and user acquisition, similar to open source on GitHub.com being marketing and user acquisition for paid accounts/Github Enterprise. Its not an unreasonable strategy to provide generous free hosting if you derive some other business benefit from it (YouTube being another example). But Docker Inc. found their moat was not that deep and other projects from the big cloud providers killed the market they saw for Docker Enterprise and they sold it off. So now they just have docker.com and Docker CE - which even that has alternatives now with other runtimes existing. So they need to make docker.com a profitable business on its own or find something else to do which changes the equation significantly.
- LockAndLol 6y agoIf people really think this is a problem, they'd contribute a non-abusive solution. Writing cron jobs to pull periodically in order to artificially reset the timer is abusive. Non-abusive solutions include: - extending docker to introduce reproducible image builds - extending docker push and pull to allow discovery from different sources that use different protocols like IPFS, TahoeLAFS, or filesharing hosts I'm sure you can come up with more solutions that don't abuse the goodwill of people.
- dgellow 6y agoIf their business model isn't working for them, it's their job to fix it in a way that does. I don't see how you can put the responsibility on users. If you say to your users their data will be deleted if they don't pull it at least once per N months, well that's exactly what they will do, and they are perfectly in their right to automate that process.
- gruez 6y ago>- extending docker to introduce reproducible image builds It's already reproducible... sort of. All you need is to eliminate any outside variables that can affect the build process. This mainly takes the form of network access (eg. to run npm install, for instance).
- Legogris 6y agodocker pull and push integrated with IPFS is a great idea!
- zoobab 6y agoIPFS is only a partial solution, if you are the only one to have a copy and you pull the plug, content is gone. You would need a bot that takes care of maintained at least 3 or 5 copies always available on the whole file system.
- stevebmark 6y agoMakes sense. I don't get how Docker could offer so much free hosting in the first place. I know storage is cheap, but not this cheap. Eventually they're going to need to make these rules more stringent.
- DJHenk 6y agoI also don't get why people put all their stuff on free services like this and expect it to work until eternity. Come on, if you stop just half a second and think about it, you know it is a stupid idea and you know that one day you will have a problem. You really don't have to be a genius for that. Same goes for all these other kinds of "services" that are bundled together with things that used to be a one-time purchase, like cars, etc. Oh, I now have a t.v. that can play Netflix and Youtube, but is otherwise not extendible. But what happens in ten years? T.v. still works fine, but Netflix has gone bust and this new video-service won't work. Too bad, gonna buy a new t.v then. I can get really mad about this stupid short-sightedness everybody has these days. Spoiler alert: one day Github will be gone too.
- judge2020 6y agoEnough billion dollar companies put their weight behind Docker that you'd think someone big is already running Docker hub pushing people to use their paid offerings, but that's not the case. Google created Kubernetes which is almost always used along with Docker, but they don't directly invest in Docker, Inc (at least, based on Crunchbase) and run their own container registry at gcr.io. The same goes for Amazon and Azure where their customers are increasingly moving to Docker instead of VMs, yet none of them directly back the company.
- globular-toast 6y agoDo you save a copy of every web page you think might be useful later? I have a small archive of things I consider to be "at risk", but there are many things I enjoy that exist only on other people's servers now. I can't keep it all on my own machines forever, so the difficulty is guessing what will disappear and what won't.
- maztaim 6y agoRelying on goodwill works until that goodwill stops. Store your images locally at least as a backup, but it has other advantages.
- GordonS 6y agoHmm, the very nature of layered images presumably means big storage savings; I wonder if block-level deduplication at the repository backend would be feasible too?
- brown9-2 6y agoRegistries already do this
- GordonS 6y agoDo you mean at the filesystem level, or higher up? Have you got any sources for this?
- binman-docker 6y agoHi, I work at Docker. Registry sees each layer as a SHA and does not store multiple copies of the same SHA for obvious reasons. This is not unique to Hub, it's part of the registry design spec. Registry is open source (https://github.com/docker/distribution https://github.com/docker/distribution) and implements the OCI Distribution Specification (https://github.com/opencontainers/distribution-spec/blob/master/spec.md https://github.com/opencontainers/distribution-spec/blob/mas...) if you want to dig into it.
- GordonS 6y agoYes, that's what I meant when I mentioned layers; clearly copies of the same layer are not kept :) My question was about block-level, or other forms of deduplication.
- binman-docker 6y agoDeduplication at the block level would be dependent on the choice of storage driver (https://docs.docker.com/registry/storage-drivers/ https://docs.docker.com/registry/storage-drivers/). In the case of Hub, S3 is the storage medium and that's an object store rather than a block store. In theory you could modify the spec/application to try to break layers down into smaller pieces but I have a feeling you would reach the point of diminishing returns for normal use cases pretty quickly.
- alexellisuk 6y agoSome thoughts / scenarios: "Fine we will just pay" - I have a personal account then 4 orgs, that's ~ 500 USD / year to keep older OSS online for users of openfaas/inlets/etc. "We'll just ping the image very 6 mos" - you have to iterate and discover every image and tag in the accounts then pull them, retry if it fails. Oh and bandwidth isn't free. "Doesn't affect me" - doesn't it? If you run a Kubernetes cluster, you'll do 100 pulls in no time from free / OSS components. The Hub will rate-limit you at 100 per 6 hours (resets every 24?). That means you need to add an image pull secret and a paid unmanned user to every Kubernetes cluster you run to prevent an outage. "You should rebuild images every 6 mo anyway!" - have you ever worked with an enterprise company? They do not upgrade like we do. "It's fair, about time they charged" - I agree with this, the costs must have been insane, but why is there no provision for OSS projects? We'll see images disappear because people can't afford to pay or to justify the costs. A thread with community responses - https://twitter.com/alexellisuk/status/1293937111956099073?s=20 https://twitter.com/alexellisuk/status/1293937111956099073?s...
- robertlagrant 6y ago> "We'll just ping the image very 6 mos" - you have to iterate and discover every image and tag in the accounts then pull them, retry if it fails. Oh and bandwidth isn't free. Set up CircleCI or similar to pull all your images once a month :)
- trey-jones 6y agoI feel like the main response should be "OK, we'll just host our own Docker Registry." This has been available as a docker image since the very beginning, which might not be good enough for everyone, but I think it will work for me and mine.
- geerlingguy 6y agoNote that for OSS images, that's a non-trivial thing to do—you have to have somewhere to run the image, and somewhere to store your images (e.g. S3), both of which are non-free, and would also require more documentation and less discoverability than Docker Hub offers.
- dataminded 6y agoI'm selling a SAAS service that will pull each of your images once every 6 months...thank you Docker.
- dewlinedew 6y agoI'll do it for free
- jmondi 6y agoAdd one more coin to the "always self host" bucket. Just another example of a service that starts free, then they pull the rug from under you and hold you hostage for their ransom.
- wilsonfiifi 6y agoIn case anyone decides to self-host their docker registry, Pluralsight has a nice course [0] on that subject. [0] https://www.pluralsight.com/courses/implementing-self-hosted-docker-registry https://www.pluralsight.com/courses/implementing-self-hosted...
- SEJeff 6y agohttps://quay.io/plans/ https://quay.io/plans/ """ Can I use Quay for free? Yes! We offer unlimited storage and serving of public repositories. We strongly believe in the open source community and will do what we can to help! """
- thrownaway954 6y agogiven their track record with developers over the years, i wouldn't be surprised if microsoft scammbles to build a competitor to docker repo service and integrates it with github.
- anderspitman 6y agoIf you've never used Singularity containers[0], I highly recommend checking them out. They make a very different set of tradeoffs compared to Docker, which can be a nice fit for some uses. Here's a few of my favorites: * Images are just files. You can copy them around (or archive them) like any other file. Docker's layer system is cool but brings a lot of complexity with it. * You can build them from Docker images (it'll even pull them directly from Dockerhub). * Containers are immutable by default. * No daemon. The runtime is just an executable. * No elevated permissions needed for running. * Easy to pipe stdin/stout through a container like any other executable. [0]: https://github.com/hpcng/singularity https://github.com/hpcng/singularity
- GordonS 6y ago> * Images are just files. You can copy them around (or archive them) like any other file Never heard of Singularity before, and it does look interesting. Wanted to point out though that you can create tarballs of Docker images, copy them around, and load them into a Docker instance. This is really common for air-gapped deployments.
- anderspitman 6y agoI've never seen this mentioned in the official Docker docs. Is it a well-supported workflow?
- jacques_chester 6y agoYes. https://docs.docker.com/engine/reference/commandline/save/ https://docs.docker.com/engine/reference/commandline/save/
- jonfw 6y agoDocker save and docker load- I use this often for air-gapped installations and it works exactly as well as docker pull in my experience.
- znpy 6y agoFor everybody complaining about having to pay actual money for goods and services: if you're not okay with this you can run a self hosted registry. The out of the box registry does very little and has a very poor user experience. But nowadays there are Harbor from VMware and Quay from RedHat that are open source and easily self-hostable. We run our own Harbor instance at work and I can tell you... Docker images are NOT light. You think they are, they are not. It's easy to have images proliferate a lot and burn a lot of disk space. Under some conditions when layers are shared among too many images (can't recall the exact details here) deleting an image may result in also deleting a lot more images (and this is not the correct/expected/wanted behaviour) and that means that under some circumstances you have to retain a lot more images or layers than you think you should. The thing is, I can only wonder how much bandwidth and disk space (oh and disk space must be replicated for fault tolerance) must cost running a public registry for everybody. It hurts the open source ecosystem a bit, I understand... Maybe some middle ground will be reached, dunno. Edit: I also run harbor at home, it's not hard to setup and operate, you should really check that out.
- gramakri 6y agoWhich IaaS do you use to selfhost for the one at work? How much does the network transfer cost you? Or are they docker pulls internal network?
- znpy 6y agoWe run on openstack, managed by a local yet fairly large openstack provider (Irideos, their devops/consulting team is top notch and has helped us adopt many cloud-native technologies while still staying fairly vendor-agnostic). I can't see the bills, but we never worry about bandwidth usage and I am fairly sure that bandwidth is free, basically. Keep in mind that since we run our own harbor instance, most of the image pulls happen within our openstack network, so that does/would not count against bandwidth usage (but image storage does). In terms of bandwidth thus, we can happily set "always" as imagePullPolicy in our kubernetes clusters. Edit: openstack works remarkably well. The horizon web interface is slow as molasses but thanks to terraform we rarely have to use it.
- 6y ago
- mrweasel 6y agoThat's fantastic, my main issue with Docker Hub is that there's a ton of unmaintained and out of date images. Some just pollutes my search result, I don't care that "yes, technically there's an image that does this thing I want, but it's Ubuntu 14.04 and 4 years old". Even better, it prevents people from using these unmaintained image as a base for new project, which they will do, because many developer don't look at the Dockerfile and actually review the images they use in shipping product. As a bonus perhaps this will mean that some a the many image of extremely low quality will go away. I think it's fair, now you can either pay or maintain your images.
- swozey 6y agoYou really shouldn't be pulling someones random images off of dockerhub. If I made a POC 4 years ago on some random kubernetes configuration/tutorial that I was testing and I decided to use dockerhub to host its images (as one typically does, and it used to not have private repos) I'm not posting that for you to come consume 4 years later out of the blue in production because you found it randomly via the search. You also tend to have no idea what's in those images and what context people are creating them under. Sure, a lot of us know to check the dockerfile, github repo, etc but I have images with 10k+ downloads from OSS contributions but as you've said a whole lot of developers just grab whatever looks fitting on there. My biggest dockerhub pull has no dockerfile, no github repo, and is a core network configuration component I put up randomly just for my own testing because no docker image for it existed years ago.
- mrweasel 6y agoYou’re right, but people tend to see Docker Hub as some master registry for quality and official images, even if it never claimed to be such a thing. Reading and understanding the Dockerfile it vital, before deciding to use it in any sort of production environment. The never policy well help clean up Docker Hub.
- swozey 6y agoTotally! I'm really worried what'll happen when these docker images I made get deleted. Dockerhub doesn't give you ANY details beyond the download number (which stops at 10k+) so I can't tell if they're still getting used or what. I'm hopeful they'll add statistics/refers when this goes live.
- pjmlp 6y agoThe complaints as expected, are the usual ones from free generation, apparently Mozilla is not enough.
- zoobab 6y agoLet's mirror dockerhub on a distributed fault tolerant file system. And IPFS sucks.
- avian 6y agoIs there a way to see when an image was last pulled? I can see the last push date, but not pull.
- jacques_chester 6y agoDocker is partly to blame for its own predicament by conflating URIs with URNs. When you give an image reference as `foo/bar`, the implicit actual name is `index.docker.io/foo/bar`. That means that "which image" is mixed with "where the image is". You can't deal with them separately. Because everyone uses the shorthand, Docker gets absolutely pummeled. Meanwhile in Java-land, private Maven repos are as ordinary as dirt and a well-smoothed path. It's time for a v3 of the Registry API, to break this accidental nexus and allow purely content-addressed references.
- pcthrowaway 6y ago> the implicit actual name is `index.docker.io/foo/bar` `index.docker.io/foo/bar:latest` to be more exact, which is a URL, but not really a URI if we're being pedantic. Docker doesn't really provide an interface to address images by URI (which would be more like the SHA), though in practice, tags other than latest should function closer to a URI
- dariusj18 6y agoThe second issue, is they purposefully do not allow you to change the default domain to point to. The only thing you can do it use a pull through a proxy.
- dewlinedew 6y agoThey did this on purpose to promote their docker hub
- nhumrich 6y agoI would delete my own images to clear up room on dockerhub, but they dont have an api to remove images. the only way is to manually click the x in the UI. So, in a lot of ways, docker forced us to "abuse" their service and store thousands of images on a free/open source account. I get this change, and it was inevitable. But its still ironic that you cant delete tour own images. The best way to delete your image is to just stop using it and let docker delete it for you in 6 months.
- lightswitch05 6y agoI agree, I have a little tool called `php-version-audit` that literally becomes useless after a few weeks without an update (you can't audit your php version without the knowledge of the latest CVEs). I have manually cleaned up old images like you say by clicking through them all, but having a way to define retention limits is a feature to me.
- Bnshsysjab 6y agoRemember that time you were looking for an answer to some obscure question, you find the perfect google result - description, page title and URL all indicate it’s going to answer your question so you click it, and... nothing.. the page cannot be found. You now have that, with docker.
- voltagex_ 6y agoI wonder what kind of account the Home Assistant images are using. This could break a whole lot of stuff - and I've seen projects that don't publish a Dockerfile anywhere.
- voltagex_ 6y agoA 2019 paper says there's 47TB of Docker images on the Hub. Get scraping.
- sebazzz 6y ago> What is an “inactive” image? > An inactive image is a container image that has not been either pushed or pulled from the image repository in 6 or months. > > How can I view the status of my images > All images in your Docker Hub repository have a “Last pushed” date and can easily be accessed in the Repositories view when logged into your account. A new dashboard will also be available in Docker Hub that offers the ability to view the status of all of your container images. That still does not tell the whole story, does it? I still don't know if my image have been pulled for the last six months. Only when I pushed it.