49 ms·
Serverless-registry: A Docker registry backed by Workers and R2
- Fire-Dragon-DoL 2y agoHow's the pricing with low usage? I suspect this is great. I wanted an image registry so that I can use it to deploy with Kamal, but the $5 plan is overpriced, given I push an image maybe once every 3 months. This could solve that
- bayesianbot 2y agoI don't use much of CloudFlare services but it seems kinda cheap, $0.015/GBmonth for storage (+10GB free), Workers are charged per request and CPU time, both of which would probably be quite low for a registry so free plan would go quite far? I just set up the official registry on a VPS (for similar usage pattern) and it was a bit of work and probably much more expensive, this seems quite attractive unless I've misunderstood something.
- Fire-Dragon-DoL 2y agoYeah it does sound great. The alternative for me is to host my own docker registry on my home server. That would cost me 0 essentially (I have good internet at home)
- Alifatisk 2y ago> regitry
- stpedgwdgfhgdd 2y agoIt is unfortunate that the cloudflare dev eco system is tight into the javascript world (e.g. npm). Wasm is not there yet to be a full replacement.
- thangngoc89 2y agoI think this is wonderful. I’m running a Gitea instance in one of our dev machine just for private registry. Keeping the instance only had been extra workflow for us. But 500MB limit of layer size is a dealbreaker for AI related workflow.
- geek_at 2y agogitea does also bring their own registry though. If you self-host you can also use LFS for unlimited filesizes
- thangngoc89 2y agoI’m self-hosting gitea just for their private docker registry. LFS is actually slow for heavy deep learning workflow with millions of small files. I’m using DVC [1] instead. [1]: https://dvc.org https://dvc.org
- arjvik 2y agoAbsolutely love DVC for data version control! What storage backend are you using with DVC?
- thangngoc89 2y agoLocal (mounted NFS) from our internal NAS
- victorbjorklund 2y agoNice. I been seriously thinking about building exactly this (but Im glad someone smarter made it already)
- jacobwg 2y agoThe annoying thing about trying to implement a Docker registry on Workers and R2 is that it's so close to having everything you need, but the 500MB request body limit means Workers is unable to accept pushes of layers larger than 500MB. The limit is even lower at 100MB on the Pro plan[0]. We are running a registry that does store content on R2[1], and today this is implemented as the unholy chimera of Cloudflare Workers, AWS CloudFront, Lambda@edge, regular Lambda, S3, and R2. Pushes go first to CloudFront+Lambda@edge, content is saved in S3 first, then moved to R2 in background jobs. Once it's transited to R2, then pulls are served from R2. I would so love for Workers + R2 to actually be able to accept pushes of large layers, unfortunately I have yet to talk to anyone at Cloudflare who believes it's possible. Especially in this era of AI/ML models, some container images can have single layers in the 10-100GB range! [0] https://developers.cloudflare.com/workers/platform/limits/#request-limits https://developers.cloudflare.com/workers/platform/limits/#r... [1] https://depot.dev/docs/guides/ephemeral-registry https://depot.dev/docs/guides/ephemeral-registry
- amenghra 2y agoCan you generate a signed url to upload directly to R2? Or perform the upload in chunks?
- jacobwg 2y agoUploading in chunks could definitely solve the issue, and the OCI Distribution Specification does actually have some language about an optional chunked push API[0]. Unfortunately very few of the registry clients actually support this, critically containerd does not[1], so this means your regular `docker push` and a whole lot of ecosystem tooling does not work. This also means that the single PUT must be able to support very large pushes as a single request, possibly even larger than what R2 or S3 would allow without using multipart upload. This means you actually need a server to accept the PUT, then do its own chunked upload to object storage or otherwise stage the content before it's finally saved in object storage. This rules out presigned URLs for push too, since the PUT request made to the presigned URL can be too large for the backing object storage to accept. There's also other processing that ideally happens on push (like hash digest verification of the pushed layer) that mean a server somewhere needs to be involved. [0] https://github.com/opencontainers/distribution-spec/blob/main/spec.md#pushing-a-blob-in-chunks https://github.com/opencontainers/distribution-spec/blob/mai... [1] https://github.com/containerd/containerd/blob/192679b05917b540cebe6282960ebc6e177870df/core/remotes/docker/pusher.go#L275-L277 https://github.com/containerd/containerd/blob/192679b05917b5...
- mikeocool 2y agoHave any container running tools just implemented basic S3 compatibility for pushing/pulling images? If your registry doesnt accept pushes from untrusted sources, it doesn't seem like there is a ton of value in having "smarts" in the registry server itself. When you push, the client could just PUT a metadata file and an object for each layer in the object store, and pulling would just read the metadata file, which would tell it where to get each layer. And could use etags to skip downloaded layers that have already been downloaded. For auth just use the standard S3 auth. Would be compatible with S3/r2/any other S3-compatible storage.
- tfolbrecht 2y agoThe docker registry container supports S3 as a storage backend you could use it locally You can also `docker image save` then write the tarball to S3, then load to use.
- champtar 2y agoI would love if the container pull protocol stopped using custom headers or content-type, so we could use any dumb http server.
- RyeCombinator 2y agoGreat feat. However I am ever more confused now on what Cloudflare does and builds. They have everything from CDN, DNS to Orange Meets and this now?
- rozenmd 2y agoThere's a developer platform: https://workers.cloudflare.com/ https://workers.cloudflare.com/
- yazaddaruvala 2y agoMy understanding is Cloudflare is a competitor of AWS, Azure, and GCP.
- yecuken 2y agoI'm using this registry with regctl[0] to chunk uploads (to circumvent 100MB limit), works just fine for huge layers with models. With regctl you will also get 'mount' query parameter for upload initialization with the proper blob name so you can skip additional R2 copy when multi-part upload finalisation which speeds up the upload (and avoids crashes on larger blobs). This is not part of docker registry API, so I never got to PR that. [0] https://github.com/regclient/regclient https://github.com/regclient/regclient
- miohtama 2y agoWhen you switch to private Docker or Github registry to Cloudflare, are you effectively just trading one vendor lock in to another, or is there more into this?
- vineyardmike 2y agoNone of these are really vendor lock in. The registry protocol is an open standard, this is just one more source that implements it. So you’re only locked in as far as your data is stored somewhere, but that data is behind an open API, so minimal risk there.
- spikey_sanju 2y agoInteresting. Just wish it handled larger image layers a bit better!
- jzelinskie 2y agoIf you are a CloudFlare employee reading this, you should get involved with the OCI Distribution group that develops the standards for the registry: https://github.com/opencontainers/distribution-spec https://github.com/opencontainers/distribution-spec
- gyre007 2y agoOCI is demonstrably broken as a specification body as demonstrated by the referrers API. Distirbution spec as it is at the moment is just a very poorly written technical doc.
- beeboobaa3 2y agoCan you explain a bit more what you mean?
- re-thc 2y ago> you should get involved with the OCI Distribution group This is likely just a sample to showcase workers. Not sure it's enough reason for CloudFlare to get involved?
- fswd 2y agousing this same architecture, it would be cool to build a serverless-git
- mdaniel 2y agoI would have thought for sure someone would have already tried that, but regrettably trying to search for "serverless git" coughs up innumerable references to the framework that is hosted on GIThub Anyway, I was curious how much work such a stunt would be and based on the git-http-backend docs <https://git.github.io/git-scm.com/docs/git-http-backend#Documentation/git-http-backend.txt-AcceleratedstaticApache2x https://git.github.io/git-scm.com/docs/git-http-backend#Docu...> it seems like there are actually a manageable number of endpoints I actually prefer their second scenario of splitting out access to the objects and pack files but since doing that would still require a function (or, ahem, a web server running) I suspect that optimization is not within the scope of what you had in mind
- jedberg 2y agoIsn’t GitHub already serverless git?
- airocker 2y agoIs there a registry that would work on extremely cheap disk storage if the use case if only push and very infrequent pulls?
- qudat 2y agoThis is pretty nice. Does it support an API for deleting images (and having it properly garbage-collected)? It looks like maybe this does it? https://github.com/cloudflare/serverless-registry/blob/13c4e88737bdadbb4cb65d82934428cfdd67837d/src/router.ts#L51 https://github.com/cloudflare/serverless-registry/blob/13c4e... We have a managed docker registry and could have definitely used this project! Slightly unrelated, but we've been experimenting with using SSH for authenticating with a docker registry if anyone is interested: https://github.com/picosh/tunkit?tab=readme-ov-file#why https://github.com/picosh/tunkit?tab=readme-ov-file#why
- ram_rattle 2y agoLooks like a neat idea, does anyone know any open source version that does just this?
- bravetraveler 2y agoInteresting approach when running one of these on your 'LAN' is relatively easy Though, to be fair, the pull-through mechanism in the reference registry been kind of goofy for years. Ask me how I know /s
- ImJasonH 2y agoI built a similar PoC using Workers+R2, before Cloudflare released theirs, in case you find it useful: https://github.com/chainguard-dev/crow-registry https://github.com/chainguard-dev/crow-registry We eventually built our own registry in Go running on Cloud Run, which now serves all our images on cgr.dev. Zero egress fees is really a game changer.