5 ms·

The annoying thing about trying to implement a Docker registry on Workers and R2 is that it's so close to having everything you need, but the 500MB request body

The annoying thing about trying to implement a Docker registry on Workers and R2 is that it's so close to having everything you need, but the 500MB request body limit means Workers is unable to accept pushes of layers larger than 500MB. The limit is even lower at 100MB on the Pro plan[0]. We are running a registry that does store content on R2[1], and today this is implemented as the unholy chimera of Cloudflare Workers, AWS CloudFront, Lambda@edge, regular Lambda, S3, and R2. Pushes go first to CloudFront+Lambda@edge, content is saved in S3 first, then moved to R2 in background jobs. Once it's transited to R2, then pulls are served from R2. I would so love for Workers + R2 to actually be able to accept pushes of large layers, unfortunately I have yet to talk to anyone at Cloudflare who believes it's possible. Especially in this era of AI/ML models, some container images can have single layers in the 10-100GB range! [0] https://developers.cloudflare.com/workers/platform/limits/#request-limits https://developers.cloudflare.com/workers/platform/limits/#r... [1] https://depot.dev/docs/guides/ephemeral-registry https://depot.dev/docs/guides/ephemeral-registry

amenghra 2y ago
Can you generate a signed url to upload directly to R2? Or perform the upload in chunks?
- jacobwg 2y ago
  Uploading in chunks could definitely solve the issue, and the OCI Distribution Specification does actually have some language about an optional chunked push API[0]. Unfortunately very few of the registry clients actually support this, critically containerd does not[1], so this means your regular `docker push` and a whole lot of ecosystem tooling does not work. This also means that the single PUT must be able to support very large pushes as a single request, possibly even larger than what R2 or S3 would allow without using multipart upload. This means you actually need a server to accept the PUT, then do its own chunked upload to object storage or otherwise stage the content before it's finally saved in object storage. This rules out presigned URLs for push too, since the PUT request made to the presigned URL can be too large for the backing object storage to accept. There's also other processing that ideally happens on push (like hash digest verification of the pushed layer) that mean a server somewhere needs to be involved. [0] https://github.com/opencontainers/distribution-spec/blob/main/spec.md#pushing-a-blob-in-chunks https://github.com/opencontainers/distribution-spec/blob/mai... [1] https://github.com/containerd/containerd/blob/192679b05917b540cebe6282960ebc6e177870df/core/remotes/docker/pusher.go#L275-L277 https://github.com/containerd/containerd/blob/192679b05917b5...
telgareith 2y ago
What am I missing such that presigned Urls aren't the solution to this issue?
- _w1tm 2y ago
  R2 is ridiculously cheap compared to S3. The price difference was more than 40x when I last looked at it.
  - compootr 2y ago
    Rediculously cheap until their sales team shows up and says otherwise!
    - kobalsky 2y ago
      Mind explaining?
      - fragmede 2y ago
        https://robindev.substack.com/p/cloudflare-took-down-our-website https://robindev.substack.com/p/cloudflare-took-down-our-web...
        judge2020 2y ago
        Once again i would like eastdakota to respond to this sales tactic. Surely this can't be a double-digit driver for revenue to where he must stay silent on it.
        alemanek 2y ago
        Yikes, looks like CF is off my list of things to use. Super scummy
        stingraycharles 2y ago
        If you read the article, the OOP was running a gambling site so it is fairly reasonable for Cloudflare not to want their IP blocks to be associated with that. It’s a communication failure on CF’s end, but it’s not an ordinary situation.
zimbatm 2y ago
Does the worker need to process the body? Sometimes, the worker can return a signed URL and have the client directly upload to R2.
- znpy 2y ago
  > Sometimes, the worker can return a signed URL and have the client directly upload to R2. That's not what client expect though, and that would be (iaas) provider-dependent.
koolba 2y ago
> We are running a registry that does store content on R2[1], and today this is implemented as the unholy chimera of Cloudflare Workers, AWS CloudFront, Lambda@edge, regular Lambda, S3, and R2. What’s the advantage over just using ECR? Cost of storage? Cost of bandwidth to read? Hosting provider genetic diversity?
- gyre007 2y ago
  Cost of storage and egress cost are wildly more expensive on ECR compared to CF+R2
- Shakahs 2y ago
  ECR is slow. Despite being a static datastore presumably backed by S3 it will only serve container image layers at around 150mbps, when dealing with large (10GB) container images this is a problem. R2 will happily serve the same data at multi-gigabit speed.
- jacobwg 2y ago
  Data transfer costs. Our registry is for our customers, and they expect to pull their images from environments outside AWS, e.g. GitHub Actions. If that customer has a 2GB image, they want to build the image and then pull it into 10 separate matrix jobs (think like parallel Cypress tests), and they have 1,000 commits in a month, then the AWS data transfer costs are $1,800/mo, just for that one customer to pull their images. With R2, it acts both as a CDN and since Cloudflare does not charge for egress, reduces the cost to only storage + the one-time transfer out of AWS.
ljm 2y ago
Live by the cloud, die by the cloud, right? It’s trivial to set up a self hosted docker registry on your own storage. I’m running the helm version of it on my own k3s cluster, the cluster itself costing 20€/mo. Why would you need all of this fancy vendor lockin when a few low cost ARM boxes can get the job done?
- rfoo 2y ago
  Because I actually want it to serve 10GB container images to our "enthusiastic" users over the Internet. Oh, and our "enthusiastic" users want to pull it on 100s of boxes at the same time. What are you going to do? Send a legal notice to them for "DDoS"-ing your poor low cost ARM boxes?
  - lstamour 2y ago
    I should preface this by saying that I figure you know all this, but I'm laying it out anyway. :) Theoretically (I haven't read the parent article yet) it's just GET and HEAD requests for a Docker Registry[1], so if CloudFlare or another CDN supports large binary files, you could cache the images from the low cost box. That said, obviously I'm suggesting adding "cloud" CDN infrastructure to your tiny ARM box. In a more normal scenario, most would probably pick a free Docker Registry and just upload a mirror of the images to multiple registries to spread out the bandwidth load. E.g. Docker Hub, GitHub, etc. For a better solution, don't serve 10GB container images. Instead, start from someone else's 10GB container image and add the layers you need on top of it. Or consider a solution where you don't need to ship 10GB of data in your application, but could perhaps side-load only the necessary data. Another workaround: because Docker images can be pushed and pulled with identical signatures, it's also possible to encourage your end users to keep their own Docker registry of the image and refer to it from an internal or private copy of the image. This is a best practice for pretty much every kind of production deployment of a Docker container, so you don't have 100s of boxes pulling from your shared infrastructure, and leads to more reliable Docker deployments. An example is, in fact, pulling an image - because it stores a copy locally that you can refer to and run multiple containers at the same time from one local image. Pretty much every container image deployment I refer to in production, I prefer to use a privately mirrored copy instead of pulling from a source repository directly. Maybe it's just me? :) [1]: https://ochagavia.nl/blog/using-s3-as-a-container-registry/ https://ochagavia.nl/blog/using-s3-as-a-container-registry/
chx 2y ago
Why R2 instead of Backblaze B2? Isn't R2 more than double the price of B2?
captainjapeng 2y ago
I was able to push to a serverless-registry deployment a 10GB docker image, I've shared my approach here https://github.com/cloudflare/serverless-registry/issues/42 https://github.com/cloudflare/serverless-registry/issues/42

amenghra 2y ago

Can you generate a signed url to upload directly to R2? Or perform the upload in chunks?

jacobwg 2y ago
Uploading in chunks could definitely solve the issue, and the OCI Distribution Specification does actually have some language about an optional chunked push API[0]. Unfortunately very few of the registry clients actually support this, critically containerd does not[1], so this means your regular `docker push` and a whole lot of ecosystem tooling does not work. This also means that the single PUT must be able to support very large pushes as a single request, possibly even larger than what R2 or S3 would allow without using multipart upload. This means you actually need a server to accept the PUT, then do its own chunked upload to object storage or otherwise stage the content before it's finally saved in object storage. This rules out presigned URLs for push too, since the PUT request made to the presigned URL can be too large for the backing object storage to accept. There's also other processing that ideally happens on push (like hash digest verification of the pushed layer) that mean a server somewhere needs to be involved. [0] https://github.com/opencontainers/distribution-spec/blob/main/spec.md#pushing-a-blob-in-chunks https://github.com/opencontainers/distribution-spec/blob/mai... [1] https://github.com/containerd/containerd/blob/192679b05917b540cebe6282960ebc6e177870df/core/remotes/docker/pusher.go#L275-L277 https://github.com/containerd/containerd/blob/192679b05917b5...

telgareith 2y ago

What am I missing such that presigned Urls aren't the solution to this issue?

_w1tm 2y ago
R2 is ridiculously cheap compared to S3. The price difference was more than 40x when I last looked at it.
- compootr 2y ago
  Rediculously cheap until their sales team shows up and says otherwise!
  - kobalsky 2y ago
    Mind explaining?
    - fragmede 2y ago
      https://robindev.substack.com/p/cloudflare-took-down-our-website https://robindev.substack.com/p/cloudflare-took-down-our-web...
      - judge2020 2y ago
        Once again i would like eastdakota to respond to this sales tactic. Surely this can't be a double-digit driver for revenue to where he must stay silent on it.
      - alemanek 2y ago
        Yikes, looks like CF is off my list of things to use. Super scummy
        stingraycharles 2y ago
        If you read the article, the OOP was running a gambling site so it is fairly reasonable for Cloudflare not to want their IP blocks to be associated with that. It’s a communication failure on CF’s end, but it’s not an ordinary situation.

zimbatm 2y ago

Does the worker need to process the body? Sometimes, the worker can return a signed URL and have the client directly upload to R2.

znpy 2y ago
> Sometimes, the worker can return a signed URL and have the client directly upload to R2. That's not what client expect though, and that would be (iaas) provider-dependent.

koolba 2y ago

> We are running a registry that does store content on R2[1], and today this is implemented as the unholy chimera of Cloudflare Workers, AWS CloudFront, Lambda@edge, regular Lambda, S3, and R2. What’s the advantage over just using ECR? Cost of storage? Cost of bandwidth to read? Hosting provider genetic diversity?

gyre007 2y ago
Cost of storage and egress cost are wildly more expensive on ECR compared to CF+R2
Shakahs 2y ago
ECR is slow. Despite being a static datastore presumably backed by S3 it will only serve container image layers at around 150mbps, when dealing with large (10GB) container images this is a problem. R2 will happily serve the same data at multi-gigabit speed.
jacobwg 2y ago
Data transfer costs. Our registry is for our customers, and they expect to pull their images from environments outside AWS, e.g. GitHub Actions. If that customer has a 2GB image, they want to build the image and then pull it into 10 separate matrix jobs (think like parallel Cypress tests), and they have 1,000 commits in a month, then the AWS data transfer costs are $1,800/mo, just for that one customer to pull their images. With R2, it acts both as a CDN and since Cloudflare does not charge for egress, reduces the cost to only storage + the one-time transfer out of AWS.

ljm 2y ago

Live by the cloud, die by the cloud, right? It’s trivial to set up a self hosted docker registry on your own storage. I’m running the helm version of it on my own k3s cluster, the cluster itself costing 20€/mo. Why would you need all of this fancy vendor lockin when a few low cost ARM boxes can get the job done?

rfoo 2y ago
Because I actually want it to serve 10GB container images to our "enthusiastic" users over the Internet. Oh, and our "enthusiastic" users want to pull it on 100s of boxes at the same time. What are you going to do? Send a legal notice to them for "DDoS"-ing your poor low cost ARM boxes?
- lstamour 2y ago
  I should preface this by saying that I figure you know all this, but I'm laying it out anyway. :) Theoretically (I haven't read the parent article yet) it's just GET and HEAD requests for a Docker Registry[1], so if CloudFlare or another CDN supports large binary files, you could cache the images from the low cost box. That said, obviously I'm suggesting adding "cloud" CDN infrastructure to your tiny ARM box. In a more normal scenario, most would probably pick a free Docker Registry and just upload a mirror of the images to multiple registries to spread out the bandwidth load. E.g. Docker Hub, GitHub, etc. For a better solution, don't serve 10GB container images. Instead, start from someone else's 10GB container image and add the layers you need on top of it. Or consider a solution where you don't need to ship 10GB of data in your application, but could perhaps side-load only the necessary data. Another workaround: because Docker images can be pushed and pulled with identical signatures, it's also possible to encourage your end users to keep their own Docker registry of the image and refer to it from an internal or private copy of the image. This is a best practice for pretty much every kind of production deployment of a Docker container, so you don't have 100s of boxes pulling from your shared infrastructure, and leads to more reliable Docker deployments. An example is, in fact, pulling an image - because it stores a copy locally that you can refer to and run multiple containers at the same time from one local image. Pretty much every container image deployment I refer to in production, I prefer to use a privately mirrored copy instead of pulling from a source repository directly. Maybe it's just me? :) [1]: https://ochagavia.nl/blog/using-s3-as-a-container-registry/ https://ochagavia.nl/blog/using-s3-as-a-container-registry/

chx 2y ago

Why R2 instead of Backblaze B2? Isn't R2 more than double the price of B2?

captainjapeng 2y ago

I was able to push to a serverless-registry deployment a 10GB docker image, I've shared my approach here https://github.com/cloudflare/serverless-registry/issues/42 https://github.com/cloudflare/serverless-registry/issues/42