10 ms·

Using S3 as a Container Registry

319 points by jandeboevrie 2y ago

tealpod 2y ago
This is such a wonderful idea, congrats. There is a real usecase for this in some high security sectors. I can't put complete info here for the security reasons, let me know if you are interested.
wofo 2y ago
Hi HN, author here. If anyone knows why layer pushes need to be sequential in the OCI specification, please tell! Is it merely a historical accident, or is there some hidden rationale behind it? Edit: to clarify, I'm talking about sequentially pushing a _single_ layer's contents. You can, of course, push multiple layers in parallel.
- rcarmo 2y ago
  Never dealt with pushes, but it’s nice to see this — back when Docker was getting started I dumped an image behind nginx and pulled from that because there was no usable private registry container, so I enjoyed reading your article.
- IanCal 2y ago
  I can't think of an obvious one, maybe load based? ~~I added parallel pushes to docker I think, unless I'm mixing up pulls & pushes, it was a while ago.~~ My stuff was around parallelising the checks not the final pushes. Edit - does a layer say which layer it goes "on top" of? If so perhaps that's the reason, so the IDs of what's being pointed to exist.
  - wofo 2y ago
    Layers are fully independent of each other in the OCI spec (which makes them reusable). They are wired together through a separate manifest file that lists the layers of a specific image. It's a mystery... Here are the bits of the OCI spec about multipart pushes (https://github.com/opencontainers/distribution-spec/blob/58d034eaee033f6f29f482817745a9b0f8b0dea2/spec.md#pushing-a-blob-in-chunks https://github.com/opencontainers/distribution-spec/blob/58d...). In short, you can only upload the next chunk after the previous one finishes, because you need to use information from the response's headers.
    - IanCal 2y ago
      Ah thanks. That's chunks of a single layer though, not multiple layers right?
      - wofo 2y ago
        Indeed, you are free to push multiple layers in parallel. But when you have a 1 GiB layer full of AI/ML stuff you can feel the pain! (I just updated my original comment to make clear I'm talking about single-layer pushes here)
        killingtime74 2y ago
        Split the layer up?
        thangngoc89 2y ago
        You can’t. Installing pytorch and supporting dependencies takes 2.2GB on debian-slim.
        fweimer 2y ago
        Surely you can have one layer per directory or something like that? Splitting along those lines works as long as everything isn't in one big file. I think it was a mistake to make layers as a storage model visible in to the end user. This should just have been an internal implementation detail, perhaps similar to how Git handles delta compression and makes it independent of branching structure. We also should have delta pushes and pulls, using global caches (for public content), and the ability to start containers while their image is still in transfer.
        password4321 2y ago
        It should be possible to split into multiple layers as long as each file is wholly within in its layer. This is the exact opposite of the work recommended combining commands to keep everything in one layer which I think is done ultimately for runtime performance reasons.
- deleted 2y ago
  [deleted]
- codethief 2y ago
  Hi, thanks for the blog post! > For the last four months I’ve been developing a custom container image builder, collaborating with Outerbounds I know you said this was something for another blog post but could you already provide some details? Maybe a link to a GitHub repo? Background: I'm looking for (or might implement myself) a way to programmatically build OCI images from within $PROGRAMMING_LANGUAGE. Think Buildah, but as an API for an actual programming language instead of a command line interface. I could of course just invoke Buildah as a subprocess but that seems a bit unwieldy (and I would have to worry about interacting with & cleaning up Buildah's internal state), plus Buildah currently doesn't support Mac.
  - wofo 2y ago
    Unfortunately, all the code is proprietary at the moment. If you are willing to get your hands dirty, the main thing to realize is that container layers are "just" tar files (see, for instance, this article: https://ochagavia.nl/blog/crafting-container-images-without-dockerfiles/ https://ochagavia.nl/blog/crafting-container-images-without-...). Contact details are in my profile, in case you'd like to chat ;)
    - codethief 2y ago
      Ah too bad :) Thanks for the link! Though I'm less worried about the tarball / OCI spec part, more about platform compatibility. I tried running runc/crun by hand at some point and let's just say I've done things before that were more fun. :)
  - cpuguy83 2y ago
    This is what buildkit is. Granted go has the only sdk I know of, but the api is purely protobuf and highly extensible.
    - cpuguy83 2y ago
      For that matter, dagger (dagger.io) provides an sdk in multiple languages and gives you the full power (and then some extra on top) of buildkit.
- abofh 2y ago
  It makes clean-up simpler - if you never got to the "last" one, it's obvious you didn't finish after N+Timeout and thus you can expunge it. It simplifies an implementation detail (how do you deal with partial uploads? make them easy to spot). Otherwise you basically have to trigger at the end of every chunk, see if all the other chunks are there and then do the 'completion'. But that's an implementation detail, and I suspect isn't one that's meaningful or intentional. Your S3 approach should work fine btw, I've done it before in a prior life when I was at a company shipping huge images and $.10/gb/month _really_ added up. You lose the 'bells and whistles' of ECR, but those are pretty limited (imho)
  - orf 2y ago
    In the case of a docker registry, isn’t the “final bit” just uploading the final manifest that actually references the layers you’re uploading? At this point you’d validate that the layers exist and have been uploaded, otherwise you’d just bail out? And those missing chunks would be handled by the normal registry GC, which evicts unreferenced layers?
    - abofh 2y ago
      It's been a long time, but I think you're correct. In my environment I didn't actually care (any failed push would be retried so the layers would always eventually complete, and anything that for whatever reason didn't retry, well, it didn't happen enough that we cared at the cost of S3 to do anything clever). I think OCI ordered manifests first to "open the flow", but then close is only when the manifests last entry was completed - which led to this ordered upload problem. If your uploader knows where the chunks are going to live (OCI is more or less CAS, so it's predictable), it can just put them there in any order as long as it's all readable before something tries to pull it.
- majewsky 2y ago
  Source: I have implemented a OCI-compliant registry [1], though for the most part I've been following the behavior of the reference implementation [2] rather than the spec, on account of its convolutedness. When the client finalizes a blob upload, they need to supply the digest of the full blob. This requirement evidently serves to enable the server side to validate the integrity of the supplied bytes. If the server only started checking the digest as part of the finalize HTTP request, it would have to read back all the blob contents that had already been written into storage in previous HTTP requests. For large layers, this can introduce an unreasonable delay. (Because of specific client requirements, I have verified my implementation to work with blobs as large as 150 GiB.) Instead, my implementation runs the digest computation throughout the entire sequence of requests. As blob data is taken in chunk by chunk, it is simultaneously streamed into the digest computation and into blob storage. Between each request, the state of the digest computation is serialized in the upload URL that is passed back to the client in the Location header. This is roughly the part where it happens in my code: https://github.com/sapcc/keppel/blob/7e43d1f6e77ca72f00206450650dd1e827460c11/internal/api/registry/uploads.go#L618 https://github.com/sapcc/keppel/blob/7e43d1f6e77ca72f0020645... I believe that this is the same approach that the reference implementation uses. Because digest computation can only work sequentially, therefore the upload has to proceed sequentially. [1] https://github.com/sapcc/keppel https://github.com/sapcc/keppel [2] https://github.com/distribution/distribution https://github.com/distribution/distribution
cpa 2y ago
Is there a good reason for not allowing parallel uploads in the spec?
- wofo 2y ago
  No idea... I asked the same question here (https://news.ycombinator.com/item?id=40943480 https://news.ycombinator.com/item?id=40943480) and am hoping we'll have a classic HN moment where someone who was involved in the design of the spec will chime in.
- benterix 2y ago
  I believe that even if there was one then, it's probably no longer valid and it's now just a performance limitation.
  - wofo 2y ago
    Other than backwards-compatibility, I can imagine simplicity being a reason. For instance, sequential pushing makes it easier to calculate the sha256 hash of the layer as it's being uploaded, without having to do it after-the-fact when the uploaded chunks are assembled.
    - jtmarmon 2y ago
      I’m no expert on docker but I thought the hashes for each layer would already be computed if your image is built
      - wofo 2y ago
        That's true, but I'd assume the server would like to double-check that the hashes are valid (for robustness / consistency)... That's something my little experiment doesn't do, obviously.
      - cpuguy83 2y ago
        It's complicated. If you are using the containerd backed image store (opt-in still) OR if you push with "build --push" then yes. The default storage backend does not keep compressed layers, so those need to be recreated and digested on push. With the new store all that stuff is kept and reused.
jaimehrubiks 2y ago
I experience everyday the slowness of pushing big images (ai related tend to be big) to ECR on our cicd.
- wofo 2y ago
  I wonder whether the folks at Cloudflare could take the ideas from the blog post and create a high-performance serverless container registry based on R2. They could call it scrubs, for "serverless container registry using blob storage" :P
  - sshine 2y ago
    https://www.youtube.com/watch?v=3QbOssRq0Gs https://www.youtube.com/watch?v=3QbOssRq0Gs [Chorus: Chilli & T-Boz] No, I don't want no scrub A scrub is a guy that can't get no love from me Hangin' out the passenger side of his best friend's ride Trying to holla at me I don't want no scrub A scrub is a guy that can't get no love from me Hangin' out the passenger side of his best friend's ride Trying to holla at me
  - zokier 2y ago
    I just point out that AWS ECR is exactly that, thin wrapper around S3. See for example https://docs.aws.amazon.com/AmazonECR/latest/APIReference/API_GetDownloadUrlForLayer.html https://docs.aws.amazon.com/AmazonECR/latest/APIReference/AP...
    - slig 2y ago
      Hopefully someone gets inspired by this and implements a thin wrapper using CF workers.
      - slig 2y ago
        Hopefully someone gets inspired by this and implements a thin wrapper using CF workers. Edit: CF already did it https://github.com/cloudflare/serverless-registry https://github.com/cloudflare/serverless-registry
stabbles 2y ago
The OCI Distribution Spec is not great, it does not read like a specification that was carefully designed. > According to the specification, a layer push must happen sequentially: even if you upload the layer in chunks, each chunk needs to finish uploading before you can move on to the next one. As far as I've tested with DockerHub and GHCR, chunked upload is broken anyways, and clients upload each blob/layer as a whole. The spec also promotes `Content-Range` value formats that do not match the RFC7233 format. (That said, there's parallelism on the level of blobs, just not per blob) Another gripe of mine is that they missed the opportunity to standardize pagination of listing tags, because they accidentally deleted some text from the standard [1]. Now different registries roll their own. [1] https://github.com/opencontainers/distribution-spec/issues/461#issuecomment-1701554264 https://github.com/opencontainers/distribution-spec/issues/4...
- mschuster91 2y ago
  On top of that, it's either the OCI spec that's broken or it's just AWS being nuts, but unlike GitLab and Nexus, AWS ECR doesn't support automatically creating folders (e.g. "<acctid>.dkr.ecr.<region>.amazonaws.com/foo/bar/baz:tag"), it can only do flat storage and either have seriously long image names or tags. Yes you can theoretically create a repository object in ECR in Terraform to mimic that behavior, but it sucks in pipelines where the result image path is dynamic - you need to give more privileges to the IAM role of the CI pipeline than I'm comfortable with, not to mention that I don't like any AWS resources managed outside of the central Terraform repository. [1] https://stackoverflow.com/questions/64232268/storing-images-in-aws-ecr-using-namespaces https://stackoverflow.com/questions/64232268/storing-images-...
  - hanikesn 2y ago
    That's seem standard AWS practice. Implement a new feature so you can check the box, but in practice it's a huge pain to actually use.
    - judge2020 2y ago
      It’s not as bad as Azure is (was?) with IPv6: https://news.ycombinator.com/item?id=29327773 https://news.ycombinator.com/item?id=29327773
rad_gruchalski 2y ago
What’s wrong with https://github.com/distribution/distribution https://github.com/distribution/distribution?
- _flux 2y ago
  I hadn't seen that before, and it indeed does support S3, but does it also offer the clients the downloads directly from S3, or does it merely use it as its own storage backend (so basically work as a proxy when pulling)?
  - vbezhenar 2y ago
    It redirects client requests to S3 endpoint, so yes, in the end all heavy traffic goes from S3.
    - est31 2y ago
      But it still means a downtime of your service is a downtime for anything which might need a docker container, whereas if it went to S3 (or cloudfront in front of S3) directly, you profit from the many nines that S3 offers without paying an arm and a leg for ECR (data in ECR costs five times as much as S3 standard tier).
victorbjorklund 2y ago
But this only works for public repos right? I assume docker pull wont use a s3 api key
- wofo 2y ago
  That's true, unfortunately. I'm thinking about ways to somehow support private repos without introducing a proxy in between... Not sure if it will be possible.
  - victorbjorklund 2y ago
    Maybe a very thin proxy layer on like cloudflare functions.
    - wofo 2y ago
      Yep, I originally thought that wouldn't work... But now I discovered (thanks to a comment here) that the registry is allowed to return an HTTP redirect instead of serving the layer blobs directly... Which opens new possibilities :)
ericpauley 2y ago
Where's the source code?
- wofo 2y ago
  The source code is proprietary, but it shouldn't take much work to replicate, fortunately (you just need to upload files at the right paths).
- seungwoolee518 2y ago
  Like treat path as a object key, and put value as a json or blob?
lofties 2y ago
This sounds very, very expensive, and I would've loved to see cost mentioned in the article too. (for both S3 and R2)
- remram 2y ago
  The cost is the S3 cost though. It depends on region and storage tier, but the storage cost per GB, the GET/PUT cost, and the bandwidth cost can be found on the AWS website: https://aws.amazon.com/s3/pricing/ https://aws.amazon.com/s3/pricing/
- est31 2y ago
  S3's standard tier costs a fifth of ECR in terms of costs per GB stored. Egress costs to the free internet are the same, with the exception that for public ECR repositories they make egress to inner-AWS usage free.
kbumsik 2y ago
Actually, Cloudflare open-sourced a container registry server using R2.[1] Anyone tried it? [1]: https://github.com/cloudflare/serverless-registry https://github.com/cloudflare/serverless-registry
- justin_oaks 2y ago
  Looks cool. Thanks for linking it. It does mention that it's limited to 500MB per layer. For some people's use case that limitation might not be a big deal, but for others that's a dealbreaker.
  - danesparza 2y ago
    From the README: * Pushing with docker is limited to images that have layers of maximum size 500MB. Refer to maximum request body sizes in your Workers plan. * To circumvent that limitation, you can manually add the layer and the manifest into the R2 bucket or use a client that is able to chunk uploads in sizes less than 500MB (or the limit that you have in your Workers plan).
phillebaba 2y ago
Interesting idea to use the file path layout as a way to control the endpoints. I do wonder though how you would deal with the Docker-Content-Digest header. While not required it is suggested that responses should include it as many clients expect it and will reject layers without the header. Another thing to consider is that you will miss out on some feature from the OCI 1.1 spec like the referrers API as that would be a bit tricky to implement.
8organicbits 2y ago
> that S3 is up to 8x faster than ECR Awesome. Developer experience is so much better when CI doesn't take ages. Every little bit counts.
- barbazoo 2y ago
  > ECR 24 MiB/s (8.2 s) > S3 115 MiB/s (1.7 s) It's great that it's faster but absolutely, it's only an improvement of 6.5s observed, as you said, on the CI server. And it means using something for a purpose that it's not intended for. I'd hate to have to spend time debugging this if it breaks for whatever reason.
  - wofo 2y ago
    To be clear, the 8x was comparing the slowest ECR throughput measurement against the fastest S3 one. In any case, the improvement is significant.
donatj 2y ago
I don't do a ton with Docker outside dev tooling, but I have never understood why private container registries even exist? It just smells like rent seeking. What real advantage does it provide over say just generating some sort of image file you manage yourself, as you please?
- JackSlateur 2y ago
  You will probably have images that you will not share to the world. Said images will probably be made available to your infrastructure (k8s clusters, CI/CD runners etc). So you have to either build your own registry or pay someone to do it for you. Of course, if you use images for dev only, all of that are worthless and you just store your images on your dev machine
  - regularfry 2y ago
    Also if your infrastructure is within AWS, you want your images to also be within AWS when the infrastructure wants them. That doesn't necessarily imply a private registry, but it's a lot less work that way.
- figmert 2y ago
  You don't have to use it. You can use docker save and docker import: docker save alpine:3.19 > alpine.tar docker load < alpine.tar But now I have to manage that tar file, have all my systems be aware of where it is, how to access it, etc. Or, I could just not re-invent the wheel and use what docker already has provided.
  - deleted 2y ago
    [deleted]
- danmur 2y ago
  Companies send young engineers (and older engineers who should know more but don't) to AWS and Microsoft for "cloud certification". They learn how to operate cloud services because thats what benefits AWS and MS, so thats what their solutions use. It's a difficult uphill battle to get people interested in how things work under the hood, which is what you need in order to know you can do things like easily host your own package repositories.
filleokus 2y ago
I've started to grow annoyed with container registry cloud products. Always surprisingly cumbersome to auto-delete old tags, deal with ACL or limit the networking. It would be nice if a Kubernetes distro took a page out of the "serverless" playbook and just embedded a registry. Or maybe I should just use GHCR
- vbezhenar 2y ago
  Kubernetes is extremely bare-bones, there's no way they'll embed a registry. Kubernetes doesn't touch images at all, AFAIK, it delegates that to the container runtime, e.g. containerd. If you want some lightweight registry, use "official" docker registry. I'm running it inside Kubernetes and it consumes it just fine.
- breatheoften 2y ago
  I'm using google's artifact registry -- aside from upload speed another thing that kills me is freakin download speed ... Why in the world should it take 2 minutes to download a 2.6 GB layer to a cloud build instance sitting in the same region as the artifact registry ... Stupidly slow networking really harms the stateless ci machine + docker registry cache which actually would be quite cool if it was fast enough ... In my case it's still faster than doing the builds would be -- but I'm definitely gonna have to get machines with persistent local cache in the mix at some point so that these operations will finish within a few seconds instead of a few minutes ...
- mdaniel 2y ago
  > Always surprisingly cumbersome to auto-delete old tags, Does this not do what you want? https://docs.aws.amazon.com/AmazonECR/latest/userguide/lifecycle_policy_examples.html https://docs.aws.amazon.com/AmazonECR/latest/userguide/lifec... I can't speak to the other "registry cloud products" except for GitLab, which is its own special UX nonsense, but they also support expiry after enough whisky consumption
  - deleted 2y ago
    [deleted]
- 2y ago
ericfrederich 2y ago
R2 in only "free" until it isn't. Cloudflare hasn't got a lot of good press recently. Not something I'd wanna build my business around.
- jgrahamc 2y ago
  R2 egress is free.
- TheMrZZ 2y ago
  Aside from the casino story (high value target that likely faces tons of attacks, therefore an expensive customer for CF), did something happen with them? I'm not aware of bad press around them in general
KronisLV 2y ago
That's a pretty cool use case! Personally, I just use Nexus because it works well enough (and supports everything from OCI images to apt packages and stuff like a custom Maven, NuGet, npm repo etc.), however the configuration and resource usage both are a bit annoying, especially when it comes to cleanup policies: https://www.sonatype.com/products/sonatype-nexus-repository https://www.sonatype.com/products/sonatype-nexus-repository That said: > More specifically, I logged the requests issued by docker pull and saw that they are “just” a bunch of HEAD and GET requests. this is immensely nice and I wish more tech out there made common sense decisions like this, just using what has worked for a long time and not overcomplicating. I am a bit surprised that there aren't more simple container repositories out there (especially with auth and cleanup support), since Nexus and Harbor are both a bit complex in practice.
- thangngoc89 2y ago
  I’m using Gitea just for packages (Docker, npm, python, ….) I’m surprised noone has mentioned this in this thread
michaelmior 2y ago
> Why can’t ECR support this kind of parallel uploads? The “problem” is that it implements the OCI Distribution Spec… I don't see any reason why ECR couldn't support parallel uploads as an optimization. Provide an alternative to `docker push` for those who care about speed that doesn't conform to the spec.
- wofo 2y ago
  Indeed, they could support it through a non-standard API... I wish they did!
lazy_moderator1 2y ago
That's neat! On that note I've been using S3 as a private registry for years now via Gitlab and couldn't be happier!
akeck 2y ago
Note that CNCF's Distribution (formerly Docker's Registry) includes support for backing a registry with Cloudfront signed URLs that pull from S3. [1] https://distribution.github.io/distribution/storage-drivers/middleware/ https://distribution.github.io/distribution/storage-drivers/...
kevin_nisbet 2y ago
It's cool to see it, I was interested in trying something similar a couple years ago but priorities changed. My interest was mainly around a hardening stand point. The base idea was the release system through IAM permissions would be the only system with any write access to the underlying S3 bucket. All the public / internet facing components could then be limited to read only access as part of the hardening. This would of course be in addition to signing the images, but I don't think many of the customers at the time knew anything about or configured any of the signature verification mechanisms.
watermelon0 2y ago
It seems that ECR is actually designed in a way to support uploading image layers in multiple parts. Related ECR APIs: - InitiateLayerUpload API: called at the beginning of upload of each image layer - UploadLayerPart API: called for each layer chunk (up to 20 MB) - PutImage API: called after layers are uploaded, to push image manifest, containing references to all image layers The only weird thing seems to be that you have to upload layer chunks in base64 encoding, which increases data for ~33%.
- wofo 2y ago
  I tried using that API directly, but it unfortunately did require ordered uploads too :')
fnord77 2y ago
> What makes S3 faster than ECR? the author is missing something huge - ECR does a security scan on upload, too.
champtar 2y ago
What I would really love is for the OCI Distribution spec to support just static files, so we can use dumb http servers directly, or even file:// (for pull). All the metadata could be/is already in the manifests, having Content-Type: octet-stream could work just fine.
dheera 2y ago
Make sure you use HTTPS, or someone could theoretically inject malicious code into your container. If you want to use your own domain you'll have to use CloudFront to wrap S3 though.
- wofo 2y ago
  That's what I thought originally, but you can actually use `https://<your-bucket>.s3.amazonaws.com https://<your-bucket>.s3.amazonaws.com` without Cloudfront or other service on top (it wasn't easy to find in the AWS docs, but it works)

319 points by jandeboevrie 2y ago

This is such a wonderful idea, congrats. There is a real usecase for this in some high security sectors. I can't put complete info here for the security reasons, let me know if you are interested.

wofo 2y ago

Hi HN, author here. If anyone knows why layer pushes need to be sequential in the OCI specification, please tell! Is it merely a historical accident, or is there some hidden rationale behind it? Edit: to clarify, I'm talking about sequentially pushing a _single_ layer's contents. You can, of course, push multiple layers in parallel.

rcarmo 2y ago
Never dealt with pushes, but it’s nice to see this — back when Docker was getting started I dumped an image behind nginx and pulled from that because there was no usable private registry container, so I enjoyed reading your article.
IanCal 2y ago
I can't think of an obvious one, maybe load based? ~~I added parallel pushes to docker I think, unless I'm mixing up pulls & pushes, it was a while ago.~~ My stuff was around parallelising the checks not the final pushes. Edit - does a layer say which layer it goes "on top" of? If so perhaps that's the reason, so the IDs of what's being pointed to exist.
- wofo 2y ago
  Layers are fully independent of each other in the OCI spec (which makes them reusable). They are wired together through a separate manifest file that lists the layers of a specific image. It's a mystery... Here are the bits of the OCI spec about multipart pushes (https://github.com/opencontainers/distribution-spec/blob/58d034eaee033f6f29f482817745a9b0f8b0dea2/spec.md#pushing-a-blob-in-chunks https://github.com/opencontainers/distribution-spec/blob/58d...). In short, you can only upload the next chunk after the previous one finishes, because you need to use information from the response's headers.
  - IanCal 2y ago
    Ah thanks. That's chunks of a single layer though, not multiple layers right?
    - wofo 2y ago
      Indeed, you are free to push multiple layers in parallel. But when you have a 1 GiB layer full of AI/ML stuff you can feel the pain! (I just updated my original comment to make clear I'm talking about single-layer pushes here)
      - killingtime74 2y ago
        Split the layer up?
        thangngoc89 2y ago
        You can’t. Installing pytorch and supporting dependencies takes 2.2GB on debian-slim.
        fweimer 2y ago
        Surely you can have one layer per directory or something like that? Splitting along those lines works as long as everything isn't in one big file. I think it was a mistake to make layers as a storage model visible in to the end user. This should just have been an internal implementation detail, perhaps similar to how Git handles delta compression and makes it independent of branching structure. We also should have delta pushes and pulls, using global caches (for public content), and the ability to start containers while their image is still in transfer.
        password4321 2y ago
        It should be possible to split into multiple layers as long as each file is wholly within in its layer. This is the exact opposite of the work recommended combining commands to keep everything in one layer which I think is done ultimately for runtime performance reasons.
deleted 2y ago
[deleted]
codethief 2y ago
Hi, thanks for the blog post! > For the last four months I’ve been developing a custom container image builder, collaborating with Outerbounds I know you said this was something for another blog post but could you already provide some details? Maybe a link to a GitHub repo? Background: I'm looking for (or might implement myself) a way to programmatically build OCI images from within $PROGRAMMING_LANGUAGE. Think Buildah, but as an API for an actual programming language instead of a command line interface. I could of course just invoke Buildah as a subprocess but that seems a bit unwieldy (and I would have to worry about interacting with & cleaning up Buildah's internal state), plus Buildah currently doesn't support Mac.
- wofo 2y ago
  Unfortunately, all the code is proprietary at the moment. If you are willing to get your hands dirty, the main thing to realize is that container layers are "just" tar files (see, for instance, this article: https://ochagavia.nl/blog/crafting-container-images-without-dockerfiles/ https://ochagavia.nl/blog/crafting-container-images-without-...). Contact details are in my profile, in case you'd like to chat ;)
  - codethief 2y ago
    Ah too bad :) Thanks for the link! Though I'm less worried about the tarball / OCI spec part, more about platform compatibility. I tried running runc/crun by hand at some point and let's just say I've done things before that were more fun. :)
- cpuguy83 2y ago
  This is what buildkit is. Granted go has the only sdk I know of, but the api is purely protobuf and highly extensible.
  - cpuguy83 2y ago
    For that matter, dagger (dagger.io) provides an sdk in multiple languages and gives you the full power (and then some extra on top) of buildkit.
abofh 2y ago
It makes clean-up simpler - if you never got to the "last" one, it's obvious you didn't finish after N+Timeout and thus you can expunge it. It simplifies an implementation detail (how do you deal with partial uploads? make them easy to spot). Otherwise you basically have to trigger at the end of every chunk, see if all the other chunks are there and then do the 'completion'. But that's an implementation detail, and I suspect isn't one that's meaningful or intentional. Your S3 approach should work fine btw, I've done it before in a prior life when I was at a company shipping huge images and $.10/gb/month _really_ added up. You lose the 'bells and whistles' of ECR, but those are pretty limited (imho)
- orf 2y ago
  In the case of a docker registry, isn’t the “final bit” just uploading the final manifest that actually references the layers you’re uploading? At this point you’d validate that the layers exist and have been uploaded, otherwise you’d just bail out? And those missing chunks would be handled by the normal registry GC, which evicts unreferenced layers?
  - abofh 2y ago
    It's been a long time, but I think you're correct. In my environment I didn't actually care (any failed push would be retried so the layers would always eventually complete, and anything that for whatever reason didn't retry, well, it didn't happen enough that we cared at the cost of S3 to do anything clever). I think OCI ordered manifests first to "open the flow", but then close is only when the manifests last entry was completed - which led to this ordered upload problem. If your uploader knows where the chunks are going to live (OCI is more or less CAS, so it's predictable), it can just put them there in any order as long as it's all readable before something tries to pull it.
majewsky 2y ago
Source: I have implemented a OCI-compliant registry [1], though for the most part I've been following the behavior of the reference implementation [2] rather than the spec, on account of its convolutedness. When the client finalizes a blob upload, they need to supply the digest of the full blob. This requirement evidently serves to enable the server side to validate the integrity of the supplied bytes. If the server only started checking the digest as part of the finalize HTTP request, it would have to read back all the blob contents that had already been written into storage in previous HTTP requests. For large layers, this can introduce an unreasonable delay. (Because of specific client requirements, I have verified my implementation to work with blobs as large as 150 GiB.) Instead, my implementation runs the digest computation throughout the entire sequence of requests. As blob data is taken in chunk by chunk, it is simultaneously streamed into the digest computation and into blob storage. Between each request, the state of the digest computation is serialized in the upload URL that is passed back to the client in the Location header. This is roughly the part where it happens in my code: https://github.com/sapcc/keppel/blob/7e43d1f6e77ca72f00206450650dd1e827460c11/internal/api/registry/uploads.go#L618 https://github.com/sapcc/keppel/blob/7e43d1f6e77ca72f0020645... I believe that this is the same approach that the reference implementation uses. Because digest computation can only work sequentially, therefore the upload has to proceed sequentially. [1] https://github.com/sapcc/keppel https://github.com/sapcc/keppel [2] https://github.com/distribution/distribution https://github.com/distribution/distribution

cpa 2y ago

Is there a good reason for not allowing parallel uploads in the spec?

wofo 2y ago
No idea... I asked the same question here (https://news.ycombinator.com/item?id=40943480 https://news.ycombinator.com/item?id=40943480) and am hoping we'll have a classic HN moment where someone who was involved in the design of the spec will chime in.
benterix 2y ago
I believe that even if there was one then, it's probably no longer valid and it's now just a performance limitation.
- wofo 2y ago
  Other than backwards-compatibility, I can imagine simplicity being a reason. For instance, sequential pushing makes it easier to calculate the sha256 hash of the layer as it's being uploaded, without having to do it after-the-fact when the uploaded chunks are assembled.
  - jtmarmon 2y ago
    I’m no expert on docker but I thought the hashes for each layer would already be computed if your image is built
    - wofo 2y ago
      That's true, but I'd assume the server would like to double-check that the hashes are valid (for robustness / consistency)... That's something my little experiment doesn't do, obviously.
    - cpuguy83 2y ago
      It's complicated. If you are using the containerd backed image store (opt-in still) OR if you push with "build --push" then yes. The default storage backend does not keep compressed layers, so those need to be recreated and digested on push. With the new store all that stuff is kept and reused.

jaimehrubiks 2y ago

I experience everyday the slowness of pushing big images (ai related tend to be big) to ECR on our cicd.

wofo 2y ago
I wonder whether the folks at Cloudflare could take the ideas from the blog post and create a high-performance serverless container registry based on R2. They could call it scrubs, for "serverless container registry using blob storage" :P
- sshine 2y ago
  https://www.youtube.com/watch?v=3QbOssRq0Gs https://www.youtube.com/watch?v=3QbOssRq0Gs [Chorus: Chilli & T-Boz] No, I don't want no scrub A scrub is a guy that can't get no love from me Hangin' out the passenger side of his best friend's ride Trying to holla at me I don't want no scrub A scrub is a guy that can't get no love from me Hangin' out the passenger side of his best friend's ride Trying to holla at me
- zokier 2y ago
  I just point out that AWS ECR is exactly that, thin wrapper around S3. See for example https://docs.aws.amazon.com/AmazonECR/latest/APIReference/API_GetDownloadUrlForLayer.html https://docs.aws.amazon.com/AmazonECR/latest/APIReference/AP...
  - slig 2y ago
    Hopefully someone gets inspired by this and implements a thin wrapper using CF workers.
    - slig 2y ago
      Hopefully someone gets inspired by this and implements a thin wrapper using CF workers. Edit: CF already did it https://github.com/cloudflare/serverless-registry https://github.com/cloudflare/serverless-registry

stabbles 2y ago

The OCI Distribution Spec is not great, it does not read like a specification that was carefully designed. > According to the specification, a layer push must happen sequentially: even if you upload the layer in chunks, each chunk needs to finish uploading before you can move on to the next one. As far as I've tested with DockerHub and GHCR, chunked upload is broken anyways, and clients upload each blob/layer as a whole. The spec also promotes `Content-Range` value formats that do not match the RFC7233 format. (That said, there's parallelism on the level of blobs, just not per blob) Another gripe of mine is that they missed the opportunity to standardize pagination of listing tags, because they accidentally deleted some text from the standard [1]. Now different registries roll their own. [1] https://github.com/opencontainers/distribution-spec/issues/461#issuecomment-1701554264 https://github.com/opencontainers/distribution-spec/issues/4...

mschuster91 2y ago
On top of that, it's either the OCI spec that's broken or it's just AWS being nuts, but unlike GitLab and Nexus, AWS ECR doesn't support automatically creating folders (e.g. "<acctid>.dkr.ecr.<region>.amazonaws.com/foo/bar/baz:tag"), it can only do flat storage and either have seriously long image names or tags. Yes you can theoretically create a repository object in ECR in Terraform to mimic that behavior, but it sucks in pipelines where the result image path is dynamic - you need to give more privileges to the IAM role of the CI pipeline than I'm comfortable with, not to mention that I don't like any AWS resources managed outside of the central Terraform repository. [1] https://stackoverflow.com/questions/64232268/storing-images-in-aws-ecr-using-namespaces https://stackoverflow.com/questions/64232268/storing-images-...
- hanikesn 2y ago
  That's seem standard AWS practice. Implement a new feature so you can check the box, but in practice it's a huge pain to actually use.
  - judge2020 2y ago
    It’s not as bad as Azure is (was?) with IPv6: https://news.ycombinator.com/item?id=29327773 https://news.ycombinator.com/item?id=29327773

rad_gruchalski 2y ago

What’s wrong with https://github.com/distribution/distribution https://github.com/distribution/distribution?

_flux 2y ago
I hadn't seen that before, and it indeed does support S3, but does it also offer the clients the downloads directly from S3, or does it merely use it as its own storage backend (so basically work as a proxy when pulling)?
- vbezhenar 2y ago
  It redirects client requests to S3 endpoint, so yes, in the end all heavy traffic goes from S3.
  - est31 2y ago
    But it still means a downtime of your service is a downtime for anything which might need a docker container, whereas if it went to S3 (or cloudfront in front of S3) directly, you profit from the many nines that S3 offers without paying an arm and a leg for ECR (data in ECR costs five times as much as S3 standard tier).

victorbjorklund 2y ago

But this only works for public repos right? I assume docker pull wont use a s3 api key

wofo 2y ago
That's true, unfortunately. I'm thinking about ways to somehow support private repos without introducing a proxy in between... Not sure if it will be possible.
- victorbjorklund 2y ago
  Maybe a very thin proxy layer on like cloudflare functions.
  - wofo 2y ago
    Yep, I originally thought that wouldn't work... But now I discovered (thanks to a comment here) that the registry is allowed to return an HTTP redirect instead of serving the layer blobs directly... Which opens new possibilities :)

ericpauley 2y ago

Where's the source code?

wofo 2y ago
The source code is proprietary, but it shouldn't take much work to replicate, fortunately (you just need to upload files at the right paths).
seungwoolee518 2y ago
Like treat path as a object key, and put value as a json or blob?

lofties 2y ago

This sounds very, very expensive, and I would've loved to see cost mentioned in the article too. (for both S3 and R2)

remram 2y ago
The cost is the S3 cost though. It depends on region and storage tier, but the storage cost per GB, the GET/PUT cost, and the bandwidth cost can be found on the AWS website: https://aws.amazon.com/s3/pricing/ https://aws.amazon.com/s3/pricing/
est31 2y ago
S3's standard tier costs a fifth of ECR in terms of costs per GB stored. Egress costs to the free internet are the same, with the exception that for public ECR repositories they make egress to inner-AWS usage free.

kbumsik 2y ago

Actually, Cloudflare open-sourced a container registry server using R2.[1] Anyone tried it? [1]: https://github.com/cloudflare/serverless-registry https://github.com/cloudflare/serverless-registry

justin_oaks 2y ago
Looks cool. Thanks for linking it. It does mention that it's limited to 500MB per layer. For some people's use case that limitation might not be a big deal, but for others that's a dealbreaker.
- danesparza 2y ago
  From the README: * Pushing with docker is limited to images that have layers of maximum size 500MB. Refer to maximum request body sizes in your Workers plan. * To circumvent that limitation, you can manually add the layer and the manifest into the R2 bucket or use a client that is able to chunk uploads in sizes less than 500MB (or the limit that you have in your Workers plan).

phillebaba 2y ago

Interesting idea to use the file path layout as a way to control the endpoints. I do wonder though how you would deal with the Docker-Content-Digest header. While not required it is suggested that responses should include it as many clients expect it and will reject layers without the header. Another thing to consider is that you will miss out on some feature from the OCI 1.1 spec like the referrers API as that would be a bit tricky to implement.

8organicbits 2y ago

> that S3 is up to 8x faster than ECR Awesome. Developer experience is so much better when CI doesn't take ages. Every little bit counts.

barbazoo 2y ago
> ECR 24 MiB/s (8.2 s) > S3 115 MiB/s (1.7 s) It's great that it's faster but absolutely, it's only an improvement of 6.5s observed, as you said, on the CI server. And it means using something for a purpose that it's not intended for. I'd hate to have to spend time debugging this if it breaks for whatever reason.
- wofo 2y ago
  To be clear, the 8x was comparing the slowest ECR throughput measurement against the fastest S3 one. In any case, the improvement is significant.

donatj 2y ago

I don't do a ton with Docker outside dev tooling, but I have never understood why private container registries even exist? It just smells like rent seeking. What real advantage does it provide over say just generating some sort of image file you manage yourself, as you please?

JackSlateur 2y ago
You will probably have images that you will not share to the world. Said images will probably be made available to your infrastructure (k8s clusters, CI/CD runners etc). So you have to either build your own registry or pay someone to do it for you. Of course, if you use images for dev only, all of that are worthless and you just store your images on your dev machine
- regularfry 2y ago
  Also if your infrastructure is within AWS, you want your images to also be within AWS when the infrastructure wants them. That doesn't necessarily imply a private registry, but it's a lot less work that way.
figmert 2y ago
You don't have to use it. You can use docker save and docker import: docker save alpine:3.19 > alpine.tar docker load < alpine.tar But now I have to manage that tar file, have all my systems be aware of where it is, how to access it, etc. Or, I could just not re-invent the wheel and use what docker already has provided.
- deleted 2y ago
  [deleted]
danmur 2y ago
Companies send young engineers (and older engineers who should know more but don't) to AWS and Microsoft for "cloud certification". They learn how to operate cloud services because thats what benefits AWS and MS, so thats what their solutions use. It's a difficult uphill battle to get people interested in how things work under the hood, which is what you need in order to know you can do things like easily host your own package repositories.

filleokus 2y ago

I've started to grow annoyed with container registry cloud products. Always surprisingly cumbersome to auto-delete old tags, deal with ACL or limit the networking. It would be nice if a Kubernetes distro took a page out of the "serverless" playbook and just embedded a registry. Or maybe I should just use GHCR

vbezhenar 2y ago
Kubernetes is extremely bare-bones, there's no way they'll embed a registry. Kubernetes doesn't touch images at all, AFAIK, it delegates that to the container runtime, e.g. containerd. If you want some lightweight registry, use "official" docker registry. I'm running it inside Kubernetes and it consumes it just fine.
breatheoften 2y ago
I'm using google's artifact registry -- aside from upload speed another thing that kills me is freakin download speed ... Why in the world should it take 2 minutes to download a 2.6 GB layer to a cloud build instance sitting in the same region as the artifact registry ... Stupidly slow networking really harms the stateless ci machine + docker registry cache which actually would be quite cool if it was fast enough ... In my case it's still faster than doing the builds would be -- but I'm definitely gonna have to get machines with persistent local cache in the mix at some point so that these operations will finish within a few seconds instead of a few minutes ...
mdaniel 2y ago
> Always surprisingly cumbersome to auto-delete old tags, Does this not do what you want? https://docs.aws.amazon.com/AmazonECR/latest/userguide/lifecycle_policy_examples.html https://docs.aws.amazon.com/AmazonECR/latest/userguide/lifec... I can't speak to the other "registry cloud products" except for GitLab, which is its own special UX nonsense, but they also support expiry after enough whisky consumption
- deleted 2y ago
  [deleted]
2y ago

ericfrederich 2y ago

R2 in only "free" until it isn't. Cloudflare hasn't got a lot of good press recently. Not something I'd wanna build my business around.

jgrahamc 2y ago
R2 egress is free.
TheMrZZ 2y ago
Aside from the casino story (high value target that likely faces tons of attacks, therefore an expensive customer for CF), did something happen with them? I'm not aware of bad press around them in general

KronisLV 2y ago

That's a pretty cool use case! Personally, I just use Nexus because it works well enough (and supports everything from OCI images to apt packages and stuff like a custom Maven, NuGet, npm repo etc.), however the configuration and resource usage both are a bit annoying, especially when it comes to cleanup policies: https://www.sonatype.com/products/sonatype-nexus-repository https://www.sonatype.com/products/sonatype-nexus-repository That said: > More specifically, I logged the requests issued by docker pull and saw that they are “just” a bunch of HEAD and GET requests. this is immensely nice and I wish more tech out there made common sense decisions like this, just using what has worked for a long time and not overcomplicating. I am a bit surprised that there aren't more simple container repositories out there (especially with auth and cleanup support), since Nexus and Harbor are both a bit complex in practice.

thangngoc89 2y ago
I’m using Gitea just for packages (Docker, npm, python, ….) I’m surprised noone has mentioned this in this thread

michaelmior 2y ago

> Why can’t ECR support this kind of parallel uploads? The “problem” is that it implements the OCI Distribution Spec… I don't see any reason why ECR couldn't support parallel uploads as an optimization. Provide an alternative to `docker push` for those who care about speed that doesn't conform to the spec.

wofo 2y ago
Indeed, they could support it through a non-standard API... I wish they did!

lazy_moderator1 2y ago

That's neat! On that note I've been using S3 as a private registry for years now via Gitlab and couldn't be happier!

akeck 2y ago

Note that CNCF's Distribution (formerly Docker's Registry) includes support for backing a registry with Cloudfront signed URLs that pull from S3. [1] https://distribution.github.io/distribution/storage-drivers/middleware/ https://distribution.github.io/distribution/storage-drivers/...

kevin_nisbet 2y ago

It's cool to see it, I was interested in trying something similar a couple years ago but priorities changed. My interest was mainly around a hardening stand point. The base idea was the release system through IAM permissions would be the only system with any write access to the underlying S3 bucket. All the public / internet facing components could then be limited to read only access as part of the hardening. This would of course be in addition to signing the images, but I don't think many of the customers at the time knew anything about or configured any of the signature verification mechanisms.

watermelon0 2y ago

It seems that ECR is actually designed in a way to support uploading image layers in multiple parts. Related ECR APIs: - InitiateLayerUpload API: called at the beginning of upload of each image layer - UploadLayerPart API: called for each layer chunk (up to 20 MB) - PutImage API: called after layers are uploaded, to push image manifest, containing references to all image layers The only weird thing seems to be that you have to upload layer chunks in base64 encoding, which increases data for ~33%.

wofo 2y ago
I tried using that API directly, but it unfortunately did require ordered uploads too :')

fnord77 2y ago

> What makes S3 faster than ECR? the author is missing something huge - ECR does a security scan on upload, too.

champtar 2y ago

What I would really love is for the OCI Distribution spec to support just static files, so we can use dumb http servers directly, or even file:// (for pull). All the metadata could be/is already in the manifests, having Content-Type: octet-stream could work just fine.

dheera 2y ago

Make sure you use HTTPS, or someone could theoretically inject malicious code into your container. If you want to use your own domain you'll have to use CloudFront to wrap S3 though.

wofo 2y ago
That's what I thought originally, but you can actually use `https://<your-bucket>.s3.amazonaws.com https://<your-bucket>.s3.amazonaws.com` without Cloudfront or other service on top (it wasn't easy to find in the AWS docs, but it works)