4 ms·

I wish containers hadn't created the abstractions of a registry and an image instead of exposing it all as tar files (which is what it kind of is under the cove

I wish containers hadn't created the abstractions of a registry and an image instead of exposing it all as tar files (which is what it kind of is under the covers) served over a glorified file server. This leads to people assuming there's some kind of magic happening and that the entire process is very arcane, when in reality it's just unpacking tar files. If you want to DIY a container with unix tools, this should help: https://containers.gitbook.io/build-containers-the-hard-way/ https://containers.gitbook.io/build-containers-the-hard-way/

westurner 5y ago
"Signing Images with Docker Content Trust" explains how cryptographic container image signatures work w/ Docker Notary (TUF) https://docs.docker.com/engine/security/trust/#signing-images-with-docker-content-trust https://docs.docker.com/engine/security/trust/#signing-image... The TUF spec (and PyPI TUF PEPs) explains why a tar over https (with optional DNSSEC, a CA cert bundle, CRL, OCSP,) isn't sufficient for secure software distribution. "#ZeroTrust DevOps"; #DevSecOps What's the favorite package format with content signatures, key distribution, a keyring of trusted (authorized) keys, and a cryptographically-signed manifest of per-file hashes, permissions, and extended file attributes? FWIW, ZIP at least does a CRC32. We now have the Linux Foundation CNCF sigstore for any artifact, including OCI container images. W3C ld-proofs is a newer web standard that unfortunately all package managers haven't yet migrated to. https://news.ycombinator.com/item?id=29355786 https://news.ycombinator.com/item?id=29355786 Because ld-proofs is RDF, it works in JSON-LD and you could merge the entire SBOM [1] and e.g. CodeMeta [2] Linked Data metadata for all of the standardized-metadata-documented components in a stack. [1] https://github.com/google/osv/issues/55 https://github.com/google/osv/issues/55 [2] https://github.com/codemeta/codemeta https://github.com/codemeta/codemeta
- denysvitali 5y ago
  Isn't this reinventing the wheel somehow? Linux distros have been using tar.gz over HTTP (not S!) and relying on GPG for years.
  - ibotty 5y ago
    But most Linux distros are not that great either. They don't implement TUF and are not safe against some vectors prevented by that. It's OK to start anew when container distribution poses a different problem anyway. https://en.wikipedia.org/wiki/The_Update_Framework_(TUF) https://en.wikipedia.org/wiki/The_Update_Framework_(TUF)
    - krageon 5y ago
      The linked page doesn't explain anything at all. It says some incredibly obvious things (remotes can be compromised and you should be prepared for that) and then goes on to say the subject fixes all that, but there's actually no explanation of how it does that or what it does to get there. It's a marketing piece, essentially.
      - westurner 5y ago
        From this about DNSSEC and software supply chain security a couple weeks ago, from this about Android packages: https://news.ycombinator.com/item?id=27700513 https://news.ycombinator.com/item?id=27700513 : >> [...] Sigstore is a free and open Linux Foundation service for asset signatures: https://sigstore.dev/what_is_sigstore/ https://sigstore.dev/what_is_sigstore/ >> The TUF Overview explains some of the risks of asset signature systems; key compromise, there's one key for everything that we all share and can't log the revocation of in a CT (Certificate Transparency) log distributed like a DLT, https://theupdateframework.io/overview/ https://theupdateframework.io/overview/ >> Certificate Transparency: https://en.wikipedia.org/wiki/Certificate_Transparency https://en.wikipedia.org/wiki/Certificate_Transparency >> Yeah, there's a channel to secure there at that layer of the software supply chain as well. >> "PEP 480 -- Surviving a Compromise of PyPI: End-to-end signing of packages" (2014-) https://www.python.org/dev/peps/pep-0480/ https://www.python.org/dev/peps/pep-0480/ >>> Proposed is an extension to PEP 458 that adds support for end-to-end signing and the maximum security model. End-to-end signing allows both PyPI and developers to sign for the distributions that are downloaded by clients. The minimum security model proposed by PEP 458 supports continuous delivery of distributions (because they are signed by online keys), but that model does not protect distributions in the event that PyPI is compromised. In the minimum security model, attackers who have compromised the signing keys stored on PyPI Infrastructure may sign for malicious distributions. The maximum security model, described in this PEP, retains the benefits of PEP 458 (e.g., immediate availability of distributions that are uploaded to PyPI), but additionally ensures that end-users are not at risk of installing forged software if PyPI is compromised. >> One W3C Linked Data way to handle https://schema.org/SoftwareApplication https://schema.org/SoftwareApplication ( https://codemeta.github.io/user-guide/ https://codemeta.github.io/user-guide/ ) cryptographic signatures of a JSON-LD manifest with per-file and whole package hashes would be with e.g. W3C ld-signatures/ld-proofs and W3C DID (Decentralized Identifiers) or x.509 certs in a CT log. > FWIU, the Fuschia team is building package signing on top of TUF. W3C Web Bundles + Linked Data for the SBOM and metadata could be a good solution for software supply chain security in general: https://news.ycombinator.com/item?id=29296573 https://news.ycombinator.com/item?id=29296573 : >> Web Bundles, more formally known as Bundled HTTP Exchanges, are part of the Web Packaging proposal. https://web.dev/web-bundles/ https://web.dev/web-bundles/ >> HTTP resources in a Web Bundle are indexed by request URLs, and can optionally come with signatures that vouch for the resources. Signatures allow browsers to understand and verify where each resource came from, and treats each as coming from its true origin. This is similar to how Signed HTTP Exchanges, a feature for signing a single HTTP resource, are handled.
- acidbaseextract 5y ago
  Wut? I just make websites man. I don't want to be dismissive, but I hate deploying my applications for this reason. I'm an application developer. I'm not averse to infrastructure as code, and containerization, and I'm happy to do ops for my preferred stack. But I can't learn all this stuff too.
  - andrewxdiamond 5y ago
    Conveniently, Docker has abstracted 99% of what she said away from you.
    - westurner 5y ago
      Does `podman trust` also work yet? Shell commands from the first Docker docs link above: docker trust key generate jeff # docker trust key load key.pem --name jeff docker trust signer add --key cert.pem jeff registry.example.com/admin/demo docker trust sign registry.example.com/admin/demo:1 export DOCKER_CONTENT_TRUST=1 docker push registry.example.com/admin/demo:1 docker trust inspect --pretty registry.example.com/admin/demo:1 docker trust revoke registry.example.com/admin/demo:1
zerd 5y ago
I hope the image registry abstraction opens up the possibility to improve the storage and transfer of images without breaking backwards compatibility. See https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar for why tar is not optimal.
- 411111111111111 5y ago
  I don't see how without breaking compatibility, as that would change the image hash.
hutrdvnj 5y ago
A .tar file on a file server wouldn't suffice, because you have to store a bunch of metadata for that .tar file. But now you could argue that the metadata could just be a file next to the tar file. True, but container registries are kind of like that + optimization + some useful extra features. If you were to start from a simple file server and think about the stuff you need, then after some iterations you would actually end up with a container registry.
- minaguib 5y ago
  AFAIK this is how it worked with classical OS packages (think rpm/yum/apt/etc.) Packages and metadata was all prepared for distribution over nothing more than static file hosting and http(s) to leverage as much existing infrastructure as possible.
- ffk 5y ago
  Docker save (and presumably podman too) adds that metadata as a file in a tar. Each layer becomes a tar that is nested in the top level tar.
  - e12e 5y ago
    Indeed, registries don't do much more than what: docker image save foo:latest \ | ssh pod docker image load does (the stream is a docker tar archive with layers and metadata).
  - hutrdvnj 5y ago
    A container registry allows you to download multiple layer simultaneously and only those that you need (that you don't have in cache).
chriswarbo 5y ago
Agree 100%. Running containers on AWS requires uploading an image to a registry/repository, but there isn't any API to actually upload the .json manifest files and .tar.gz layer files they need, even if I have them sat in a folder in front of me (compare this to Lambda, where we can upload a .zip to S3). The worst part is that the "official" way to upload artifacts to a repository actually requires installing the 'docker' command, running some sort of "login" command in that, piping around some secret tokens generated by the AWS CLI, etc. The only other approach I could find is their "low level" chunk-based upload API. I wrote a Python script around that (pretty much just a 'while' loop), so I can avoid this docker sillyness. https://docs.aws.amazon.com/AmazonECR/latest/userguide/getting-started-cli.html https://docs.aws.amazon.com/AmazonECR/latest/userguide/getti...
- Nullabillity 5y ago
  Looks like Docker's registry API (what the docker CLI uses) supports doing single-chunk uploads: https://docs.docker.com/registry/spec/api/#monolithic-upload https://docs.docker.com/registry/spec/api/#monolithic-upload.
  - chriswarbo 5y ago
    I think accessing AWS's Docker API is what requires the "fake login dance" to begin with.
    - Nullabillity 5y ago
      Yeah, but "fake login dance" is mostly just a consequence of the Docker CLI's design. As an API client you just get an API credential and use it to authenticate against the API.
formerly_proven 5y ago
The docker/OCI container format is incredibly clunky. .wh. files? S_IFWHT? Never heard of 'er! Layers as tars that need extracting? #gzipallthethings (what is that you're saying? gzip is slow?) They did add mime types for layers, so in theory you could do something better, but that's not going to happen because it's not backwards compatible.
juriansluiman 5y ago
Actually it's the process how I (DIY) deploy several static sites from my local machine. I build the Docker image (hugo sites with nginx, expose a single port for HTTP traffic) and save it as tar. Ironically the base images do come from a registry, but I can't deploy to a public registry and don't want to host my own. On the server where I need to run that site, I just transfer the tar, load the image and run the docker image. It's so straightforward I much more prefer this way than being dependant on external registry sites for deployments.
- jffry 5y ago
  Why wouldn't you just run a vanilla nginx image and mount in the folder of site content? Then a deploy would be as simple as rsyncing the built folder from your laptop to the server, no need for sending a whole image every time
grappler 5y ago
I don't mind that registries are a thing but I hate that they're so baked into the tools, which I think is what you're getting at. If I'm starting a project, I want everything small, simple, and self-contained within my project directory. If, right out of the gate, I'm supposed to already have a daemon running (as root it seemed for a while?) which will do the building and execution, and a "registry" set up somewhere, I've already got pieces I likely don't understand and magic commands and high level abstractions for interacting with them, and introductory material telling me "this magic command is all you have to do! look how much you don't have to worry about!" But I am worrying about it. And that's why I love when people write these "hard way" guides. So thank you for that link.

westurner 5y ago

"Signing Images with Docker Content Trust" explains how cryptographic container image signatures work w/ Docker Notary (TUF) https://docs.docker.com/engine/security/trust/#signing-images-with-docker-content-trust https://docs.docker.com/engine/security/trust/#signing-image... The TUF spec (and PyPI TUF PEPs) explains why a tar over https (with optional DNSSEC, a CA cert bundle, CRL, OCSP,) isn't sufficient for secure software distribution. "#ZeroTrust DevOps"; #DevSecOps What's the favorite package format with content signatures, key distribution, a keyring of trusted (authorized) keys, and a cryptographically-signed manifest of per-file hashes, permissions, and extended file attributes? FWIW, ZIP at least does a CRC32. We now have the Linux Foundation CNCF sigstore for any artifact, including OCI container images. W3C ld-proofs is a newer web standard that unfortunately all package managers haven't yet migrated to. https://news.ycombinator.com/item?id=29355786 https://news.ycombinator.com/item?id=29355786 Because ld-proofs is RDF, it works in JSON-LD and you could merge the entire SBOM [1] and e.g. CodeMeta [2] Linked Data metadata for all of the standardized-metadata-documented components in a stack. [1] https://github.com/google/osv/issues/55 https://github.com/google/osv/issues/55 [2] https://github.com/codemeta/codemeta https://github.com/codemeta/codemeta

denysvitali 5y ago
Isn't this reinventing the wheel somehow? Linux distros have been using tar.gz over HTTP (not S!) and relying on GPG for years.
- ibotty 5y ago
  But most Linux distros are not that great either. They don't implement TUF and are not safe against some vectors prevented by that. It's OK to start anew when container distribution poses a different problem anyway. https://en.wikipedia.org/wiki/The_Update_Framework_(TUF) https://en.wikipedia.org/wiki/The_Update_Framework_(TUF)
  - krageon 5y ago
    The linked page doesn't explain anything at all. It says some incredibly obvious things (remotes can be compromised and you should be prepared for that) and then goes on to say the subject fixes all that, but there's actually no explanation of how it does that or what it does to get there. It's a marketing piece, essentially.
    - westurner 5y ago
      From this about DNSSEC and software supply chain security a couple weeks ago, from this about Android packages: https://news.ycombinator.com/item?id=27700513 https://news.ycombinator.com/item?id=27700513 : >> [...] Sigstore is a free and open Linux Foundation service for asset signatures: https://sigstore.dev/what_is_sigstore/ https://sigstore.dev/what_is_sigstore/ >> The TUF Overview explains some of the risks of asset signature systems; key compromise, there's one key for everything that we all share and can't log the revocation of in a CT (Certificate Transparency) log distributed like a DLT, https://theupdateframework.io/overview/ https://theupdateframework.io/overview/ >> Certificate Transparency: https://en.wikipedia.org/wiki/Certificate_Transparency https://en.wikipedia.org/wiki/Certificate_Transparency >> Yeah, there's a channel to secure there at that layer of the software supply chain as well. >> "PEP 480 -- Surviving a Compromise of PyPI: End-to-end signing of packages" (2014-) https://www.python.org/dev/peps/pep-0480/ https://www.python.org/dev/peps/pep-0480/ >>> Proposed is an extension to PEP 458 that adds support for end-to-end signing and the maximum security model. End-to-end signing allows both PyPI and developers to sign for the distributions that are downloaded by clients. The minimum security model proposed by PEP 458 supports continuous delivery of distributions (because they are signed by online keys), but that model does not protect distributions in the event that PyPI is compromised. In the minimum security model, attackers who have compromised the signing keys stored on PyPI Infrastructure may sign for malicious distributions. The maximum security model, described in this PEP, retains the benefits of PEP 458 (e.g., immediate availability of distributions that are uploaded to PyPI), but additionally ensures that end-users are not at risk of installing forged software if PyPI is compromised. >> One W3C Linked Data way to handle https://schema.org/SoftwareApplication https://schema.org/SoftwareApplication ( https://codemeta.github.io/user-guide/ https://codemeta.github.io/user-guide/ ) cryptographic signatures of a JSON-LD manifest with per-file and whole package hashes would be with e.g. W3C ld-signatures/ld-proofs and W3C DID (Decentralized Identifiers) or x.509 certs in a CT log. > FWIU, the Fuschia team is building package signing on top of TUF. W3C Web Bundles + Linked Data for the SBOM and metadata could be a good solution for software supply chain security in general: https://news.ycombinator.com/item?id=29296573 https://news.ycombinator.com/item?id=29296573 : >> Web Bundles, more formally known as Bundled HTTP Exchanges, are part of the Web Packaging proposal. https://web.dev/web-bundles/ https://web.dev/web-bundles/ >> HTTP resources in a Web Bundle are indexed by request URLs, and can optionally come with signatures that vouch for the resources. Signatures allow browsers to understand and verify where each resource came from, and treats each as coming from its true origin. This is similar to how Signed HTTP Exchanges, a feature for signing a single HTTP resource, are handled.
acidbaseextract 5y ago
Wut? I just make websites man. I don't want to be dismissive, but I hate deploying my applications for this reason. I'm an application developer. I'm not averse to infrastructure as code, and containerization, and I'm happy to do ops for my preferred stack. But I can't learn all this stuff too.
- andrewxdiamond 5y ago
  Conveniently, Docker has abstracted 99% of what she said away from you.
  - westurner 5y ago
    Does `podman trust` also work yet? Shell commands from the first Docker docs link above: docker trust key generate jeff # docker trust key load key.pem --name jeff docker trust signer add --key cert.pem jeff registry.example.com/admin/demo docker trust sign registry.example.com/admin/demo:1 export DOCKER_CONTENT_TRUST=1 docker push registry.example.com/admin/demo:1 docker trust inspect --pretty registry.example.com/admin/demo:1 docker trust revoke registry.example.com/admin/demo:1

zerd 5y ago

I hope the image registry abstraction opens up the possibility to improve the storage and transfer of images without breaking backwards compatibility. See https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar for why tar is not optimal.

411111111111111 5y ago
I don't see how without breaking compatibility, as that would change the image hash.

hutrdvnj 5y ago

A .tar file on a file server wouldn't suffice, because you have to store a bunch of metadata for that .tar file. But now you could argue that the metadata could just be a file next to the tar file. True, but container registries are kind of like that + optimization + some useful extra features. If you were to start from a simple file server and think about the stuff you need, then after some iterations you would actually end up with a container registry.

minaguib 5y ago
AFAIK this is how it worked with classical OS packages (think rpm/yum/apt/etc.) Packages and metadata was all prepared for distribution over nothing more than static file hosting and http(s) to leverage as much existing infrastructure as possible.
ffk 5y ago
Docker save (and presumably podman too) adds that metadata as a file in a tar. Each layer becomes a tar that is nested in the top level tar.
- e12e 5y ago
  Indeed, registries don't do much more than what: docker image save foo:latest \ | ssh pod docker image load does (the stream is a docker tar archive with layers and metadata).
- hutrdvnj 5y ago
  A container registry allows you to download multiple layer simultaneously and only those that you need (that you don't have in cache).

chriswarbo 5y ago

Agree 100%. Running containers on AWS requires uploading an image to a registry/repository, but there isn't any API to actually upload the .json manifest files and .tar.gz layer files they need, even if I have them sat in a folder in front of me (compare this to Lambda, where we can upload a .zip to S3). The worst part is that the "official" way to upload artifacts to a repository actually requires installing the 'docker' command, running some sort of "login" command in that, piping around some secret tokens generated by the AWS CLI, etc. The only other approach I could find is their "low level" chunk-based upload API. I wrote a Python script around that (pretty much just a 'while' loop), so I can avoid this docker sillyness. https://docs.aws.amazon.com/AmazonECR/latest/userguide/getting-started-cli.html https://docs.aws.amazon.com/AmazonECR/latest/userguide/getti...

Nullabillity 5y ago
Looks like Docker's registry API (what the docker CLI uses) supports doing single-chunk uploads: https://docs.docker.com/registry/spec/api/#monolithic-upload https://docs.docker.com/registry/spec/api/#monolithic-upload.
- chriswarbo 5y ago
  I think accessing AWS's Docker API is what requires the "fake login dance" to begin with.
  - Nullabillity 5y ago
    Yeah, but "fake login dance" is mostly just a consequence of the Docker CLI's design. As an API client you just get an API credential and use it to authenticate against the API.

formerly_proven 5y ago

The docker/OCI container format is incredibly clunky. .wh. files? S_IFWHT? Never heard of 'er! Layers as tars that need extracting? #gzipallthethings (what is that you're saying? gzip is slow?) They did add mime types for layers, so in theory you could do something better, but that's not going to happen because it's not backwards compatible.

juriansluiman 5y ago

Actually it's the process how I (DIY) deploy several static sites from my local machine. I build the Docker image (hugo sites with nginx, expose a single port for HTTP traffic) and save it as tar. Ironically the base images do come from a registry, but I can't deploy to a public registry and don't want to host my own. On the server where I need to run that site, I just transfer the tar, load the image and run the docker image. It's so straightforward I much more prefer this way than being dependant on external registry sites for deployments.

jffry 5y ago
Why wouldn't you just run a vanilla nginx image and mount in the folder of site content? Then a deploy would be as simple as rsyncing the built folder from your laptop to the server, no need for sending a whole image every time

grappler 5y ago

I don't mind that registries are a thing but I hate that they're so baked into the tools, which I think is what you're getting at. If I'm starting a project, I want everything small, simple, and self-contained within my project directory. If, right out of the gate, I'm supposed to already have a daemon running (as root it seemed for a while?) which will do the building and execution, and a "registry" set up somewhere, I've already got pieces I likely don't understand and magic commands and high level abstractions for interacting with them, and introductory material telling me "this magic command is all you have to do! look how much you don't have to worry about!" But I am worrying about it. And that's why I love when people write these "hard way" guides. So thank you for that link.