13 ms·

Podman can transfer container images without a registry

149 points by kukx 5y ago

leetbulb 5y ago
You can do this with Docker easily as well docker save <image> | ssh <remote host> docker load
- realPubkey 5y ago
  Also with docker-compose there is a remote option to deploy local containers via ssh.
  - Hasnep 5y ago
    Could you explain more about this? I can't seem to find anything online about it
- ollien 5y ago
  Heh - in college I took a network security class that required us to set up our own attack/defend environments, and one of the important parts of the report was to explain to the TAs how to set up our environment. We were able to save ourselves a ton of writing by just exporting docker images like this and giving instructions on how to load those onto the VMs :) Someone else I knew made .deb packages, which might have been smarter, since there were some hosts we didn't containerize (mainly ones that handled routing and such; I know now we might have been able to get away with doing it, but at the time I didn't and I thought it might be too much hassle for an already complicated project)
  - ffk 5y ago
    Thank you for mentioning this! I wrote docker save and load, and I’m happy to hear that it helped you!
- matt_kantor 5y ago
  As a step beyond that, I use this script: https://github.com/mkantor/docker-pushmi-pullyu https://github.com/mkantor/docker-pushmi-pullyu `docker save` archives the entire (often huge) image. `docker-pushmi-pullyu` uses an ephemeral registry as an intermediary, so it only needs to transfer layers that have changed. It saves me a lot of time.
  - mst 5y ago
    Heh, the last time I saw something called pushmi/pullyu it was clkao's svn repo sync stuff. Is that where you got the names from or somewhere else? (the blast-from-the-past aspect of seeing those names again has got me wondering about the etymology in general :)
    - matt_kantor 5y ago
      It's a creature from Doctor Dolittle: https://en.wikipedia.org/wiki/List_of_Doctor_Dolittle_characters#Pushmi-Pullyu https://en.wikipedia.org/wiki/List_of_Doctor_Dolittle_charac...
      - mst 5y ago
        Aha! Thank you.
nawgz 5y ago
I use this trick to push to servers in an unnecessarily tight network I have to deploy to sometimes that can't see my source control / container registry. But I do it for Docker. I have overall the sense that Podman is trying to accomplish feature parity with Docker but isn't there yet. Feedback on this formulation?
- scheme271 5y ago
  There's a few places where Podman probably has to catch up with docker but conversely, I think Docker is still trying to catch up with Podman in regards to running rootless (i.e. running containers using a user account without having root privileges).
  - nawgz 5y ago
    Yes, having to set up the docker group is quite painful, and especially the membership requirement can cause issues when the account has to log in to the machine to be eligible for membership.
    - _flux 5y ago
      Oftentimes exec newgrp docker can help. Of course fixes it only for that particular process.
    - rhatdan 5y ago
      Podman works with docker-compose in rootless mode, no need to set up the security questionable DOCKER Group.
  - stavros 5y ago
    Does Podman have enough parity that I can run our Compose stack locally? Last I tried (months ago), enough things failed that I gave up, but it would be great if developers could easily develop locally without root (which causes a bunch of permissions problems).
    - 5y ago
qbasic_forever 5y ago
Containerd and nerdctl have the neat trick of (experimental) support for distributing images over IPFS: https://github.com/containerd/nerdctl/blob/master/docs/ipfs.md https://github.com/containerd/nerdctl/blob/master/docs/ipfs....
candiddevmike 5y ago
I wish containers hadn't created the abstractions of a registry and an image instead of exposing it all as tar files (which is what it kind of is under the covers) served over a glorified file server. This leads to people assuming there's some kind of magic happening and that the entire process is very arcane, when in reality it's just unpacking tar files. If you want to DIY a container with unix tools, this should help: https://containers.gitbook.io/build-containers-the-hard-way/ https://containers.gitbook.io/build-containers-the-hard-way/
- westurner 5y ago
  "Signing Images with Docker Content Trust" explains how cryptographic container image signatures work w/ Docker Notary (TUF) https://docs.docker.com/engine/security/trust/#signing-images-with-docker-content-trust https://docs.docker.com/engine/security/trust/#signing-image... The TUF spec (and PyPI TUF PEPs) explains why a tar over https (with optional DNSSEC, a CA cert bundle, CRL, OCSP,) isn't sufficient for secure software distribution. "#ZeroTrust DevOps"; #DevSecOps What's the favorite package format with content signatures, key distribution, a keyring of trusted (authorized) keys, and a cryptographically-signed manifest of per-file hashes, permissions, and extended file attributes? FWIW, ZIP at least does a CRC32. We now have the Linux Foundation CNCF sigstore for any artifact, including OCI container images. W3C ld-proofs is a newer web standard that unfortunately all package managers haven't yet migrated to. https://news.ycombinator.com/item?id=29355786 https://news.ycombinator.com/item?id=29355786 Because ld-proofs is RDF, it works in JSON-LD and you could merge the entire SBOM [1] and e.g. CodeMeta [2] Linked Data metadata for all of the standardized-metadata-documented components in a stack. [1] https://github.com/google/osv/issues/55 https://github.com/google/osv/issues/55 [2] https://github.com/codemeta/codemeta https://github.com/codemeta/codemeta
  - denysvitali 5y ago
    Isn't this reinventing the wheel somehow? Linux distros have been using tar.gz over HTTP (not S!) and relying on GPG for years.
    - 5y ago
m463 5y ago
It's unclear if this will be faster if the image already exists (layers exist). a docker push/docker pull can skip layers that already exist.
muhehe 5y ago
Anybody know of some simple/lighweight registry for local usage? Quay boasts itself as a super duper enterprisey all solution. I'm looking for something more of a 'simple http server with basic acl' solution.
- briggers 5y ago
  Perhaps you are looking for something like this? docker run -d -p 5000:5000 --name registry registry:2 https://docs.docker.com/registry/#:~:text=The%20Registry%20is%20a%20stateless,under%20the%20permissive%20Apache%20license https://docs.docker.com/registry/#:~:text=The%20Registry%20i....
- gangstead 5y ago
  I don't know if this is light weight enough but I have some experience with [Harbor](https://goharbor.io/ https://goharbor.io/) for our company. The ACL it presents is simple enough. Maybe it was just our implementation but it ended running a lot of components on our cluster so I can't vouch for local use. I ended up replacing it with AWS ECR. We only have a couple of container repos so ECR only ends up costing a few dollars per month. Not local, but very easy and almost free.
technofiend 5y ago
It's interesting skopeo [1] hasn't popped up in this discussion, partially because it is part of redhat's container tools along with podman, and partially because although it started out as a tool to examine remote containers it too supports container migration, but not just between registries. From the linked website "Skopeo is a tool for moving container images between different types of container storages. It allows you to copy container images between container registries like docker.io, quay.io, and your internal container registry or different types of storage on your local system". Perhaps redhat plan to roll up skopeo functionality into Podman at some point? https://www.redhat.com/en/blog/skopeo-10-released#:~:text=Skopeo%20is%20a%20tool%20for,storage%20on%20your%20local%20system https://www.redhat.com/en/blog/skopeo-10-released#:~:text=Sk....

149 points by kukx 5y ago

leetbulb 5y ago

You can do this with Docker easily as well docker save <image> | ssh <remote host> docker load

realPubkey 5y ago
Also with docker-compose there is a remote option to deploy local containers via ssh.
- Hasnep 5y ago
  Could you explain more about this? I can't seem to find anything online about it
ollien 5y ago
Heh - in college I took a network security class that required us to set up our own attack/defend environments, and one of the important parts of the report was to explain to the TAs how to set up our environment. We were able to save ourselves a ton of writing by just exporting docker images like this and giving instructions on how to load those onto the VMs :) Someone else I knew made .deb packages, which might have been smarter, since there were some hosts we didn't containerize (mainly ones that handled routing and such; I know now we might have been able to get away with doing it, but at the time I didn't and I thought it might be too much hassle for an already complicated project)
- ffk 5y ago
  Thank you for mentioning this! I wrote docker save and load, and I’m happy to hear that it helped you!
matt_kantor 5y ago
As a step beyond that, I use this script: https://github.com/mkantor/docker-pushmi-pullyu https://github.com/mkantor/docker-pushmi-pullyu `docker save` archives the entire (often huge) image. `docker-pushmi-pullyu` uses an ephemeral registry as an intermediary, so it only needs to transfer layers that have changed. It saves me a lot of time.
- mst 5y ago
  Heh, the last time I saw something called pushmi/pullyu it was clkao's svn repo sync stuff. Is that where you got the names from or somewhere else? (the blast-from-the-past aspect of seeing those names again has got me wondering about the etymology in general :)
  - matt_kantor 5y ago
    It's a creature from Doctor Dolittle: https://en.wikipedia.org/wiki/List_of_Doctor_Dolittle_characters#Pushmi-Pullyu https://en.wikipedia.org/wiki/List_of_Doctor_Dolittle_charac...
    - mst 5y ago
      Aha! Thank you.

nawgz 5y ago

I use this trick to push to servers in an unnecessarily tight network I have to deploy to sometimes that can't see my source control / container registry. But I do it for Docker. I have overall the sense that Podman is trying to accomplish feature parity with Docker but isn't there yet. Feedback on this formulation?

scheme271 5y ago
There's a few places where Podman probably has to catch up with docker but conversely, I think Docker is still trying to catch up with Podman in regards to running rootless (i.e. running containers using a user account without having root privileges).
- nawgz 5y ago
  Yes, having to set up the docker group is quite painful, and especially the membership requirement can cause issues when the account has to log in to the machine to be eligible for membership.
  - _flux 5y ago
    Oftentimes exec newgrp docker can help. Of course fixes it only for that particular process.
  - rhatdan 5y ago
    Podman works with docker-compose in rootless mode, no need to set up the security questionable DOCKER Group.
- stavros 5y ago
  Does Podman have enough parity that I can run our Compose stack locally? Last I tried (months ago), enough things failed that I gave up, but it would be great if developers could easily develop locally without root (which causes a bunch of permissions problems).
  - 5y ago

qbasic_forever 5y ago

Containerd and nerdctl have the neat trick of (experimental) support for distributing images over IPFS: https://github.com/containerd/nerdctl/blob/master/docs/ipfs.md https://github.com/containerd/nerdctl/blob/master/docs/ipfs....

candiddevmike 5y ago

I wish containers hadn't created the abstractions of a registry and an image instead of exposing it all as tar files (which is what it kind of is under the covers) served over a glorified file server. This leads to people assuming there's some kind of magic happening and that the entire process is very arcane, when in reality it's just unpacking tar files. If you want to DIY a container with unix tools, this should help: https://containers.gitbook.io/build-containers-the-hard-way/ https://containers.gitbook.io/build-containers-the-hard-way/

westurner 5y ago
"Signing Images with Docker Content Trust" explains how cryptographic container image signatures work w/ Docker Notary (TUF) https://docs.docker.com/engine/security/trust/#signing-images-with-docker-content-trust https://docs.docker.com/engine/security/trust/#signing-image... The TUF spec (and PyPI TUF PEPs) explains why a tar over https (with optional DNSSEC, a CA cert bundle, CRL, OCSP,) isn't sufficient for secure software distribution. "#ZeroTrust DevOps"; #DevSecOps What's the favorite package format with content signatures, key distribution, a keyring of trusted (authorized) keys, and a cryptographically-signed manifest of per-file hashes, permissions, and extended file attributes? FWIW, ZIP at least does a CRC32. We now have the Linux Foundation CNCF sigstore for any artifact, including OCI container images. W3C ld-proofs is a newer web standard that unfortunately all package managers haven't yet migrated to. https://news.ycombinator.com/item?id=29355786 https://news.ycombinator.com/item?id=29355786 Because ld-proofs is RDF, it works in JSON-LD and you could merge the entire SBOM [1] and e.g. CodeMeta [2] Linked Data metadata for all of the standardized-metadata-documented components in a stack. [1] https://github.com/google/osv/issues/55 https://github.com/google/osv/issues/55 [2] https://github.com/codemeta/codemeta https://github.com/codemeta/codemeta
- denysvitali 5y ago
  Isn't this reinventing the wheel somehow? Linux distros have been using tar.gz over HTTP (not S!) and relying on GPG for years.
  - 5y ago

m463 5y ago

It's unclear if this will be faster if the image already exists (layers exist). a docker push/docker pull can skip layers that already exist.

muhehe 5y ago

Anybody know of some simple/lighweight registry for local usage? Quay boasts itself as a super duper enterprisey all solution. I'm looking for something more of a 'simple http server with basic acl' solution.

briggers 5y ago
Perhaps you are looking for something like this? docker run -d -p 5000:5000 --name registry registry:2 https://docs.docker.com/registry/#:~:text=The%20Registry%20is%20a%20stateless,under%20the%20permissive%20Apache%20license https://docs.docker.com/registry/#:~:text=The%20Registry%20i....
gangstead 5y ago
I don't know if this is light weight enough but I have some experience with [Harbor](https://goharbor.io/ https://goharbor.io/) for our company. The ACL it presents is simple enough. Maybe it was just our implementation but it ended running a lot of components on our cluster so I can't vouch for local use. I ended up replacing it with AWS ECR. We only have a couple of container repos so ECR only ends up costing a few dollars per month. Not local, but very easy and almost free.

technofiend 5y ago

It's interesting skopeo [1] hasn't popped up in this discussion, partially because it is part of redhat's container tools along with podman, and partially because although it started out as a tool to examine remote containers it too supports container migration, but not just between registries. From the linked website "Skopeo is a tool for moving container images between different types of container storages. It allows you to copy container images between container registries like docker.io, quay.io, and your internal container registry or different types of storage on your local system". Perhaps redhat plan to roll up skopeo functionality into Podman at some point? https://www.redhat.com/en/blog/skopeo-10-released#:~:text=Skopeo%20is%20a%20tool%20for,storage%20on%20your%20local%20system https://www.redhat.com/en/blog/skopeo-10-released#:~:text=Sk....