5 ms·

I've been trying to use podman in production and it is not working very well. I'm excited for this technology but it is not ready. - podman layer caching durin

by yobert 6y ago

I've been trying to use podman in production and it is not working very well. I'm excited for this technology but it is not ready. - podman layer caching during builds do not work. When we switched back to docker-ce, builds went from 45 minutes to 3 minutes with no changes to our Dockerfile - fuse-overlayfs sits at 100% CPU all day, every day on our production servers - podman loses track of containers. Sometimes they are running but don't show in podman ps - sometimes podman just gets stuck and can't prune or stop a container (edit: formatting)

yobert 6y ago
Reading through the comments though it looks like I should try making a BTRFS mount for the container storage and that might help the fuse-overlayfs problems though.
- tw04 6y ago
  I know it tends to be a controversial technology around here, but I've had great luck with OpenZFS for container storage.
- znpy 6y ago
  It feels weird that a containerisation technology imposes requirements on the underlying storage technology to work well, tho. Now I understand that btrfs is not really a requirement. But I've been thinking about using podman to replace docker-in-docker on our fleet of gitlab runners where we build our images, and running btrfs is a deal breaker. We really don't want to add complexity to the mix. Fuse-overlay burning through the whole CPU is not really good looking either. Mh. I still have to properly dig deep into this, but I'm keeping my hopes high.
  - carlosf 6y ago
    Have you tried Kaniko? That's how I got hid of DinD in my Gitlab Runners.
    - horstmeyer 6y ago
      Could you maybe provide a small example of the relevant part of the gitlab CI definition? I assume you run kaniko inside a Docker container on the runner?
  - tw04 6y ago
    >It feels weird that a containerisation technology imposes requirements on the underlying storage technology to work well, tho. Why? Storage is literally the backbone of technology. From databases to webservers there are unique and real requirements. There's a reason why the enterprise storage market is still worth several billion dollars, and it's not because storage is easy. In 2021, with billions of dollars of R&D at their disposal - neither Amazon, Google, or Microsoft have even a mediocre NAS stack in comparison to the likes of Dell/EMC (Isilon) or NetApp (ONTAP). Heck they can't even compete with startups like Qumulo or Vast.
    - windexh8er 6y ago
      Are you sure Amazon, Google and Microsoft don't have this competitive stack because they don't want to enter that market or are in that market solely for themselves? Storage may be a multi-billion dollar market - but it's also razor thin margin compared to where those brands focus their R&D spend. I'd argue those you've named don't have interest selling storage to the enterprise in the form of legacy boxes. They make far more revenue leasing storage to the enterprise. Cloud storage is also recurring revenue where enterprise storage is considered perpetual. The latter is generally frowned upon from the Street's perspective these days (good, bad or otherwise).
      - tw04 6y ago
        >Are you sure Amazon, Google and Microsoft don't have this competitive stack because they don't want to enter that market or are in that market solely for themselves They have "competition" in the market, it's just horrible. Because storage is hard. >Storage may be a multi-billion dollar market - but it's also razor thin margin compared to where those brands focus their R&D spend. I'm going to let you go back and review the earnings reports from the aforementioned companies. Razor thin margins? You wouldn't survive in the market with razor thin margins. > I'd argue those you've named don't have interest selling storage to the enterprise in the form of legacy boxes. I'd argue Azure Stack, AWS Outpost, and Google Anthos readily prove you wrong. > Cloud storage is also recurring revenue where enterprise storage is considered perpetual. Just... no. Enterprise storage has a shelf life of 3-5 years before maintenance or technology make it obsolete. >The latter is generally frowned upon from the Street's perspective these days (good, bad or otherwise). https://www.google.com/finance/quote/NTAP:NASDAQ https://www.google.com/finance/quote/NTAP:NASDAQ 5 years ago they were at $20/share, today they're at $69/share. Reality doesn't appear to match your claim as to the street's opinion of enterprise storage.
dastx 6y ago
I've been running multiple services through podman on a pi without any issues. Certainly not the same as production though. Layer caching seems to work for me. Note that rootless stores images (and everything else), in different place from rootfull. It may be that you're caching the wrong directory.
nednar 6y ago
What are the Github issue IDs for these 4 problems?
- bartvk 6y ago
  What are you trying to say here? Your comment feels a bit passive-aggressive.
  - nednar 6y ago
    > What are you trying to say here? I want to look at the problems. But I don't want to do a guessing game with the 159 open issues, which one may relate to one of these points or not.
- qbasic_forever 6y ago
  Where are the four unit tests in podman's codebase that should be catching these issues? I find it very strange that your reaction is, "the software is infallible, it is _YOU_ who have failed the software by not logging bugs!" It is perfectly reasonable to have a conversation about the quality or issues with a piece of software SEPARATE TO the logging or finding of bugs in said software.
- yobert 6y ago
  I agree with the sentiment that I should go find them. But I also agree with the sentiment that if RHEL 8 is going to make it hard to install docker and promote podman instead as a drop-in replacement, it's a bit frustrating.
  - nednar 6y ago
    > I agree with the sentiment that I should go find them. I can sympathize with feeling overwhelmed when searching for similar Issues in a bug database, as I was in a similar situation in the past. What happend was this: I went to the central starting point of the repo's database (e.g., https://github.com/containers/podman/issues https://github.com/containers/podman/issues), removed the filters like `is:open` from the search bar, and entered one or two keywords that for me sounded similar to the problem I faced. As I didn't find any Issues that seemed similar, I just created a new one. Of course, I was worried about creating a possible duplicate, but I figured that it would still be a valuable contribution by adding more keywords (with the maintainers linking my duplicate to the ticket where the problem gets resolved) for others to search for. And last but not least, I thought this would also add additional debugging information for the developers, as well as an indication of the community-wide impact of the problem for the release management team. In the end, I was quite happy I went this way, because I didn't just make a small impact on a project that would be useful for me, but it also enabled me to connect to the amazing people that build the tools I use on a daily basis. I hope that you can also find a way that gives you some happiness in dealing with your situation, yobert.
windexh8er 6y ago
I could very well be wrong but Podman seems to have missed the time-frame of opportunity. It was always a knife fight between Red Hat and Docker with regard to tooling. Red Hat wanting to own the toolchain for containers so they didn't have to deal with, and so they could box out, competitors like (now basically defunct) Docker Enterprise. I've taken a look at podman from time to time over the years but it seems like it's just never formalized, never been polished and almost always has been sub-par in execution. On this list the builds and container control are things that I've run across. I guess - what's the point? The rootless argument leaned on so heavily is pretty much gone, the quality of Podman hasn't (seemingly) improved and now IBM owns Red Hat (subjective, but a viable concern/consideration given what's recently happened with CentOS). You're more than safe leveraging Docker and buildkit (when and where needed). Quite honestly, given the relatively poor execution of Red Hat with these tools over the years, I don't see the point. I'm sure there are some niche use cases for Podman/Buildah, but overall it just seems to come up as an agenda more than an exponentially better product at this point. Red Hat could have made things better, instead they just created a distraction and worked against the broader effort in the container ecosystem.
- stock_toaster 6y ago
  Not to mention podman's historic[1] lack of compose support (or similar), which is frequently used in CI and local dev environments. I think has really held it back for a lot of folks. [1]: Apparently initial support planned in version 3.0?
  - dastx 6y ago
    I don't know if I'm misunderstanding, but if by 'compose support' you mean docker-compose, then podman-compose has existed for a lot longer than v3.0 (which got released yesterday). First commit was nearly two years ago. Sadly it doesn't feel as polished as docker-compose, I can only assume it's due to the lack of API. Specifically, podman-compose just translates all the docker-compose.yml file directives into podman cli commands, which doesn't seem to be handled gracefully.
- gbraad 6y ago
paulfurtado 6y ago
I'm surprised it's using fuse-overlayfs instead of the overlay2 driver. Is it that you're using rootless containers in production? For rootful containers I think you'd have the same experience as docker here (or at least much closer).
mekster 6y ago
What are you exactly excited about? It's just minor difference to me except it doesn't work well. Not sure what the laziness was about from RedHat on podman development.

by yobert 6y ago

yobert 6y ago

Reading through the comments though it looks like I should try making a BTRFS mount for the container storage and that might help the fuse-overlayfs problems though.

tw04 6y ago
I know it tends to be a controversial technology around here, but I've had great luck with OpenZFS for container storage.
znpy 6y ago
It feels weird that a containerisation technology imposes requirements on the underlying storage technology to work well, tho. Now I understand that btrfs is not really a requirement. But I've been thinking about using podman to replace docker-in-docker on our fleet of gitlab runners where we build our images, and running btrfs is a deal breaker. We really don't want to add complexity to the mix. Fuse-overlay burning through the whole CPU is not really good looking either. Mh. I still have to properly dig deep into this, but I'm keeping my hopes high.
- carlosf 6y ago
  Have you tried Kaniko? That's how I got hid of DinD in my Gitlab Runners.
  - horstmeyer 6y ago
    Could you maybe provide a small example of the relevant part of the gitlab CI definition? I assume you run kaniko inside a Docker container on the runner?
- tw04 6y ago
  >It feels weird that a containerisation technology imposes requirements on the underlying storage technology to work well, tho. Why? Storage is literally the backbone of technology. From databases to webservers there are unique and real requirements. There's a reason why the enterprise storage market is still worth several billion dollars, and it's not because storage is easy. In 2021, with billions of dollars of R&D at their disposal - neither Amazon, Google, or Microsoft have even a mediocre NAS stack in comparison to the likes of Dell/EMC (Isilon) or NetApp (ONTAP). Heck they can't even compete with startups like Qumulo or Vast.
  - windexh8er 6y ago
    Are you sure Amazon, Google and Microsoft don't have this competitive stack because they don't want to enter that market or are in that market solely for themselves? Storage may be a multi-billion dollar market - but it's also razor thin margin compared to where those brands focus their R&D spend. I'd argue those you've named don't have interest selling storage to the enterprise in the form of legacy boxes. They make far more revenue leasing storage to the enterprise. Cloud storage is also recurring revenue where enterprise storage is considered perpetual. The latter is generally frowned upon from the Street's perspective these days (good, bad or otherwise).
    - tw04 6y ago
      >Are you sure Amazon, Google and Microsoft don't have this competitive stack because they don't want to enter that market or are in that market solely for themselves They have "competition" in the market, it's just horrible. Because storage is hard. >Storage may be a multi-billion dollar market - but it's also razor thin margin compared to where those brands focus their R&D spend. I'm going to let you go back and review the earnings reports from the aforementioned companies. Razor thin margins? You wouldn't survive in the market with razor thin margins. > I'd argue those you've named don't have interest selling storage to the enterprise in the form of legacy boxes. I'd argue Azure Stack, AWS Outpost, and Google Anthos readily prove you wrong. > Cloud storage is also recurring revenue where enterprise storage is considered perpetual. Just... no. Enterprise storage has a shelf life of 3-5 years before maintenance or technology make it obsolete. >The latter is generally frowned upon from the Street's perspective these days (good, bad or otherwise). https://www.google.com/finance/quote/NTAP:NASDAQ https://www.google.com/finance/quote/NTAP:NASDAQ 5 years ago they were at $20/share, today they're at $69/share. Reality doesn't appear to match your claim as to the street's opinion of enterprise storage.

dastx 6y ago

I've been running multiple services through podman on a pi without any issues. Certainly not the same as production though. Layer caching seems to work for me. Note that rootless stores images (and everything else), in different place from rootfull. It may be that you're caching the wrong directory.

nednar 6y ago

What are the Github issue IDs for these 4 problems?

bartvk 6y ago
What are you trying to say here? Your comment feels a bit passive-aggressive.
- nednar 6y ago
  > What are you trying to say here? I want to look at the problems. But I don't want to do a guessing game with the 159 open issues, which one may relate to one of these points or not.
qbasic_forever 6y ago
Where are the four unit tests in podman's codebase that should be catching these issues? I find it very strange that your reaction is, "the software is infallible, it is _YOU_ who have failed the software by not logging bugs!" It is perfectly reasonable to have a conversation about the quality or issues with a piece of software SEPARATE TO the logging or finding of bugs in said software.
yobert 6y ago
I agree with the sentiment that I should go find them. But I also agree with the sentiment that if RHEL 8 is going to make it hard to install docker and promote podman instead as a drop-in replacement, it's a bit frustrating.
- nednar 6y ago
  > I agree with the sentiment that I should go find them. I can sympathize with feeling overwhelmed when searching for similar Issues in a bug database, as I was in a similar situation in the past. What happend was this: I went to the central starting point of the repo's database (e.g., https://github.com/containers/podman/issues https://github.com/containers/podman/issues), removed the filters like `is:open` from the search bar, and entered one or two keywords that for me sounded similar to the problem I faced. As I didn't find any Issues that seemed similar, I just created a new one. Of course, I was worried about creating a possible duplicate, but I figured that it would still be a valuable contribution by adding more keywords (with the maintainers linking my duplicate to the ticket where the problem gets resolved) for others to search for. And last but not least, I thought this would also add additional debugging information for the developers, as well as an indication of the community-wide impact of the problem for the release management team. In the end, I was quite happy I went this way, because I didn't just make a small impact on a project that would be useful for me, but it also enabled me to connect to the amazing people that build the tools I use on a daily basis. I hope that you can also find a way that gives you some happiness in dealing with your situation, yobert.

windexh8er 6y ago

I could very well be wrong but Podman seems to have missed the time-frame of opportunity. It was always a knife fight between Red Hat and Docker with regard to tooling. Red Hat wanting to own the toolchain for containers so they didn't have to deal with, and so they could box out, competitors like (now basically defunct) Docker Enterprise. I've taken a look at podman from time to time over the years but it seems like it's just never formalized, never been polished and almost always has been sub-par in execution. On this list the builds and container control are things that I've run across. I guess - what's the point? The rootless argument leaned on so heavily is pretty much gone, the quality of Podman hasn't (seemingly) improved and now IBM owns Red Hat (subjective, but a viable concern/consideration given what's recently happened with CentOS). You're more than safe leveraging Docker and buildkit (when and where needed). Quite honestly, given the relatively poor execution of Red Hat with these tools over the years, I don't see the point. I'm sure there are some niche use cases for Podman/Buildah, but overall it just seems to come up as an agenda more than an exponentially better product at this point. Red Hat could have made things better, instead they just created a distraction and worked against the broader effort in the container ecosystem.

stock_toaster 6y ago
Not to mention podman's historic[1] lack of compose support (or similar), which is frequently used in CI and local dev environments. I think has really held it back for a lot of folks. [1]: Apparently initial support planned in version 3.0?
- dastx 6y ago
  I don't know if I'm misunderstanding, but if by 'compose support' you mean docker-compose, then podman-compose has existed for a lot longer than v3.0 (which got released yesterday). First commit was nearly two years ago. Sadly it doesn't feel as polished as docker-compose, I can only assume it's due to the lack of API. Specifically, podman-compose just translates all the docker-compose.yml file directives into podman cli commands, which doesn't seem to be handled gracefully.
gbraad 6y ago

paulfurtado 6y ago

I'm surprised it's using fuse-overlayfs instead of the overlay2 driver. Is it that you're using rootless containers in production? For rootful containers I think you'd have the same experience as docker here (or at least much closer).

mekster 6y ago

What are you exactly excited about? It's just minor difference to me except it doesn't work well. Not sure what the laziness was about from RedHat on podman development.