8 ms·

Reading through the comments though it looks like I should try making a BTRFS mount for the container storage and that might help the fuse-overlayfs problems th

by yobert 6y ago

Reading through the comments though it looks like I should try making a BTRFS mount for the container storage and that might help the fuse-overlayfs problems though.

tw04 6y ago
I know it tends to be a controversial technology around here, but I've had great luck with OpenZFS for container storage.
znpy 6y ago
It feels weird that a containerisation technology imposes requirements on the underlying storage technology to work well, tho. Now I understand that btrfs is not really a requirement. But I've been thinking about using podman to replace docker-in-docker on our fleet of gitlab runners where we build our images, and running btrfs is a deal breaker. We really don't want to add complexity to the mix. Fuse-overlay burning through the whole CPU is not really good looking either. Mh. I still have to properly dig deep into this, but I'm keeping my hopes high.
- carlosf 6y ago
  Have you tried Kaniko? That's how I got hid of DinD in my Gitlab Runners.
  - horstmeyer 6y ago
    Could you maybe provide a small example of the relevant part of the gitlab CI definition? I assume you run kaniko inside a Docker container on the runner?
- tw04 6y ago
  >It feels weird that a containerisation technology imposes requirements on the underlying storage technology to work well, tho. Why? Storage is literally the backbone of technology. From databases to webservers there are unique and real requirements. There's a reason why the enterprise storage market is still worth several billion dollars, and it's not because storage is easy. In 2021, with billions of dollars of R&D at their disposal - neither Amazon, Google, or Microsoft have even a mediocre NAS stack in comparison to the likes of Dell/EMC (Isilon) or NetApp (ONTAP). Heck they can't even compete with startups like Qumulo or Vast.
  - windexh8er 6y ago
    Are you sure Amazon, Google and Microsoft don't have this competitive stack because they don't want to enter that market or are in that market solely for themselves? Storage may be a multi-billion dollar market - but it's also razor thin margin compared to where those brands focus their R&D spend. I'd argue those you've named don't have interest selling storage to the enterprise in the form of legacy boxes. They make far more revenue leasing storage to the enterprise. Cloud storage is also recurring revenue where enterprise storage is considered perpetual. The latter is generally frowned upon from the Street's perspective these days (good, bad or otherwise).
    - tw04 6y ago
      >Are you sure Amazon, Google and Microsoft don't have this competitive stack because they don't want to enter that market or are in that market solely for themselves They have "competition" in the market, it's just horrible. Because storage is hard. >Storage may be a multi-billion dollar market - but it's also razor thin margin compared to where those brands focus their R&D spend. I'm going to let you go back and review the earnings reports from the aforementioned companies. Razor thin margins? You wouldn't survive in the market with razor thin margins. > I'd argue those you've named don't have interest selling storage to the enterprise in the form of legacy boxes. I'd argue Azure Stack, AWS Outpost, and Google Anthos readily prove you wrong. > Cloud storage is also recurring revenue where enterprise storage is considered perpetual. Just... no. Enterprise storage has a shelf life of 3-5 years before maintenance or technology make it obsolete. >The latter is generally frowned upon from the Street's perspective these days (good, bad or otherwise). https://www.google.com/finance/quote/NTAP:NASDAQ https://www.google.com/finance/quote/NTAP:NASDAQ 5 years ago they were at $20/share, today they're at $69/share. Reality doesn't appear to match your claim as to the street's opinion of enterprise storage.
      - windexh8er 6y ago
        First, if you review the financials of the storage companies compared to earnings of all cloud companies they're abysmal. Yes, razor thin margins - comparatively. The cost of buying physical storage is commoditized. I've worked in technology from the pre-sale engineering side for a number of years. Margins on hardware are fractional compared to software and subscription offerings. Stack, Outpost and Anthos are not revenue drivers today. They're vendor lock in tools. 3-5 years is a horrible argument in the position you're attempting to make considering it's an eternity in technology. Consider that you sell the storage once in that 3-5 years. Your cloud provider bills you monthly and you don't own it. It's recurring revenue vs a singular sales event. NTAP over 5 years is a decent return. 215% over that timeframe. Amazon returns 543% in the same timeframe. And let's use a hot space that's 100% subscription based, like Crowdstrike: 265% in less than a year. Software and subscription margins crush the legacy perpetual hardware model. Storage is an old, stable, and commoditized market. There's no huge growth (any storage earnings report show that very clearly). And while the demand for storage continues to increase, the NetApps of the world aren't capturing where that growth is.

by yobert 6y ago

Reading through the comments though it looks like I should try making a BTRFS mount for the container storage and that might help the fuse-overlayfs problems though.

tw04 6y ago

I know it tends to be a controversial technology around here, but I've had great luck with OpenZFS for container storage.

znpy 6y ago

It feels weird that a containerisation technology imposes requirements on the underlying storage technology to work well, tho. Now I understand that btrfs is not really a requirement. But I've been thinking about using podman to replace docker-in-docker on our fleet of gitlab runners where we build our images, and running btrfs is a deal breaker. We really don't want to add complexity to the mix. Fuse-overlay burning through the whole CPU is not really good looking either. Mh. I still have to properly dig deep into this, but I'm keeping my hopes high.

carlosf 6y ago
Have you tried Kaniko? That's how I got hid of DinD in my Gitlab Runners.
- horstmeyer 6y ago
  Could you maybe provide a small example of the relevant part of the gitlab CI definition? I assume you run kaniko inside a Docker container on the runner?
tw04 6y ago
>It feels weird that a containerisation technology imposes requirements on the underlying storage technology to work well, tho. Why? Storage is literally the backbone of technology. From databases to webservers there are unique and real requirements. There's a reason why the enterprise storage market is still worth several billion dollars, and it's not because storage is easy. In 2021, with billions of dollars of R&D at their disposal - neither Amazon, Google, or Microsoft have even a mediocre NAS stack in comparison to the likes of Dell/EMC (Isilon) or NetApp (ONTAP). Heck they can't even compete with startups like Qumulo or Vast.
- windexh8er 6y ago
  Are you sure Amazon, Google and Microsoft don't have this competitive stack because they don't want to enter that market or are in that market solely for themselves? Storage may be a multi-billion dollar market - but it's also razor thin margin compared to where those brands focus their R&D spend. I'd argue those you've named don't have interest selling storage to the enterprise in the form of legacy boxes. They make far more revenue leasing storage to the enterprise. Cloud storage is also recurring revenue where enterprise storage is considered perpetual. The latter is generally frowned upon from the Street's perspective these days (good, bad or otherwise).
  - tw04 6y ago
    >Are you sure Amazon, Google and Microsoft don't have this competitive stack because they don't want to enter that market or are in that market solely for themselves They have "competition" in the market, it's just horrible. Because storage is hard. >Storage may be a multi-billion dollar market - but it's also razor thin margin compared to where those brands focus their R&D spend. I'm going to let you go back and review the earnings reports from the aforementioned companies. Razor thin margins? You wouldn't survive in the market with razor thin margins. > I'd argue those you've named don't have interest selling storage to the enterprise in the form of legacy boxes. I'd argue Azure Stack, AWS Outpost, and Google Anthos readily prove you wrong. > Cloud storage is also recurring revenue where enterprise storage is considered perpetual. Just... no. Enterprise storage has a shelf life of 3-5 years before maintenance or technology make it obsolete. >The latter is generally frowned upon from the Street's perspective these days (good, bad or otherwise). https://www.google.com/finance/quote/NTAP:NASDAQ https://www.google.com/finance/quote/NTAP:NASDAQ 5 years ago they were at $20/share, today they're at $69/share. Reality doesn't appear to match your claim as to the street's opinion of enterprise storage.
    - windexh8er 6y ago
      First, if you review the financials of the storage companies compared to earnings of all cloud companies they're abysmal. Yes, razor thin margins - comparatively. The cost of buying physical storage is commoditized. I've worked in technology from the pre-sale engineering side for a number of years. Margins on hardware are fractional compared to software and subscription offerings. Stack, Outpost and Anthos are not revenue drivers today. They're vendor lock in tools. 3-5 years is a horrible argument in the position you're attempting to make considering it's an eternity in technology. Consider that you sell the storage once in that 3-5 years. Your cloud provider bills you monthly and you don't own it. It's recurring revenue vs a singular sales event. NTAP over 5 years is a decent return. 215% over that timeframe. Amazon returns 543% in the same timeframe. And let's use a hot space that's 100% subscription based, like Crowdstrike: 265% in less than a year. Software and subscription margins crush the legacy perpetual hardware model. Storage is an old, stable, and commoditized market. There's no huge growth (any storage earnings report show that very clearly). And while the demand for storage continues to increase, the NetApps of the world aren't capturing where that growth is.