4 ms·

I've done Kubernetes consulting with some other people and for their on prem solutions I always recommend just buying a TrueNAS with it's s3 api. The other peo

I've done Kubernetes consulting with some other people and for their on prem solutions I always recommend just buying a TrueNAS with it's s3 api. The other people I tend to end up with always try to sell people on minio+rook-ceph and then offer support along the way. So basically you buy their kubernetes deployment and then you have to pay them for the rest of your life for troubleshooting. The TrueNAS seems cheaper to me. I don't see a good backup story, but maybe I just don't know it.

ochoseis 5y ago
Truenas with S3 piqued my interest. Does that use minio under the hood to serve objects? This doc indicates it might at least use the browser component: https://www.truenas.com/docs/core/services/s3/ https://www.truenas.com/docs/core/services/s3/
- Havoc 5y ago
  Pretty sure it’s minio for everything. Mainly use it for gitlab object storage current. Logs docker images etc
hardwaresofton 5y ago
Have you ever seen people using rook-ceph and the built in object gateway Ceph provides? AFAIK Ceph's object store powers some official cloud solutions out there like Digital Ocean spaces
- rjzzleep 5y ago
  Yes, one of my clients uses it. It seems okay, but it's all pretty low volume and most people have no real concept of disaster recovery. I think backups and DR are things that most people take for granted and don't really think about.
  - hardwaresofton 5y ago
    Yep reasonable -- most people these days don't have to, since the clouds do the hard work for most people. The knowledge on how to run those kinds of systems becomes harder to find (in the community) by the year. Also by the time most people account for 2/3 copies of the data + 1 completely offsite backup of data I think their eyes might start to water at the cost if you want reasonable performance as well. I experiment with this kind of stuff a lot of the time and always am surprised at how much more than you'd expect it costs to get drive, node, and region level redundancy with backups. You need essentially at a minimum 3TB for every 1TB of usable storage assuming regular RAID1.
    - willis936 5y ago
      That's the cost of privacy until there a byopk cloud service.
bogomipz 5y ago
I'm curious is the use case for Kubernetes and MinIO mainly just backups or are there are good use-cases for Kubernetes and object storage you are seeing?
- ckdarby 5y ago
  Hedge bets against an s3 outage. Cloud agnostic solution with abstraction of object storage. Reducing the cost of s3 by using minio as a cache. Plenty of good use-cases.
  - fnord123 5y ago
    Devs can also spin up a local object store to test with instead of hitting some common remote bucket.
nwmcsween 5y ago
So ZFS on truenas/freebsd is in some weird twilight zone of great stability and horribly broken, we ended up having VM corruption after removing an l2arc cache drive on truenas. I do agree with the Ceph being much more complex which is why I hope OpenEBS implements zfs-localpv send recv for a poor man's replication.
- carterschonwald 5y ago
  Could you elaborate more? I’ve been evaluating using zfs for my new home work station and would like to understand what failure modes I need to be aware of
  - nwmcsween 5y ago
    ZFS is still the best FS out there but a good rule of thumb is to look over the open and closed issues and see how things are handled and the problems for a given project, a sort of issue-driven-assessment.
    - carterschonwald 5y ago
      Ah, like the hibernate/suspend and swap file issue for mobile computers?
moondev 5y ago
TrueNAS just packages minio, so it would seem to be the same result but with less flexibility and visibility to the cluster. Although if the strategy is to not hyperconverge storage that could be desired I guess
- rjzzleep 5y ago
  You are indeed correct, but minio is actually a lot of components. And in the case of TrueNAS they handle most of the storage related components with a minio object gateway running on top. A pure kubernetes deployment is more complex(although it's all part of the same binary i think). I could be completely wrong though.
  - moondev 5y ago
    In truenas scale (based on linux not freebsd) they actually run most integrated and 3rd party "apps" (including minio) inside a k3s cluster running on truenas itself
    - rjzzleep 5y ago
      Is TrueNAS Scale stable yet?
      - moondev 5y ago
        It seems to be under pretty active development but I do like their strategy of releasing early and often. I haven't run into any issues yet with it in my lab. My biggest wishes for Truenas Scale: 1. NVMe-oF support (over RDMA or TCP) 2. aarch64 uefi iso I could see Truenas Scale overtaking Proxmox soon in the KVM space, their API and UI are already more enjoyable to use IMO.
      - willis936 5y ago
        They're in RC now. In terms of data loss: I'm not concerned with what I've seen. Some features are still missing, such as SED control. I think most of things I wish were there aren't present on core either, such as VM VNC clipboard and solutions to the web UI logging out often when 2FA is enabled and some mysterious directory locking issue when pulling dropbox.
ignoramous 5y ago
> I don't see a good backup story, but maybe I just don't know it. Where do data-management SaaS companies like Rubrik.com and Druva.com fit in? Are they not popular enough solution to secure min.io deployments?

ochoseis 5y ago

Truenas with S3 piqued my interest. Does that use minio under the hood to serve objects? This doc indicates it might at least use the browser component: https://www.truenas.com/docs/core/services/s3/ https://www.truenas.com/docs/core/services/s3/

Havoc 5y ago
Pretty sure it’s minio for everything. Mainly use it for gitlab object storage current. Logs docker images etc

hardwaresofton 5y ago

Have you ever seen people using rook-ceph and the built in object gateway Ceph provides? AFAIK Ceph's object store powers some official cloud solutions out there like Digital Ocean spaces

rjzzleep 5y ago
Yes, one of my clients uses it. It seems okay, but it's all pretty low volume and most people have no real concept of disaster recovery. I think backups and DR are things that most people take for granted and don't really think about.
- hardwaresofton 5y ago
  Yep reasonable -- most people these days don't have to, since the clouds do the hard work for most people. The knowledge on how to run those kinds of systems becomes harder to find (in the community) by the year. Also by the time most people account for 2/3 copies of the data + 1 completely offsite backup of data I think their eyes might start to water at the cost if you want reasonable performance as well. I experiment with this kind of stuff a lot of the time and always am surprised at how much more than you'd expect it costs to get drive, node, and region level redundancy with backups. You need essentially at a minimum 3TB for every 1TB of usable storage assuming regular RAID1.
  - willis936 5y ago
    That's the cost of privacy until there a byopk cloud service.

bogomipz 5y ago

I'm curious is the use case for Kubernetes and MinIO mainly just backups or are there are good use-cases for Kubernetes and object storage you are seeing?

ckdarby 5y ago
Hedge bets against an s3 outage. Cloud agnostic solution with abstraction of object storage. Reducing the cost of s3 by using minio as a cache. Plenty of good use-cases.
- fnord123 5y ago
  Devs can also spin up a local object store to test with instead of hitting some common remote bucket.

nwmcsween 5y ago

So ZFS on truenas/freebsd is in some weird twilight zone of great stability and horribly broken, we ended up having VM corruption after removing an l2arc cache drive on truenas. I do agree with the Ceph being much more complex which is why I hope OpenEBS implements zfs-localpv send recv for a poor man's replication.

carterschonwald 5y ago
Could you elaborate more? I’ve been evaluating using zfs for my new home work station and would like to understand what failure modes I need to be aware of
- nwmcsween 5y ago
  ZFS is still the best FS out there but a good rule of thumb is to look over the open and closed issues and see how things are handled and the problems for a given project, a sort of issue-driven-assessment.
  - carterschonwald 5y ago
    Ah, like the hibernate/suspend and swap file issue for mobile computers?

moondev 5y ago

TrueNAS just packages minio, so it would seem to be the same result but with less flexibility and visibility to the cluster. Although if the strategy is to not hyperconverge storage that could be desired I guess

rjzzleep 5y ago
You are indeed correct, but minio is actually a lot of components. And in the case of TrueNAS they handle most of the storage related components with a minio object gateway running on top. A pure kubernetes deployment is more complex(although it's all part of the same binary i think). I could be completely wrong though.
- moondev 5y ago
  In truenas scale (based on linux not freebsd) they actually run most integrated and 3rd party "apps" (including minio) inside a k3s cluster running on truenas itself
  - rjzzleep 5y ago
    Is TrueNAS Scale stable yet?
    - moondev 5y ago
      It seems to be under pretty active development but I do like their strategy of releasing early and often. I haven't run into any issues yet with it in my lab. My biggest wishes for Truenas Scale: 1. NVMe-oF support (over RDMA or TCP) 2. aarch64 uefi iso I could see Truenas Scale overtaking Proxmox soon in the KVM space, their API and UI are already more enjoyable to use IMO.
    - willis936 5y ago
      They're in RC now. In terms of data loss: I'm not concerned with what I've seen. Some features are still missing, such as SED control. I think most of things I wish were there aren't present on core either, such as VM VNC clipboard and solutions to the web UI logging out often when 2FA is enabled and some mysterious directory locking issue when pulling dropbox.

ignoramous 5y ago

> I don't see a good backup story, but maybe I just don't know it. Where do data-management SaaS companies like Rubrik.com and Druva.com fit in? Are they not popular enough solution to secure min.io deployments?