5 ms·

I love CoreOS and they've done some super impressive engineering. But really, a new storage system? Rewriting in Go and using etcd for central state management

by sysexit 10y ago

I love CoreOS and they've done some super impressive engineering. But really, a new storage system? Rewriting in Go and using etcd for central state management makes things easier, but this is still a hard problem. Some things that need to be solved sooner or later: data replication so that N faults of X entities are protected against (X can be disks, enclosures, racks, data centers, regions, ..), recovery from failed disks, scrubbing, data management, backups, some kind of storage orchestration and centralized management. If you look at Ceph which IMHO represents the state of the art in software defined storage, it took many years to get it to a point that it was usable. I hate to be cynical but in this case I would be surprised if CoreOS can pull this of. I wish them all the best though and would be happy for them if I am wrong.

agentultra 10y ago
> this is still a hard problem Which is why I'm rather surprised that these systems continue to be released without any formal specification or model checking. It's hard to get these things right... worth the effort, IMO, to create formal specifications.
ams6110 10y ago
It's like any time we take a step forward in one area we have to reinvent the last 50 years of computing to support it. Persistent storage is a "hard problem" Really?
- vidarh 10y ago
  Persistent storage isn't a hard problem. Distributed, well performing, scalable and consistent storage is a hard problem.
  - dap 10y ago
    Actually, persistent storage is fairly hard in itself. Look at what ZFS does to ensure data integrity in the face of phantom writes, dropped writes, bad controllers, and other implicit, non-fatal failures.
    - vidarh 10y ago
      ZFS still also have the issue of having to perform well. You have a point, but ZFS is still trivial compared to a proper distributed filesystem, and you could achieve the same reliability much easier than ZFS if you sacrificed the performance.
      - dap 10y ago
        Absolutely -- I didn't mean to imply that ZFS even comes close to solving the distributed parts of the problem, but rather that a distributed storage system does have to address the problems of putting bits on disk.
      - thinkersilver 10y ago
        The ClusterHQ guys behind the FlockerHQ found this out the hard way [0]. Initially Flocker was meant to provide a container data migration tool on top of ZFS, now it is a front-end to more established storage systems like Cinder,vSan,EBS,GPD and so on. [0] https://docs.clusterhq.com/en/latest/faq/#what-happened-to-zfs-support https://docs.clusterhq.com/en/latest/faq/#what-happened-to-z...
    - zzzcpan 10y ago
      > Actually, persistent storage is fairly hard in itself Don't we have distributed data storages precisely because it's impossible to guaranty persistence locally? It's kind of a way to not bother trying to solve the impossible, but to achieve some guarantees on a different level.
      - wnoise 10y ago
        That's one use case, but far more common is reducing latency and increasing bandwidth for distributed computing acting on a shared set of data.
- nickpsecurity 10y ago
  Persistent storage on today's filesystems and complex hardware is a hard problem. All kinds of failures can happen during any write. Some are obvious with some silently corrupting data. There's been decades of work on approaches to dealing with this with a variety of tradeoffs. Picking the right one for a widely-deployed, portable, distributed app is tricky by itself. They're aiming to do a lot more than that. ;)
schmichael 10y ago
> But really, a new storage system? I think the post outlines quite well why they chose to build a new distributed storage layer. From a product perspective it makes a lot of sense for them to have an offering in this space. Also, the fact that it's a hard problem is what makes it a compelling product offering. You can't make much money solving easy problems.
- sysexit 10y ago
  > Also, the fact that it's a hard problem is what makes it a compelling product offering. You can't make much money solving easy problems. If they can solve the hard problem, yes. Which I believe is unlikely.
tveita 10y ago
I've been trying to find performance evaluations of distributed file systems, and in most tests I've seen Ceph is a lot slower than alternatives like GlusterFS. When you say Ceph is "state of the art in software defined storage", are you including performance, or are there advantages in features or reliability where you think it outclasses the competition?

by sysexit 10y ago

agentultra 10y ago

> this is still a hard problem Which is why I'm rather surprised that these systems continue to be released without any formal specification or model checking. It's hard to get these things right... worth the effort, IMO, to create formal specifications.

ams6110 10y ago

It's like any time we take a step forward in one area we have to reinvent the last 50 years of computing to support it. Persistent storage is a "hard problem" Really?

vidarh 10y ago
Persistent storage isn't a hard problem. Distributed, well performing, scalable and consistent storage is a hard problem.
- dap 10y ago
  Actually, persistent storage is fairly hard in itself. Look at what ZFS does to ensure data integrity in the face of phantom writes, dropped writes, bad controllers, and other implicit, non-fatal failures.
  - vidarh 10y ago
    ZFS still also have the issue of having to perform well. You have a point, but ZFS is still trivial compared to a proper distributed filesystem, and you could achieve the same reliability much easier than ZFS if you sacrificed the performance.
    - dap 10y ago
      Absolutely -- I didn't mean to imply that ZFS even comes close to solving the distributed parts of the problem, but rather that a distributed storage system does have to address the problems of putting bits on disk.
    - thinkersilver 10y ago
      The ClusterHQ guys behind the FlockerHQ found this out the hard way [0]. Initially Flocker was meant to provide a container data migration tool on top of ZFS, now it is a front-end to more established storage systems like Cinder,vSan,EBS,GPD and so on. [0] https://docs.clusterhq.com/en/latest/faq/#what-happened-to-zfs-support https://docs.clusterhq.com/en/latest/faq/#what-happened-to-z...
  - zzzcpan 10y ago
    > Actually, persistent storage is fairly hard in itself Don't we have distributed data storages precisely because it's impossible to guaranty persistence locally? It's kind of a way to not bother trying to solve the impossible, but to achieve some guarantees on a different level.
    - wnoise 10y ago
      That's one use case, but far more common is reducing latency and increasing bandwidth for distributed computing acting on a shared set of data.
nickpsecurity 10y ago
Persistent storage on today's filesystems and complex hardware is a hard problem. All kinds of failures can happen during any write. Some are obvious with some silently corrupting data. There's been decades of work on approaches to dealing with this with a variety of tradeoffs. Picking the right one for a widely-deployed, portable, distributed app is tricky by itself. They're aiming to do a lot more than that. ;)

schmichael 10y ago

> But really, a new storage system? I think the post outlines quite well why they chose to build a new distributed storage layer. From a product perspective it makes a lot of sense for them to have an offering in this space. Also, the fact that it's a hard problem is what makes it a compelling product offering. You can't make much money solving easy problems.

sysexit 10y ago
> Also, the fact that it's a hard problem is what makes it a compelling product offering. You can't make much money solving easy problems. If they can solve the hard problem, yes. Which I believe is unlikely.

tveita 10y ago

I've been trying to find performance evaluations of distributed file systems, and in most tests I've seen Ceph is a lot slower than alternatives like GlusterFS. When you say Ceph is "state of the art in software defined storage", are you including performance, or are there advantages in features or reliability where you think it outclasses the competition?