8 ms·

It's hard to take this seriously: storage is an excruciatingly hard problem, yet this cheerful description of a nascent and aspirational effort seems blissfull

It's hard to take this seriously: storage is an excruciatingly hard problem, yet this cheerful description of a nascent and aspirational effort seems blissfully unaware of how difficult it is to even just reliably get bits to and from stable storage, let alone string that into a distributed system that must make CAP tradeoffs. There is not so much of a whisper as to what the data path actually looks like other than "the design includes the ability to support [...] Reed-Solomon error correction in the near future" -- and the fact that such an empty system hails itself as pioneering an unsolved problem in storage is galling in its ignorance of prior work (much of it open source). Take it from someone who has been involved in both highly durable local filesystems[1] and highly available object storage systems[2][3]: this is such a hard, nasty problem with so many dark, hidden and dire failure modes, that it takes years of running in production to get these systems to the level of reliability and operability that the data path demands. Given that (according to the repo, though not the breathless blog entry) its creators "do not recommend its use in production", Torus is -- in the famous words of Wolfgang Pauli -- not even wrong. [1] http://dtrace.org/blogs/bmc/2008/11/10/fishworks-now-it-can-be-told/ http://dtrace.org/blogs/bmc/2008/11/10/fishworks-now-it-can-... [2] http://dtrace.org/blogs/bmc/2013/06/25/manta-from-revelation-to-product/ http://dtrace.org/blogs/bmc/2013/06/25/manta-from-revelation... [3] http://dtrace.org/blogs/dap/2013/07/03/fault-tolerance-in-manta/ http://dtrace.org/blogs/dap/2013/07/03/fault-tolerance-in-ma...

nickpsecurity 10y ago
I totally agree with you. I also liked how they said the motivation is to make Google infrastructure for everyone else. How did Google do this? They basically imitated and improved on clustered filesystems developed in HPC. There were a lot of lessons to learn for emerging cloud market in all tooling done in HPC. Some was FOSS. Whereas, many companies seem to be doing the opposite in their work on these cloud filesystems. They don't build on proven, already-OSS components that have been battle-tested for a long time. They lack features and wisdom from prior deployments. They duplicate effort. They are also prone to using popular components, languages, whatever with implicitly-higher risk due to fact they were never built for fault-tolerant systems. Actually, many of them often assume something will watch and help them out in time of failure. If it's high-assurance or fault-tolerance, my manta is "tried and true beats novel and new." Just repurpose what's known to work while improving its capabilities and code quality. Knocks out risks you know about plus others you don't since they were never documents. Has that been your experience, too? Should be a maxim in IT given how often problem plays out.
- pinewurst 10y ago
  While I strongly agree with bcantrill, I have to disagree with your HPC origins model in the most strenuous fashion. The HPC world can be thought of as an alternative universe, one where dinosaurs came to some sort of lazy sentience instead us mammals. HPC RAS (reliability, availability, serviceability) requirements and demands are profoundly different and profoundly lower from what enterprise or even SMB (small & medium sized businesses) would find acceptable. There are or were interesting ideas in GPFS but that was long ago, far away, and sterile besides - not that it stops IBM squeezing revenue from locked-in customers. Some people in HPC do realize their insular state, but mostly there are hopes for things to be done better that were done better earlier (e.g. "declustered RAID") in the "real" world. Their ongoing hype is for yet-another-meta-filesystem built with feet of straw upon a pile of manure. The likes of Google infrastructure, are built instead on academic storage R&D, which has been a great source of ideas and interesting test implementations - log structured file systems, distributed hash tables, etc. There is some small overlap, for example in the work of Garth Gibson, but it's not a joint evolution at all.
  - nickpsecurity 10y ago
    HPC as an alternative universe is a fine metaphor. Yet, I don't see how there's no comparison given HPC already had more RAS than I could get out of any Linux box for years. IBM SP2, SGI, and some Beowulf-style clusters had high-performance I/O, single-system image, distributed filesystems, and fault-tolerance where you wanted. That sounds a lot like what cloud market aimed for. On top of it, you had grid and metacomputing groups that spread it out across multiple sites with ease of use for developers. They developed in parallel and separately but there's lots of similarities. Many startups in HPC were taking HPC stuff to meet cloud-style requirements for years in terms of management, reliability, and performance. That it wasn't up to the standards of businesses is countered by how many bought those NUMA machines and clusters for mission-critical apps. A huge chunk of work in Windows and Linux in 90's and early 2000's went to trying to match their metrics in order to replace them on the cheap with x86 servers and Ethernet. So, yes, HPC was making stuff to meet tough, business needs that commercial servers and regular stacks couldn't compare to. That's despite number-crunching supercomputers being their main market.
zzzcpan 10y ago
> unaware of how difficult it is to even just reliably get bits to and from stable storage It's not difficult, it's actually impossible. And any system relying on the reliability of that would be broken by design. > let alone string that into a distributed system that must make CAP tradeoffs No. Distributed systems and "CAP tradeoffs" actually exist to solve the problem above with predictable certainty.
- halayli 10y ago
  It seems you're disagreeing with yourself.
  - zzzcpan 10y ago
    Care to point out where? I can explain if something seems too ambiguous. It is impossible to reliably store something in your local storage. It is possible to achieve some probability of data retention and availability in a distributed system.
halayli 10y ago
Looking at dtrace, fishwork, zfs and Solaris (imo was the best OS technologically speaking) it's interesting to see how strong and innovative its engineering team was while the business was just going down. How did that happen? How can the engineering team be that productive and functional while the business vision was so lacking?
- bcantrill 10y ago
  Even when commercially misguided, Sun always had terrific engineering talent -- and my farewell to the company captures some of that.[1] In terms of why did the company fail, the short answer is probably that SPARC was disrupted by x86, and by the time the company figured that out, it was too late to recover.[2] [1] http://dtrace.org/blogs/bmc/2010/07/25/good-bye-sun/ http://dtrace.org/blogs/bmc/2010/07/25/good-bye-sun/ [2] Longer answer: https://news.ycombinator.com/item?id=2287033 https://news.ycombinator.com/item?id=2287033
  - halayli 10y ago
    I've always been very impressed by Sun's talent, the team was . And thank you for dtrace! :) If only Linux community / leaders would be less arrogant and adopt technologies from Solaris/BSD that are order of magnitude better (kqueue, netgraph and more) instead of coming up with new ways to screw up. Sun should be resurrected now given that risc is leading the way and build everything on top of ARM! :). if only.
    - jacques_chester 10y ago
      Linux can't be blamed for the decision to place Sun's technologies under the CDDL.
      - halayli 10y ago
        there's no license for kqueue's interface or netgraph, and many other great interfaces. but they decide to create square wheels and not leverage from errors that others have made before them.
notacoward 10y ago
Agree 100% on this, Bryan. Their claims about how existing storage solutions are a poor fit for this use case are completely false. As too often happens, they address only high-margin commercial systems and ignore the fact that open-source solutions are out there as well. (Disclaimer: I'm a Gluster developer). Then they leave both files and objects as "exercises for the reader" which shows a total lack of understanding for the problem or respect for people already working on it. Their announcement is clearly more about staking a claim to the "storage for containers" space, before the serious players get there, than it is about realistic expectations for how the project might grow. Particularly galling are their comments about the difficulty of building a community for something like this, when such communities already exist and their announcement actually harms those communities. Until now, I've thought quite highly of the CoreOS team. Now, not so much. They're playing a "freeze the competition" game instead of trying to compete on merit.
- mevile 10y ago
  How is them developing their own technology not competing on merit? Did they steal the technology? Did they claim that anyone ought to use this in production? Why all the negativity? Yeah never mind let's just discourage everyone from trying to build new technology. Nobody is forcing you to use this. If it's not for you, move on. Going out on a rant about what you think their intentions are is ridiculous. Really you come off as defensive. I don't understand how CoreOS has disrespected anyone by coming up with their own approach to the problem. They're not writing blog posts or comments on HN, they're writing code.
  - notacoward 10y ago
    > They're not writing blog posts or comments on HN, they're writing code. Actually, the problem is that 90% of the code still remains to be written, while others (including me) have already done so. They've addressed only the very simplest part of the problem, not even far enough to show any performance comparisons, in a manner strongly reminiscent of Sheepdog (belying your "own approach" claim). That's a poor basis from which to promise so much. It's like writing an interpreter for a simple programming language and claiming it'll be a full optimizing C++ compiler soon. Just a few little pieces remaining, right? It's perfectly fine for them to start their own project and have high hopes for it. The more the merrier. However, I have little patience for people who blur the lines between what's there and what might hypothetically exist some time in the future. That's far too often used to stifle real innovation that's occurring elsewhere. Maybe it's more common in storage than whatever your specialty is, but it's a well known part of the playbook. It's important to be crystal clear about what's real vs. what's seriously thought out vs. what's total blue-sky. Users and fellow developers deserve nothing less.
merb 10y ago
Actually the biggest problem is actual either Full Posix or Mutable Storage. If there is just one way in and never a way out, it's actually way easier. So practically if you don't run your database on shared storage etc and manage it "the old way". you could have something reliable by a 'never delete'. However CAS and immutable file systems aren't that common.
cyphar 10y ago
Just as a separate question: why are you so bitter about btrfs? ;)
stevelandiss 10y ago
If I could also add that NBD is extremely notorious in Linux. I am a block storage developer and NBD has been one of the main reasons why so many openstack storage products (like formation data and others) have really struggled. There are many known Linux kernel issues with NBD, for example, if an NBD provider (user space daemon) exits for any reason, the kernel panics. Here is an example of a long outstanding bug that has plagued the NBD community and openstack community for years (https://www.mail-archive.com/nbd-general@lists.sourceforge.net/msg02388.html https://www.mail-archive.com/nbd-general@lists.sourceforge.n...). It won't get addressed any time soon. I also looked at the repo and it looks like a very simplistic approach to a hard problem. BTW I have used etcd and if this is anything like etcd, I'd really be worried. etcd snapshots bring an entire cluster down for a while.
- stevelandis 10y ago
  Here is an example script that will panic anyone using NBD to server a block device (replace qemu-nbd with the nbd export provided by torus) qemu-img create -f qcow2 f.img 1G mkfs.ext4 f.img modprobe nbd || true qemu-nbd -c /dev/nbd0 f.img mount /dev/nbd0 k killall -KILL qemu-nbd sleep 1 ls k
  - electrum 10y ago
    Above script with correct formatting: qemu-img create -f qcow2 f.img 1G mkfs.ext4 f.img modprobe nbd || true qemu-nbd -c /dev/nbd0 f.img mount /dev/nbd0 k killall -KILL qemu-nbd sleep 1 ls k
- fulafel 10y ago
  But: that's not a kernel panic, just a BUG() stack trace from kernel which doesn't halt the system.
regularfry 10y ago
Their one saving grace might be that NBD is such a simple protocol. By not trying to make it a full-fledged FS, they might actually have hit on a tractable problem.
curtisptrsn 10y ago
Just how many things is CoreOS trying to do? Last I counted, they want to a) Build a distributed OS b) Build a distributed scheduler (Fleet) c) Build a distributed key value system (etcd) d) Build a new container engine (Rocket) e) Build a network fabric (Flannel) f) Now embark on building a brand new distributed storage system. Holy cow that's some goal list. Sounds like something my kids would make up for their Christmas wish list. I really don't get how their board and investors let them get away with such a childish imagination. Each one of those is a company effort on its own.
- jsmthrowaway 10y ago
  Fleet died once Kubernetes became a thing, even though they serve entirely different use cases.

nickpsecurity 10y ago

I totally agree with you. I also liked how they said the motivation is to make Google infrastructure for everyone else. How did Google do this? They basically imitated and improved on clustered filesystems developed in HPC. There were a lot of lessons to learn for emerging cloud market in all tooling done in HPC. Some was FOSS. Whereas, many companies seem to be doing the opposite in their work on these cloud filesystems. They don't build on proven, already-OSS components that have been battle-tested for a long time. They lack features and wisdom from prior deployments. They duplicate effort. They are also prone to using popular components, languages, whatever with implicitly-higher risk due to fact they were never built for fault-tolerant systems. Actually, many of them often assume something will watch and help them out in time of failure. If it's high-assurance or fault-tolerance, my manta is "tried and true beats novel and new." Just repurpose what's known to work while improving its capabilities and code quality. Knocks out risks you know about plus others you don't since they were never documents. Has that been your experience, too? Should be a maxim in IT given how often problem plays out.

pinewurst 10y ago
While I strongly agree with bcantrill, I have to disagree with your HPC origins model in the most strenuous fashion. The HPC world can be thought of as an alternative universe, one where dinosaurs came to some sort of lazy sentience instead us mammals. HPC RAS (reliability, availability, serviceability) requirements and demands are profoundly different and profoundly lower from what enterprise or even SMB (small & medium sized businesses) would find acceptable. There are or were interesting ideas in GPFS but that was long ago, far away, and sterile besides - not that it stops IBM squeezing revenue from locked-in customers. Some people in HPC do realize their insular state, but mostly there are hopes for things to be done better that were done better earlier (e.g. "declustered RAID") in the "real" world. Their ongoing hype is for yet-another-meta-filesystem built with feet of straw upon a pile of manure. The likes of Google infrastructure, are built instead on academic storage R&D, which has been a great source of ideas and interesting test implementations - log structured file systems, distributed hash tables, etc. There is some small overlap, for example in the work of Garth Gibson, but it's not a joint evolution at all.
- nickpsecurity 10y ago
  HPC as an alternative universe is a fine metaphor. Yet, I don't see how there's no comparison given HPC already had more RAS than I could get out of any Linux box for years. IBM SP2, SGI, and some Beowulf-style clusters had high-performance I/O, single-system image, distributed filesystems, and fault-tolerance where you wanted. That sounds a lot like what cloud market aimed for. On top of it, you had grid and metacomputing groups that spread it out across multiple sites with ease of use for developers. They developed in parallel and separately but there's lots of similarities. Many startups in HPC were taking HPC stuff to meet cloud-style requirements for years in terms of management, reliability, and performance. That it wasn't up to the standards of businesses is countered by how many bought those NUMA machines and clusters for mission-critical apps. A huge chunk of work in Windows and Linux in 90's and early 2000's went to trying to match their metrics in order to replace them on the cheap with x86 servers and Ethernet. So, yes, HPC was making stuff to meet tough, business needs that commercial servers and regular stacks couldn't compare to. That's despite number-crunching supercomputers being their main market.

zzzcpan 10y ago

> unaware of how difficult it is to even just reliably get bits to and from stable storage It's not difficult, it's actually impossible. And any system relying on the reliability of that would be broken by design. > let alone string that into a distributed system that must make CAP tradeoffs No. Distributed systems and "CAP tradeoffs" actually exist to solve the problem above with predictable certainty.

halayli 10y ago
It seems you're disagreeing with yourself.
- zzzcpan 10y ago
  Care to point out where? I can explain if something seems too ambiguous. It is impossible to reliably store something in your local storage. It is possible to achieve some probability of data retention and availability in a distributed system.

halayli 10y ago

Looking at dtrace, fishwork, zfs and Solaris (imo was the best OS technologically speaking) it's interesting to see how strong and innovative its engineering team was while the business was just going down. How did that happen? How can the engineering team be that productive and functional while the business vision was so lacking?

bcantrill 10y ago
Even when commercially misguided, Sun always had terrific engineering talent -- and my farewell to the company captures some of that.[1] In terms of why did the company fail, the short answer is probably that SPARC was disrupted by x86, and by the time the company figured that out, it was too late to recover.[2] [1] http://dtrace.org/blogs/bmc/2010/07/25/good-bye-sun/ http://dtrace.org/blogs/bmc/2010/07/25/good-bye-sun/ [2] Longer answer: https://news.ycombinator.com/item?id=2287033 https://news.ycombinator.com/item?id=2287033
- halayli 10y ago
  I've always been very impressed by Sun's talent, the team was . And thank you for dtrace! :) If only Linux community / leaders would be less arrogant and adopt technologies from Solaris/BSD that are order of magnitude better (kqueue, netgraph and more) instead of coming up with new ways to screw up. Sun should be resurrected now given that risc is leading the way and build everything on top of ARM! :). if only.
  - jacques_chester 10y ago
    Linux can't be blamed for the decision to place Sun's technologies under the CDDL.
    - halayli 10y ago
      there's no license for kqueue's interface or netgraph, and many other great interfaces. but they decide to create square wheels and not leverage from errors that others have made before them.

notacoward 10y ago

Agree 100% on this, Bryan. Their claims about how existing storage solutions are a poor fit for this use case are completely false. As too often happens, they address only high-margin commercial systems and ignore the fact that open-source solutions are out there as well. (Disclaimer: I'm a Gluster developer). Then they leave both files and objects as "exercises for the reader" which shows a total lack of understanding for the problem or respect for people already working on it. Their announcement is clearly more about staking a claim to the "storage for containers" space, before the serious players get there, than it is about realistic expectations for how the project might grow. Particularly galling are their comments about the difficulty of building a community for something like this, when such communities already exist and their announcement actually harms those communities. Until now, I've thought quite highly of the CoreOS team. Now, not so much. They're playing a "freeze the competition" game instead of trying to compete on merit.

mevile 10y ago
How is them developing their own technology not competing on merit? Did they steal the technology? Did they claim that anyone ought to use this in production? Why all the negativity? Yeah never mind let's just discourage everyone from trying to build new technology. Nobody is forcing you to use this. If it's not for you, move on. Going out on a rant about what you think their intentions are is ridiculous. Really you come off as defensive. I don't understand how CoreOS has disrespected anyone by coming up with their own approach to the problem. They're not writing blog posts or comments on HN, they're writing code.
- notacoward 10y ago
  > They're not writing blog posts or comments on HN, they're writing code. Actually, the problem is that 90% of the code still remains to be written, while others (including me) have already done so. They've addressed only the very simplest part of the problem, not even far enough to show any performance comparisons, in a manner strongly reminiscent of Sheepdog (belying your "own approach" claim). That's a poor basis from which to promise so much. It's like writing an interpreter for a simple programming language and claiming it'll be a full optimizing C++ compiler soon. Just a few little pieces remaining, right? It's perfectly fine for them to start their own project and have high hopes for it. The more the merrier. However, I have little patience for people who blur the lines between what's there and what might hypothetically exist some time in the future. That's far too often used to stifle real innovation that's occurring elsewhere. Maybe it's more common in storage than whatever your specialty is, but it's a well known part of the playbook. It's important to be crystal clear about what's real vs. what's seriously thought out vs. what's total blue-sky. Users and fellow developers deserve nothing less.

merb 10y ago

Actually the biggest problem is actual either Full Posix or Mutable Storage. If there is just one way in and never a way out, it's actually way easier. So practically if you don't run your database on shared storage etc and manage it "the old way". you could have something reliable by a 'never delete'. However CAS and immutable file systems aren't that common.

cyphar 10y ago

Just as a separate question: why are you so bitter about btrfs? ;)

stevelandiss 10y ago

If I could also add that NBD is extremely notorious in Linux. I am a block storage developer and NBD has been one of the main reasons why so many openstack storage products (like formation data and others) have really struggled. There are many known Linux kernel issues with NBD, for example, if an NBD provider (user space daemon) exits for any reason, the kernel panics. Here is an example of a long outstanding bug that has plagued the NBD community and openstack community for years (https://www.mail-archive.com/nbd-general@lists.sourceforge.net/msg02388.html https://www.mail-archive.com/nbd-general@lists.sourceforge.n...). It won't get addressed any time soon. I also looked at the repo and it looks like a very simplistic approach to a hard problem. BTW I have used etcd and if this is anything like etcd, I'd really be worried. etcd snapshots bring an entire cluster down for a while.

stevelandis 10y ago
Here is an example script that will panic anyone using NBD to server a block device (replace qemu-nbd with the nbd export provided by torus) qemu-img create -f qcow2 f.img 1G mkfs.ext4 f.img modprobe nbd || true qemu-nbd -c /dev/nbd0 f.img mount /dev/nbd0 k killall -KILL qemu-nbd sleep 1 ls k
- electrum 10y ago
  Above script with correct formatting: qemu-img create -f qcow2 f.img 1G mkfs.ext4 f.img modprobe nbd || true qemu-nbd -c /dev/nbd0 f.img mount /dev/nbd0 k killall -KILL qemu-nbd sleep 1 ls k
fulafel 10y ago
But: that's not a kernel panic, just a BUG() stack trace from kernel which doesn't halt the system.

regularfry 10y ago

Their one saving grace might be that NBD is such a simple protocol. By not trying to make it a full-fledged FS, they might actually have hit on a tractable problem.

curtisptrsn 10y ago

Just how many things is CoreOS trying to do? Last I counted, they want to a) Build a distributed OS b) Build a distributed scheduler (Fleet) c) Build a distributed key value system (etcd) d) Build a new container engine (Rocket) e) Build a network fabric (Flannel) f) Now embark on building a brand new distributed storage system. Holy cow that's some goal list. Sounds like something my kids would make up for their Christmas wish list. I really don't get how their board and investors let them get away with such a childish imagination. Each one of those is a company effort on its own.

jsmthrowaway 10y ago
Fleet died once Kubernetes became a thing, even though they serve entirely different use cases.