6 ms·

Ignored the internet in terms of distributed systems, which I assumed was obvious from the wording. Look at that 1994 paper, none of the things in the paper sh

Ignored the internet in terms of distributed systems, which I assumed was obvious from the wording. Look at that 1994 paper, none of the things in the paper show any background to build a distributed key-value storage or really anything for the cloud. I'm not sure why people are trying to claim otherwise. Even ZFS from a decade later reflects both a complete lack of experience in distributed systems and ignorance towards the internet.

notacoward 7y ago
I believe ZFS represented a missed opportunity by spending all that effort on a local filesystem instead of trying to do something interesting in the distributed space. I even said so at the time, since I was working on distributed filesystems at the time, but that's just a strategic issue (and one that's quite arguable). It's not evidence of ignorance or lack of experience.
- zzzcpan 7y ago
  Well, on that part I was referring more specifically to the vdev mess and raidz, how you cannot just add a disk to the file system. Which is pretty silly, considering that at the time there were plenty of ideas how to do it much better, although not for disks, but for nodes in distributed systems. But apparently no one on ZFS team had any idea about any of it.
  - notacoward 7y ago
    That was a pretty common pattern at Sun. Engineers there did a lot of great work, no denying it. Like many other projects, ZFS solved some pretty difficult and important problems while ignoring others. Which ones get talked about ad nauseam? We all make mistakes, of course, but some of those ignored problems were well known at the time to be worth solving. Did those self-professed "complete engineers" learn anything from those times they missed the target? The fact that Sun doesn't exist any more suggests an answer.
    - Annatar 7y ago
      The problem is they cannot all be solved with just one single thing. Jeff Bonwick and his team knew that. They knew what was left on the table and they had to make that call. A completely fault tolerant, distributed system which can correctly arbitrate a write to a single shared resource (usually a file or even a block) from multiple nodes at the exact same time is a problem which has not been solved correctly yet, to the best of my knowledge and belief. I myself have spent considerable amount of time (20 years, to be precise) trying to solve that particular problem correctly for all edge cases. That problem is terra incognita in computer science and yet to be throughly researched, let alone understood. All attempts at solving that particular problem 100% correctly have not been successful, by anyone, but if you know of someone, please do tell, I'd love to be corrected on this and start using such technology immediately. The ZFS team was well aware of this and that's why they made the trade-offs they did. They picked a subset of problems and they solved them. Even in hindsight, to me at least, it's obvious why. But to suggest that Sun Microsystems doesn't exist because the company's engineers picked only a subset of problems they knew they could solve is... either terribly ignorant or wilfully malicious, or both, since malice often stems from ignorance. Sun Microsystems lost because they were too expensive, more expensive than garbage intel personal computer servers and because Solaris wasn't open source code in 1993. It's that simple.
  - Annatar 7y ago
    ZFS doesn't work that way: it works on pools of storage. Disks are added to the pool. Filesystems (or datasets) are carved out of a pool, not out of physical disks directly. There are vdevs on top of physical disks. That's like people who are stuck back in the '80's because of ext3 arguing that ZFS is crap because it doesn't come with fsck or "some sort of repair tool" -- they just can't wrap their head around the ZFS concepts.
timClicks 7y ago
"The network is the computer" doesn't feel like a slogan that a company ignoring distributed systems would use
- mpweiher 7y ago
  Ahh yes, knew I was forgetting an important one... Oh, and SUN was the Stanford University Network.
- zzzcpan 7y ago
  I guess I rephrase: they didn't do any of the distributed systems stuff necessary for the cloud if people so desperately want to call all the network stuff they did "distributed systems". Technically they were distributed systems, sure, just like anything client-server, but irrelevant to the challenges of fault tolerant geographically distributed asynchronously communicating systems.
  - aramallo 7y ago
    I completely disagree, see my comment above. I was doing asynchronous distributed data replication and compute over a grid in 2001 using Jini and Javaspaces, and I was the one late to the party.
    - zzzcpan 7y ago
      So old broken networking tech (Jini and Javaspaces) over a fast reliable local network (a grid) in 2001. How could this knowledge help anyone with fault tolerance, high-availability, partitioning/sharding, CAP consistency/availability trade offs, etc., which internet companies at the time were figuring out?
  - csirac2 7y ago
    ... Does this statement come from having actually used things like Hadoop on Sun Grid Engine or any of Sun's other products? Or... what? What knowledge are you drawing on? Edit: I mean, this is the company that gave us the 8 Fallacies of distributed computing, as they were 20yrs ago
analogkid 7y ago
So you're going to pick up that shovel and keep digging?
KirinDave 7y ago
Why does ZFS reflect a lack of experience in the internet and/or distributed systems? Simply because it didn't follow a questionable tech trend of "everything is a DHT?"
- zzzcpan 7y ago
  Because they didn't seem to know any of the simple and nice ideas from distributed systems to solve pain points of managing space, since adding, removing nodes, sharding/partitioning is something you have to figure out pretty much from the start.
  - Annatar 7y ago
    Distributed systems and write arbitration on the same file are still terra incognita. This has never been solved because no one, and I really mean literally no one has been able to solve write arbitration to the same file or even the same block from multiple sources at the same time yet. Especially not in a generic way. Which node's update is the correct one in that instance? There is an entire world of computer science research waiting there. I'm sorry to have to write this, I really am, but you have no idea what the problems of the systems you are referring to are. It would be enough for me to cut off electricity to one or even all of your systems and then you would see all hell break loose on your "simple and nice distributed system ideas". You are just blissfully unaware of all the issues because you assume the network and the hardware are infallible and apparently you gave no thought to write arbitration in a true clustering filesystem. You have no idea what lurks beneath the surface and how fragile your "simple and nice distributed ideas" are, especially when implemented in software.
    - zzzcpan 7y ago
      Flagged your comment for this. You should at least glance over my comment history on distributed computing before attacking me on things you don't understand.
      - Annatar 7y ago
Annatar 7y ago
You seem to believe a shitty general key-value store with no SQL interface and no ACID guarantees qualifies as a distributed system, which is terribly shallow and misguided. I worked at a company circa 2005 where we had a platform as a service built on top of Solaris 10 and zones, on-premise infrastructure which you now call cloud. Fully configured and serving Sun clusters would come into existence by merely DHCP booting from the OpenBoot PROM (Sun had to give us patches for the OBP for DHCP booting). I happen to know this, because I was one of the senior system engineers who helped architect it and then built it. We had it powering tens of thousands of servers. We were one of the first users of IP Multipathing in Solaris ("IPMP"), a technology Sun engineers invented to provide redundancy for network connectivity for the clusters. Amazon wasn't even a smear yet. That was in 2005. Which year is it now? How many years ago was that?
cbsmith 7y ago
I'm sorry, but you don't know what you are talking about. JavaSpaces was a distributed key-value store. JXTA was all about a resilient distributed mesh. I hate on NFS all the time, but WebNFS demonstrated a much better understanding of how to do reliable data delivery over the Internet than HTTP ever did. Heck, even the old stuff around NIS+/ONC or NeWS or USENET covers distributed systems pretty well. If you compare Sun to their competitors at the time: IBM, DEC, SGI, HP, and even Microsoft. Sun was way ahead on distributed systems.
- Annatar 7y ago
  "If you compare Sun to their competitors at the time: IBM, DEC, SGI, HP, and even Microsoft. Sun was way ahead on distributed systems." Careful now, you're on thin ice: Sun Microsystems might have been a pioneer in distributed cloud computing, but nobody did it better than SGI: once they had the NUMALink technology from CRAY, they invented cluster routing in hardware which would linearly scale with added router modules. I had Origin 2000's hooked up and scaled them linearly thanks to NUMALink, the hardware routing bricks and the crossbar switch on the motherboard. And IRIX 6.5 was aware of it all and used it! I'd hit the power button on one server and the other one would come up; after the systems booted, each having 8 GB of memory and 8 processors, I logged into a single IRIX instance with 16 GB of RAM and 16 processors. The only company to come even close to that was DEC, years prior, but nobody perfected it so throughly like SGI engineers. What CRAY started, SGI engineers finished. Sun only came with hardware which had the crossbar switch after SGI, and they never had their own hardware router bricks. By router I don't mean network router, I mean cluster scalability router which had nothing to do with TCP/IP networking.
  - cbsmith 7y ago
    That's not the distributed computing that the OP was referring to. Explicit reference was made to WAN-style networking that makes the crossbar switch effectively seem like motherboard traces.
    - Annatar 7y ago
      Doesn't matter (and I don't care) what they referred to, only the end result counts: SGI designed and implemented cloud computing in hardware. Color me impressed. Haven't seen anyone successfully do it since, hardware- or software-wise (and everyone is sure as hell trying!)

notacoward 7y ago

I believe ZFS represented a missed opportunity by spending all that effort on a local filesystem instead of trying to do something interesting in the distributed space. I even said so at the time, since I was working on distributed filesystems at the time, but that's just a strategic issue (and one that's quite arguable). It's not evidence of ignorance or lack of experience.

zzzcpan 7y ago
Well, on that part I was referring more specifically to the vdev mess and raidz, how you cannot just add a disk to the file system. Which is pretty silly, considering that at the time there were plenty of ideas how to do it much better, although not for disks, but for nodes in distributed systems. But apparently no one on ZFS team had any idea about any of it.
- notacoward 7y ago
  That was a pretty common pattern at Sun. Engineers there did a lot of great work, no denying it. Like many other projects, ZFS solved some pretty difficult and important problems while ignoring others. Which ones get talked about ad nauseam? We all make mistakes, of course, but some of those ignored problems were well known at the time to be worth solving. Did those self-professed "complete engineers" learn anything from those times they missed the target? The fact that Sun doesn't exist any more suggests an answer.
  - Annatar 7y ago
    The problem is they cannot all be solved with just one single thing. Jeff Bonwick and his team knew that. They knew what was left on the table and they had to make that call. A completely fault tolerant, distributed system which can correctly arbitrate a write to a single shared resource (usually a file or even a block) from multiple nodes at the exact same time is a problem which has not been solved correctly yet, to the best of my knowledge and belief. I myself have spent considerable amount of time (20 years, to be precise) trying to solve that particular problem correctly for all edge cases. That problem is terra incognita in computer science and yet to be throughly researched, let alone understood. All attempts at solving that particular problem 100% correctly have not been successful, by anyone, but if you know of someone, please do tell, I'd love to be corrected on this and start using such technology immediately. The ZFS team was well aware of this and that's why they made the trade-offs they did. They picked a subset of problems and they solved them. Even in hindsight, to me at least, it's obvious why. But to suggest that Sun Microsystems doesn't exist because the company's engineers picked only a subset of problems they knew they could solve is... either terribly ignorant or wilfully malicious, or both, since malice often stems from ignorance. Sun Microsystems lost because they were too expensive, more expensive than garbage intel personal computer servers and because Solaris wasn't open source code in 1993. It's that simple.
- Annatar 7y ago
  ZFS doesn't work that way: it works on pools of storage. Disks are added to the pool. Filesystems (or datasets) are carved out of a pool, not out of physical disks directly. There are vdevs on top of physical disks. That's like people who are stuck back in the '80's because of ext3 arguing that ZFS is crap because it doesn't come with fsck or "some sort of repair tool" -- they just can't wrap their head around the ZFS concepts.

timClicks 7y ago

"The network is the computer" doesn't feel like a slogan that a company ignoring distributed systems would use

mpweiher 7y ago
Ahh yes, knew I was forgetting an important one... Oh, and SUN was the Stanford University Network.
zzzcpan 7y ago
I guess I rephrase: they didn't do any of the distributed systems stuff necessary for the cloud if people so desperately want to call all the network stuff they did "distributed systems". Technically they were distributed systems, sure, just like anything client-server, but irrelevant to the challenges of fault tolerant geographically distributed asynchronously communicating systems.
- aramallo 7y ago
  I completely disagree, see my comment above. I was doing asynchronous distributed data replication and compute over a grid in 2001 using Jini and Javaspaces, and I was the one late to the party.
  - zzzcpan 7y ago
    So old broken networking tech (Jini and Javaspaces) over a fast reliable local network (a grid) in 2001. How could this knowledge help anyone with fault tolerance, high-availability, partitioning/sharding, CAP consistency/availability trade offs, etc., which internet companies at the time were figuring out?
- csirac2 7y ago
  ... Does this statement come from having actually used things like Hadoop on Sun Grid Engine or any of Sun's other products? Or... what? What knowledge are you drawing on? Edit: I mean, this is the company that gave us the 8 Fallacies of distributed computing, as they were 20yrs ago

analogkid 7y ago

So you're going to pick up that shovel and keep digging?

KirinDave 7y ago

Why does ZFS reflect a lack of experience in the internet and/or distributed systems? Simply because it didn't follow a questionable tech trend of "everything is a DHT?"

zzzcpan 7y ago
Because they didn't seem to know any of the simple and nice ideas from distributed systems to solve pain points of managing space, since adding, removing nodes, sharding/partitioning is something you have to figure out pretty much from the start.
- Annatar 7y ago
  Distributed systems and write arbitration on the same file are still terra incognita. This has never been solved because no one, and I really mean literally no one has been able to solve write arbitration to the same file or even the same block from multiple sources at the same time yet. Especially not in a generic way. Which node's update is the correct one in that instance? There is an entire world of computer science research waiting there. I'm sorry to have to write this, I really am, but you have no idea what the problems of the systems you are referring to are. It would be enough for me to cut off electricity to one or even all of your systems and then you would see all hell break loose on your "simple and nice distributed system ideas". You are just blissfully unaware of all the issues because you assume the network and the hardware are infallible and apparently you gave no thought to write arbitration in a true clustering filesystem. You have no idea what lurks beneath the surface and how fragile your "simple and nice distributed ideas" are, especially when implemented in software.
  - zzzcpan 7y ago
    Flagged your comment for this. You should at least glance over my comment history on distributed computing before attacking me on things you don't understand.
    - Annatar 7y ago

Annatar 7y ago

You seem to believe a shitty general key-value store with no SQL interface and no ACID guarantees qualifies as a distributed system, which is terribly shallow and misguided. I worked at a company circa 2005 where we had a platform as a service built on top of Solaris 10 and zones, on-premise infrastructure which you now call cloud. Fully configured and serving Sun clusters would come into existence by merely DHCP booting from the OpenBoot PROM (Sun had to give us patches for the OBP for DHCP booting). I happen to know this, because I was one of the senior system engineers who helped architect it and then built it. We had it powering tens of thousands of servers. We were one of the first users of IP Multipathing in Solaris ("IPMP"), a technology Sun engineers invented to provide redundancy for network connectivity for the clusters. Amazon wasn't even a smear yet. That was in 2005. Which year is it now? How many years ago was that?

cbsmith 7y ago

I'm sorry, but you don't know what you are talking about. JavaSpaces was a distributed key-value store. JXTA was all about a resilient distributed mesh. I hate on NFS all the time, but WebNFS demonstrated a much better understanding of how to do reliable data delivery over the Internet than HTTP ever did. Heck, even the old stuff around NIS+/ONC or NeWS or USENET covers distributed systems pretty well. If you compare Sun to their competitors at the time: IBM, DEC, SGI, HP, and even Microsoft. Sun was way ahead on distributed systems.

Annatar 7y ago
"If you compare Sun to their competitors at the time: IBM, DEC, SGI, HP, and even Microsoft. Sun was way ahead on distributed systems." Careful now, you're on thin ice: Sun Microsystems might have been a pioneer in distributed cloud computing, but nobody did it better than SGI: once they had the NUMALink technology from CRAY, they invented cluster routing in hardware which would linearly scale with added router modules. I had Origin 2000's hooked up and scaled them linearly thanks to NUMALink, the hardware routing bricks and the crossbar switch on the motherboard. And IRIX 6.5 was aware of it all and used it! I'd hit the power button on one server and the other one would come up; after the systems booted, each having 8 GB of memory and 8 processors, I logged into a single IRIX instance with 16 GB of RAM and 16 processors. The only company to come even close to that was DEC, years prior, but nobody perfected it so throughly like SGI engineers. What CRAY started, SGI engineers finished. Sun only came with hardware which had the crossbar switch after SGI, and they never had their own hardware router bricks. By router I don't mean network router, I mean cluster scalability router which had nothing to do with TCP/IP networking.
- cbsmith 7y ago
  That's not the distributed computing that the OP was referring to. Explicit reference was made to WAN-style networking that makes the crossbar switch effectively seem like motherboard traces.
  - Annatar 7y ago
    Doesn't matter (and I don't care) what they referred to, only the end result counts: SGI designed and implemented cloud computing in hardware. Color me impressed. Haven't seen anyone successfully do it since, hardware- or software-wise (and everyone is sure as hell trying!)