6 ms·

A Note on Distributed Computing (1994) [pdf]

88 points by vaughan 4y ago

DonHopkins 4y ago
It was nice to see the later Java people at Sun (and even later the XMLHTTP/AJAX people at Microsoft) finally pick up some of the ideas the earlier NeWS people were talking about until we were blue in the face, and implementing in the 80's with PostScript, long before JavaScript or even Java. In 1990 Owen Densmore rightfully complained, "This has been particularly difficult for me to get across to others here at Sun." James Gosling designed NeWS long before he designed Java. This is his 1986 paper about it, when it was originally called "SunDew". http://www.chilton-computing.org.uk/inf/literature/books/wm/p005.htm http://www.chilton-computing.org.uk/inf/literature/books/wm/... NFS Version 3 (NeFS) was all about applying NeWS's distributed computing architecture of sending PostScript programs instead of using a fixed protocol to interact with the file system as well as the window system. In fact John Warnock originally intended PostScript to be a "linguistic motherboard" that supported network services like file storage and other features as well as graphics, which Owen Densmore recounted in his "Swiss Army NeWS" paper. https://news.ycombinator.com/item?id=20891291 https://news.ycombinator.com/item?id=20891291 DonHopkins on Sept 5, 2019 | parent | context | favorite | on: Samsung Announces Key-Value SSD Prototype It's like NFS 3 (NeFS) in hardware! (NeWS for disks: A PostScript interpreter in the kernel as a file system API.) https://www.donhopkins.com/home/nfs3_0.pdf https://www.donhopkins.com/home/nfs3_0.pdf Network Extensible File System Protocol Specification (2/12/90) Comments to: sun!nfs3 nfs3@SUN.COM Sun Microsystems, Inc. 2550 Garcia Ave. Mountain View, CA 94043 1.0 Introduction The Network Extensible File System protocol (NeFS) provides transparent remote access to shared file systems over networks. The NeFS protocol is designed to be machine, operating system, network architecture, and transport protocol independent. This document is the draft specification for the protocol. It will remain in draft form during a period of public review. Italicized comments in the document are intended to present the rationale behind elements of the design and to raise questions where there are doubts. Comments and suggestions on this draft specification are most welcome. [...] Although it has features in common with NFS, NeFS is a radical departure from NFS. The NFS protocol is built according to a Remote Procedure Call model (RPC) where filesystem operations are mapped across the network as remote procedure calls. The NeFS protocol abandons this model in favor of an interpretive model in which the filesystem operations become operators in an interpreted language. Clients send their requests to the server as programs to be interpreted. Execution of the request by the server’s interpreter results in the filesystem operations being invoked and results returned to the client. Using the interpretive model, filesystem operations can be defined more simply. Clients can build arbitrarily complex requests from these simple operations. https://news.ycombinator.com/item?id=22456710 https://news.ycombinator.com/item?id=22456710 DonHopkins on March 1, 2020 | parent | context | favorite | on: Sun's NeWS was a mistake, as are all toolkit-in-se... Owen Densmore recounted John Warnock's idea that PostScript was actually a "linguistic motherboard". (This was part of a discussion with Owen about NeFS, which was a proposal for the next version of NFS to run a PostScript interpreter in the kernel. More about that here:) https://news.ycombinator.com/item?id=17077721 https://news.ycombinator.com/item?id=17077721 Owen Densmore's discussion of John Warnock's "Linguistic Motherboard" idea for PostScript: https://donhopkins.com/home/archive/NeWS/linguistic-motherboard-owen.txt https://donhopkins.com/home/archive/NeWS/linguistic-motherbo... Date: Tue, 20 Feb 90 15:20:52 PST From: owen@Sun.COM (Owen Densmore) To: don@cs.UMD.EDU Subject: Re: NeFS > They changed the meaning of some of the standard PostScript operators, > like read and write, which I don't think was a good idea, for several > reasons... They should have used different names, or at least made .. Agreed. And I DO see reasons for the old operators. They could be optimized as a local cache for NeFS to use in its own calcs. > Basically, NeFS is a *particular* application of an abstraction of > NeWS. The abstract idea is that of having a server with a dynamically > extensible interpreter as an interface to whatever library or resource > you want to make available over the network (let's call it a generic > Ne* server). Very true. This has been particularly difficult for me to get across to others here at Sun. I recently wrote it up for Steve MacKay and include it at the end of the message. > It's not clear to me if NeFS supports multiple light weight PostScript > processes like NeWS. I asked Brent about this, and he agreed that it's an issue. Brent has been talking to a guy here who's interested in re-writing the NeWS interpreter to be much easier to program and debug. I'd love to see them come up with a NeWS Core that could be used as a generic NetWare core. I think you should send your comments off to nfs3 & see what happens! I agree with most of your points. Owen Here's the memo I consed up for MacKay: Window System? ..NeWS ain' no stinkin' Window System! -or- Swiss Army NeWS: A Programmable Network Facility Introduction NeWS is difficult to understand simply because it is not just a window system. It is a "Swiss Army Knife" containing several components, some of which contribute to its use as a window system, others which provide the networking facilities for implementing the client-server model, all embedded in a programmable substrate allowing extremely flexible and creative combination of these elements. During the initial implementation phase of the Macintosh LaserWriter software, I temporarily transfered from Apple to Adobe working closely with John Warnock and other Adobe engineers. At lunch one day, I asked: "John, what do you plan to do after LaserWriter?" His answer was interesting: PostScript is a linguistic "mother board", which has "slots" for several "cards". The first card we (Adobe) built was a graphics card. We're considering other cards. In particular, we've thought about other network services, such as a file server card. He went on to say how a programmable network was really his goal, and that the printing work was just the first component. His mentioning using PostScript for a file server is particularly interesting: Sun's next version of NFS is going to use PostScript with file extentions as the client-server protocol! This paper explores NeWS in this light: as a Programmable Network Facility, a major part of Sun's future networking strategy. [...]
vaughan 4y ago
Original title: A Note on Distributed Computing > A better approach is to accept that there are irreconcilable differences between local and distributed computing, and to be conscious of those differences at all stages of the design and implementation of distributed applications. Rather than trying to merge local and remote objects, engineers need to be constantly reminded of the differences between the two, and know when it is appropriate to use each kind of object. Aka. Transparent remoting.
- deleted 4y ago
  [deleted]
- dboreham 4y ago
  > Transparent remoting Presumably you mean "Transparent remoting isn't a thing"? Paper authors were at Sun, so had experience with NFS (the design of which assumed that transparent remoting was a thing).
  - ninkendo 4y ago
    NFS is my favorite example of how to do distributed computing incorrectly. Take a filesystem API that assumes fast, local access, and that individual I/O operations are cheap and immediate (like stat()’ing a file or resolving a symlink), and suddenly make it so all these operations may have to block for hundreds/thousands of milliseconds while accessing something over a remote server potentially on the other side of the world. It causes so many issues: syscalls that can’t be interrupted (and thus you can’t ctrl+c a process), stuck mounts that can’t be unmounted, hanging for ages on `ls`’ing a folder with a lot of symlinks… NFS locking up is the most common answer to the typical interview question “what could cause a load average in the hundreds on a machine with zero CPU usage?”: because there’s a stuck NFS server and the pending IO operations put the processes in the run queue. I’m glad there’s a paper that articulates my thoughts about this subject better than I can.
    - zozbot234 4y ago
      > It causes so many issues: syscalls that can’t be interrupted (and thus you can’t ctrl+c a process), stuck mounts that can’t be unmounted, hanging for ages on `ls`’ing a folder with a lot of symlinks… These things have nothing to do with distributed systems, they can happen on any machine. That's what the "Not ready reading drive A. Abort, Retry, Fail?_" prompt is for.
samsquire 4y ago
Interactive Distributed systems are only slow if you command them to do things sequentially and wait for a round trip. If you send the computation plan with the initial input request, they can be fast. We can also program distributed systems to run a switch statement after each response. A state machine for every response. What if you could coordinate a computer to do things depending on result? And you can do this similar to Capnproto promise pipelining? With promise pipelining you can say what operations you want to do when the operation completes against the output of the response from the server. One of my ideas is the thought puzzle of extremely high performance computer with high latency. You design sessions which decide what to do and what to do given what output. Then the computer automatically follows the sequence of commands depending on the output of the remote system. I see it as chaining together promises that are typed. Imagine an advanced option Monad that captures every possible output state and let's you pattern match it. When you next create a change for your CI system which needs to build a docker image, upload, prepare a Kubernetes manifest, run a data migration, run a redacted data import you can use this architecture. Admittedly we need an elegant GUI for queuing responses of servers. My idea was GUI thunking that is inspired by Haskell's thunks https://github.com/samsquire/gui-thunks https://github.com/samsquire/gui-thunks
- mcguire 4y ago
  Depends on how embarrassingly parallel (https://en.wikipedia.org/wiki/Embarrassingly_parallel https://en.wikipedia.org/wiki/Embarrassingly_parallel) your problem is. If many individual operations depend on the results of previous operations, it'll be slow.
  - samsquire 4y ago
    You are of course completely right. How do you parallelise building a docker image, compilation, uploading to a server and deploying to Kubernetes? I was trying to remove all round trips from the equation.
    - Jtsummers 4y ago
      If they depend on each other, you don't. The best you can do is get all the data in the right place(s) to reduce latency and use caches to reduce duplication of effort.
rileyphone 4y ago
Waldo went on to be the chief architect of Jini at Sun, which IMO is a very elegant solution to the problems of distributed computing. Unfortunately Jini never found success in the real world, but there are a lit of great ideas contained within, like resource leasing, tuplespaces, and interface-based proxying. Here’s a later interview with Waldo that’s a bit of a retrospective [0]. With 20 years passed since then, I don’t think we have yet found a satisfactory approach to distributed systems, so maybe it’s time to revisit these older ones. 0. https://www.artima.com/articles/jim-waldo-on-distributed-computing#part10 https://www.artima.com/articles/jim-waldo-on-distributed-com...
NelsonMinar 4y ago
Hard to believe this paper is 28 years old now! I still forward it to folks who tell me excitedly about how they've found some new system that makes it so that remote resources look just like local resources. Fortunately that fantasy is less common now that so much is remote, folks are more used to experiencing dropouts and jitter and delay when interacting with things online.
Upvoter33 4y ago
This is an old paper that is not super meaningful these days. Distributed computing has transformed radically in the past three decades. In the 80s and 90s, there were few useful distributed systems, and we only had a rudimentary idea as to how to build them. Now we run planet-wide distributed systems for billions of users. The lessons from the 90s are no longer very useful.
- mcguire 4y ago
  The vast majority of modern distributed systems are (possibly stacked) client/server architectures, possibly with a significant minority of publish/subscribe systems. And even so, latency, partial failure, and concurrency are still major issues.
w10-1 4y ago
"The hard problems in distributed computing are not the problems of how to get things on and off the wire. The hard problems in distributed computing concern dealing with partial failure and the lack of a central resource manager. The hard problems in distributed computing concern insuring adequate performance and dealing with problems of concurrency. The hard problems have to do with differences in memory access paradigms between local and distributed entities. People attempting to write distributed applications quickly discover that they are spending all of their efforts in these areas and not on the communications protocol programming interface." They got that right. Most of the value of Java, and the market value of Google and Amazon, come from solving these problems. The viability of javascript in the browser has depended entirely on the communications protocol not being that important. One thing the paper misses is the value of encryption. Distributed computing is severely limiting if it can't be encrypted. Indeed, the rate-limiting factor to uptake might be not protocols or distributed algorithms, but confidence in encryption and distributed security -- and that's as much a social problem as a technological one.
- mcguire 4y ago
  At the time this was written (IIRC), public-key encryption was still under the DSA/RSA patents and therefore not commonly available. And (possibly) the machines didn't have the horsepower to do a lot of cryptography.
  - pjdesno 4y ago
    Encryption was export-restricted until 1996. Actually that isn't really relevant - most of the systems the authors are talking about (and that they worked on) would have been running on enterprise or campus LANs, and it wasn't totally clear (at least to the sysadmins at the place where I worked) whether it was even legal to connect corporate networks to the internet at that point in time if you weren't a govt contractor. Based on what I recall, most of the security threats we were worried about had to do with corporate espionage and information theft, and would have required physical access to the network. In that environment, encryption isn't important in the same way as it is on today's Internet.
bradneuberg 4y ago
Foundational paper IMHO that all computer programmers should read.

88 points by vaughan 4y ago

It was nice to see the later Java people at Sun (and even later the XMLHTTP/AJAX people at Microsoft) finally pick up some of the ideas the earlier NeWS people were talking about until we were blue in the face, and implementing in the 80's with PostScript, long before JavaScript or even Java. In 1990 Owen Densmore rightfully complained, "This has been particularly difficult for me to get across to others here at Sun." James Gosling designed NeWS long before he designed Java. This is his 1986 paper about it, when it was originally called "SunDew". http://www.chilton-computing.org.uk/inf/literature/books/wm/p005.htm http://www.chilton-computing.org.uk/inf/literature/books/wm/... NFS Version 3 (NeFS) was all about applying NeWS's distributed computing architecture of sending PostScript programs instead of using a fixed protocol to interact with the file system as well as the window system. In fact John Warnock originally intended PostScript to be a "linguistic motherboard" that supported network services like file storage and other features as well as graphics, which Owen Densmore recounted in his "Swiss Army NeWS" paper. https://news.ycombinator.com/item?id=20891291 https://news.ycombinator.com/item?id=20891291 DonHopkins on Sept 5, 2019 | parent | context | favorite | on: Samsung Announces Key-Value SSD Prototype It's like NFS 3 (NeFS) in hardware! (NeWS for disks: A PostScript interpreter in the kernel as a file system API.) https://www.donhopkins.com/home/nfs3_0.pdf https://www.donhopkins.com/home/nfs3_0.pdf Network Extensible File System Protocol Specification (2/12/90) Comments to: sun!nfs3 nfs3@SUN.COM Sun Microsystems, Inc. 2550 Garcia Ave. Mountain View, CA 94043 1.0 Introduction The Network Extensible File System protocol (NeFS) provides transparent remote access to shared file systems over networks. The NeFS protocol is designed to be machine, operating system, network architecture, and transport protocol independent. This document is the draft specification for the protocol. It will remain in draft form during a period of public review. Italicized comments in the document are intended to present the rationale behind elements of the design and to raise questions where there are doubts. Comments and suggestions on this draft specification are most welcome. [...] Although it has features in common with NFS, NeFS is a radical departure from NFS. The NFS protocol is built according to a Remote Procedure Call model (RPC) where filesystem operations are mapped across the network as remote procedure calls. The NeFS protocol abandons this model in favor of an interpretive model in which the filesystem operations become operators in an interpreted language. Clients send their requests to the server as programs to be interpreted. Execution of the request by the server’s interpreter results in the filesystem operations being invoked and results returned to the client. Using the interpretive model, filesystem operations can be defined more simply. Clients can build arbitrarily complex requests from these simple operations. https://news.ycombinator.com/item?id=22456710 https://news.ycombinator.com/item?id=22456710 DonHopkins on March 1, 2020 | parent | context | favorite | on: Sun's NeWS was a mistake, as are all toolkit-in-se... Owen Densmore recounted John Warnock's idea that PostScript was actually a "linguistic motherboard". (This was part of a discussion with Owen about NeFS, which was a proposal for the next version of NFS to run a PostScript interpreter in the kernel. More about that here:) https://news.ycombinator.com/item?id=17077721 https://news.ycombinator.com/item?id=17077721 Owen Densmore's discussion of John Warnock's "Linguistic Motherboard" idea for PostScript: https://donhopkins.com/home/archive/NeWS/linguistic-motherboard-owen.txt https://donhopkins.com/home/archive/NeWS/linguistic-motherbo... Date: Tue, 20 Feb 90 15:20:52 PST From: owen@Sun.COM (Owen Densmore) To: don@cs.UMD.EDU Subject: Re: NeFS > They changed the meaning of some of the standard PostScript operators, > like read and write, which I don't think was a good idea, for several > reasons... They should have used different names, or at least made .. Agreed. And I DO see reasons for the old operators. They could be optimized as a local cache for NeFS to use in its own calcs. > Basically, NeFS is a *particular* application of an abstraction of > NeWS. The abstract idea is that of having a server with a dynamically > extensible interpreter as an interface to whatever library or resource > you want to make available over the network (let's call it a generic > Ne* server). Very true. This has been particularly difficult for me to get across to others here at Sun. I recently wrote it up for Steve MacKay and include it at the end of the message. > It's not clear to me if NeFS supports multiple light weight PostScript > processes like NeWS. I asked Brent about this, and he agreed that it's an issue. Brent has been talking to a guy here who's interested in re-writing the NeWS interpreter to be much easier to program and debug. I'd love to see them come up with a NeWS Core that could be used as a generic NetWare core. I think you should send your comments off to nfs3 & see what happens! I agree with most of your points. Owen Here's the memo I consed up for MacKay: Window System? ..NeWS ain' no stinkin' Window System! -or- Swiss Army NeWS: A Programmable Network Facility Introduction NeWS is difficult to understand simply because it is not just a window system. It is a "Swiss Army Knife" containing several components, some of which contribute to its use as a window system, others which provide the networking facilities for implementing the client-server model, all embedded in a programmable substrate allowing extremely flexible and creative combination of these elements. During the initial implementation phase of the Macintosh LaserWriter software, I temporarily transfered from Apple to Adobe working closely with John Warnock and other Adobe engineers. At lunch one day, I asked: "John, what do you plan to do after LaserWriter?" His answer was interesting: PostScript is a linguistic "mother board", which has "slots" for several "cards". The first card we (Adobe) built was a graphics card. We're considering other cards. In particular, we've thought about other network services, such as a file server card. He went on to say how a programmable network was really his goal, and that the printing work was just the first component. His mentioning using PostScript for a file server is particularly interesting: Sun's next version of NFS is going to use PostScript with file extentions as the client-server protocol! This paper explores NeWS in this light: as a Programmable Network Facility, a major part of Sun's future networking strategy. [...]

vaughan 4y ago

Original title: A Note on Distributed Computing > A better approach is to accept that there are irreconcilable differences between local and distributed computing, and to be conscious of those differences at all stages of the design and implementation of distributed applications. Rather than trying to merge local and remote objects, engineers need to be constantly reminded of the differences between the two, and know when it is appropriate to use each kind of object. Aka. Transparent remoting.

deleted 4y ago
[deleted]
dboreham 4y ago
> Transparent remoting Presumably you mean "Transparent remoting isn't a thing"? Paper authors were at Sun, so had experience with NFS (the design of which assumed that transparent remoting was a thing).
- ninkendo 4y ago
  NFS is my favorite example of how to do distributed computing incorrectly. Take a filesystem API that assumes fast, local access, and that individual I/O operations are cheap and immediate (like stat()’ing a file or resolving a symlink), and suddenly make it so all these operations may have to block for hundreds/thousands of milliseconds while accessing something over a remote server potentially on the other side of the world. It causes so many issues: syscalls that can’t be interrupted (and thus you can’t ctrl+c a process), stuck mounts that can’t be unmounted, hanging for ages on `ls`’ing a folder with a lot of symlinks… NFS locking up is the most common answer to the typical interview question “what could cause a load average in the hundreds on a machine with zero CPU usage?”: because there’s a stuck NFS server and the pending IO operations put the processes in the run queue. I’m glad there’s a paper that articulates my thoughts about this subject better than I can.
  - zozbot234 4y ago
    > It causes so many issues: syscalls that can’t be interrupted (and thus you can’t ctrl+c a process), stuck mounts that can’t be unmounted, hanging for ages on `ls`’ing a folder with a lot of symlinks… These things have nothing to do with distributed systems, they can happen on any machine. That's what the "Not ready reading drive A. Abort, Retry, Fail?_" prompt is for.

samsquire 4y ago

Interactive Distributed systems are only slow if you command them to do things sequentially and wait for a round trip. If you send the computation plan with the initial input request, they can be fast. We can also program distributed systems to run a switch statement after each response. A state machine for every response. What if you could coordinate a computer to do things depending on result? And you can do this similar to Capnproto promise pipelining? With promise pipelining you can say what operations you want to do when the operation completes against the output of the response from the server. One of my ideas is the thought puzzle of extremely high performance computer with high latency. You design sessions which decide what to do and what to do given what output. Then the computer automatically follows the sequence of commands depending on the output of the remote system. I see it as chaining together promises that are typed. Imagine an advanced option Monad that captures every possible output state and let's you pattern match it. When you next create a change for your CI system which needs to build a docker image, upload, prepare a Kubernetes manifest, run a data migration, run a redacted data import you can use this architecture. Admittedly we need an elegant GUI for queuing responses of servers. My idea was GUI thunking that is inspired by Haskell's thunks https://github.com/samsquire/gui-thunks https://github.com/samsquire/gui-thunks

mcguire 4y ago
Depends on how embarrassingly parallel (https://en.wikipedia.org/wiki/Embarrassingly_parallel https://en.wikipedia.org/wiki/Embarrassingly_parallel) your problem is. If many individual operations depend on the results of previous operations, it'll be slow.
- samsquire 4y ago
  You are of course completely right. How do you parallelise building a docker image, compilation, uploading to a server and deploying to Kubernetes? I was trying to remove all round trips from the equation.
  - Jtsummers 4y ago
    If they depend on each other, you don't. The best you can do is get all the data in the right place(s) to reduce latency and use caches to reduce duplication of effort.

rileyphone 4y ago

Waldo went on to be the chief architect of Jini at Sun, which IMO is a very elegant solution to the problems of distributed computing. Unfortunately Jini never found success in the real world, but there are a lit of great ideas contained within, like resource leasing, tuplespaces, and interface-based proxying. Here’s a later interview with Waldo that’s a bit of a retrospective [0]. With 20 years passed since then, I don’t think we have yet found a satisfactory approach to distributed systems, so maybe it’s time to revisit these older ones. 0. https://www.artima.com/articles/jim-waldo-on-distributed-computing#part10 https://www.artima.com/articles/jim-waldo-on-distributed-com...

NelsonMinar 4y ago

Hard to believe this paper is 28 years old now! I still forward it to folks who tell me excitedly about how they've found some new system that makes it so that remote resources look just like local resources. Fortunately that fantasy is less common now that so much is remote, folks are more used to experiencing dropouts and jitter and delay when interacting with things online.

Upvoter33 4y ago

This is an old paper that is not super meaningful these days. Distributed computing has transformed radically in the past three decades. In the 80s and 90s, there were few useful distributed systems, and we only had a rudimentary idea as to how to build them. Now we run planet-wide distributed systems for billions of users. The lessons from the 90s are no longer very useful.

mcguire 4y ago
The vast majority of modern distributed systems are (possibly stacked) client/server architectures, possibly with a significant minority of publish/subscribe systems. And even so, latency, partial failure, and concurrency are still major issues.

w10-1 4y ago

"The hard problems in distributed computing are not the problems of how to get things on and off the wire. The hard problems in distributed computing concern dealing with partial failure and the lack of a central resource manager. The hard problems in distributed computing concern insuring adequate performance and dealing with problems of concurrency. The hard problems have to do with differences in memory access paradigms between local and distributed entities. People attempting to write distributed applications quickly discover that they are spending all of their efforts in these areas and not on the communications protocol programming interface." They got that right. Most of the value of Java, and the market value of Google and Amazon, come from solving these problems. The viability of javascript in the browser has depended entirely on the communications protocol not being that important. One thing the paper misses is the value of encryption. Distributed computing is severely limiting if it can't be encrypted. Indeed, the rate-limiting factor to uptake might be not protocols or distributed algorithms, but confidence in encryption and distributed security -- and that's as much a social problem as a technological one.

mcguire 4y ago
At the time this was written (IIRC), public-key encryption was still under the DSA/RSA patents and therefore not commonly available. And (possibly) the machines didn't have the horsepower to do a lot of cryptography.
- pjdesno 4y ago
  Encryption was export-restricted until 1996. Actually that isn't really relevant - most of the systems the authors are talking about (and that they worked on) would have been running on enterprise or campus LANs, and it wasn't totally clear (at least to the sysadmins at the place where I worked) whether it was even legal to connect corporate networks to the internet at that point in time if you weren't a govt contractor. Based on what I recall, most of the security threats we were worried about had to do with corporate espionage and information theft, and would have required physical access to the network. In that environment, encryption isn't important in the same way as it is on today's Internet.

bradneuberg 4y ago

Foundational paper IMHO that all computer programmers should read.