3 ms·

One interesting idea that got lost in the last 10 years was "push compute to the data", saving a network transfer. It seems like these days most systems prefer

One interesting idea that got lost in the last 10 years was "push compute to the data", saving a network transfer. It seems like these days most systems prefer the flexibility of separate data/compute and are willing to pay the network tax for it.

jeffbee 3y ago
It wasn't an interesting idea, it was a footgun. There's nothing naturally advantageous about the machine your process happens to be running on. In the era of very large machines there may be substantial contention for the block device that by coincidence is in the same chassis as your CPU.
- mbreese 3y ago
  It was useful at the time though. Both network and disk were substantially slower at the time. And servers had far fewer cores. So which server stored the data could make your processing faster. And you’d have less risk for IO contention within the same chassis. Fast forward a few years and compute servers can have hundreds of cores and the network is significantly faster. I think it was a “good enough” design for the time and those assumptions. And batch processing. Now? I think there are different assumptions to be made. (And I think there is a good argument that by the time Hadoop/HDFS had stabilized the assumptions had already shifted)
- woooooo 3y ago
  It's a footgun if your cluster is providing data/compute to other systems, absolutely, the case you mentioned is a risk worth extra network traffic to avoid. But theoretically, if we're trying to beat a benchmark or something, the "move compute to data" idea has something to it. Nobody transfers the LLM weights off of card memory, let alone to other machines for compute.
woadwarrior01 3y ago
It's still around in things like Spanner. Although, compute and storage are mostly disaggregated into different classes of servers now.
the-rc 3y ago
BigTable did have coprocessors, but they were only ever described with not a lot of detail in a Jeff Dean talk. That didn't stop HBase from implementing their own take. See https://www.cs.cornell.edu/projects/ladis2009/talks/dean-keynote-ladis2009.pdf https://www.cs.cornell.edu/projects/ladis2009/talks/dean-key...
DonHopkins 3y ago
It got lost a lot longer than 10 years ago. Sun published the NFS 3 spec (NeFS -- Network extensible File System) 34 years ago in 1990, which was in turn based on John Warnock's earlier ideas of using PostScript as a "linguistic motherboard" with "slots" for different kinds of "cards" like graphics and file systems (drawing from his experience at Adobe and Xerox PARC): PostScript is a linguistic "mother board", which has "slots" for several "cards". The first card we (Adobe) built was a graphics card. We're considering other cards. In particular, we've thought about other network services, such as a file server card. https://news.ycombinator.com/item?id=34256293 https://news.ycombinator.com/item?id=34256293 [...] DonHopkins on Sept 5, 2019 | parent | context | favorite | on: Samsung Announces Key-Value SSD Prototype It's like NFS 3 (NeFS) in hardware! (NeWS for disks: A PostScript interpreter in the kernel as a file system API.) https://www.donhopkins.com/home/nfs3_0.pdf https://www.donhopkins.com/home/nfs3_0.pdf Network Extensible File System Protocol Specification (2/12/90) Comments to: sun!nfs3 nfs3@SUN.COM Sun Microsystems, Inc. 2550 Garcia Ave. Mountain View, CA 94043 1.0 Introduction The Network Extensible File System protocol (NeFS) provides transparent remote access to shared file systems over networks. The NeFS protocol is designed to be machine, operating system, network architecture, and transport protocol independent. This document is the draft specification for the protocol. It will remain in draft form during a period of public review. Italicized comments in the document are intended to present the rationale behind elements of the design and to raise questions where there are doubts. Comments and suggestions on this draft specification are most welcome. [...] Although it has features in common with NFS, NeFS is a radical departure from NFS. The NFS protocol is built according to a Remote Procedure Call model (RPC) where filesystem operations are mapped across the network as remote procedure calls. The NeFS protocol abandons this model in favor of an interpretive model in which the filesystem operations become operators in an interpreted language. Clients send their requests to the server as programs to be interpreted. Execution of the request by the server’s interpreter results in the filesystem operations being invoked and results returned to the client. Using the interpretive model, filesystem operations can be defined more simply. Clients can build arbitrarily complex requests from these simple operations. [...] https://news.ycombinator.com/item?id=22456710 https://news.ycombinator.com/item?id=22456710 DonHopkins on March 1, 2020 | parent | context | favorite | on: Sun's NeWS was a mistake, as are all toolkit-in-se... Owen Densmore recounted John Warnock's idea that PostScript was actually a "linguistic motherboard". (This was part of a discussion with Owen about NeFS, which was a proposal for the next version of NFS to run a PostScript interpreter in the kernel. More about that here:) https://news.ycombinator.com/item?id=17077721 https://news.ycombinator.com/item?id=17077721 Owen Densmore's discussion of John Warnock's "Linguistic Motherboard" idea for PostScript: https://donhopkins.com/home/archive/NeWS/linguistic-motherbo https://donhopkins.com/home/archive/NeWS/linguistic-motherbo... Date: Tue, 20 Feb 90 15:20:52 PST From: owen@Sun.COM (Owen Densmore) To: don@cs.UMD.EDU Subject: Re: NeFS > They changed the meaning of some of the standard PostScript operators, > like read and write, which I don't think was a good idea, for several > reasons... They should have used different names, or at least made .. Agreed. And I DO see reasons for the old operators. They could be optimized as a local cache for NeFS to use in its own calcs. > Basically, NeFS is a *particular* application of an abstraction of > NeWS. The abstract idea is that of having a server with a dynamically > extensible interpreter as an interface to whatever library or resource > you want to make available over the network (let's call it a generic > Ne* server). Very true. This has been particularly difficult for me to get across to others here at Sun. I recently wrote it up for Steve MacKay and include it at the end of the message. > It's not clear to me if NeFS supports multiple light weight PostScript > processes like NeWS. I asked Brent about this, and he agreed that it's an issue. Brent has been talking to a guy here who's interested in re-writing the NeWS interpreter to be much easier to program and debug. I'd love to see them come up with a NeWS Core that could be used as a generic NetWare core. I think you should send your comments off to nfs3 & see what happens! I agree with most of your points. Owen Here's the memo I consed up for MacKay: More: https://news.ycombinator.com/item?id=22456710 https://news.ycombinator.com/item?id=22456710

jeffbee 3y ago

It wasn't an interesting idea, it was a footgun. There's nothing naturally advantageous about the machine your process happens to be running on. In the era of very large machines there may be substantial contention for the block device that by coincidence is in the same chassis as your CPU.

mbreese 3y ago
It was useful at the time though. Both network and disk were substantially slower at the time. And servers had far fewer cores. So which server stored the data could make your processing faster. And you’d have less risk for IO contention within the same chassis. Fast forward a few years and compute servers can have hundreds of cores and the network is significantly faster. I think it was a “good enough” design for the time and those assumptions. And batch processing. Now? I think there are different assumptions to be made. (And I think there is a good argument that by the time Hadoop/HDFS had stabilized the assumptions had already shifted)
woooooo 3y ago
It's a footgun if your cluster is providing data/compute to other systems, absolutely, the case you mentioned is a risk worth extra network traffic to avoid. But theoretically, if we're trying to beat a benchmark or something, the "move compute to data" idea has something to it. Nobody transfers the LLM weights off of card memory, let alone to other machines for compute.

woadwarrior01 3y ago

It's still around in things like Spanner. Although, compute and storage are mostly disaggregated into different classes of servers now.

the-rc 3y ago

BigTable did have coprocessors, but they were only ever described with not a lot of detail in a Jeff Dean talk. That didn't stop HBase from implementing their own take. See https://www.cs.cornell.edu/projects/ladis2009/talks/dean-keynote-ladis2009.pdf https://www.cs.cornell.edu/projects/ladis2009/talks/dean-key...

DonHopkins 3y ago

It got lost a lot longer than 10 years ago. Sun published the NFS 3 spec (NeFS -- Network extensible File System) 34 years ago in 1990, which was in turn based on John Warnock's earlier ideas of using PostScript as a "linguistic motherboard" with "slots" for different kinds of "cards" like graphics and file systems (drawing from his experience at Adobe and Xerox PARC): PostScript is a linguistic "mother board", which has "slots" for several "cards". The first card we (Adobe) built was a graphics card. We're considering other cards. In particular, we've thought about other network services, such as a file server card. https://news.ycombinator.com/item?id=34256293 https://news.ycombinator.com/item?id=34256293 [...] DonHopkins on Sept 5, 2019 | parent | context | favorite | on: Samsung Announces Key-Value SSD Prototype It's like NFS 3 (NeFS) in hardware! (NeWS for disks: A PostScript interpreter in the kernel as a file system API.) https://www.donhopkins.com/home/nfs3_0.pdf https://www.donhopkins.com/home/nfs3_0.pdf Network Extensible File System Protocol Specification (2/12/90) Comments to: sun!nfs3 nfs3@SUN.COM Sun Microsystems, Inc. 2550 Garcia Ave. Mountain View, CA 94043 1.0 Introduction The Network Extensible File System protocol (NeFS) provides transparent remote access to shared file systems over networks. The NeFS protocol is designed to be machine, operating system, network architecture, and transport protocol independent. This document is the draft specification for the protocol. It will remain in draft form during a period of public review. Italicized comments in the document are intended to present the rationale behind elements of the design and to raise questions where there are doubts. Comments and suggestions on this draft specification are most welcome. [...] Although it has features in common with NFS, NeFS is a radical departure from NFS. The NFS protocol is built according to a Remote Procedure Call model (RPC) where filesystem operations are mapped across the network as remote procedure calls. The NeFS protocol abandons this model in favor of an interpretive model in which the filesystem operations become operators in an interpreted language. Clients send their requests to the server as programs to be interpreted. Execution of the request by the server’s interpreter results in the filesystem operations being invoked and results returned to the client. Using the interpretive model, filesystem operations can be defined more simply. Clients can build arbitrarily complex requests from these simple operations. [...] https://news.ycombinator.com/item?id=22456710 https://news.ycombinator.com/item?id=22456710 DonHopkins on March 1, 2020 | parent | context | favorite | on: Sun's NeWS was a mistake, as are all toolkit-in-se... Owen Densmore recounted John Warnock's idea that PostScript was actually a "linguistic motherboard". (This was part of a discussion with Owen about NeFS, which was a proposal for the next version of NFS to run a PostScript interpreter in the kernel. More about that here:) https://news.ycombinator.com/item?id=17077721 https://news.ycombinator.com/item?id=17077721 Owen Densmore's discussion of John Warnock's "Linguistic Motherboard" idea for PostScript: https://donhopkins.com/home/archive/NeWS/linguistic-motherbo https://donhopkins.com/home/archive/NeWS/linguistic-motherbo... Date: Tue, 20 Feb 90 15:20:52 PST From: owen@Sun.COM (Owen Densmore) To: don@cs.UMD.EDU Subject: Re: NeFS > They changed the meaning of some of the standard PostScript operators, > like read and write, which I don't think was a good idea, for several > reasons... They should have used different names, or at least made .. Agreed. And I DO see reasons for the old operators. They could be optimized as a local cache for NeFS to use in its own calcs. > Basically, NeFS is a *particular* application of an abstraction of > NeWS. The abstract idea is that of having a server with a dynamically > extensible interpreter as an interface to whatever library or resource > you want to make available over the network (let's call it a generic > Ne* server). Very true. This has been particularly difficult for me to get across to others here at Sun. I recently wrote it up for Steve MacKay and include it at the end of the message. > It's not clear to me if NeFS supports multiple light weight PostScript > processes like NeWS. I asked Brent about this, and he agreed that it's an issue. Brent has been talking to a guy here who's interested in re-writing the NeWS interpreter to be much easier to program and debug. I'd love to see them come up with a NeWS Core that could be used as a generic NetWare core. I think you should send your comments off to nfs3 & see what happens! I agree with most of your points. Owen Here's the memo I consed up for MacKay: More: https://news.ycombinator.com/item?id=22456710 https://news.ycombinator.com/item?id=22456710