5 ms·

Remote procedure calls (or whatever it is called) simply can never really feel like local functions. It has been tried again and again for decades. Maybe call

Remote procedure calls (or whatever it is called) simply can never really feel like local functions. It has been tried again and again for decades. Maybe call it "removes as much friction as possible compared to local functions" but please don't call it "feels [exactly] like local functions". It's simply impossible and pretending so will just cause trouble.

DrDeadCrash 3y ago
I once wrote a system for distributed data processing (8-10 years ago). There was a base class that when inherited from would, via Interception reroute a call to any virtual method to a server with the same interface loaded (plug-in architecture). The server would then delegate the method to one of its worker clients, which had the (same) required plugin(s) loaded. It worked over TCP on a local intranet. Once the admin-server-worker plugin system was figured out, projecting a virtual method was fairly transparent. There's always going to be some boiler-plate!
- calebpeterson 3y ago
  I suspect parent was referring more to the runtime characteristics of latency, error handling, retries, load balancing, etc… more than the syntax. Nonetheless, what you built sounds quite interesting: can you say anything more about the language, tech stack, etc…?
  - photonthug 3y ago
    Not OP but I too have built this, basically a few hundred lines of python to implement something like json rpc (plus introspection to answer help requests / publish typed api info) on top of AWS lambda.
- osigurdson 3y ago
  I had to replace a .NET remoting based system with thousands of individual calls, with all manner of in process like conventions, with an http layer without changing client code at all in a few weeks. The result was good but I still don't like transparent rpc (on large dev teams anyway).
mietek 3y ago
Yes. More on that from the creator of Erlang: http://armstrongonsoftware.blogspot.com/2008/05/road-we-didnt-go-down.html http://armstrongonsoftware.blogspot.com/2008/05/road-we-didn... Bye, Joe.
helloooooooo 3y ago
Try DCOM
- hmeh 3y ago
  Yeah, exactly. DCOM was a failure for the same reason that every other RPC thing is a failure. It turns out that introducing cascading failure or partial failure opportunities due to synchronous remote calls was never a good idea. It was always, and still is, treating the fallacies of distributed computing as first principles.
osigurdson 3y ago
Agree. Hard to abstract away a 3-4 orders of magnitude latency difference. However, even the most obvious /absurd / ugly shell out to curl and hand parse results like code, is going to get wrapped in a normal looking function. Maybe the solution is naming or tooling where: 1) it is easy for a developer to see if a network call exists anywhere in the stack and 2) it isn't possible to introduce a network call into an existing function without breaking contract. This could be as simple as a viral naming convention ("_rpc" suffix for example).
mumblemumble 3y ago
And even then, I no longer believe that "removes as much friction as possible compared to local functions" is a desirable goal in the first place. Over the course of my short 20-year career, I've encountered far too many applications that accumulated terrible performance or reliability problems that were directly attributable to their authors feeling a bit too comfortable about getting chatty with the slow, unreliable remote procedure calls. By contrast, the most successful microservices shop I ever worked at was very conscientious about doing nothing to hide the messiness of RPC. (Ease the implementation, yes, but not hide the messiness.) And that seemed to foster a culture of being much more thoughtful about interaction patterns and data flows and how they might contribute to or detract from the overall performance and stability of the system.
- lunarcave 3y ago
  One of the author here. I definitely agree that you get markedly lower reliability and performance one you move a local function call to be remote. So, we're definitely not advocating that moving a local function call to be remote is inherently virtuous. However, I've found the abstraction to be useful when someone _has_ already made the decision to refactor their service out to be a separate service.
JonChesterfield 3y ago
There's an exception to this rule if the underlying network does not go wrong. In particular, if the "network" is something local to one machine, pcie or some cross-chip fabric thing, where the failure modes of the "network" are the box falls over, then RPC works as one might hope. You don't have to mangle functions to take account of extra network errors because there are none. The specific use case I'm interested in is heterogenous compute on shared memory machines. Code running on a GPU can call `int rc = fprintf(stderr, "oh dear");` and have no idea whether that runs code on a CPU or not. Code running on a CPU can call some linear algebra function and either run it locally or on a GPU, unobservable to the caller. That is a good trick. It does make for an interesting debugging experience where you lose track of where code is executing. ---- There's a less esoteric application in the area of the OP for pure functions where you don't need the result immediately. That goes something like: 1. Call the function and get back a promise or similar. All runs locally, no failures 2. Something in the background maybe does the work for you 3. Ask for the result. If it's not available yet, do the work locally That has the same zero failure modes property because it'll ultimately run all the work locally if the network offloading never works out. It only really makes sense if the functions can be called multiple times (e.g. locally and remote at the same time) without doing harm, i.e. pure functions. It costs latency and gains throughput. For some batch computation work that is great.
- ColonelPhantom 3y ago
  Won't the proposed implementation risk doing duplicate work? e.g. another machine starts on your work, but then you ask for it before it's done. So you'd probably need the "not available, but already 'stolen' case". For example, the current machine could look for work to steal itself. But this probably will increase latency a lot, so it would only make sense if the batches are huge.
  - lunarcave 3y ago
    It's really hard to have exactly-once delivery[1]. But exactly once processing is viable. With Differential, the implementation does allow for idempotency in cases where you need exactly-once processing. And it does handle the "not available, but already 'stolen' case". Disclaimer: I'm an author. [1] https://news.ycombinator.com/item?id=34986691 https://news.ycombinator.com/item?id=34986691
lunarcave 3y ago
One of the authors here. Thanks for the insight. I generally agree with the fact pretending it's like a local function will cause trouble down the track. After all, you're introducing a number of extra failure modes. The client library however does give some protections to the developer through the type system - although it's not perfect. Hence the wording "feel like". It's not actually local function calls. But it abstracts away a lot of the complexity.
kuschku 3y ago
I'll disagree with that. In fact, I'd argue the vast majority of "local" functions are no better than RPC calls and shouldn't be treated any different. Sure, `sprintf` will run synchronously and finish in known time, but `read` may as well end up waiting for seconds for an HDD to spin up, or for a mounted NFS filesystem to respond. `read` is not any different from an RPC call. And, in fact, `read` can fail in just the same ways as an RPC call can. So can `write`, which is why `sync` exists. Software ignoring these potential issues, and pretending they don't exist, is the source of many unnecessary software hangs and crashes, such as Windows Explorer crashing if your SMB share responds too slowly or Linux being unable to unmount an NFS share if the connection breaks badly. The solution IMO is that we should finally stop hiding asynchronicity behind synchronous facades. If coroutines, promises, async programming, etc are first-class features, it becomes much easier to cleanly model these potential errors and in turn handle them correctly. And once you have that, an rpc call async fn rpcMakePurchase(user: &User, order: &Order) -> Result<Purchase> does feel just like a local function async fn read(path: impl AsRef<Path>) -> Result<Vec<u8>>
- valenterry 3y ago
  Actually, I think we are on the same side. :-) There is still a small difference though: when I execute a write and sync, then I'm pretty sure it either succeeded or didn't. I'm not sure if there is any way for the disk to say "don't know" which the OS will then pass to my program. Even if, I guess such a case is so rare that it is probably excluded in any error-model. Like cosmic rays. And I can always retry, since it's very very unlikely that there is a connection issue only between my program and the disk but not between other programs and that disk - since they run on the same machine. Over a network things are different, because the connection might be gone for a long time and know I'm not aware of the state and I can't check it. I can also not tell if the connection might come back, the machine I was talking to might have burned down. If I, myself, burn down, then I don't need to worry anyways. That's the difference. This is specific for write/sync though - as you explained it, just because things are running on the same machine does not mean things are necessarily more reliable than a "remote" call.
  - kuschku 3y ago
lunarcave 3y ago
One of the authors here. We didn't know we were going to be posted on HN, and we're still iterating on copy that "lands". I don't think we've found the words yet. I agree with the premise that RPC has markedly different characteristics. The actual implementation [1] makes it clear that you're connecting through a client interface. Thanks for the feedback on calling it "removes as much friction as possible compared to local functions". I think we'll trend this way. [1] https://docs.differential.dev/getting-started/quick-start/#3-change-the-greet-command-to-call-the-new-function https://docs.differential.dev/getting-started/quick-start/#3...

DrDeadCrash 3y ago

I once wrote a system for distributed data processing (8-10 years ago). There was a base class that when inherited from would, via Interception reroute a call to any virtual method to a server with the same interface loaded (plug-in architecture). The server would then delegate the method to one of its worker clients, which had the (same) required plugin(s) loaded. It worked over TCP on a local intranet. Once the admin-server-worker plugin system was figured out, projecting a virtual method was fairly transparent. There's always going to be some boiler-plate!

calebpeterson 3y ago
I suspect parent was referring more to the runtime characteristics of latency, error handling, retries, load balancing, etc… more than the syntax. Nonetheless, what you built sounds quite interesting: can you say anything more about the language, tech stack, etc…?
- photonthug 3y ago
  Not OP but I too have built this, basically a few hundred lines of python to implement something like json rpc (plus introspection to answer help requests / publish typed api info) on top of AWS lambda.
osigurdson 3y ago
I had to replace a .NET remoting based system with thousands of individual calls, with all manner of in process like conventions, with an http layer without changing client code at all in a few weeks. The result was good but I still don't like transparent rpc (on large dev teams anyway).

mietek 3y ago

Yes. More on that from the creator of Erlang: http://armstrongonsoftware.blogspot.com/2008/05/road-we-didnt-go-down.html http://armstrongonsoftware.blogspot.com/2008/05/road-we-didn... Bye, Joe.

helloooooooo 3y ago

Try DCOM

hmeh 3y ago
Yeah, exactly. DCOM was a failure for the same reason that every other RPC thing is a failure. It turns out that introducing cascading failure or partial failure opportunities due to synchronous remote calls was never a good idea. It was always, and still is, treating the fallacies of distributed computing as first principles.

osigurdson 3y ago

Agree. Hard to abstract away a 3-4 orders of magnitude latency difference. However, even the most obvious /absurd / ugly shell out to curl and hand parse results like code, is going to get wrapped in a normal looking function. Maybe the solution is naming or tooling where: 1) it is easy for a developer to see if a network call exists anywhere in the stack and 2) it isn't possible to introduce a network call into an existing function without breaking contract. This could be as simple as a viral naming convention ("_rpc" suffix for example).

mumblemumble 3y ago

And even then, I no longer believe that "removes as much friction as possible compared to local functions" is a desirable goal in the first place. Over the course of my short 20-year career, I've encountered far too many applications that accumulated terrible performance or reliability problems that were directly attributable to their authors feeling a bit too comfortable about getting chatty with the slow, unreliable remote procedure calls. By contrast, the most successful microservices shop I ever worked at was very conscientious about doing nothing to hide the messiness of RPC. (Ease the implementation, yes, but not hide the messiness.) And that seemed to foster a culture of being much more thoughtful about interaction patterns and data flows and how they might contribute to or detract from the overall performance and stability of the system.

lunarcave 3y ago
One of the author here. I definitely agree that you get markedly lower reliability and performance one you move a local function call to be remote. So, we're definitely not advocating that moving a local function call to be remote is inherently virtuous. However, I've found the abstraction to be useful when someone _has_ already made the decision to refactor their service out to be a separate service.

JonChesterfield 3y ago

There's an exception to this rule if the underlying network does not go wrong. In particular, if the "network" is something local to one machine, pcie or some cross-chip fabric thing, where the failure modes of the "network" are the box falls over, then RPC works as one might hope. You don't have to mangle functions to take account of extra network errors because there are none. The specific use case I'm interested in is heterogenous compute on shared memory machines. Code running on a GPU can call `int rc = fprintf(stderr, "oh dear");` and have no idea whether that runs code on a CPU or not. Code running on a CPU can call some linear algebra function and either run it locally or on a GPU, unobservable to the caller. That is a good trick. It does make for an interesting debugging experience where you lose track of where code is executing. ---- There's a less esoteric application in the area of the OP for pure functions where you don't need the result immediately. That goes something like: 1. Call the function and get back a promise or similar. All runs locally, no failures 2. Something in the background maybe does the work for you 3. Ask for the result. If it's not available yet, do the work locally That has the same zero failure modes property because it'll ultimately run all the work locally if the network offloading never works out. It only really makes sense if the functions can be called multiple times (e.g. locally and remote at the same time) without doing harm, i.e. pure functions. It costs latency and gains throughput. For some batch computation work that is great.

ColonelPhantom 3y ago
Won't the proposed implementation risk doing duplicate work? e.g. another machine starts on your work, but then you ask for it before it's done. So you'd probably need the "not available, but already 'stolen' case". For example, the current machine could look for work to steal itself. But this probably will increase latency a lot, so it would only make sense if the batches are huge.
- lunarcave 3y ago
  It's really hard to have exactly-once delivery[1]. But exactly once processing is viable. With Differential, the implementation does allow for idempotency in cases where you need exactly-once processing. And it does handle the "not available, but already 'stolen' case". Disclaimer: I'm an author. [1] https://news.ycombinator.com/item?id=34986691 https://news.ycombinator.com/item?id=34986691

lunarcave 3y ago

One of the authors here. Thanks for the insight. I generally agree with the fact pretending it's like a local function will cause trouble down the track. After all, you're introducing a number of extra failure modes. The client library however does give some protections to the developer through the type system - although it's not perfect. Hence the wording "feel like". It's not actually local function calls. But it abstracts away a lot of the complexity.

kuschku 3y ago

I'll disagree with that. In fact, I'd argue the vast majority of "local" functions are no better than RPC calls and shouldn't be treated any different. Sure, `sprintf` will run synchronously and finish in known time, but `read` may as well end up waiting for seconds for an HDD to spin up, or for a mounted NFS filesystem to respond. `read` is not any different from an RPC call. And, in fact, `read` can fail in just the same ways as an RPC call can. So can `write`, which is why `sync` exists. Software ignoring these potential issues, and pretending they don't exist, is the source of many unnecessary software hangs and crashes, such as Windows Explorer crashing if your SMB share responds too slowly or Linux being unable to unmount an NFS share if the connection breaks badly. The solution IMO is that we should finally stop hiding asynchronicity behind synchronous facades. If coroutines, promises, async programming, etc are first-class features, it becomes much easier to cleanly model these potential errors and in turn handle them correctly. And once you have that, an rpc call async fn rpcMakePurchase(user: &User, order: &Order) -> Result<Purchase> does feel just like a local function async fn read(path: impl AsRef<Path>) -> Result<Vec<u8>>

valenterry 3y ago
Actually, I think we are on the same side. :-) There is still a small difference though: when I execute a write and sync, then I'm pretty sure it either succeeded or didn't. I'm not sure if there is any way for the disk to say "don't know" which the OS will then pass to my program. Even if, I guess such a case is so rare that it is probably excluded in any error-model. Like cosmic rays. And I can always retry, since it's very very unlikely that there is a connection issue only between my program and the disk but not between other programs and that disk - since they run on the same machine. Over a network things are different, because the connection might be gone for a long time and know I'm not aware of the state and I can't check it. I can also not tell if the connection might come back, the machine I was talking to might have burned down. If I, myself, burn down, then I don't need to worry anyways. That's the difference. This is specific for write/sync though - as you explained it, just because things are running on the same machine does not mean things are necessarily more reliable than a "remote" call.
- kuschku 3y ago

lunarcave 3y ago

One of the authors here. We didn't know we were going to be posted on HN, and we're still iterating on copy that "lands". I don't think we've found the words yet. I agree with the premise that RPC has markedly different characteristics. The actual implementation [1] makes it clear that you're connecting through a client interface. Thanks for the feedback on calling it "removes as much friction as possible compared to local functions". I think we'll trend this way. [1] https://docs.differential.dev/getting-started/quick-start/#3-change-the-greet-command-to-call-the-new-function https://docs.differential.dev/getting-started/quick-start/#3...