4 ms·

Is there a name for the pattern where your client retrieves a subset of the available instances via service discovery, establishes connections to every instance

by strken 5y ago

Is there a name for the pattern where your client retrieves a subset of the available instances via service discovery, establishes connections to every instance in the subset, and spreads requests between them? E.g. you have 200 clients connected to 100 servers, your service discovery sends each client a list of 10 servers, and 2000 connections are established instead of 20000. "Partial mesh" is a good description, but it's mostly used to describe routing, not load balancing.

herval 5y ago
Finagle calls this "aperture" (https://twitter.github.io/finagle/guide/ApertureLoadBalancers.html https://twitter.github.io/finagle/guide/ApertureLoadBalancer...)
- morelisp 5y ago
  This seems to still rely on (or at least, describes) the client making the determination of what to talk to, not the mesh / service discovery.
  - herval 5y ago
    Well, it does exactly what op wrote: "your client retrieves a subset of the available instances via service discovery, establishes connections to every instance in the subset, and spreads requests between them"
    - morelisp 5y ago
      The article you linked describes retrieving all instances and establishing connections to a subset of them, not retrieving a subset of them. To achieve deterministic apertures it further needs both the complete serverset and the complete peerset.
jayd16 5y ago
Is this not the Look-Aside pattern described in TFA?
morelisp 5y ago
What would be the advantage of doing this at the service discovery level (filtering what the client sees) as opposed to the client level (filtering what the client uses)?
- jayd16 5y ago
  You can usually push server changes faster then client changes.
  - morelisp 5y ago
    I haven't been in the app development space in ages, are people really exposing their service meshes to public clients? I have trouble imagining a case where external clients could cause sufficient load for this to matter, but also where relying on the client to do it right wouldn't be a DoS risk.
    - AtNightWeCode 5y ago
      At least one reversed proxy layer between the client and the server is standard I would say.
jeffbee 5y ago
Nobody should be afraid of 20000 connections or even 20 billion connections. The cost of having a lot of TCP connections hanging around are wildly over-estimated. You should not even think about subsetting unless/until you can measure the cost in terms of CPU spent on keepalive messages or memory spent on connection state, or you're getting more than millions of file descriptors on a single machine. Even if you are subsetting, very small subsets (like 10) are definitely harmful, and hiding the full set of candidate backends is also harmful. If you are subsetting, allow your clients to observe the global state and make their own decisions. What's it going to cost you? A few thousand IP addresses and port numbers in memory? That's basically nothing.
- morelisp 5y ago
  I also struggle to see the case the parent is trying to solve, but I would be concerned purely on a memory basis if my service opens even millions of unnecessary connections, let alone 20 billion.
  - jeffbee 5y ago
    Across what size of fleet? It's meaningless to say you'd be concerned about millions of connected sockets, except to the extent that it was a significant fraction of total system cost.
    - mumblemumble 5y ago
      You've got at least the size of the read and write for every open connection to contend with. Let's be real conservative and clamp that down to 4KiB each. So that's 8K per connection. Meaning that a rough lower bound on the memory usage under this scenario would be something like ~8GiB per million open connections, and ~8TiB per billion open connections. You're right that it comes down to the extent to which that's a significant fraction of total system cost. I think I can say with reasonable confidence, though, that this would be a significant fraction of total system cost for the vast majority of systems.
      - 5y ago
AtNightWeCode 5y ago
Not sure why you want to do this. In normal RPC over messaging you would use the competing consumers pattern which also solves the connection issues. But I don't think it exists in gRPC.

by strken 5y ago

herval 5y ago

Finagle calls this "aperture" (https://twitter.github.io/finagle/guide/ApertureLoadBalancers.html https://twitter.github.io/finagle/guide/ApertureLoadBalancer...)

morelisp 5y ago
This seems to still rely on (or at least, describes) the client making the determination of what to talk to, not the mesh / service discovery.
- herval 5y ago
  Well, it does exactly what op wrote: "your client retrieves a subset of the available instances via service discovery, establishes connections to every instance in the subset, and spreads requests between them"
  - morelisp 5y ago
    The article you linked describes retrieving all instances and establishing connections to a subset of them, not retrieving a subset of them. To achieve deterministic apertures it further needs both the complete serverset and the complete peerset.

jayd16 5y ago

Is this not the Look-Aside pattern described in TFA?

morelisp 5y ago

What would be the advantage of doing this at the service discovery level (filtering what the client sees) as opposed to the client level (filtering what the client uses)?

jayd16 5y ago
You can usually push server changes faster then client changes.
- morelisp 5y ago
  I haven't been in the app development space in ages, are people really exposing their service meshes to public clients? I have trouble imagining a case where external clients could cause sufficient load for this to matter, but also where relying on the client to do it right wouldn't be a DoS risk.
  - AtNightWeCode 5y ago
    At least one reversed proxy layer between the client and the server is standard I would say.

jeffbee 5y ago

Nobody should be afraid of 20000 connections or even 20 billion connections. The cost of having a lot of TCP connections hanging around are wildly over-estimated. You should not even think about subsetting unless/until you can measure the cost in terms of CPU spent on keepalive messages or memory spent on connection state, or you're getting more than millions of file descriptors on a single machine. Even if you are subsetting, very small subsets (like 10) are definitely harmful, and hiding the full set of candidate backends is also harmful. If you are subsetting, allow your clients to observe the global state and make their own decisions. What's it going to cost you? A few thousand IP addresses and port numbers in memory? That's basically nothing.

morelisp 5y ago
I also struggle to see the case the parent is trying to solve, but I would be concerned purely on a memory basis if my service opens even millions of unnecessary connections, let alone 20 billion.
- jeffbee 5y ago
  Across what size of fleet? It's meaningless to say you'd be concerned about millions of connected sockets, except to the extent that it was a significant fraction of total system cost.
  - mumblemumble 5y ago
    You've got at least the size of the read and write for every open connection to contend with. Let's be real conservative and clamp that down to 4KiB each. So that's 8K per connection. Meaning that a rough lower bound on the memory usage under this scenario would be something like ~8GiB per million open connections, and ~8TiB per billion open connections. You're right that it comes down to the extent to which that's a significant fraction of total system cost. I think I can say with reasonable confidence, though, that this would be a significant fraction of total system cost for the vast majority of systems.
    - 5y ago

AtNightWeCode 5y ago

Not sure why you want to do this. In normal RPC over messaging you would use the competing consumers pattern which also solves the connection issues. But I don't think it exists in gRPC.