5 ms·

If the idea is to leverage the Redis protocol for an “API”, what are the benefits to this approach over using the built-in `pub-sub` and the regular old C imple

by dljsjr 5y ago

If the idea is to leverage the Redis protocol for an “API”, what are the benefits to this approach over using the built-in `pub-sub` and the regular old C implementation of Redis Server? I could maybe see some really specific use-cases for this but for probably 90% of cases implementing a distributed process API should be able to go over pubsub fine shouldn’t it? Or does pubsub have some sort of massive overhead? EDIT: Just because it’s tangential to the topic at hand, and since I’m up here at the top, I also wanna throw out a nod to libraries like `nng` and `nanomsg` which are spiritual successors to ZeroMQ and they have functionality like brokerless RPC/pubsub built in as messaging models. I don’t see tools like this talked about a lot in this space cuz systems software isn’t the sexiest but if you need to embed a small and lightweight messaging endpoint in your backend stuff then look at those as well. No horse in the race, just like sharing useful tools with people. https://nng.nanomsg.org https://nng.nanomsg.org

mst 5y ago
If it's straight RPC there's no need for the extra moving parts, and the simpler your topology the fewer ways for it to go wrong. I often build TPC based "protocols" that are just newline delimited json in each direction, it's a nice middle ground between using an HTTP POST and something like gRPC.
- dljsjr 5y ago
  I’m not 100% sure I’m understanding but if the idea is to implement a “traditional”/synchronous RPC mechanism you can do that using the PubSub functionality trivially and I’m not sure which additional moving parts it adds unless you’re in a situation where you have to cluster Redis (which has a reputation for being a PITA but isn’t that hard honestly). The only use case that really jumps out is when you don’t want a broker because you don’t want a single point of failure but now you’re embedding a Redis server implementation in every one of the services in your mesh and I’m not convinced that’s much better but I can see where it might be helpful.
  - anonymousDan 5y ago
    I don't think this is anything to do with Redis server, you are simply using the Redis wire protocol/parser? I could be wrong...
    - hansonkd13 5y ago
      Correct this post is about the Redis Protocol.
    - dljsjr 5y ago
      That’s correct but my point was that Redis has an out-of-the-box solution for “message passing arbitrary data using the Redis protocol over TCP to named endpoints”, and Redis Server is a very lightweight piece of software. Even if you aren’t using it as an in-memory key-value DB it’s not a big problem to pull it in to your stack even if you’re only gonna use it for PubSub or RPC/IPC. This is a super cool write-up and I’m not saying anything negative about what the author did. I like it a lot. I’m just asking from a technical and curiosity perspective what the advantages are to this over using the stuff Redis Server already provides which can do the same thing.
      - hansonkd13 5y ago
        I suppose the main advantages would be less dependencies (no Redis server). Less overhead (you aren’t routing though anything) I haven’t looked at the exact performance characteristics but it would be fun! You would have built in load balancing!
      - mrkurt 5y ago
        With Elixir specifically, you have the option of clustering these things. We (fly.io) send messages between servers using NATS. This works well with geographically distributed infrastructure, messages are somewhat peer to peer. If we were using Redis, we'd need a round trip to a centralized server. And we'd need the internet to always work well. You can do a similar thing with Elixir and your own protocol (or the Redis protocol).
      - skyde 5y ago
hansonkd13 5y ago
This article is about implementing the Redis Protocol on a socket yourself. Not about the Redis Database. Sorry for the confusion.
- dljsjr 5y ago
  I actually did understand that and I think it’s super cool from a technical perspective. I was just curious as to the use case where this was a strong contender to beat out the built-in stuff you can get from Redis server. It’s a “small” implementation but you still have to test it and own it throughtout the lifecycle of the product that’s using it.
  - ellyagg 5y ago
    It's pretty common that you won't have a specific use case in mind when learning something new, but I've often found that later it ends up fitting in somehow.
derefr 5y ago
> and the regular old C implementation of Redis Server? To be clear, this article is talking about implementing your own synchronous-RPC-request server, i.e. a network service that other services talk to through an API over some network wire protocol, to make requests over a socket and then wait for responses to those requests over that same socket. This article assumes that you already know that that's what you need. This article then offers an additional alternative to the traditional wire protocols one might expose to clients in a synchronous-RPC-request server (RESTful HTTP, gRPC, JSON-RPC over HTTP, JSON-RPC over TCP, etc.); namely, mimicking the wire protocol Redis uses, but exposing your own custom Redis commands. This choice allows you to use existing Redis client libraries as your RPC clients, just as writing a RESTful HTTP server allows you to use existing HTTP client libraries as your RPC clients. The alternative to doing so, if you want to call it that, would be to write these custom commands as a Redis module in C. But then you have to structure your code to live inside a Redis server, when that might not be at-all what you want, especially if your code already lives inside some other kind of framework, or is written in a managed-runtime language unsuited to plugging into a C server. Or think of it like this: this article is about taking an existing daemon, written in some arbitrary language (in this case Elixir), where that daemon already speaks some other, slower RPC protocol (e.g. REST over HTTP); and adding an additional Redis-protocol RPC listener to that daemon, so that you can use a Redis client as a drop-in replacement for an HTTP client for doing RPC against the daemon, thus (presumably) lowering per-request protocol overhead for clients that need to pump through a lot of RPC requests. I do realize that you're suggesting that you could use Redis as an event-bus between two processes that each connect to it via the Redis protocol; and then use a "fire an async request as an event over the bus, and then await a response event containing the request's ref to show up on the bus" RPC strategy, ala Erlang's own gen_server:call messaging strategy. All I can say is that, due to there being three processes and two separate RPC sessions involved, with their own wire-protocol encoding/decoding phases, that's likely higher-overhead than even a direct RESTful-HTTP RPC session between the client and the relevant daemon; let alone a direct Redis RPC session between the client and the daemon.
- dljsjr 5y ago
  That’s fair. Although I think the characterization of doing RPC over Redis’s built-in pubsub is a little uncharitable. You’d just fire off a PUBLISH command w/ the channel and payload, and you’d receive the response to the RPC request on a subscriber that you can immediately drop (a pseudo-one-shot channel). It doesn’t have to be an event bus (even though that’s close to how Redis does pubsub internally).

by dljsjr 5y ago

mst 5y ago

If it's straight RPC there's no need for the extra moving parts, and the simpler your topology the fewer ways for it to go wrong. I often build TPC based "protocols" that are just newline delimited json in each direction, it's a nice middle ground between using an HTTP POST and something like gRPC.

dljsjr 5y ago
I’m not 100% sure I’m understanding but if the idea is to implement a “traditional”/synchronous RPC mechanism you can do that using the PubSub functionality trivially and I’m not sure which additional moving parts it adds unless you’re in a situation where you have to cluster Redis (which has a reputation for being a PITA but isn’t that hard honestly). The only use case that really jumps out is when you don’t want a broker because you don’t want a single point of failure but now you’re embedding a Redis server implementation in every one of the services in your mesh and I’m not convinced that’s much better but I can see where it might be helpful.
- anonymousDan 5y ago
  I don't think this is anything to do with Redis server, you are simply using the Redis wire protocol/parser? I could be wrong...
  - hansonkd13 5y ago
    Correct this post is about the Redis Protocol.
  - dljsjr 5y ago
    That’s correct but my point was that Redis has an out-of-the-box solution for “message passing arbitrary data using the Redis protocol over TCP to named endpoints”, and Redis Server is a very lightweight piece of software. Even if you aren’t using it as an in-memory key-value DB it’s not a big problem to pull it in to your stack even if you’re only gonna use it for PubSub or RPC/IPC. This is a super cool write-up and I’m not saying anything negative about what the author did. I like it a lot. I’m just asking from a technical and curiosity perspective what the advantages are to this over using the stuff Redis Server already provides which can do the same thing.
    - hansonkd13 5y ago
      I suppose the main advantages would be less dependencies (no Redis server). Less overhead (you aren’t routing though anything) I haven’t looked at the exact performance characteristics but it would be fun! You would have built in load balancing!
    - mrkurt 5y ago
      With Elixir specifically, you have the option of clustering these things. We (fly.io) send messages between servers using NATS. This works well with geographically distributed infrastructure, messages are somewhat peer to peer. If we were using Redis, we'd need a round trip to a centralized server. And we'd need the internet to always work well. You can do a similar thing with Elixir and your own protocol (or the Redis protocol).
    - skyde 5y ago

hansonkd13 5y ago

This article is about implementing the Redis Protocol on a socket yourself. Not about the Redis Database. Sorry for the confusion.

dljsjr 5y ago
I actually did understand that and I think it’s super cool from a technical perspective. I was just curious as to the use case where this was a strong contender to beat out the built-in stuff you can get from Redis server. It’s a “small” implementation but you still have to test it and own it throughtout the lifecycle of the product that’s using it.
- ellyagg 5y ago
  It's pretty common that you won't have a specific use case in mind when learning something new, but I've often found that later it ends up fitting in somehow.

derefr 5y ago

> and the regular old C implementation of Redis Server? To be clear, this article is talking about implementing your own synchronous-RPC-request server, i.e. a network service that other services talk to through an API over some network wire protocol, to make requests over a socket and then wait for responses to those requests over that same socket. This article assumes that you already know that that's what you need. This article then offers an additional alternative to the traditional wire protocols one might expose to clients in a synchronous-RPC-request server (RESTful HTTP, gRPC, JSON-RPC over HTTP, JSON-RPC over TCP, etc.); namely, mimicking the wire protocol Redis uses, but exposing your own custom Redis commands. This choice allows you to use existing Redis client libraries as your RPC clients, just as writing a RESTful HTTP server allows you to use existing HTTP client libraries as your RPC clients. The alternative to doing so, if you want to call it that, would be to write these custom commands as a Redis module in C. But then you have to structure your code to live inside a Redis server, when that might not be at-all what you want, especially if your code already lives inside some other kind of framework, or is written in a managed-runtime language unsuited to plugging into a C server. Or think of it like this: this article is about taking an existing daemon, written in some arbitrary language (in this case Elixir), where that daemon already speaks some other, slower RPC protocol (e.g. REST over HTTP); and adding an additional Redis-protocol RPC listener to that daemon, so that you can use a Redis client as a drop-in replacement for an HTTP client for doing RPC against the daemon, thus (presumably) lowering per-request protocol overhead for clients that need to pump through a lot of RPC requests. I do realize that you're suggesting that you could use Redis as an event-bus between two processes that each connect to it via the Redis protocol; and then use a "fire an async request as an event over the bus, and then await a response event containing the request's ref to show up on the bus" RPC strategy, ala Erlang's own gen_server:call messaging strategy. All I can say is that, due to there being three processes and two separate RPC sessions involved, with their own wire-protocol encoding/decoding phases, that's likely higher-overhead than even a direct RESTful-HTTP RPC session between the client and the relevant daemon; let alone a direct Redis RPC session between the client and the daemon.

dljsjr 5y ago
That’s fair. Although I think the characterization of doing RPC over Redis’s built-in pubsub is a little uncharitable. You’d just fire off a PUBLISH command w/ the channel and payload, and you’d receive the response to the RPC request on a subscriber that you can immediately drop (a pseudo-one-shot channel). It doesn’t have to be an event bus (even though that’s close to how Redis does pubsub internally).