9 ms·

Thanks for your insights. Clearly, I should have put more thought about the whole project before showing it to the world. You're absolutely right on most points

Thanks for your insights. Clearly, I should have put more thought about the whole project before showing it to the world. You're absolutely right on most points, this introduces more moving parts, another unnecessary service, and defeats the purpose of an RPC. In fact, I have had to implement my own XML-RPC client (and an RPC-over-HTTP client too) on a certain occasion, and I tried to remember my gripes on that occasion and avoid them with this approach. Regarding your last line: > So you say Redis acts as a magic faerie dust that eases integration process? How? I guess what I said was too vague. It should have been "it makes an RPC-like system easier to integrate when you already have Redis". Shisetsu is not a magic bullet. It won't make things magically easier for everyone. All it is really is a niche project that, given you already use Python and Redis, is something for you to consider. If not, then you can move along, there's absolutely nothing to see here. :)

dozzie 10y ago
>> So you say Redis acts as a magic faerie dust that eases integration process? How? > I guess what I said was too vague. It should have been "it makes an RPC-like system easier to integrate when you already have Redis". OK, let's work with this: in what way is Redis helping here over XML-RPC or JSON-RPC? What kind of problems does it solve that are pain in the ass with the other RPC protocols? (And now don't take my word that it doesn't, try to justify your choice, and next time start your README with that.) I'm not against custom protocols or adding daemons to a system (I've recently(ish) written a set of RPC daemons with their own protocols; the daemons are replacing Celery task queue, which itself is a widely tested software), they just have a decent reason to be introduced. Additional complexity just needs to pay for itself somehow.
- kixpanganiban 10y ago
  Thanks man, I really appreciate you giving me points and guiding me here. > OK, let's work with this: in what way is Redis helping here over XML-RPC or JSON-RPC? What kind of problems does it solve that are pain in the ass with the other RPC protocols? Redis vs XML-RPC/JSON-RPC is a mismatch. It's more of a msgpack-RPC with Redis as broker vs XML-RPC or JSON-RPC: 1. Most brokerless RPC protocols make it difficult to inspect the message on the fly. With Redis, you can open up redis-cli and subscribe to the appropriate channels to extract the raw messages. For most users this might not mean anything, but for others this could mean easier debugging, measuring performance, and so on. 2. msgpack's serialized format is small. This makes it faster to send it across a network especially when the message is considerably large (vs, say, JSON or XML). That, and msgpack itself performs serialization/deserialization quickly on its own. see: http://webcache.googleusercontent.com/search?q=cache:hYaRwEUpbdwJ:maxondev.com/serialization-performance-comparison-c-net-formats-frameworks-xmldatacontractserializer-xmlserializer-binaryformatter-json-newtonsoft-servicestack-text/+&cd=2&hl=en&ct=clnk&gl=ph http://webcache.googleusercontent.com/search?q=cache:hYaRwEU... (I'm using a Google Cached link because main link seems to be down) > I'm not against custom protocols or adding daemons to a system (I've recently(ish) written a set of RPC daemons with their own protocols; the daemons are replacing Celery task queue, which itself is a widely tested software), they just have a decent reason to be introduced. Additional complexity just needs to pay for itself somehow I totally agree. However, the idea of introducing Redis into the equation to use as the broker didn't seem like additional complexity to me because a lot of other network messaging protocols rely on a broker anyway. For example, the widely popular Web App Messaging Protocol (WAMP) (http://wamp-proto.org/why/ http://wamp-proto.org/why/) relies on brokers to decouple callee/caller on an RPC operation. ZeroRPC, another popular RPC choice, uses ZMQ. For Shisetsu, what I see is the main advantage of Redis is that I can delegate multiplexing work to it instead of having to code myself, making the protocol and its implementation easier to manage, just by relying on Redis' PubSub. The client and server don't even have to know each other's address and port, they just have to know which Redis instance to connect to and assume that there exists one of the other to work with. Redis also provides the channel buffer and the queue instead of me having to write those as part of the client/server code, which means anyone else looking to implement the protocol in another language wouldn't have to deal with those as well.
  - dozzie 10y ago
    > Redis vs XML-RPC/JSON-RPC is a mismatch. It's more of a msgpack-RPC with Redis as broker vs XML-RPC or JSON-RPC: I used it as a sort of mental shortcut. MessagePack is very similar in data model to JSON or what XML-RPC allows, so I treat them as essentially the same. I was comparing the architectures where the client connects directly to RPC server, as opposed to connecting to a third party (broker, Redis in this case), which is supposed to relay the communication. > 1. Most brokerless RPC protocols make it difficult to inspect the message on the fly. With Redis, you can open up redis-cli and subscribe to the appropriate channels to extract the raw messages. For most users this might not mean anything, but for others this could mean easier debugging, measuring performance, and so on. This sort of things is usually implemented using logging, but I agree that while usually it's not an important feature, there are cases when it is extremely useful. > 2. msgpack's serialized format is small. [...] Though this doesn't concern Redis, and could be a wire protocol over raw TCP. > [...] the idea of introducing Redis into the equation to use as the broker didn't seem like additional complexity to me because a lot of other network messaging protocols rely on a broker anyway. But we're not talking about messaging protocols, we're talking about RPC protocols/systems. You introduce two services (Redis + RPC server) for a task that usually was carried out by one. This is why I claim it's greater complexity than usual. > ZeroRPC, another popular RPC choice, uses ZMQ. I'm not sure it is that popular. Thrift, Protocol Buffers, and gRPC seem much more widely used, and XML-RPC is a little longer on the market, so it probably got a fair share of RPC use. And ZeroRPC doesn't seem use broker, just a ZeroMQ protocol. ZMQ is in this setting more like a transport protocol than a separate service to dispatch requests and responses. > The client and server don't even have to know each other's address and port, they just have to know which Redis instance to connect to and assume that there exists one of the other to work with. So instead of knowing an address and port of an RPC server, client needs to know address and port of Redis. This doesn't seem to be much of a gain. At best, Redis acts as something analogous to port mapper in this setting (though it relays all the communication instead of telling the client where to connect to). > Redis also provides the channel buffer and the queue instead of me having to write those as part of the client/server code, which means anyone else looking to implement the protocol in another language wouldn't have to deal with those as well. Job queueing is somewhat outside of the RPC protocol, though I agree it can be useful. The RPC system I mentioned I wrote does have a queue and we use it. Though I don't quite understand what you mean under "channel buffer" term.

dozzie 10y ago

>> So you say Redis acts as a magic faerie dust that eases integration process? How? > I guess what I said was too vague. It should have been "it makes an RPC-like system easier to integrate when you already have Redis". OK, let's work with this: in what way is Redis helping here over XML-RPC or JSON-RPC? What kind of problems does it solve that are pain in the ass with the other RPC protocols? (And now don't take my word that it doesn't, try to justify your choice, and next time start your README with that.) I'm not against custom protocols or adding daemons to a system (I've recently(ish) written a set of RPC daemons with their own protocols; the daemons are replacing Celery task queue, which itself is a widely tested software), they just have a decent reason to be introduced. Additional complexity just needs to pay for itself somehow.

kixpanganiban 10y ago
Thanks man, I really appreciate you giving me points and guiding me here. > OK, let's work with this: in what way is Redis helping here over XML-RPC or JSON-RPC? What kind of problems does it solve that are pain in the ass with the other RPC protocols? Redis vs XML-RPC/JSON-RPC is a mismatch. It's more of a msgpack-RPC with Redis as broker vs XML-RPC or JSON-RPC: 1. Most brokerless RPC protocols make it difficult to inspect the message on the fly. With Redis, you can open up redis-cli and subscribe to the appropriate channels to extract the raw messages. For most users this might not mean anything, but for others this could mean easier debugging, measuring performance, and so on. 2. msgpack's serialized format is small. This makes it faster to send it across a network especially when the message is considerably large (vs, say, JSON or XML). That, and msgpack itself performs serialization/deserialization quickly on its own. see: http://webcache.googleusercontent.com/search?q=cache:hYaRwEUpbdwJ:maxondev.com/serialization-performance-comparison-c-net-formats-frameworks-xmldatacontractserializer-xmlserializer-binaryformatter-json-newtonsoft-servicestack-text/+&cd=2&hl=en&ct=clnk&gl=ph http://webcache.googleusercontent.com/search?q=cache:hYaRwEU... (I'm using a Google Cached link because main link seems to be down) > I'm not against custom protocols or adding daemons to a system (I've recently(ish) written a set of RPC daemons with their own protocols; the daemons are replacing Celery task queue, which itself is a widely tested software), they just have a decent reason to be introduced. Additional complexity just needs to pay for itself somehow I totally agree. However, the idea of introducing Redis into the equation to use as the broker didn't seem like additional complexity to me because a lot of other network messaging protocols rely on a broker anyway. For example, the widely popular Web App Messaging Protocol (WAMP) (http://wamp-proto.org/why/ http://wamp-proto.org/why/) relies on brokers to decouple callee/caller on an RPC operation. ZeroRPC, another popular RPC choice, uses ZMQ. For Shisetsu, what I see is the main advantage of Redis is that I can delegate multiplexing work to it instead of having to code myself, making the protocol and its implementation easier to manage, just by relying on Redis' PubSub. The client and server don't even have to know each other's address and port, they just have to know which Redis instance to connect to and assume that there exists one of the other to work with. Redis also provides the channel buffer and the queue instead of me having to write those as part of the client/server code, which means anyone else looking to implement the protocol in another language wouldn't have to deal with those as well.
- dozzie 10y ago
  > Redis vs XML-RPC/JSON-RPC is a mismatch. It's more of a msgpack-RPC with Redis as broker vs XML-RPC or JSON-RPC: I used it as a sort of mental shortcut. MessagePack is very similar in data model to JSON or what XML-RPC allows, so I treat them as essentially the same. I was comparing the architectures where the client connects directly to RPC server, as opposed to connecting to a third party (broker, Redis in this case), which is supposed to relay the communication. > 1. Most brokerless RPC protocols make it difficult to inspect the message on the fly. With Redis, you can open up redis-cli and subscribe to the appropriate channels to extract the raw messages. For most users this might not mean anything, but for others this could mean easier debugging, measuring performance, and so on. This sort of things is usually implemented using logging, but I agree that while usually it's not an important feature, there are cases when it is extremely useful. > 2. msgpack's serialized format is small. [...] Though this doesn't concern Redis, and could be a wire protocol over raw TCP. > [...] the idea of introducing Redis into the equation to use as the broker didn't seem like additional complexity to me because a lot of other network messaging protocols rely on a broker anyway. But we're not talking about messaging protocols, we're talking about RPC protocols/systems. You introduce two services (Redis + RPC server) for a task that usually was carried out by one. This is why I claim it's greater complexity than usual. > ZeroRPC, another popular RPC choice, uses ZMQ. I'm not sure it is that popular. Thrift, Protocol Buffers, and gRPC seem much more widely used, and XML-RPC is a little longer on the market, so it probably got a fair share of RPC use. And ZeroRPC doesn't seem use broker, just a ZeroMQ protocol. ZMQ is in this setting more like a transport protocol than a separate service to dispatch requests and responses. > The client and server don't even have to know each other's address and port, they just have to know which Redis instance to connect to and assume that there exists one of the other to work with. So instead of knowing an address and port of an RPC server, client needs to know address and port of Redis. This doesn't seem to be much of a gain. At best, Redis acts as something analogous to port mapper in this setting (though it relays all the communication instead of telling the client where to connect to). > Redis also provides the channel buffer and the queue instead of me having to write those as part of the client/server code, which means anyone else looking to implement the protocol in another language wouldn't have to deal with those as well. Job queueing is somewhat outside of the RPC protocol, though I agree it can be useful. The RPC system I mentioned I wrote does have a queue and we use it. Though I don't quite understand what you mean under "channel buffer" term.