3 ms·

Here's what I've seen: 1. HTTP/JSON, with resiliency semantics layered on via SmartStack, Hystrix, or DIY library. 2. GRPC/protobuf. Haven't seen anyone other

by rdli 11y ago

Here's what I've seen: 1. HTTP/JSON, with resiliency semantics layered on via SmartStack, Hystrix, or DIY library. 2. GRPC/protobuf. Haven't seen anyone other than Google run this in production, and Google doesn't really use gRPC (they use Stubby). 3. Finagle/Thrift. Twitter and some other folks use this. 4. Async message bus. Not as popular because it's not point-to-point, but it works. It's technically not RPC so it requires a different architectural approach that lots of people aren't willing to adopt. 5. Super fancy custom RPC protocol. Uber does this, for example. As does Google. You need lots of engineering to do this well, though. <plug> We've seen these design patterns over and over again, which is why we're building Quark (github.com/datawire/quark) which is a language for you to build your own custom RPC protocol (case #5 or to a lesser extent #1). We're also building Datawire Connect on top of Quark which gives you resilient RPC semantics in a way that's compatible with your existing tech stack. </plug>

lobster_johnson 11y ago
Quark looks quite interesting, if odd — it's an entire language, with classes and generics and everything? Just for implementing protocol glue? I think it's something I'd like to revisit once it's matured, as your set of languages is quite small at the moment, and I think it's overkill for us at the moment. In particular, I'm much more concerned about connection and discovery semantics than about things like wire protocol and how to express the "schema".
- meddlepal 11y ago
  Disclosure: Datawire employee Quark is a protocol implementation language. If you think of a protocol as being defined by something like an RFC (or RFC equivalent) then RFC's are fairly complex because of English and ambiguity. Quark is trying to help you define protocol's by encoding them in a language more familiar to programmers and computers and at the same time generate language-native bindings.
- rdli 11y ago
  If you're more worried about connection and discovery semantics, you'd want to use Datawire Connect which is built on top of Quark. Quark is being used by companies that for one reason or another need/want to build their own protocol. This is admittedly overkill for most people. Datawire Connect integrates discovery as well as connection semantics (next week we'll have a release that supports full Hystrix semantics, e.g., circuit breakers and timeouts and load balancing).
  - lobster_johnson 11y ago
    Looks interesting. As with Quark, I think we'd hve to wait until you have reached a higher level of maturity and a broader selection of languages. We need at least Go, Ruby and Node.js to get going, which is what our current apps our built with. We're also evaluating Elixir and Swift as possible future languages.
    - rdli 11y ago
      Great feedback! We will have Ruby in a few weeks (code is sitting on a GitHub branch), and we have Node already. Go is on the roadmap, but we also have a proxy that lets you interface with languages we don't support natively. I'd be super interested in what tech you find that works well with your language set of Go, Ruby, and NodeJS. GRPC won't. Thrift might (don't know offhand). NATS might, but there's lots of coding there you need as you mention and it's a SPOF.
      - lobster_johnson 11y ago
        gRPC does have native bindings for all of Go, Ruby and Node.js. Not sure why you say it won't work well? NATS being a SPOF isn't necessarily a problem for us if it's stable, which it seems to be. We are already completely dependent on HAProxy as a SPOF, anyway. Moreover, most client bindings seem to support providing a host list they use for failover. I will take a closer look at Quark and Datawire Connect when I have a spare moment. Do you have it running in production? Happy to hear that Go is on the roadmap.

by rdli 11y ago

lobster_johnson 11y ago

Quark looks quite interesting, if odd — it's an entire language, with classes and generics and everything? Just for implementing protocol glue? I think it's something I'd like to revisit once it's matured, as your set of languages is quite small at the moment, and I think it's overkill for us at the moment. In particular, I'm much more concerned about connection and discovery semantics than about things like wire protocol and how to express the "schema".

meddlepal 11y ago
Disclosure: Datawire employee Quark is a protocol implementation language. If you think of a protocol as being defined by something like an RFC (or RFC equivalent) then RFC's are fairly complex because of English and ambiguity. Quark is trying to help you define protocol's by encoding them in a language more familiar to programmers and computers and at the same time generate language-native bindings.
rdli 11y ago
If you're more worried about connection and discovery semantics, you'd want to use Datawire Connect which is built on top of Quark. Quark is being used by companies that for one reason or another need/want to build their own protocol. This is admittedly overkill for most people. Datawire Connect integrates discovery as well as connection semantics (next week we'll have a release that supports full Hystrix semantics, e.g., circuit breakers and timeouts and load balancing).
- lobster_johnson 11y ago
  Looks interesting. As with Quark, I think we'd hve to wait until you have reached a higher level of maturity and a broader selection of languages. We need at least Go, Ruby and Node.js to get going, which is what our current apps our built with. We're also evaluating Elixir and Swift as possible future languages.
  - rdli 11y ago
    Great feedback! We will have Ruby in a few weeks (code is sitting on a GitHub branch), and we have Node already. Go is on the roadmap, but we also have a proxy that lets you interface with languages we don't support natively. I'd be super interested in what tech you find that works well with your language set of Go, Ruby, and NodeJS. GRPC won't. Thrift might (don't know offhand). NATS might, but there's lots of coding there you need as you mention and it's a SPOF.
    - lobster_johnson 11y ago
      gRPC does have native bindings for all of Go, Ruby and Node.js. Not sure why you say it won't work well? NATS being a SPOF isn't necessarily a problem for us if it's stable, which it seems to be. We are already completely dependent on HAProxy as a SPOF, anyway. Moreover, most client bindings seem to support providing a host list they use for failover. I will take a closer look at Quark and Datawire Connect when I have a spare moment. Do you have it running in production? Happy to hear that Go is on the roadmap.