4 ms·
Vice: Go channels across many machines
- chenzhekl 9y agoThere was once a network channel in the standard library. But it was later deprecated. Does anyone know the reason?
- kyrra 9y agoRob didn't like the design. https://softwareengineering.stackexchange.com/questions/154004/why-did-golang-discontinue-the-netchan-package https://softwareengineering.stackexchange.com/questions/1540...
- dullgiulio 9y agoIt's not just Rob. Channels don't return errors and network operations can always produce errors.
- ramshanker 9y agoI see a breewing home super computer cluster here.
- packetized 9y agoget go channels to work well on one machine first. also see: heka, deprecation of
- owaislone 9y agoThey are meant to provide easier concurrency mechanism. A way to coordinate concurrent routines. They are not meant to build a data pipeline for processing. I agree that Heka's use of channel was not optimal.
- chewxy 9y agonetchan is back! Let's see how this one fares
- yiyus 9y agoIt looks quite nice. An alternative solution, which requires a modified compiler but avoids the need for a context and errors channel, are the channels used by Clive: https://lsub.org/export/golsub.html https://lsub.org/export/golsub.html I'd really like these channels (or some modified version of them) made it into Go2.
- i_have_to_speak 9y agonRPC [1] provides a higher-level abstraction but only over NATS. Similar to gRPC. [1] http://github.com/rapidloop/nrpc http://github.com/rapidloop/nrpc
- oelmekki 9y agoNot sure I would use it, it seems a bit too much of magic to me (coming from ruby, I really love how everything is explicit in go), but this is certainly a clean and well executed idea. Good job, guys!
- PaulRobinson 9y agoWe have an abstraction library at work we plan to release in coming weeks (although we've been saying that for a year now) called "drumbeat". We have libraries in Ruby, Go and Java, and the concept is that developers should not have to worry about transports or how they work: you make it as simple as possible for most developers in the org to work with distributed systems, and they'll start working with distributed systems, and the arcane understanding of exactly what the trade-offs are between different vendors and messaging technologies can be centralised to a few devs with ops support instead of it being bikeshedded to death. This is nice in that it becomes scalable and transparent with little refactoring in Go, but having multi-language support is useful, otherwise we'd probably drop drumbeat and use this instead. It does suggest we should think about how best to do this in Go again though - I like the pattern a lot.
- lobster_johnson 9y agoHow does this compare to, say, gRPC? One of the ideas in the gRPC space is that the protocol, at the client and server level, is always plain HTTP/2 and DNS. To do discovery (that is, find the IP of a peer), load balancing, routing, retrying, throttling, circuit-breaking etc. you inject a proxy such as Istio [1] in the middle that modifies the connection with all the necessary intelligence. This way, the app stays super simple -- it needs no configuration. If it wants to talk a service called "foo", it just naively dials http://foo/ http://foo/ and does gRPC. If you write lots of apps in different languages, they only need the gRPC client/server glue, nothing else. [1] https://istio.io/ https://istio.io/
- lobster_johnson 9y agoThe idea of using channels for network RPC has come up before, and it's still a bad idea. As jerf points out in the /r/golang thread [1] and this [2] older thread, Go channel semantics don't match network semantics. In particular, Go channels have exactly-once delivery, which is impossible to do over the network. Go channels are designed for communicating between goroutines; they work more like a signaling mechanism than a data pipe mechanism. They have too many design trade-offs to be practical for anything else. They're terrible as general-purpose queues and pub/sub-style topologies, for example. It's tempting to use channels are "iterators" because of range{}, but that's also a terrible idea. Buffered channels should be avoided unless you know exactly what you're doing. And since unbuffered channels block on send, you have to be super careful about goroutine interdependencies; the only way to avoid causing receivers to block senders is to wrap the send call itself in a goroutine, but if you do have slowness, you can accidentally fork millions of goroutines this way. Channel ownership is tricky to get right. And so on. Channels seem trivial on the surface, but they aren't. They look like Unix pipes, but they aren't. (That only one party to the communication can close a channelI consider to be a wart. Trying to close a closed channel panics.) The reason we see people abusing Go channels like this is simple -- channels are generic and support type-safe type matching (with select{}). They sound like a perfect match for certain things. But they really aren't. Channels are really a great argument for why Go could benefit from generics. [1] https://www.reddit.com/r/golang/comments/6q4p1j/comment/dkv6u8v https://www.reddit.com/r/golang/comments/6q4p1j/comment/dkv6... [2] https://www.reddit.com/r/golang/comments/2gecvq/netchan_go_channels_over_a_network/ckix68f/ https://www.reddit.com/r/golang/comments/2gecvq/netchan_go_c...
- MoOmer 9y agoOn the other hand, I use channels successfully for queuing and processing data processing pipelines in many data ingestion applications. There are patterns that work well, and appropriate use of the context package, error receiver channels, and sanity work very well for some problems.