3 ms·

I don't think choosing channel itself is an issue, if it works for the said platform, that is great. What I don't understand is why there should be such a blog

by tw1984 3y ago

I don't think choosing channel itself is an issue, if it works for the said platform, that is great. What I don't understand is why there should be such a blog article telling people that you decided to use channel in your go based program? This is like writing an article telling people that you decided to use the MMU when building an OS.

saisrirampur 3y ago
Thanks for posting this question. As explained in the article we started off with pull/push model (with configurable batching) and it worked well for streaming to data warehouses, where 30s+ was acceptable latency. We added Queues as a supported target, where one of our users wanted single digit second latency. This is when we introduced channels. We agree that it is a small change. But the latency improvements were significant and wanted to share it with broader programming/Go community of Go Channels and the objective impact they can have.
- tw1984 3y ago
  Pull/Push itself is not the problem unless you have stats to back such implied claims. It is far more likely that your highly inefficient Pull/Push implementation caused the problem. Without any real insight on why the pull/push based approach is slower, there is nothing the "Go community" can get. All valuable details were omitted, e.g. 1) how much data is being moved? a few megabytes? what is the state of art latency? maybe 0.001 second for that? when latency is still seconds huge, what can be further improved, in the go runtime or in the apps? etc A highly efficient pull/push model without using go channel can easily beat your 1-5 seconds latency. A crappy implementation using channel can get worse. Delay of several seconds is when you pass on information mouth to mouth, in 2023, that kind of delay doesn't justify a blog article on such fancy implementation. I'd suggest you to delete the implementation to rebuild from ground zero.
  - saisrirampur 3y ago
    Thanks for the comment. We will aim to write a more deeper blog on this in the future. We were observing 60<>40 ratio for Pull/Push, as specified in the blog and majority of the Push was target data-store (network) bound. So there was less room of optimization on the Go side. The test was done with ~10-15K TPS on Postgres and the target was Azure Event Hubs, which has a limit of 1MB write batch size.
  - earthboundkid 3y ago
    Yes, thank you.

by tw1984 3y ago

saisrirampur 3y ago

Thanks for posting this question. As explained in the article we started off with pull/push model (with configurable batching) and it worked well for streaming to data warehouses, where 30s+ was acceptable latency. We added Queues as a supported target, where one of our users wanted single digit second latency. This is when we introduced channels. We agree that it is a small change. But the latency improvements were significant and wanted to share it with broader programming/Go community of Go Channels and the objective impact they can have.

tw1984 3y ago
Pull/Push itself is not the problem unless you have stats to back such implied claims. It is far more likely that your highly inefficient Pull/Push implementation caused the problem. Without any real insight on why the pull/push based approach is slower, there is nothing the "Go community" can get. All valuable details were omitted, e.g. 1) how much data is being moved? a few megabytes? what is the state of art latency? maybe 0.001 second for that? when latency is still seconds huge, what can be further improved, in the go runtime or in the apps? etc A highly efficient pull/push model without using go channel can easily beat your 1-5 seconds latency. A crappy implementation using channel can get worse. Delay of several seconds is when you pass on information mouth to mouth, in 2023, that kind of delay doesn't justify a blog article on such fancy implementation. I'd suggest you to delete the implementation to rebuild from ground zero.
- saisrirampur 3y ago
  Thanks for the comment. We will aim to write a more deeper blog on this in the future. We were observing 60<>40 ratio for Pull/Push, as specified in the blog and majority of the Push was target data-store (network) bound. So there was less room of optimization on the Go side. The test was done with ~10-15K TPS on Postgres and the target was Azure Event Hubs, which has a limit of 1MB write batch size.
- earthboundkid 3y ago
  Yes, thank you.