6 ms·

Building a Streaming Platform in Go for Postgres

167 points by saisrirampur 3y ago

earthboundkid 3y ago
Moving to channels seems like a weird choice. Channels should basically always be an implementation detail not exposed in public APIs.
- 38 3y ago
  Seems fine to me. What's the problem?
- cauchyk 3y ago
  Author of the blog here, curious what a better alternative would be in this context. The channel has to be passed around for the producer and consumer to interface with each other. Are there better patterns for this?
  - candiddevmike 3y ago
    Why have consumers and producers vs doing it all in one goroutine, utilizing some kind of connection pool?
    - cauchyk 3y ago
      To get higher throughput we would need one goroutine to pull from the replication slot while the other is pushing to the target. The idea is to keep the Postgres connection useful and reading the slot while also pushing to the target asynchronously.
    - reactordev 3y ago
      Because then you are consuming, or producing, you can’t do both at the same time. You are either reading from a stream of data, or you are writing it. Using goroutines to separate these allows you to do both at the same time, as soon as data is available on the channel or you receive the signal to stop.
  - atombender 3y ago
    Not the parent, but I personally dislike it when Go libraries use channels in their public APIs, as it forces a specific concurrency model on the consumer; in particular, channels are quite slow, being protected by an internal mutex, so you're always paying for the overhead no matter if you need it or not. You also have to be very careful about managing the channel lifecycle. If you're not pulling (selecting from) the channel, the library will be permanently stuck. So you must now have a way to tell the library to stop sending, and it must cancel any in-flight send operations if you call producer.Stop() or whatever. In my experience libraries often have bugs in their channel code. It's far too easy to get deadlocks with channels that have interdependencies, and you have to be very careful about buffered versus unbuffered channels, as they behave differently. A better API, in my opinion, is to offer a callback or single-method interface. Then the implementer of that callback or interface can choose to use channels internally if they desire, or they can use something else. You get the same backpressure support since you can treat it as synchronous. After all, a channel's send interface is essentially just: type Channel[T any] interface { Send(T) } But a "chan T" doesn't offer this flexibility. My rule of thumb for channels is that they're goroutine glue, not an API primitive. Build APIs out of interfaces, not channels. The only thing that uses channels should be the one that's controlling the goroutines, because it's the thing that orchestrates them. That said, it's not a hard rule. There are places where channels may have their place in a public API, though I'm not sure I can think of any examples off-hand.
    - foobiekr 3y ago
      this breaks select to send and is a terrible reduction in capability. you can always wrap channels to make them worse and less capable, but your API should expose the more capable option.
      - __turbobrew__ 3y ago
        I think it is a matter of preference. For me personally I use raw channels and goroutines all day every day and I really like using them. Channels are a core primitive in golang so I think it is worth getting familiar with them. As you say being able to select is really nice too.
  - skybrian 3y ago
    Looks like the channel field is private in CDCRecordStream, but exposed by GetRecords. The callers mostly loop over Record objects. [1] If I wanted to encapsulate iterating over a channel of Records, maybe it would be something like Go's io.Pipe function [2], which returns a PipeReader and PipeWriter? Except that it would work on Records rather than byte streams. I don't have enough context to know if the extra encapsulation is a good idea in this case, though. [1] https://github.com/search?q=repo%3APeerDB-io%2Fpeerdb%20GetRecords&type=code https://github.com/search?q=repo%3APeerDB-io%2Fpeerdb%20GetR... [2] https://pkg.go.dev/io#Pipe https://pkg.go.dev/io#Pipe
  - JyB 3y ago
    Please see this great talk by Bryan C. Mills touching on the subject: https://youtu.be/5zXAHh5tJqQ?t=421 https://youtu.be/5zXAHh5tJqQ?t=421
  - earthboundkid 3y ago
    Use an iterator object that can use channels behind the scenes.
- reactordev 3y ago
  Using channels to talk between goroutines is the perfect choice. There’s plenty of go API’s that require a chan of a type.
  - treyd 3y ago
    Go's channels also have a lot of footguns and should be abstracted in a handle type in a public API.
    - reactordev 3y ago
      Except it itself is exposed via a public API. Context interface also exposes a chan Done() <- chan struct{} What are you trying to say? That using a chan as a function argument can shoot yourself in the foot? The whole idea behind channels is to not shoot yourself in the foot with mutex's, deadlocks, etc. What are you trying to say? Wrapping a chan in a type handle is safer than passing a chan of a type? Nope.
      - foobiekr 3y ago
        exactly. you can't abstract over channels easily unless you're missing the point of channels.
        earthboundkid 3y ago
        I think the rule is: don't use a channel unless at some point you need a select with 2+ branches. The context.Done is a case in point. You're not just waiting around for a context to finish. You do two things: try some network request and wait to see if the context cancels before it finishes. Because it's made to work in a select block like that, there wasn't another, better thing to use than a channel for context.Done. But for just iterating through some stream, you take a context in and return an iterator object that will do all the channel stuff internally.
      - treyd 3y ago
        That's not at all what I was referring to. It's not merely having channels as function arguments. Channels broadly have footguns around how closed and nil channels are treated and you can run into work contention/starvation issues accidentally, since they do have their own throughput limits. And wrapping them in your own handle types that's specific to the library's use case allows you to deal with those issues and swap out the implementation of the cross-goroutine bits without breaking the downstream API.
- deleted 3y ago
  [deleted]
- xarope 3y ago
  channels seemed to fit very nicely in a gRPC streaming API call. But gRPC is not for everyone.
- AlphaSite 3y ago
  I think there are exceptions, for example Gos contexts expose a channel.
DonnyV 3y ago
I wonder if they have a setup without Docker. Not a fan, wonder why they even bother with Docker if they're using GO. Thats one of the great things about GO. Easy distributable that works on all platforms.
- cauchyk 3y ago
  Not all of our code is in Go. PeerDB has multiple components: the workers, the UI and the query layer. Some of which is in Rust, Go and Typescript. While it would certainly be possible to package them into individual binaries, I found it significantly easier to define the stack in a docker compose file with the requisite environment setup.
steno132 3y ago
[flagged]
- the_gipsy 3y ago
  Java more productive than go?
  - bashtoni 3y ago
    If you judge productivity by lines of code, absolutely. https://github.com/Hello-World-EE/Java-Hello-World-Enterprise-Edition https://github.com/Hello-World-EE/Java-Hello-World-Enterpris... is an excellent demonstration of this.
    - steno132 3y ago
      Java developers tend to produce more value since they can write the code in a more concise way using Lambdas, functional interfaces, record classes, etc. Overengineering like in your example above can be done in any language, but shouldn't.
      - fsaintjacques 3y ago
        If only go had lambdas, functional interfaces and record classes...
- bashtoni 3y ago
  I strongly disagree with this view for applications, and mildly disagree for infrastructure projects. There is one thing I believe Go is by far and away the leader at: keeping codebases maintainable over time. Go is designed very clearly with this goal and it's excellent at it. The verbosity you complain about makes it much, much clearer what something is doing. That coupled with some of the baked in opinions means it is significantly more straightforward to pick up code that you or someone else wrote a few years ago and modify it.
  - 3y ago
tw1984 3y ago
I don't think choosing channel itself is an issue, if it works for the said platform, that is great. What I don't understand is why there should be such a blog article telling people that you decided to use channel in your go based program? This is like writing an article telling people that you decided to use the MMU when building an OS.
- saisrirampur 3y ago
  Thanks for posting this question. As explained in the article we started off with pull/push model (with configurable batching) and it worked well for streaming to data warehouses, where 30s+ was acceptable latency. We added Queues as a supported target, where one of our users wanted single digit second latency. This is when we introduced channels. We agree that it is a small change. But the latency improvements were significant and wanted to share it with broader programming/Go community of Go Channels and the objective impact they can have.
  - tw1984 3y ago
    Pull/Push itself is not the problem unless you have stats to back such implied claims. It is far more likely that your highly inefficient Pull/Push implementation caused the problem. Without any real insight on why the pull/push based approach is slower, there is nothing the "Go community" can get. All valuable details were omitted, e.g. 1) how much data is being moved? a few megabytes? what is the state of art latency? maybe 0.001 second for that? when latency is still seconds huge, what can be further improved, in the go runtime or in the apps? etc A highly efficient pull/push model without using go channel can easily beat your 1-5 seconds latency. A crappy implementation using channel can get worse. Delay of several seconds is when you pass on information mouth to mouth, in 2023, that kind of delay doesn't justify a blog article on such fancy implementation. I'd suggest you to delete the implementation to rebuild from ground zero.
    - saisrirampur 3y ago
      Thanks for the comment. We will aim to write a more deeper blog on this in the future. We were observing 60<>40 ratio for Pull/Push, as specified in the blog and majority of the Push was target data-store (network) bound. So there was less room of optimization on the Go side. The test was done with ~10-15K TPS on Postgres and the target was Azure Event Hubs, which has a limit of 1MB write batch size.

167 points by saisrirampur 3y ago

Moving to channels seems like a weird choice. Channels should basically always be an implementation detail not exposed in public APIs.

38 3y ago
Seems fine to me. What's the problem?
cauchyk 3y ago
Author of the blog here, curious what a better alternative would be in this context. The channel has to be passed around for the producer and consumer to interface with each other. Are there better patterns for this?
- candiddevmike 3y ago
  Why have consumers and producers vs doing it all in one goroutine, utilizing some kind of connection pool?
  - cauchyk 3y ago
    To get higher throughput we would need one goroutine to pull from the replication slot while the other is pushing to the target. The idea is to keep the Postgres connection useful and reading the slot while also pushing to the target asynchronously.
  - reactordev 3y ago
    Because then you are consuming, or producing, you can’t do both at the same time. You are either reading from a stream of data, or you are writing it. Using goroutines to separate these allows you to do both at the same time, as soon as data is available on the channel or you receive the signal to stop.
- atombender 3y ago
  Not the parent, but I personally dislike it when Go libraries use channels in their public APIs, as it forces a specific concurrency model on the consumer; in particular, channels are quite slow, being protected by an internal mutex, so you're always paying for the overhead no matter if you need it or not. You also have to be very careful about managing the channel lifecycle. If you're not pulling (selecting from) the channel, the library will be permanently stuck. So you must now have a way to tell the library to stop sending, and it must cancel any in-flight send operations if you call producer.Stop() or whatever. In my experience libraries often have bugs in their channel code. It's far too easy to get deadlocks with channels that have interdependencies, and you have to be very careful about buffered versus unbuffered channels, as they behave differently. A better API, in my opinion, is to offer a callback or single-method interface. Then the implementer of that callback or interface can choose to use channels internally if they desire, or they can use something else. You get the same backpressure support since you can treat it as synchronous. After all, a channel's send interface is essentially just: type Channel[T any] interface { Send(T) } But a "chan T" doesn't offer this flexibility. My rule of thumb for channels is that they're goroutine glue, not an API primitive. Build APIs out of interfaces, not channels. The only thing that uses channels should be the one that's controlling the goroutines, because it's the thing that orchestrates them. That said, it's not a hard rule. There are places where channels may have their place in a public API, though I'm not sure I can think of any examples off-hand.
  - foobiekr 3y ago
    this breaks select to send and is a terrible reduction in capability. you can always wrap channels to make them worse and less capable, but your API should expose the more capable option.
    - __turbobrew__ 3y ago
      I think it is a matter of preference. For me personally I use raw channels and goroutines all day every day and I really like using them. Channels are a core primitive in golang so I think it is worth getting familiar with them. As you say being able to select is really nice too.
- skybrian 3y ago
  Looks like the channel field is private in CDCRecordStream, but exposed by GetRecords. The callers mostly loop over Record objects. [1] If I wanted to encapsulate iterating over a channel of Records, maybe it would be something like Go's io.Pipe function [2], which returns a PipeReader and PipeWriter? Except that it would work on Records rather than byte streams. I don't have enough context to know if the extra encapsulation is a good idea in this case, though. [1] https://github.com/search?q=repo%3APeerDB-io%2Fpeerdb%20GetRecords&type=code https://github.com/search?q=repo%3APeerDB-io%2Fpeerdb%20GetR... [2] https://pkg.go.dev/io#Pipe https://pkg.go.dev/io#Pipe
- JyB 3y ago
  Please see this great talk by Bryan C. Mills touching on the subject: https://youtu.be/5zXAHh5tJqQ?t=421 https://youtu.be/5zXAHh5tJqQ?t=421
- earthboundkid 3y ago
  Use an iterator object that can use channels behind the scenes.
reactordev 3y ago
Using channels to talk between goroutines is the perfect choice. There’s plenty of go API’s that require a chan of a type.
- treyd 3y ago
  Go's channels also have a lot of footguns and should be abstracted in a handle type in a public API.
  - reactordev 3y ago
    Except it itself is exposed via a public API. Context interface also exposes a chan Done() <- chan struct{} What are you trying to say? That using a chan as a function argument can shoot yourself in the foot? The whole idea behind channels is to not shoot yourself in the foot with mutex's, deadlocks, etc. What are you trying to say? Wrapping a chan in a type handle is safer than passing a chan of a type? Nope.
    - foobiekr 3y ago
      exactly. you can't abstract over channels easily unless you're missing the point of channels.
      - earthboundkid 3y ago
        I think the rule is: don't use a channel unless at some point you need a select with 2+ branches. The context.Done is a case in point. You're not just waiting around for a context to finish. You do two things: try some network request and wait to see if the context cancels before it finishes. Because it's made to work in a select block like that, there wasn't another, better thing to use than a channel for context.Done. But for just iterating through some stream, you take a context in and return an iterator object that will do all the channel stuff internally.
    - treyd 3y ago
      That's not at all what I was referring to. It's not merely having channels as function arguments. Channels broadly have footguns around how closed and nil channels are treated and you can run into work contention/starvation issues accidentally, since they do have their own throughput limits. And wrapping them in your own handle types that's specific to the library's use case allows you to deal with those issues and swap out the implementation of the cross-goroutine bits without breaking the downstream API.
deleted 3y ago
[deleted]
xarope 3y ago
channels seemed to fit very nicely in a gRPC streaming API call. But gRPC is not for everyone.
AlphaSite 3y ago
I think there are exceptions, for example Gos contexts expose a channel.

DonnyV 3y ago

I wonder if they have a setup without Docker. Not a fan, wonder why they even bother with Docker if they're using GO. Thats one of the great things about GO. Easy distributable that works on all platforms.

cauchyk 3y ago
Not all of our code is in Go. PeerDB has multiple components: the workers, the UI and the query layer. Some of which is in Rust, Go and Typescript. While it would certainly be possible to package them into individual binaries, I found it significantly easier to define the stack in a docker compose file with the requisite environment setup.

steno132 3y ago

[flagged]

the_gipsy 3y ago
Java more productive than go?
- bashtoni 3y ago
  If you judge productivity by lines of code, absolutely. https://github.com/Hello-World-EE/Java-Hello-World-Enterprise-Edition https://github.com/Hello-World-EE/Java-Hello-World-Enterpris... is an excellent demonstration of this.
  - steno132 3y ago
    Java developers tend to produce more value since they can write the code in a more concise way using Lambdas, functional interfaces, record classes, etc. Overengineering like in your example above can be done in any language, but shouldn't.
    - fsaintjacques 3y ago
      If only go had lambdas, functional interfaces and record classes...
bashtoni 3y ago
I strongly disagree with this view for applications, and mildly disagree for infrastructure projects. There is one thing I believe Go is by far and away the leader at: keeping codebases maintainable over time. Go is designed very clearly with this goal and it's excellent at it. The verbosity you complain about makes it much, much clearer what something is doing. That coupled with some of the baked in opinions means it is significantly more straightforward to pick up code that you or someone else wrote a few years ago and modify it.
- 3y ago

tw1984 3y ago

I don't think choosing channel itself is an issue, if it works for the said platform, that is great. What I don't understand is why there should be such a blog article telling people that you decided to use channel in your go based program? This is like writing an article telling people that you decided to use the MMU when building an OS.

saisrirampur 3y ago
Thanks for posting this question. As explained in the article we started off with pull/push model (with configurable batching) and it worked well for streaming to data warehouses, where 30s+ was acceptable latency. We added Queues as a supported target, where one of our users wanted single digit second latency. This is when we introduced channels. We agree that it is a small change. But the latency improvements were significant and wanted to share it with broader programming/Go community of Go Channels and the objective impact they can have.
- tw1984 3y ago
  Pull/Push itself is not the problem unless you have stats to back such implied claims. It is far more likely that your highly inefficient Pull/Push implementation caused the problem. Without any real insight on why the pull/push based approach is slower, there is nothing the "Go community" can get. All valuable details were omitted, e.g. 1) how much data is being moved? a few megabytes? what is the state of art latency? maybe 0.001 second for that? when latency is still seconds huge, what can be further improved, in the go runtime or in the apps? etc A highly efficient pull/push model without using go channel can easily beat your 1-5 seconds latency. A crappy implementation using channel can get worse. Delay of several seconds is when you pass on information mouth to mouth, in 2023, that kind of delay doesn't justify a blog article on such fancy implementation. I'd suggest you to delete the implementation to rebuild from ground zero.
  - saisrirampur 3y ago
    Thanks for the comment. We will aim to write a more deeper blog on this in the future. We were observing 60<>40 ratio for Pull/Push, as specified in the blog and majority of the Push was target data-store (network) bound. So there was less room of optimization on the Go side. The test was done with ~10-15K TPS on Postgres and the target was Azure Event Hubs, which has a limit of 1MB write batch size.