9 ms·

Event Sourcing in Go: From Zero to Production

60 points by tdom 11mo ago

xlii 11mo ago
I'm going to have a word with my ISP. It seems that sites SSL certificates has expired. That's not a good thing, but my ISP decided I'm an idiot and gave me a condescending message about accepting expired certificate - unacceptable in my book. VPN helped. Too much dry code for my taste and not many remarks/explanations - that's not bad because for prose I'd recommend Martin's Fowler articles on Event processing, but _could be better_ ;-) WRT to tech itself - personally I think Go is one of the best languages to go for Event Sourcing today (with Haskell maybe being second). I've been doing complexity analysis for ES in various languages and Go implementation was mostly free (due to Event being an interface and not a concrete structure).
- mrsmrtss 11mo ago
  Have you also considered C# for Event Sourcing? We've built many successful ES projects with C# and the awesome Marten library (https://martendb.io/ https://martendb.io/). It's a real productivity multiplier for us.
- azkalam 11mo ago
  > Go is one of the best languages to go for Event Sourcing toda Can you explain this? Go has a very limited type system.
azkalam 11mo ago
How does event sourcing handle aggregates that may be larger than memory?
- mexicocitinluez 11mo ago
  Smaller aggregates. You really don't want your streams/aggs to come close to being that large.
zknill 11mo ago
Anyone who's built, run, evolved, and operated any reasonably sized event sourced system will know it's a total nightmare. Immutable history sounds like a good idea, until you're writing code to support every event schema you ever published. And all the edge cases that inevitably creates. CQRS sounds good, until you just want to read a value that you know has been written. Event sourcing probably has some legitimate applications, but I'm convinced the hype around it is predominantly just excellent marketing of an inappropriate technology by folks and companies who host queueing technologies (like Kafka).
- anthonylevine 11mo ago
  > CQRS sounds good, until you just want to read a value that you know has been written. This is for you and the author apparently: Prating CQRS does not mean you're splitting up databases. CQRS is simply using different models for reading and writing. That's it. Nothing about different databases or projections or event sourcing. This quote from the article is just flat out false: > CQRS introduces eventual consistency between write and read models: No it doesn't. Eventual consistency is a design decision made independent of using CQRS. Just because CQRS might make it easier to split, it doesn't in any way have an opinion on whether you should or not. > by folks and companies who host queueing technologies (like Kafka). Well that's good because Kafka isn't an event-sourcing technology and shouldn't be used as one.
  - zknill 11mo ago
    Please explain how you intend to use different models for reading and writing without there being some temporal separation between the two? Most all CQRS designs have some read view or projection built off consuming the write side. If this is not the case, and you're just writing your "read models" in the write path; where is the 'S' from CQRS (s for segregation). You wouldn't have a CQRS system here. You'd just be writing read optimised data.
    - azkalam 11mo ago
      - Write side is a Postgres INSERT - Read side is a SELECT on a Postgres view
fleahunter 11mo ago
[flagged]
- mexicocitinluez 11mo ago
  > or a third team starts depending on your event stream as an integration API. > vents stop being an internal persistence detail and become a public contract. You can't blame event sourcing for people not doing it correctly, though. The events aren't a public contract and shouldn't be treated as such. Treating them that way will result in issues. > Used as a generic CRUD replacement it’s a complexity bomb with a 12-18 month fuse. This is true, but all you're really saying it "Use the right tool for the right job".
  - zknill 11mo ago
    > You can't blame event sourcing for people not doing it correctly, though. Perhaps not, but you can criticise articles like this that suggest that CQRS will solve many problems for you, without touching on _any_ of its difficulties or downsides, or the mistakes that many people end up making when implementing these systems.
    - anthonylevine 11mo ago
      CQRS is simply splitting your read and write models. That's it. It's not complicated or complex.
  - simonw 11mo ago
    > You can't blame event sourcing for people not doing it correctly, though. You really can. If there's a technology or approach which the majority of people apply incorrectly that's a problem with that technology or approach.
    - anthonylevine 11mo ago
      No you can't. You can blame the endless amount of people that jump in these threads with hot takes about technologies they neither understand or have experience with. How many event sourced systems have you built? If the answer is 0, I'd have a real hard time understanding how you can even make that judgement. In fact, half of this thread can't even be bothered to look up the definition of CQRS, so the idea that "Storing facts" is to blame for people abusing it is a bit of a stretch, no?
quapster 11mo ago
[flagged]
- simonw 11mo ago
  This comment just made it finally click for me why event sourcing sounds so good on paper but rarely seems to work out for real-world projects: it expects a level of correct-design-up-front which isn't realistic for most teams.
  - hermanradtke 11mo ago
    > it expects a level of correct-design-up-front which isn't realistic for most teams It requires a business that is willing to pay the maintenance cost of event sourcing in order to get capabilities needed capabilities (like an audit trail or replayability).
  - mrkeen 11mo ago
    > it expects a level of correct-design-up-front which isn't realistic for most teams. The opposite is true. A non-ES system is an ES system where you are so sure about being correct-up-front that you perform your reduce/fold step when any new input arrives, and throw away the input. It's like not keeping your receipts around for tax time (because they might get crinkled or hard to read, or someone might want to change them).
    - hu3 11mo ago
      Most systems just need to be correct-enough-up-front though.
- pdhborges 11mo ago
  I would upvote this comment more if I could. I already refrained from introducing event sourcing to tackle wierd dependecies multiple time just by justaposing the amount of discipline that the team has that lead to the current state vs the discipline that is required to keep the event source solution going.
- 11mo ago
liampulles 11mo ago
If you are considering event sourcing, run an event/audit log for a while and see if that does not get you most of the way there. You get similar levels of historical insight, with the disadvantage that to replay things you might need to put a little CLI or script together to infer commands out of the audit log (which if you do a lot, you can make a little library to make building those one off tools quite simple - I've done that). But you avoid all the many well documented footguns that come from trying to run an event sourced system in a typical evolving business.
- mrkeen 11mo ago
  I've done this. We have a customer whom we bill for feature X. Does he actually have feature X or are we billing him for nothing? With ES: We see his Subscriptions and Cancellations and know if he has feature X. Without ES: We don't know if he subscribed or cancelled. With audit log: We almost know whether he subscribed or cancelled.
  - deleted 11mo ago
    [deleted]
techn00 11mo ago
Good article, you might like my lib https://github.com/DeluxeOwl/chronicle https://github.com/DeluxeOwl/chronicle - covers a lot of event sourcing pains for Go
gethly 11mo ago
I posted similar thing 6 months ago, https://news.ycombinator.com/item?id=43870318 https://news.ycombinator.com/item?id=43870318, and got only one upvote and this gets to front page after few hours. Ah, C'est la vie.
tamnd 11mo ago
I don't think this design in the article works in practice. A single `events` table falls apart as the system grows, and untyped JSONB data in `event_data` column just moves the mess into code. Event payloads drift, handlers fill with branching logic, and replaying or migrating old events becomes slow and risky. The pattern promises clarity but eventually turns into a pile of conditionals trying to decode years of inconsistent data. A simpler and more resilient approach is using the database features already built for this. Stored procedures can record both business data and audit records in a controlled way. CDC provides a clean stream for the tables that actually need downstream consumers. And even carefully designed triggers give you consistent invariants and auditability without maintaining a separate projection system that can lag or break. Event sourcing works when the domain truly centers on events, but for most systems these database driven tools stay cleaner, cheaper, and far more predictable over time.
- tamnd 11mo ago
  The only place this kind of append-only event log consistently works well is clickstream-style workloads. You rarely replay them to reconstruct business state; you just pump them into analytics or enrichment pipelines.
cedws 11mo ago
Stuffing data into JSONB columns always makes me feel uncomfortable. Not necessarily for performance/efficiency reasons. You also loose the strong schema that SQL gives you, you don't get to use constraints. You might as well use Mongo, no? How can you be sure that the data stuffed into JSONB fits a particular schema, and that future changes are backwards compatible with rows added long ago?
- LVB 11mo ago
  JSONB can have constraints. I think with an extension you can do full JSON Schema validation, too.
  - cedws 11mo ago
    Yes, I watched a video[0] about using CHECKs and pg_jsonschema to do this in the past. However this only checks for conformance at insert/update time. As time goes on you'll inevitably need to evolve your structures but you won't be able to have any assurances to whether past data conforms to the new structure. The way this article suggests using JSONB would also be problematic because you're stuffing potentially varying structures into one column. You could technically create one massive jsonschema that uses oneOf to validate that the event conforms to one of your structures, but I think it would be horrible for performance. [0]: https://www.youtube.com/watch?v=F6X60ln2VNc https://www.youtube.com/watch?v=F6X60ln2VNc
coredev_ 11mo ago
There are a lot of voices against event sourcing in the comments. I'd just want to balance things a bit. For a mature domain (like when you rebuild an existing system), event sourcing can work really well and make so much sense. But yes, dicipline is a must, as well is thinking things through before you implement a new event. Currently working on a DDDd event sourced system with CQRS and really enjoy it.
hamdouni 11mo ago
What about "just" having an history table aside ? Or perhaps a better name would be a mouvement table ?

60 points by tdom 11mo ago

xlii 11mo ago

I'm going to have a word with my ISP. It seems that sites SSL certificates has expired. That's not a good thing, but my ISP decided I'm an idiot and gave me a condescending message about accepting expired certificate - unacceptable in my book. VPN helped. Too much dry code for my taste and not many remarks/explanations - that's not bad because for prose I'd recommend Martin's Fowler articles on Event processing, but _could be better_ ;-) WRT to tech itself - personally I think Go is one of the best languages to go for Event Sourcing today (with Haskell maybe being second). I've been doing complexity analysis for ES in various languages and Go implementation was mostly free (due to Event being an interface and not a concrete structure).

mrsmrtss 11mo ago
Have you also considered C# for Event Sourcing? We've built many successful ES projects with C# and the awesome Marten library (https://martendb.io/ https://martendb.io/). It's a real productivity multiplier for us.
azkalam 11mo ago
> Go is one of the best languages to go for Event Sourcing toda Can you explain this? Go has a very limited type system.

azkalam 11mo ago

How does event sourcing handle aggregates that may be larger than memory?

mexicocitinluez 11mo ago
Smaller aggregates. You really don't want your streams/aggs to come close to being that large.

zknill 11mo ago

Anyone who's built, run, evolved, and operated any reasonably sized event sourced system will know it's a total nightmare. Immutable history sounds like a good idea, until you're writing code to support every event schema you ever published. And all the edge cases that inevitably creates. CQRS sounds good, until you just want to read a value that you know has been written. Event sourcing probably has some legitimate applications, but I'm convinced the hype around it is predominantly just excellent marketing of an inappropriate technology by folks and companies who host queueing technologies (like Kafka).

anthonylevine 11mo ago
> CQRS sounds good, until you just want to read a value that you know has been written. This is for you and the author apparently: Prating CQRS does not mean you're splitting up databases. CQRS is simply using different models for reading and writing. That's it. Nothing about different databases or projections or event sourcing. This quote from the article is just flat out false: > CQRS introduces eventual consistency between write and read models: No it doesn't. Eventual consistency is a design decision made independent of using CQRS. Just because CQRS might make it easier to split, it doesn't in any way have an opinion on whether you should or not. > by folks and companies who host queueing technologies (like Kafka). Well that's good because Kafka isn't an event-sourcing technology and shouldn't be used as one.
- zknill 11mo ago
  Please explain how you intend to use different models for reading and writing without there being some temporal separation between the two? Most all CQRS designs have some read view or projection built off consuming the write side. If this is not the case, and you're just writing your "read models" in the write path; where is the 'S' from CQRS (s for segregation). You wouldn't have a CQRS system here. You'd just be writing read optimised data.
  - azkalam 11mo ago
    - Write side is a Postgres INSERT - Read side is a SELECT on a Postgres view

fleahunter 11mo ago

[flagged]

mexicocitinluez 11mo ago
> or a third team starts depending on your event stream as an integration API. > vents stop being an internal persistence detail and become a public contract. You can't blame event sourcing for people not doing it correctly, though. The events aren't a public contract and shouldn't be treated as such. Treating them that way will result in issues. > Used as a generic CRUD replacement it’s a complexity bomb with a 12-18 month fuse. This is true, but all you're really saying it "Use the right tool for the right job".
- zknill 11mo ago
  > You can't blame event sourcing for people not doing it correctly, though. Perhaps not, but you can criticise articles like this that suggest that CQRS will solve many problems for you, without touching on _any_ of its difficulties or downsides, or the mistakes that many people end up making when implementing these systems.
  - anthonylevine 11mo ago
    CQRS is simply splitting your read and write models. That's it. It's not complicated or complex.
- simonw 11mo ago
  > You can't blame event sourcing for people not doing it correctly, though. You really can. If there's a technology or approach which the majority of people apply incorrectly that's a problem with that technology or approach.
  - anthonylevine 11mo ago
    No you can't. You can blame the endless amount of people that jump in these threads with hot takes about technologies they neither understand or have experience with. How many event sourced systems have you built? If the answer is 0, I'd have a real hard time understanding how you can even make that judgement. In fact, half of this thread can't even be bothered to look up the definition of CQRS, so the idea that "Storing facts" is to blame for people abusing it is a bit of a stretch, no?

quapster 11mo ago

[flagged]

simonw 11mo ago
This comment just made it finally click for me why event sourcing sounds so good on paper but rarely seems to work out for real-world projects: it expects a level of correct-design-up-front which isn't realistic for most teams.
- hermanradtke 11mo ago
  > it expects a level of correct-design-up-front which isn't realistic for most teams It requires a business that is willing to pay the maintenance cost of event sourcing in order to get capabilities needed capabilities (like an audit trail or replayability).
- mrkeen 11mo ago
  > it expects a level of correct-design-up-front which isn't realistic for most teams. The opposite is true. A non-ES system is an ES system where you are so sure about being correct-up-front that you perform your reduce/fold step when any new input arrives, and throw away the input. It's like not keeping your receipts around for tax time (because they might get crinkled or hard to read, or someone might want to change them).
  - hu3 11mo ago
    Most systems just need to be correct-enough-up-front though.
pdhborges 11mo ago
I would upvote this comment more if I could. I already refrained from introducing event sourcing to tackle wierd dependecies multiple time just by justaposing the amount of discipline that the team has that lead to the current state vs the discipline that is required to keep the event source solution going.
11mo ago

liampulles 11mo ago

If you are considering event sourcing, run an event/audit log for a while and see if that does not get you most of the way there. You get similar levels of historical insight, with the disadvantage that to replay things you might need to put a little CLI or script together to infer commands out of the audit log (which if you do a lot, you can make a little library to make building those one off tools quite simple - I've done that). But you avoid all the many well documented footguns that come from trying to run an event sourced system in a typical evolving business.

mrkeen 11mo ago
I've done this. We have a customer whom we bill for feature X. Does he actually have feature X or are we billing him for nothing? With ES: We see his Subscriptions and Cancellations and know if he has feature X. Without ES: We don't know if he subscribed or cancelled. With audit log: We almost know whether he subscribed or cancelled.
- deleted 11mo ago
  [deleted]

techn00 11mo ago

Good article, you might like my lib https://github.com/DeluxeOwl/chronicle https://github.com/DeluxeOwl/chronicle - covers a lot of event sourcing pains for Go

gethly 11mo ago

I posted similar thing 6 months ago, https://news.ycombinator.com/item?id=43870318 https://news.ycombinator.com/item?id=43870318, and got only one upvote and this gets to front page after few hours. Ah, C'est la vie.

tamnd 11mo ago

I don't think this design in the article works in practice. A single `events` table falls apart as the system grows, and untyped JSONB data in `event_data` column just moves the mess into code. Event payloads drift, handlers fill with branching logic, and replaying or migrating old events becomes slow and risky. The pattern promises clarity but eventually turns into a pile of conditionals trying to decode years of inconsistent data. A simpler and more resilient approach is using the database features already built for this. Stored procedures can record both business data and audit records in a controlled way. CDC provides a clean stream for the tables that actually need downstream consumers. And even carefully designed triggers give you consistent invariants and auditability without maintaining a separate projection system that can lag or break. Event sourcing works when the domain truly centers on events, but for most systems these database driven tools stay cleaner, cheaper, and far more predictable over time.

tamnd 11mo ago
The only place this kind of append-only event log consistently works well is clickstream-style workloads. You rarely replay them to reconstruct business state; you just pump them into analytics or enrichment pipelines.

cedws 11mo ago

Stuffing data into JSONB columns always makes me feel uncomfortable. Not necessarily for performance/efficiency reasons. You also loose the strong schema that SQL gives you, you don't get to use constraints. You might as well use Mongo, no? How can you be sure that the data stuffed into JSONB fits a particular schema, and that future changes are backwards compatible with rows added long ago?

LVB 11mo ago
JSONB can have constraints. I think with an extension you can do full JSON Schema validation, too.
- cedws 11mo ago
  Yes, I watched a video[0] about using CHECKs and pg_jsonschema to do this in the past. However this only checks for conformance at insert/update time. As time goes on you'll inevitably need to evolve your structures but you won't be able to have any assurances to whether past data conforms to the new structure. The way this article suggests using JSONB would also be problematic because you're stuffing potentially varying structures into one column. You could technically create one massive jsonschema that uses oneOf to validate that the event conforms to one of your structures, but I think it would be horrible for performance. [0]: https://www.youtube.com/watch?v=F6X60ln2VNc https://www.youtube.com/watch?v=F6X60ln2VNc

coredev_ 11mo ago

There are a lot of voices against event sourcing in the comments. I'd just want to balance things a bit. For a mature domain (like when you rebuild an existing system), event sourcing can work really well and make so much sense. But yes, dicipline is a must, as well is thinking things through before you implement a new event. Currently working on a DDDd event sourced system with CQRS and really enjoy it.

hamdouni 11mo ago

What about "just" having an history table aside ? Or perhaps a better name would be a mouvement table ?