5 ms·

Where's the hitchhikers guide to moving back to relational data after our whiz bang dev made us event driven and left for a new opportunity?

tarruda 3y ago
This reminds me of https://youtu.be/b2F-DItXtZs https://youtu.be/b2F-DItXtZs
jtwebman 3y ago
He tried at the beginning to say only when the DB gets too big should you even think about this but there are things you can do there as well.
- Nextgrid 3y ago
  The way I see it, either your business domain requires querying over a large amount of data, or it doesn't. If an application allows someone to be able to enter let's say an order number from anywhere in the world from the last 10 years and be able to find the order, there is no magic - some server out there is going to have to scan a huge amount of data to find a match. Tricks such as indexes, partitioned tables, etc can be employed, but those tricks have nothing to do with event-sourcing and are independent of it.
  - mrkeen 3y ago
    > Tricks such as indexes, partitioned tables, etc can be employed, but those tricks have nothing to do with event-sourcing and are independent of it. You might want to use different tricks in different situations. Different situations means different services, and different tricks means different storage/query technologies. So how do you get your data into three systems - and more crucially - keep them in sync? Webhooks? Triggers? Some bidirectional sync magic app that claims to beat CAP? Just use event-sourcing (append-only, disallow modification) and the multiple systems will stay in sync as long as they know how to process one more message.
    - Nextgrid 3y ago
      Agreed with your points but this article seems to present event-sourcing as a replacement for your database(s) and even makes claims about saving storage space, thus at least hinting at not using databases anymore.
      - mrkeen 3y ago
        It's a replacement for your source-of-truth, not your database(s). Although you're right about the article not explicitly mentioning slurping the events back into a DB. I suspect the reason is that there are plenty of articles which explain how to event-source from greenfield, but this is the first one I've seen which focuses on existing brownfield relational data - see the title. > makes claims about saving storage space I don't think that was the right reading about saving storage space: > We’ve been trying to optimise the storage size; we’ve made some sins of overriding and losing our precious business data I think it's his strawman RDBMS developer who optimised for saving storage space, and lost business data as a result. The suggested approach is: > We can optimise for information quality instead of its size.
- seanhunter 3y ago
  There is no “too big” in databases and in particular size of data is not the criterion for deciding on using something like event sourcing. It’s a really niche paradigm that is only ever going to be useful in quite unusual circumstances. Most of the time people don’t need it, and most of the time when people do it, they find their immutable event source is very inconvenient for lots of the normal things you want to do with your data, so they end up doing something like CQRS[1] (ie having a database as well). This is one of those Martin Fowler[2] type things that looks good on a whiteboard but most people would be better off avoiding most of the time. [1] https://en.wikipedia.org/wiki/Command_Query_Responsibility_Segregation https://en.wikipedia.org/wiki/Command_Query_Responsibility_S... [2] This Martin Fowler, https://en.wikipedia.org/wiki/Martin_Fowler_(software_engineer) https://en.wikipedia.org/wiki/Martin_Fowler_(software_engine... not this Martin Fowler https://metro.co.uk/2023/11/06/eastenders-star-reveals-why-massive-christmas-episode-is-one-of-the-best-ever-19778635/?ico=trending-module_tag_bbc-2_item-5 https://metro.co.uk/2023/11/06/eastenders-star-reveals-why-m...
blowski 3y ago
So long, and thanks for all the eventual consistency issues
- corethree 3y ago
  Actually event driven stuff is highly consistent. Event insertion is fast as hell and all of it is timestamped and tagged with a unique Id so there's no consistency issues. The problem is availability. You have to aggregate those events to get anything meaningful about it and aggregation can be really really slow.
  - blowski 3y ago
    Exactly. Because the aggregation is so slow, you cache the aggregates, but then the caching layer becomes slow to invalidate, and you get eventual consistency problems.
    - corethree 3y ago
      I wouldn't call that a consistency issue. That's just lag. The aggregation is valid for a specific time point. Caches aren't the source of truth. The source of truth remains consistent here.
      - deleted 3y ago
        [deleted]
      - jabradoodle 3y ago
        It's hardly a cache if it's the only way to query the data, it's a materialized view
        corethree 3y ago
        Then it's an outdated materialized view. If you don't refresh your view from a browser is it a consistency issue? No. Not in in the way the term is usually used.
tkiolp4 3y ago
Exactly. These kind of devs (like OP) cannot be quiet and need to constantly introduce something “cool” so that they can get a good salary raise. The moment they cannot introduce more BS, they move on to another company. The poor other “average” devs need to maintain all the crap.
mhd 3y ago
Next to the 97 part video tutorial on how to get back from your semi-hydrated micro-frontend to something sane for your 12 person company ;)
- revskill 3y ago
  THe problem is everyone's interpretation and implementation of micro-frontend is different.
deleted 3y ago
[deleted]
mrkeen 3y ago
Do it like an accountant would: Throw away all invoices and receipts, and just represent each customer's balance as a single number. When it changes, get your crayon, cross it out, and write the new number.

tarruda 3y ago

This reminds me of https://youtu.be/b2F-DItXtZs https://youtu.be/b2F-DItXtZs

jtwebman 3y ago

He tried at the beginning to say only when the DB gets too big should you even think about this but there are things you can do there as well.

Nextgrid 3y ago
The way I see it, either your business domain requires querying over a large amount of data, or it doesn't. If an application allows someone to be able to enter let's say an order number from anywhere in the world from the last 10 years and be able to find the order, there is no magic - some server out there is going to have to scan a huge amount of data to find a match. Tricks such as indexes, partitioned tables, etc can be employed, but those tricks have nothing to do with event-sourcing and are independent of it.
- mrkeen 3y ago
  > Tricks such as indexes, partitioned tables, etc can be employed, but those tricks have nothing to do with event-sourcing and are independent of it. You might want to use different tricks in different situations. Different situations means different services, and different tricks means different storage/query technologies. So how do you get your data into three systems - and more crucially - keep them in sync? Webhooks? Triggers? Some bidirectional sync magic app that claims to beat CAP? Just use event-sourcing (append-only, disallow modification) and the multiple systems will stay in sync as long as they know how to process one more message.
  - Nextgrid 3y ago
    Agreed with your points but this article seems to present event-sourcing as a replacement for your database(s) and even makes claims about saving storage space, thus at least hinting at not using databases anymore.
    - mrkeen 3y ago
      It's a replacement for your source-of-truth, not your database(s). Although you're right about the article not explicitly mentioning slurping the events back into a DB. I suspect the reason is that there are plenty of articles which explain how to event-source from greenfield, but this is the first one I've seen which focuses on existing brownfield relational data - see the title. > makes claims about saving storage space I don't think that was the right reading about saving storage space: > We’ve been trying to optimise the storage size; we’ve made some sins of overriding and losing our precious business data I think it's his strawman RDBMS developer who optimised for saving storage space, and lost business data as a result. The suggested approach is: > We can optimise for information quality instead of its size.
seanhunter 3y ago
There is no “too big” in databases and in particular size of data is not the criterion for deciding on using something like event sourcing. It’s a really niche paradigm that is only ever going to be useful in quite unusual circumstances. Most of the time people don’t need it, and most of the time when people do it, they find their immutable event source is very inconvenient for lots of the normal things you want to do with your data, so they end up doing something like CQRS[1] (ie having a database as well). This is one of those Martin Fowler[2] type things that looks good on a whiteboard but most people would be better off avoiding most of the time. [1] https://en.wikipedia.org/wiki/Command_Query_Responsibility_Segregation https://en.wikipedia.org/wiki/Command_Query_Responsibility_S... [2] This Martin Fowler, https://en.wikipedia.org/wiki/Martin_Fowler_(software_engineer) https://en.wikipedia.org/wiki/Martin_Fowler_(software_engine... not this Martin Fowler https://metro.co.uk/2023/11/06/eastenders-star-reveals-why-massive-christmas-episode-is-one-of-the-best-ever-19778635/?ico=trending-module_tag_bbc-2_item-5 https://metro.co.uk/2023/11/06/eastenders-star-reveals-why-m...

blowski 3y ago

So long, and thanks for all the eventual consistency issues

corethree 3y ago
Actually event driven stuff is highly consistent. Event insertion is fast as hell and all of it is timestamped and tagged with a unique Id so there's no consistency issues. The problem is availability. You have to aggregate those events to get anything meaningful about it and aggregation can be really really slow.
- blowski 3y ago
  Exactly. Because the aggregation is so slow, you cache the aggregates, but then the caching layer becomes slow to invalidate, and you get eventual consistency problems.
  - corethree 3y ago
    I wouldn't call that a consistency issue. That's just lag. The aggregation is valid for a specific time point. Caches aren't the source of truth. The source of truth remains consistent here.
    - deleted 3y ago
      [deleted]
    - jabradoodle 3y ago
      It's hardly a cache if it's the only way to query the data, it's a materialized view
      - corethree 3y ago
        Then it's an outdated materialized view. If you don't refresh your view from a browser is it a consistency issue? No. Not in in the way the term is usually used.

tkiolp4 3y ago

Exactly. These kind of devs (like OP) cannot be quiet and need to constantly introduce something “cool” so that they can get a good salary raise. The moment they cannot introduce more BS, they move on to another company. The poor other “average” devs need to maintain all the crap.

mhd 3y ago

Next to the 97 part video tutorial on how to get back from your semi-hydrated micro-frontend to something sane for your 12 person company ;)

revskill 3y ago
THe problem is everyone's interpretation and implementation of micro-frontend is different.

deleted 3y ago

[deleted]

mrkeen 3y ago

Do it like an accountant would: Throw away all invoices and receipts, and just represent each customer's balance as a single number. When it changes, get your crayon, cross it out, and write the new number.