5 ms·

Making mistakes means the author delivered a lot of features to users that they wouldn’t have otherwise. RethinkDB sounds terrifying though, I’d hate for my DB

Making mistakes means the author delivered a lot of features to users that they wouldn’t have otherwise. RethinkDB sounds terrifying though, I’d hate for my DB errors in production to literally have no idea what to do, it makes me feel sick thinking about :-) I think we should aim for kludgy but boring working solutions first - for example I’m tempted to write a redux middleware that sends dispatched actions to the server and manually sync, order and manage conflicts in these events to build a “perfect” system where front end state management and backend CQRS perfectly marry up. It’s a terrible idea because I know how to build the Podcast app I’m making perfectly well without any of this nonsense.

arvinsim 8y ago
> front end state management and backend CQRS perfectly marry up. I am currently researching how to do this properly. What would you think would be the "boring but reliable" way to do it?
- andy_ppp 8y ago
  Well I want to do this but the point is I think it's research and unknown; I'm actually saying don't do this! Have an endpoint and call it for changes. No stream of events perfectly queued and ordered between systems. Just accept your app won't be as clever and well designed but it might actually just work :-)
threeseed 8y ago
> I think we should aim for kludgy but boring working solutions first Nice idea in theory. Can be awful in practice. For example in enterprises it's very common to see applications last years or even decades. And so that kludgy but boring solution would be the foundation on which you will have hundreds of engineers and years of work built on top of. And because your kludgy but boring solution worked in the beginning you rarely go back and redo it. Hence why you see a lot of upfront developers built these over-architected monstrosities. It's because they are trying to cater for a decade of possible new features.
- Swizec 8y ago
  You either die a startup or live long enough for everyone to hate the madman who picked the stack.
  - WrtCdEvrydy 8y ago
    In my case, you either die a startup or end up having to rewrite the whole thing in React... all 30 years of technical debt.
    - fiddlerwoaroof 8y ago
      This is always the best because you get to reimplement misfeatures in your shiny new codebase because “we don’t want to change how customers are used to working”
      - Swizec 8y ago
        Bug for bug rewrites are best because “You never know what system out there is relying on current behavior”
- jbigelow76 8y ago
  This doesn't sound as bad a situation as you make it out to be... Dev Person: "Hey boss I finished that X, but I used a framework that hasn't been cool for at least 5 years but it will probably last a decade without a rewrite and everyone can build on top of it." CEO Person: "That sounds great, please keep doing more of that!" Admittedly, I like playing with the new shinies as much as the next dev, the above scenario sounds horrible to me to :)
- syastrov 8y ago
  I think boring could also be described as easily understandable. Kludgy could be interpreted to mean that all possible theoretical scenarios are not handled. Working could mean that the product works for the use cases it was intended for. In that case, I see this as a very positive pragmatic approach. You don’t waste time implementing features you “know” you will need in the future (you don’t know). You don’t waste time handling scenarios that make your feature theoretically complete and elegant but that have no significance to the users of the product. You go quicker to market and can more quickly iterate on your product. I think a much better approach is to implement these kludgy solutions that apply to the specific problem being solved. Then, you can revisit the code when it makes sense (e.g. the scope of the feature is being expanded, or just do it regularly to keep things healthy). Generally holding off on implementing things you don’t need will put you in a better position to implement the things you will need when you reach that point in the future.
virgilp 8y ago
For me, the fact that "They allow you to listen to live updates on (almost) any query." would've been a huge yellow (red?) flag. If your database tech does this_amazing_thing_that_nobody_else_does, you'd better be goddamn sure you understand the reason why nobody else does it, AND understand what are the compromises that this brand_new_tech did in order to support that. You probably pay somehow for that amazing feature - what is the cost, really?
- platform 8y ago
  WRT "… If your database tech does this_amazing_thing_that_nobody_else_does, you'd better be goddamn sure you understand the reason why nobody else does it …" I think several databases actually do this. For example, Alerting in MarkLogic https://docs.marklogic.com/guide/search-dev/alerts#id_29203 https://docs.marklogic.com/guide/search-dev/alerts#id_29203 Another example is Percolations in Elastic Search https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-percolate-query.html https://www.elastic.co/guide/en/elasticsearch/reference/curr... --- My personal view is that those are useful features, however, often they indicate poor architecture where developers did not create proper APIs that put data into the database, and instead rely on database 'triggers' (or percolations/alerts) to notify the 'world' about changes. Those kinds of notifications, should really be coming from either data consumer or data producer APIs, -- not from the database engines.
- fanf2 8y ago
  You can do that with PostgreSQL logical decoding, which listens to the replication stream. You can write a plugin in C to emit the updates in whatever form you need.
  - andy_ppp 8y ago
    Why do you need to do that... https://www.postgresql.org/docs/9.0/sql-notify.html https://www.postgresql.org/docs/9.0/sql-notify.html Should be okay for 99% of needs no?
    - alecbenzer 8y ago
hashhar 8y ago
Think about the use-case where you want all your Relational data to be denormalised and written to a S3 bucket and then loaded into something like Redshift for analytical loads. If you start pushing the data from your application itself then you introduce the problem of dual-writes (what if S3 write failed? what if RDBMS failed?). If instead you treat your RDBMS as the source of truth and use something like PostgresSQL's logical decoding feature to listen to the replication stream and then push the events from that stream to S3 you can achieve guaranteed-eventual-consistency. The best way would've obviously been to make your application write to a Message Bus (Kafka etc.) and then some other application would persist that message to your RDBMS. This approach requires more development effort and introduces all the issues that distributed systems face: Read your writes, read follow writes, write follow reads etc.
- andy_ppp 8y ago
  Yes, I’ve worked on systems that use Kafka to do exactly this...
  - hashhar 8y ago
    I'd love to discuss more on this topic with you since I am currently working on implementing a proof of concept for the same. Since I've just had almost 1 year of software development experience (2017 grad here) the task feels too daunting to me. Also it's just a one-man team since its a proof of concept.
  - hashhar 8y ago
    Oh man, rereading the comments above made me realise i replied to the wrong comment. Doesn't matter though.

arvinsim 8y ago

> front end state management and backend CQRS perfectly marry up. I am currently researching how to do this properly. What would you think would be the "boring but reliable" way to do it?

andy_ppp 8y ago
Well I want to do this but the point is I think it's research and unknown; I'm actually saying don't do this! Have an endpoint and call it for changes. No stream of events perfectly queued and ordered between systems. Just accept your app won't be as clever and well designed but it might actually just work :-)

threeseed 8y ago

> I think we should aim for kludgy but boring working solutions first Nice idea in theory. Can be awful in practice. For example in enterprises it's very common to see applications last years or even decades. And so that kludgy but boring solution would be the foundation on which you will have hundreds of engineers and years of work built on top of. And because your kludgy but boring solution worked in the beginning you rarely go back and redo it. Hence why you see a lot of upfront developers built these over-architected monstrosities. It's because they are trying to cater for a decade of possible new features.

Swizec 8y ago
You either die a startup or live long enough for everyone to hate the madman who picked the stack.
- WrtCdEvrydy 8y ago
  In my case, you either die a startup or end up having to rewrite the whole thing in React... all 30 years of technical debt.
  - fiddlerwoaroof 8y ago
    This is always the best because you get to reimplement misfeatures in your shiny new codebase because “we don’t want to change how customers are used to working”
    - Swizec 8y ago
      Bug for bug rewrites are best because “You never know what system out there is relying on current behavior”
jbigelow76 8y ago
This doesn't sound as bad a situation as you make it out to be... Dev Person: "Hey boss I finished that X, but I used a framework that hasn't been cool for at least 5 years but it will probably last a decade without a rewrite and everyone can build on top of it." CEO Person: "That sounds great, please keep doing more of that!" Admittedly, I like playing with the new shinies as much as the next dev, the above scenario sounds horrible to me to :)
syastrov 8y ago
I think boring could also be described as easily understandable. Kludgy could be interpreted to mean that all possible theoretical scenarios are not handled. Working could mean that the product works for the use cases it was intended for. In that case, I see this as a very positive pragmatic approach. You don’t waste time implementing features you “know” you will need in the future (you don’t know). You don’t waste time handling scenarios that make your feature theoretically complete and elegant but that have no significance to the users of the product. You go quicker to market and can more quickly iterate on your product. I think a much better approach is to implement these kludgy solutions that apply to the specific problem being solved. Then, you can revisit the code when it makes sense (e.g. the scope of the feature is being expanded, or just do it regularly to keep things healthy). Generally holding off on implementing things you don’t need will put you in a better position to implement the things you will need when you reach that point in the future.

virgilp 8y ago

For me, the fact that "They allow you to listen to live updates on (almost) any query." would've been a huge yellow (red?) flag. If your database tech does this_amazing_thing_that_nobody_else_does, you'd better be goddamn sure you understand the reason why nobody else does it, AND understand what are the compromises that this brand_new_tech did in order to support that. You probably pay somehow for that amazing feature - what is the cost, really?

platform 8y ago
WRT "… If your database tech does this_amazing_thing_that_nobody_else_does, you'd better be goddamn sure you understand the reason why nobody else does it …" I think several databases actually do this. For example, Alerting in MarkLogic https://docs.marklogic.com/guide/search-dev/alerts#id_29203 https://docs.marklogic.com/guide/search-dev/alerts#id_29203 Another example is Percolations in Elastic Search https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-percolate-query.html https://www.elastic.co/guide/en/elasticsearch/reference/curr... --- My personal view is that those are useful features, however, often they indicate poor architecture where developers did not create proper APIs that put data into the database, and instead rely on database 'triggers' (or percolations/alerts) to notify the 'world' about changes. Those kinds of notifications, should really be coming from either data consumer or data producer APIs, -- not from the database engines.
fanf2 8y ago
You can do that with PostgreSQL logical decoding, which listens to the replication stream. You can write a plugin in C to emit the updates in whatever form you need.
- andy_ppp 8y ago
  Why do you need to do that... https://www.postgresql.org/docs/9.0/sql-notify.html https://www.postgresql.org/docs/9.0/sql-notify.html Should be okay for 99% of needs no?
  - alecbenzer 8y ago

hashhar 8y ago

Think about the use-case where you want all your Relational data to be denormalised and written to a S3 bucket and then loaded into something like Redshift for analytical loads. If you start pushing the data from your application itself then you introduce the problem of dual-writes (what if S3 write failed? what if RDBMS failed?). If instead you treat your RDBMS as the source of truth and use something like PostgresSQL's logical decoding feature to listen to the replication stream and then push the events from that stream to S3 you can achieve guaranteed-eventual-consistency. The best way would've obviously been to make your application write to a Message Bus (Kafka etc.) and then some other application would persist that message to your RDBMS. This approach requires more development effort and introduces all the issues that distributed systems face: Read your writes, read follow writes, write follow reads etc.

andy_ppp 8y ago
Yes, I’ve worked on systems that use Kafka to do exactly this...
- hashhar 8y ago
  I'd love to discuss more on this topic with you since I am currently working on implementing a proof of concept for the same. Since I've just had almost 1 year of software development experience (2017 grad here) the task feels too daunting to me. Also it's just a one-man team since its a proof of concept.
- hashhar 8y ago
  Oh man, rereading the comments above made me realise i replied to the wrong comment. Doesn't matter though.