5 ms·

Completely agree with the article. If each piece of code has its own data store, you lose all the advantages of the DBMS: you have to handwrite your joins, your

by taffer 8y ago

Completely agree with the article. If each piece of code has its own data store, you lose all the advantages of the DBMS: you have to handwrite your joins, your transaction system and get all sorts of problems with cache invalidation, data inconsistencies and n+1 fetching. Encapsulation is important, but there are better ways to do it, such as using the authorization system of your DBMS.

ChicagoDave 8y ago
The only thing that needs a relational data store are reports. Operational data stores should only be concerned with their own domain and expose behaviors and events. There are exceptions, but if you’re building a complex system, monolithic architectures have a known lifetime while MSA’s tend to mitigate long term coderot. It’s hard to see this until you’ve built domain driven micro service based systems properly.
- taffer 8y ago
  > monolithic architectures have a known lifetime while MSA’s tend to mitigate long term coderot I'd say it's exactly the other way around: the programming languages and technologies change every five years or so, but the data stays forever, so why couple the data with the technology du jour? I'd rather have my data managed in one place so I can adapt the technology layers that sit on top whenever requirements change, rather than writing joins and transactions in my application code. I'd rather not bother with all the complexity and keep it simple as long as I don't need an Amazon or Netflix scale.
  - ChicagoDave 8y ago
    Because business complexity modeled relationally becomes stale over time, reduces agility, and develops massive amounts of code rot/tech debt. Relational databases are tools to be used where it makes sense. NoSQL tables are tools. Graph databases are tools. Configuration files are tools. Given your statement, Linux’s entire configuration footprint should be in a relational database. As any good architect will say in response to a design question, “It depends.” MSA’s aren’t a panacea either. Event Streams are an interesting development and new patterns are likely to emerge. But today? I’d focus on reducing complexity with domain-driven design and choosing patterns that enable agility and support separation of concerns. MSA fits very well into that philosophy.
    - wrmsr 8y ago
      Given your statement we should all be running GHU Hurd.
    - taffer 8y ago
      > As any good architect will say in response to a design question, “It depends.” I absolutely agree with that. However I don't like being strawmanned. I never said linux configuration "should be in a relational database". I said I'd like to have my data in one place to reduce complexity and decoupled from the layers above.
      - gcufhg 8y ago
        You can for the start use a single database as long as the different services don't use / know about each others tables. Then again you could just use separate db's.
        taffer 8y ago
        Sure, but as I said, if you split your database, you have to deal with transactions, joins, consistency, over- / underfetching. All this leads to complexity and performace issues you wouldn't have otherwise.
        ChicagoDave 8y ago
        Reliable messaging and eventual consistency eliminates the need for transactions. Performance can be a mitigating factor. Joins are for reporting. In the cloud, performance is just a cost factor.
        taffer 8y ago
        Eventual consistency makes everything even more complicated and I am by far not the only one who says so [1]. [1]: https://cloud.google.com/blog/products/gcp/why-you-should-pick-strong-consistency-whenever-possible https://cloud.google.com/blog/products/gcp/why-you-should-pi...
- gcufhg 8y ago
  You could build a separate ReportService which replicates only the data that's needed for a report in RDBMS.
  - HelloNurse 8y ago
    With a sane database design, the Report service can read user details of all users of interest from the user service and ask the Orders service for the orders of each user (a presumably efficient query), without replicating data.
    - tatersolid 8y ago
      So you’re guaranteeing O(n^2) runtime? Relational DBs are really just fantastic at high-performance sorts, merges, and joins. Why do you want to pay the continual ongoing cost of replicating that poorly and inefficiently in your application logic? For each microservice that refers to another in some way?
- barrkel 8y ago
  The trouble is, if you push state into different services behind APIs, you end up reinventing 50 to 80% of RDBMS anyhow. You need transactions, replication, indexing, integrity, backups etc. etc. You can use an event stream (a substitute for a commit log) but then you need to migrate your event stream, be able to undo events in the stream, etc. It's complexity you really ought to avoid unless you need it for serious business reasons. Monolithic architectures have tended to last for decades in mature industries. I don't think we have a good handle on the lifetime of microservice architectures yet - typically, when they come into being out of necessity rather than fashion, they're part of a startup that hit a major hockey-stick and needed multiple teams working concurrently without stepping on each other, and microservices serve an organizational purpose rather than an architecture purpose. I will second what other people say: the database tends to stick around. Code may rot, but the data is reused by the next generation.
  - NicoJuicy 8y ago
    > multiple teams working concurrently without stepping on each other, and microservices serve an organizational purpose rather than an architecture purpose. I so hard agree with this.
  - quizotic 8y ago
    Agree you need all of these things and more... but if you could do this above the microservice level, as a piece of K8S or whatever the MSA, then you might have an advance over RDBMS. The result would be the same features/guaranteed you have with current DBMS, but you get independent deployability, elastic scalability, redundant availability, and other advantages of microservices that are missing with traditional DBMS.
sauceop 8y ago
Agreed, I think the issues you're touching on are fundamentally difficult problems that don't have a single solution. For example. * storing complex data and supporting multiple access patterns efficiently (like joins) is hard and involves trade-offs (e.g. normalisation vs denormalisation, benefit vs overhead of indices). * data consistency is hard, particularly once you are at a scale that requires distributed storage * durability and disaster recovery is hard * failover and availability, particularly for a stateful system, is hard The right solution is completely application dependent. Usually the best approach is to avoid hard problems to the extent your application allows it, e.g. by accepting relaxed consistency, partitioning the data, or limiting the use cases the system supports. Then you can outsource the hard problems that remain to an existing solution (e.g. a transactional database). Microservices help address some of those problems, but you need to recognize when they don't (e.g. you need strong consistency across multiple services) and adjust your designs accordingly
nqzero 8y ago
could you give an example of what data would belong in distinct microservices but need to be joined or need transactional semantics ?
- jayd16 8y ago
  Foreign key constraints seems like an easy example. Not every service needs the power to create and manage user accounts but its nice know that every user id in your datamodel is valid.

by taffer 8y ago

ChicagoDave 8y ago

The only thing that needs a relational data store are reports. Operational data stores should only be concerned with their own domain and expose behaviors and events. There are exceptions, but if you’re building a complex system, monolithic architectures have a known lifetime while MSA’s tend to mitigate long term coderot. It’s hard to see this until you’ve built domain driven micro service based systems properly.

taffer 8y ago
> monolithic architectures have a known lifetime while MSA’s tend to mitigate long term coderot I'd say it's exactly the other way around: the programming languages and technologies change every five years or so, but the data stays forever, so why couple the data with the technology du jour? I'd rather have my data managed in one place so I can adapt the technology layers that sit on top whenever requirements change, rather than writing joins and transactions in my application code. I'd rather not bother with all the complexity and keep it simple as long as I don't need an Amazon or Netflix scale.
- ChicagoDave 8y ago
  Because business complexity modeled relationally becomes stale over time, reduces agility, and develops massive amounts of code rot/tech debt. Relational databases are tools to be used where it makes sense. NoSQL tables are tools. Graph databases are tools. Configuration files are tools. Given your statement, Linux’s entire configuration footprint should be in a relational database. As any good architect will say in response to a design question, “It depends.” MSA’s aren’t a panacea either. Event Streams are an interesting development and new patterns are likely to emerge. But today? I’d focus on reducing complexity with domain-driven design and choosing patterns that enable agility and support separation of concerns. MSA fits very well into that philosophy.
  - wrmsr 8y ago
    Given your statement we should all be running GHU Hurd.
  - taffer 8y ago
    > As any good architect will say in response to a design question, “It depends.” I absolutely agree with that. However I don't like being strawmanned. I never said linux configuration "should be in a relational database". I said I'd like to have my data in one place to reduce complexity and decoupled from the layers above.
    - gcufhg 8y ago
      You can for the start use a single database as long as the different services don't use / know about each others tables. Then again you could just use separate db's.
      - taffer 8y ago
        Sure, but as I said, if you split your database, you have to deal with transactions, joins, consistency, over- / underfetching. All this leads to complexity and performace issues you wouldn't have otherwise.
        ChicagoDave 8y ago
        Reliable messaging and eventual consistency eliminates the need for transactions. Performance can be a mitigating factor. Joins are for reporting. In the cloud, performance is just a cost factor.
        taffer 8y ago
        Eventual consistency makes everything even more complicated and I am by far not the only one who says so [1]. [1]: https://cloud.google.com/blog/products/gcp/why-you-should-pick-strong-consistency-whenever-possible https://cloud.google.com/blog/products/gcp/why-you-should-pi...
gcufhg 8y ago
You could build a separate ReportService which replicates only the data that's needed for a report in RDBMS.
- HelloNurse 8y ago
  With a sane database design, the Report service can read user details of all users of interest from the user service and ask the Orders service for the orders of each user (a presumably efficient query), without replicating data.
  - tatersolid 8y ago
    So you’re guaranteeing O(n^2) runtime? Relational DBs are really just fantastic at high-performance sorts, merges, and joins. Why do you want to pay the continual ongoing cost of replicating that poorly and inefficiently in your application logic? For each microservice that refers to another in some way?
barrkel 8y ago
The trouble is, if you push state into different services behind APIs, you end up reinventing 50 to 80% of RDBMS anyhow. You need transactions, replication, indexing, integrity, backups etc. etc. You can use an event stream (a substitute for a commit log) but then you need to migrate your event stream, be able to undo events in the stream, etc. It's complexity you really ought to avoid unless you need it for serious business reasons. Monolithic architectures have tended to last for decades in mature industries. I don't think we have a good handle on the lifetime of microservice architectures yet - typically, when they come into being out of necessity rather than fashion, they're part of a startup that hit a major hockey-stick and needed multiple teams working concurrently without stepping on each other, and microservices serve an organizational purpose rather than an architecture purpose. I will second what other people say: the database tends to stick around. Code may rot, but the data is reused by the next generation.
- NicoJuicy 8y ago
  > multiple teams working concurrently without stepping on each other, and microservices serve an organizational purpose rather than an architecture purpose. I so hard agree with this.
- quizotic 8y ago
  Agree you need all of these things and more... but if you could do this above the microservice level, as a piece of K8S or whatever the MSA, then you might have an advance over RDBMS. The result would be the same features/guaranteed you have with current DBMS, but you get independent deployability, elastic scalability, redundant availability, and other advantages of microservices that are missing with traditional DBMS.

sauceop 8y ago

Agreed, I think the issues you're touching on are fundamentally difficult problems that don't have a single solution. For example. * storing complex data and supporting multiple access patterns efficiently (like joins) is hard and involves trade-offs (e.g. normalisation vs denormalisation, benefit vs overhead of indices). * data consistency is hard, particularly once you are at a scale that requires distributed storage * durability and disaster recovery is hard * failover and availability, particularly for a stateful system, is hard The right solution is completely application dependent. Usually the best approach is to avoid hard problems to the extent your application allows it, e.g. by accepting relaxed consistency, partitioning the data, or limiting the use cases the system supports. Then you can outsource the hard problems that remain to an existing solution (e.g. a transactional database). Microservices help address some of those problems, but you need to recognize when they don't (e.g. you need strong consistency across multiple services) and adjust your designs accordingly

nqzero 8y ago

could you give an example of what data would belong in distinct microservices but need to be joined or need transactional semantics ?

jayd16 8y ago
Foreign key constraints seems like an easy example. Not every service needs the power to create and manage user accounts but its nice know that every user id in your datamodel is valid.