4 ms·
I think it’s the natural evolution of software development in an age where decoupled architectures make more and more sense. I’ve been around long enough to see
by devjab 3y ago
I think it’s the natural evolution of software development in an age where decoupled architectures make more and more sense. I’ve been around long enough to see a few pendulum swings, but I’m not sure we’re ever going back to a world where the monolith makes sense again. I know some people will disagree with me, but I think there is a fundamental issue with the OOP line of thought, to no fault of the philosophy itself. It’s simply that people don’t handle complexity well in the real world. The way I see it, you have two modes of operating. The right way and whatever the heck you’re doing when it’s Thursday afternoon and you’ve slept poorly all week. Sure there are ways around this, but in most cases your code-reviewer and quality control isn’t going to be allocated enough time (and neither are you for that matter) to actually deny things that work. Which is how you end up with 9000 different tables, views, stored procedures and what not in your giant mess of a database.
Which was sort of fine in a world where nobody cared about data security. But it’s almost impossible to build a “monolith” database that lives up to the modern legislation requirements, at least within the EU. So we’re basically living in a world where the way we’ve done databases for the past 50 years, doesn’t work for the business. Unless you separate everything into business related domains in small kingdoms with total sovereignty over who comes and goes. Which is basically what micro services are.
At the same time we’re also living in a world where it’s often cheaper (and performant enough) to put 95% of what you do in a container using volumes for persistence. If SQLite was ready for it, or maybe, if all the ORMs were ready for SQLite, you could probably do 90% or your database needs in it. Since that’s not the case, Postgres reigns supreme because it’s the next best thing (yes, you’re allowed to think it’s better). I view Vector DBs sort of similarity to this. We operate almost all of our databases in containers, the one exception is for the one services everyone calls where the performance loss of the container is too much. Well, that’s not entirely true, our dataware house isn’t operated in containers, at least not by us, but I rarely interact with the BI side of things except for data architecture so I’m actually not sure how their external partners do Ops. Anyway, I think Postgres will do 90% or (y)our vector needs and then we will need dedicated vector databases for those last bits that won’t fit into the “everything” box. I don’t think this is bad either, maybe Oracle wanted it to be theirs, but then Oracle should not have Oracles governance. I’m sure they didn’t foresee this future when they bought MySQL though, I know I didn’t. Because who could’ve foreseen that that way we used to do Databases would become sort of obsolete because of containers and EU legislation… and other things?
- gaocegege 3y agoDoes `container` mean `Docker/OCI container`?
- fauigerzigerk 3y ago>But it’s almost impossible to build a “monolith” database that lives up to the modern legislation requirements, at least within the EU. So we’re basically living in a world where the way we’ve done databases for the past 50 years, doesn’t work for the business. Not sure I agree. It seems far simpler to comply with most regulations if you store your data in a centralised data store with good management features. If each and every microservice handles its own data, how would you implement rules affecting all of them, such as retention periods, deletion requests, permissions, access logs, etc? It's not just that the data is stored all over the place. You may also have to use microservice specific APIs to access it. If you're referring to laws that require data storage in a particular jurisdiction, it would still be easier to centralise data management in each jursdiction rather than distribute it across scores of services within each of those jurisdictions.
- robertlagrant 3y agoI think the idea is e.g. you can just have opaque keys (e.g. UUIDs) to a person in all services except one, which has their name in. Destroying that record in one place pseud/anonymisises the record across the system.
- TheNewsIsHere 3y agoThis is a fairly standard approach to solving that issue. There are even some applications built for the purpose of housing that data, encrypting it, maintaining audit and use records, processing DSARs, and interacting with the data in useful ways via API.
- devjab 3y ago> Not sure I agree. It seems far simpler to comply with most regulations if you store your data in a centralised data store with good management features. I don't think we disagree, it's just that we soon won't be allowed to physically store our data on the same servers because the EU deems it too risky. A couple of years ago you could operate 600 energy plants from the same few servers, but by January 2024 that will no longer be legal. Aside from that, we're now entering a world where you need to do comply with regular audits on data field levels as well as an explosion of access roles. I'm not sure how you would manage that in a traditional centralized data store, you probably could, but we do it with OPA and once you do that, there is no difference between using a monolith or micro services because it all goes through the central policies. You could argue that the energy sector is a bit of a special case, but it's even sticter in anything related to the financial sectors.