5 ms·

Reading these comments make me feel like a huge outlier. Practically every project I ever worked on included AT LEAST one database swap. That includes startups

by herval 6y ago

Reading these comments make me feel like a huge outlier. Practically every project I ever worked on included AT LEAST one database swap. That includes startups and big tech, for all sorts of different reasons.

geofft 6y ago
I've worked on projects with database swaps, too, but I find it hard to believe that use of abstraction in advance would have helped them. There's a couple of cases. One is that you're using two SQL databases and you're not using any advanced features. You started dev on MySQL, and then the company says "Thou shalt use Postgres" (or whatever). You don't need anything fancy in your own code to handle this. You're still making SQL queries, you're just swapping out the database engine. Technically, this is an example of dependency inversion (depend on the SQL abstraction), but you also didn't set out to do it - basically any programming language you're likely to use has the common database libraries use a common abstraction for sending queries. And you didn't specifically make sure you were writing generic SQL, you just happened not to need anything. More commonly, you're switching databases for a specific feature. Maybe you realize PostGIS (or whatever) is going to solve a problem for you very well. But then you're changing how you model data, what your schemata are, and even how your code is architected and accesses things in the database. You're deciding to move certain logic from your code to the database engine - might even decide to move certain logic from the frontend into the backend, or change how request routing works, or something. This is a fantastic reason to move databases, but no amount of abstraction can prepare you for it, because you're fundamentally changing what the abstraction is. And you're deliberately abandoning SOLID because you're picking up a dependency on a concrete database. But the real case I've seen is where you're switching databases (or data storage layers, more generically) to a different model - MongoDB to not-MongoDB, a C/P database to an A/P one, a relational one to a key-value store, etc. This is the above case but even larger. There is no abstraction you could possibly write that could encompass the old and new cases. It requires rearchitecting how your code works. And then there are the most boring of cases - the ones where a database swap sounds doable in theory and the code is supposedly using an abstraction layer, but no one has ever verified that the code doesn't make assumptions about what database it's on and the code has gotten too big, so we just get an architectural exemption from "Thou shalt" and we run our own instance of the wrong database, because the overhead of running our own DB costs the business less than getting the swap wrong. (Some public examples of these sorts of database migrations that come to mind: https://slack.engineering/scaling-datastores-at-slack-with-vitess/ https://slack.engineering/scaling-datastores-at-slack-with-v... is about how Slack couldn't move away from MySQL and the architectural assumptions they made about it and had to rule out migrations to non-relational databases out of hand, and https://about.gitlab.com/blog/2018/09/12/the-road-to-gitaly-1-0/ https://about.gitlab.com/blog/2018/09/12/the-road-to-gitaly-... talks about how GitLab moved from NFS storage to an RPC service, requiring a lot of refactoring of callers.)
- herval 6y ago
  I’ve been through plenty of cases where companies successfully swap out databases exactly as you described - a document storage for an Rdbms or vice-versa. The bird social network is an example where the db was so well abstracted, they managed to swap these out with no need to rewrite any application code. So does Facebook, for instance. Slack is a clear example of what the complete lack of forethought on this leads to. (Disclaimer: I’m familiar with all 3, but obviously can’t talk details - there’s plenty of public posts on these cases, though) FWIW, every single db abstraction I’ve ever witnessed was worth it - if only so that one could run tests in a sqlite and run prod in something else, or as a way to contain vendor lockin in the code (I’ve seen projects successfully migrate from a plsql-heavy system to mysql because the code was well segregated, and I worked at a startup that literally imploded bc the database was metastasized all over the place) Anyway, as I put it, abstracting data storage is a no-brainer for me, and it saved my skin every single time. I don’t expect to convince anyone here to go do it. :-)
  - ivan_gammel 6y ago
    >>the db was so well abstracted, they managed to swap these out with no need to rewrite any application code What were the reasons for the swap and what were the outcomes of it besides that it actually happened? Costs, performance, scalability etc? I think hardly anyone would argue that those transitions are impossible, because they are indeed possible. The main question is usually if your abstraction can leverage the benefits of underlying implementation or it is just a common denominator. I can imagine a clever technical strategy in a startup, foreseeing future growth and starting with a simple DB before moving to a more complex in maintenance but scalable solution. This may work. Often it doesn't.
    - herval 6y ago
      Varies from project to project of course, but it’s never “just because”. If you’re swapping dbs without reasonable expectations to improve SOMETHING, it is indeed a waste of time (one that I also fought against in my career many times - been through many, many “let’s adopt MongoDB bc it’s hot” cases)

by herval 6y ago

geofft 6y ago

I've worked on projects with database swaps, too, but I find it hard to believe that use of abstraction in advance would have helped them. There's a couple of cases. One is that you're using two SQL databases and you're not using any advanced features. You started dev on MySQL, and then the company says "Thou shalt use Postgres" (or whatever). You don't need anything fancy in your own code to handle this. You're still making SQL queries, you're just swapping out the database engine. Technically, this is an example of dependency inversion (depend on the SQL abstraction), but you also didn't set out to do it - basically any programming language you're likely to use has the common database libraries use a common abstraction for sending queries. And you didn't specifically make sure you were writing generic SQL, you just happened not to need anything. More commonly, you're switching databases for a specific feature. Maybe you realize PostGIS (or whatever) is going to solve a problem for you very well. But then you're changing how you model data, what your schemata are, and even how your code is architected and accesses things in the database. You're deciding to move certain logic from your code to the database engine - might even decide to move certain logic from the frontend into the backend, or change how request routing works, or something. This is a fantastic reason to move databases, but no amount of abstraction can prepare you for it, because you're fundamentally changing what the abstraction is. And you're deliberately abandoning SOLID because you're picking up a dependency on a concrete database. But the real case I've seen is where you're switching databases (or data storage layers, more generically) to a different model - MongoDB to not-MongoDB, a C/P database to an A/P one, a relational one to a key-value store, etc. This is the above case but even larger. There is no abstraction you could possibly write that could encompass the old and new cases. It requires rearchitecting how your code works. And then there are the most boring of cases - the ones where a database swap sounds doable in theory and the code is supposedly using an abstraction layer, but no one has ever verified that the code doesn't make assumptions about what database it's on and the code has gotten too big, so we just get an architectural exemption from "Thou shalt" and we run our own instance of the wrong database, because the overhead of running our own DB costs the business less than getting the swap wrong. (Some public examples of these sorts of database migrations that come to mind: https://slack.engineering/scaling-datastores-at-slack-with-vitess/ https://slack.engineering/scaling-datastores-at-slack-with-v... is about how Slack couldn't move away from MySQL and the architectural assumptions they made about it and had to rule out migrations to non-relational databases out of hand, and https://about.gitlab.com/blog/2018/09/12/the-road-to-gitaly-1-0/ https://about.gitlab.com/blog/2018/09/12/the-road-to-gitaly-... talks about how GitLab moved from NFS storage to an RPC service, requiring a lot of refactoring of callers.)

herval 6y ago
I’ve been through plenty of cases where companies successfully swap out databases exactly as you described - a document storage for an Rdbms or vice-versa. The bird social network is an example where the db was so well abstracted, they managed to swap these out with no need to rewrite any application code. So does Facebook, for instance. Slack is a clear example of what the complete lack of forethought on this leads to. (Disclaimer: I’m familiar with all 3, but obviously can’t talk details - there’s plenty of public posts on these cases, though) FWIW, every single db abstraction I’ve ever witnessed was worth it - if only so that one could run tests in a sqlite and run prod in something else, or as a way to contain vendor lockin in the code (I’ve seen projects successfully migrate from a plsql-heavy system to mysql because the code was well segregated, and I worked at a startup that literally imploded bc the database was metastasized all over the place) Anyway, as I put it, abstracting data storage is a no-brainer for me, and it saved my skin every single time. I don’t expect to convince anyone here to go do it. :-)
- ivan_gammel 6y ago
  >>the db was so well abstracted, they managed to swap these out with no need to rewrite any application code What were the reasons for the swap and what were the outcomes of it besides that it actually happened? Costs, performance, scalability etc? I think hardly anyone would argue that those transitions are impossible, because they are indeed possible. The main question is usually if your abstraction can leverage the benefits of underlying implementation or it is just a common denominator. I can imagine a clever technical strategy in a startup, foreseeing future growth and starting with a simple DB before moving to a more complex in maintenance but scalable solution. This may work. Often it doesn't.
  - herval 6y ago
    Varies from project to project of course, but it’s never “just because”. If you’re swapping dbs without reasonable expectations to improve SOMETHING, it is indeed a waste of time (one that I also fought against in my career many times - been through many, many “let’s adopt MongoDB bc it’s hot” cases)