5 ms·

One thing I would note is that it's not always necessary to abstract away your database layer. It is if you are developing software for installation into an exi

One thing I would note is that it's not always necessary to abstract away your database layer. It is if you are developing software for installation into an existing infrastructure at a customer's datacenter. It's not necessary if you're building your own application on your own infrastructure. In my experience I've never seen an enterprise company change databases over night. Uber perhaps being the exception to the rule. If you're choosing the target platform that your software runs on you should exploit that platform for all it's worth to get the best benefit from it. Yet I've seen plenty of software shops that insist on writing/running heaps of code to abstract away the database server on some imagined future point where someone decides they're going to run it on Mongo now instead of MySQL.

sullenpaladin 10y ago
Abstracting the database has another benefit you are neglecting. It is easy to mock the database and test your application use cases if the database layer is abstracted. This is nearly impossible if database access is baked into the app.
- Rickasaurus 10y ago
  But do you really need to mock the database when you can just deploy a containerized one for testing?
  - mac01021 10y ago
    We used a real database for testing. In the short term it was less work than architecting the application in a way that allowed the DB to be mocked. Long term, the result was a 45-minute test suite which spent most of its time setting up records in the DB, and which could not be parallelized except by adding more instances of the real DB (otherwise the tests would interfere with one-another's assertions about the DB state).
    - Rickasaurus 10y ago
      but now with containers you can have a snapshot that's already set up in the right state. A separate build.
      - theptip 10y ago
        This is a powerful technique, though I'd say it's more useful for the build pipeline than for developers running UTs while coding (since the schema can change more frequently while doing so). To provide a counterpoint though, an advantage touted by advocates of complete decoupling from the DB is that your UT suite can run in O(one minute), rather than O(ten minutes). E.g. see https://www.youtube.com/watch?v=tg5RFeSfBM4 https://www.youtube.com/watch?v=tg5RFeSfBM4. I'm not 100% sold on this approach yet (separating your domain objects entirely from the ORM wrapper is an uphill struggle), but it's interesting.
        Rickasaurus 10y ago
        We run our tests on databases and it's on the order of one minute. Maybe it's a hardware issue? We all have beefy computers with fast SSDs. Also since we strive to make our databases upgradable, it's important that the actual schema update scripts themselves are tested and used directly.
- UK-AL 10y ago
  Another benefit is getting database stuff out of your domain code, makes solving the business problem a lot cleaner. In fact i just write a domain model, solve the problem, then write database adapter to persist the domain model. I always think about modeling the domain first, then persistence is an after thought. The persistence adapter can be done however you want, raw sql orm, nosql. So it's not really "abstracted away", it's just no the focus of the application. Everything just plugs into the domain model.
  - agentultra 10y ago
    Your data is your domain. If you don't understand your data then you don't understand the problem. If there's a business rule for how that data is handled then it's most consistent to define it in the server that is purpose built to manage your data. Especially when it has built in facilities to constrain possible states. I've seen too many programmer bugs to trust putting business logic outside of the DB. Separation of concerns here too just on different lines of concern.
    - theptip 10y ago
      > If you don't understand your data then you don't understand the problem. That's a rather sweeping statement, that many in the industry do not agree with. https://en.wikipedia.org/wiki/Domain-driven_design https://en.wikipedia.org/wiki/Domain-driven_design https://martinfowler.com/ https://martinfowler.com/ > If there's a business rule for how that data is handled then it's most consistent to define it in the [db] To be clear, you're proposing that all business rule validation should be implemented as stored procedures in the DB? One of my domain model aggregates consists of thousands of lines of code just enforcing business rules and constraints. Am I to put that all in the DB?
      - agentultra 10y ago
        > That's a rather sweeping statement, that many in the industry do not agree with. And some who do https://dataorientedprogramming.wordpress.com/tag/mike-acton/ https://dataorientedprogramming.wordpress.com/tag/mike-acton... I'm not going to appeal to authority here. I'm speaking from experience. We all know how source code gets over time with hundreds of programmers working on it. A clear, consistent specification of your data model, rules, invariants, and transformations is far more valuable than the abstract-soup of trying to model your business domain in source code. I think we can all agree that the less code there is to understand then the easier it is to verify it is correct. Verifying the requirement that "when record A is written to the database then B is appended with the delta change if such and such is True" is guaranteed at the database level along with all of the other constraints on those relations. If it's nested in one of these rings behind an abstract factory somewhere it's harder to verify. > Am I to put that all in the DB? That's where I would start. But I'm not you and I don't understand the problem you're trying to solve. My original point was that abstracting out the platform if you're not really concerned about switching platforms is a form of premature pessimization and a source of errors. If you control the platform, target the platform and don't bother with the abstractions.
- agentultra 10y ago
  Design from the data first. You can build testing into your change management at the database layer. In my experience building applications this way reduces the chance for the database to enter into an invalid state. When I build web applications on top of this they have less to do. They literally parse HTTP and shuttle data. No big MVC framework needed. When the data model changes we change the data model. In the database. And we use the abstraction facilities in our server to keep the public schema clean. Premature abstraction is just as dangerous as premature optimization. Maybe even worse in my experience.
UK-AL 10y ago
I don't think the main advantage of this is switching databases. It's switching mindset. Lots of developers think database first, then build on top of that. That's good for simple crud apps. For complicated apps you should think domain or business problem first. Model your domain, make it good for reasoning about your business issues, and then afterwords think about persisting that to a database. You domain is your application core, database persistence is a technical issue outside of that core.
- jl6 10y ago
  Can you give an example of an application core where the database is just a technical issue?
  - UK-AL 10y ago
    Anything where there is calculations, multi-step business process, regulations, complicated business language which takes time to understand is in use. You first step in implementing in a feature is understanding the business domain with the help of people who know it well, then modeling it in the core of the application. I'm currently working in the retail and warehouse distribution domain.
    - dfox 10y ago
      It is surprising how often all these things get reduced to bunch of relational tables with slightly non trivial integrity constraints because sooner than later there will be something that is outside of normal bussiness processes and thus requires exceptions in all these rules. Essentially all big ERP packages follow the model of bunch of relational tables directly exposed to user with bussiness rules and processes as an afterthough. One can say that this stems from historical reasons, but unforeseen usecases that have to be somehow handled right now are also significant reason.
      - UK-AL 10y ago
        Having worked on ERP systems, I find the dB often becomes a major bottleneck because of this reason. You end up creating a really beefy single database, with tons of ram, infinityio etc We have a bunch of different DBs for specific purposes. Some have to be super fast for reading, some just store streams of events, others need to maintain consistency.
regulation_d 10y ago
I think a more common occurrence than "let's move everything from MySQL to Mongo", is "we need these user preferences really fast, so let's cache them in Redis". If you need to do something like that, you can handle all the implementation details in that domain/service layer without changing the API to the rest of your application(s).

sullenpaladin 10y ago

Abstracting the database has another benefit you are neglecting. It is easy to mock the database and test your application use cases if the database layer is abstracted. This is nearly impossible if database access is baked into the app.

Rickasaurus 10y ago
But do you really need to mock the database when you can just deploy a containerized one for testing?
- mac01021 10y ago
  We used a real database for testing. In the short term it was less work than architecting the application in a way that allowed the DB to be mocked. Long term, the result was a 45-minute test suite which spent most of its time setting up records in the DB, and which could not be parallelized except by adding more instances of the real DB (otherwise the tests would interfere with one-another's assertions about the DB state).
  - Rickasaurus 10y ago
    but now with containers you can have a snapshot that's already set up in the right state. A separate build.
    - theptip 10y ago
      This is a powerful technique, though I'd say it's more useful for the build pipeline than for developers running UTs while coding (since the schema can change more frequently while doing so). To provide a counterpoint though, an advantage touted by advocates of complete decoupling from the DB is that your UT suite can run in O(one minute), rather than O(ten minutes). E.g. see https://www.youtube.com/watch?v=tg5RFeSfBM4 https://www.youtube.com/watch?v=tg5RFeSfBM4. I'm not 100% sold on this approach yet (separating your domain objects entirely from the ORM wrapper is an uphill struggle), but it's interesting.
      - Rickasaurus 10y ago
        We run our tests on databases and it's on the order of one minute. Maybe it's a hardware issue? We all have beefy computers with fast SSDs. Also since we strive to make our databases upgradable, it's important that the actual schema update scripts themselves are tested and used directly.
UK-AL 10y ago
Another benefit is getting database stuff out of your domain code, makes solving the business problem a lot cleaner. In fact i just write a domain model, solve the problem, then write database adapter to persist the domain model. I always think about modeling the domain first, then persistence is an after thought. The persistence adapter can be done however you want, raw sql orm, nosql. So it's not really "abstracted away", it's just no the focus of the application. Everything just plugs into the domain model.
- agentultra 10y ago
  Your data is your domain. If you don't understand your data then you don't understand the problem. If there's a business rule for how that data is handled then it's most consistent to define it in the server that is purpose built to manage your data. Especially when it has built in facilities to constrain possible states. I've seen too many programmer bugs to trust putting business logic outside of the DB. Separation of concerns here too just on different lines of concern.
  - theptip 10y ago
    > If you don't understand your data then you don't understand the problem. That's a rather sweeping statement, that many in the industry do not agree with. https://en.wikipedia.org/wiki/Domain-driven_design https://en.wikipedia.org/wiki/Domain-driven_design https://martinfowler.com/ https://martinfowler.com/ > If there's a business rule for how that data is handled then it's most consistent to define it in the [db] To be clear, you're proposing that all business rule validation should be implemented as stored procedures in the DB? One of my domain model aggregates consists of thousands of lines of code just enforcing business rules and constraints. Am I to put that all in the DB?
    - agentultra 10y ago
      > That's a rather sweeping statement, that many in the industry do not agree with. And some who do https://dataorientedprogramming.wordpress.com/tag/mike-acton/ https://dataorientedprogramming.wordpress.com/tag/mike-acton... I'm not going to appeal to authority here. I'm speaking from experience. We all know how source code gets over time with hundreds of programmers working on it. A clear, consistent specification of your data model, rules, invariants, and transformations is far more valuable than the abstract-soup of trying to model your business domain in source code. I think we can all agree that the less code there is to understand then the easier it is to verify it is correct. Verifying the requirement that "when record A is written to the database then B is appended with the delta change if such and such is True" is guaranteed at the database level along with all of the other constraints on those relations. If it's nested in one of these rings behind an abstract factory somewhere it's harder to verify. > Am I to put that all in the DB? That's where I would start. But I'm not you and I don't understand the problem you're trying to solve. My original point was that abstracting out the platform if you're not really concerned about switching platforms is a form of premature pessimization and a source of errors. If you control the platform, target the platform and don't bother with the abstractions.
agentultra 10y ago
Design from the data first. You can build testing into your change management at the database layer. In my experience building applications this way reduces the chance for the database to enter into an invalid state. When I build web applications on top of this they have less to do. They literally parse HTTP and shuttle data. No big MVC framework needed. When the data model changes we change the data model. In the database. And we use the abstraction facilities in our server to keep the public schema clean. Premature abstraction is just as dangerous as premature optimization. Maybe even worse in my experience.

UK-AL 10y ago

I don't think the main advantage of this is switching databases. It's switching mindset. Lots of developers think database first, then build on top of that. That's good for simple crud apps. For complicated apps you should think domain or business problem first. Model your domain, make it good for reasoning about your business issues, and then afterwords think about persisting that to a database. You domain is your application core, database persistence is a technical issue outside of that core.

jl6 10y ago
Can you give an example of an application core where the database is just a technical issue?
- UK-AL 10y ago
  Anything where there is calculations, multi-step business process, regulations, complicated business language which takes time to understand is in use. You first step in implementing in a feature is understanding the business domain with the help of people who know it well, then modeling it in the core of the application. I'm currently working in the retail and warehouse distribution domain.
  - dfox 10y ago
    It is surprising how often all these things get reduced to bunch of relational tables with slightly non trivial integrity constraints because sooner than later there will be something that is outside of normal bussiness processes and thus requires exceptions in all these rules. Essentially all big ERP packages follow the model of bunch of relational tables directly exposed to user with bussiness rules and processes as an afterthough. One can say that this stems from historical reasons, but unforeseen usecases that have to be somehow handled right now are also significant reason.
    - UK-AL 10y ago
      Having worked on ERP systems, I find the dB often becomes a major bottleneck because of this reason. You end up creating a really beefy single database, with tons of ram, infinityio etc We have a bunch of different DBs for specific purposes. Some have to be super fast for reading, some just store streams of events, others need to maintain consistency.

regulation_d 10y ago

I think a more common occurrence than "let's move everything from MySQL to Mongo", is "we need these user preferences really fast, so let's cache them in Redis". If you need to do something like that, you can handle all the implementation details in that domain/service layer without changing the API to the rest of your application(s).