4 ms·

Abstract the persistence layer, document the contract, build a conformance suite, and write your one impl that uses sqlite. I know sometimes there seems to be a

Abstract the persistence layer, document the contract, build a conformance suite, and write your one impl that uses sqlite. I know sometimes there seems to be an impedance mismatch between callers and databases, but often it's not that bad. And no, don't use an ORM or existing DB abstraction to do this. Sure it takes work, but pluggable storage is nice on some of these types of things.

sametmax 9y ago
Congratulation, you just rewrote the base for an orm.
- kodablah 9y ago
  Sorry, put in my no-ORM edit as you may have been writing this. No, I mean higher level than that. ORMs suck as solving your problems specifically. Think more like an API than an ORM. I just started a side project and here's the beginning of an example of the iface I mean [0] and here's the beginning of an example of the sqlite impl I mean [1] (all untested and really early stage of course, but the idea is there). 0 - https://github.com/cretz/yukup/blob/eecb3b24b33f40e6a1eefaa5a3a6ebcd1d2cb084/yukup/storage/storage.go https://github.com/cretz/yukup/blob/eecb3b24b33f40e6a1eefaa5... 1 - https://github.com/cretz/yukup/tree/eecb3b24b33f40e6a1eefaa5a3a6ebcd1d2cb084/yukup/storage/sqlite https://github.com/cretz/yukup/tree/eecb3b24b33f40e6a1eefaa5...
  - sametmax 9y ago
    But they abstract the underlying db very well and have an uniform api that you can rely on. It's also tested, documented and battletested. It has drawbacks but if accepting several db is an important goal, you don't want to handcraft this unless you can measure inaceptable perfs.
    - kodablah 9y ago
      > they abstract the underlying db very well I am admittedly unfamiliar with this specific case, but even a cursory glance showed me some sqlite-specific code embedded in the main library portion [0]. 0 - https://github.com/beetbox/beets/blob/4f7c1c9beda6c2a5b0705e868fb61e9b0af0e16d/beets/library.py#L107 https://github.com/beetbox/beets/blob/4f7c1c9beda6c2a5b0705e...
- rekwah 9y ago
  More of a DAO than an ORM.
  - sametmax 9y ago
    Hence "base". Good orm like sqlalchemy have several layers, each of them building on each others, and you can use each if them independantly depending how much abstraction you want.
jpitz 9y ago
But what is the value add in this specific situation?
- kodablah 9y ago
  I am admittedly unfamiliar with this specific situation, but if this is a common request it can alleviate the requests by allowing others to write their own impls without decoupling what exists. Again though, maybe that's already present here and I am ignorant of the details.
Analemma_ 9y ago
Why? I know this is kind of a snarky response, but seriously... why? This would take developer resources away from real feature work, introduce boatloads of complexity, add more points of failure, make testing and validation harder and make it harder to reason about the entire system end-to-end. Not that that isn't sometimes necessary, but you need a compelling reason, not just jumping into it because it sounds like a good idea. Programmers have this strange compulsion to design swappable interfaces even when they're totally unnecessary and more often than not just break everything while– ironically– only one implementation is ever used. YAGNI.
- kodablah 9y ago
  > Why? To allow others to impl the data store they want and to avoid having to write blog posts like these. It's not that much real work to quickly pull out an iface on a stable system. Assuming it would be just mostly pass-through to your main impl anyways and this is a common approach in refactorization. You don't need some huge system, just drop it back a layer and abstract it. Often, what you end up learning about your system when you do this minimal work (again, nothing big) is that you screwed up and buried SQL and string concatenation and a bunch of other hard-to-review pieces all throughout your app because you keep a stranglehold on terms like YAGNI when separation of concerns has real value. Sometimes you also end up learning that your tests don't cover what you thought they did. Nobody wants bloat, but this adds very little and helps devs clamoring for alternative stores.
  - jimktrains2 9y ago
    > To allow others to impl the data store they want and to avoid having to write blog posts like these. But why? For the longest time MySQL didn't support window function or CTEs. What do you do if you're using those? (I'm not sure of SQLite's window support, but I'm pretty sure it has CTEs.) If you're using postgres, should you not use pg_trgm (trigram indexer to use an index for regex searches on text), postgis (GIS tools), pg-routing (routing tools), ossp_uuid (uuid support), better types (boolean, range, inet, geometric, &c), or features such as RETURNING just because someone doesn't like my choice of database? No, I'm going to do what makes my life easier and my application faster and more featureful.
mamcx 9y ago
This was for long time the "correct" way to think about this. Let me show you why is insane. Imagine you work on Java. Do you build a DSL/Transpiler to Java so you MAYBE could change later to C#? A RDBMS is even more important that the glue code. Why hide it, why think is something "easy" to trow away? Why think is nut to code most or all the code on it?
- kodablah 9y ago
  > Imagine you work on Java. Do you build a DSL/Transpiler to Java so you MAYBE could change later to C#? Why would I do such a thing? I rarely ever promote abstraction for abstraction's sake and will in almost all cases disagree with premature abstraction. There are levels and for simple CRUD things, a thin contract over your DB is a reasonable tradeoff. > A RDBMS is even more important that the glue code. Why hide it, why think is something "easy" to trow away? I don't think you even need glue code. You just need a well-specified contract. You don't want to throw away anything of course. > Why think is nut to code most or all the code on it? I don't think that. Often it can be prudent to use plsql/triggers for things. Doesn't mean the frontend of your web app needs to build the SQL in the JS though, right? Just a high level "DoThing" contract with a description of what it should do is fine.
  - dagss 9y ago
    In many projects the database is way more important than the "backend" or what have you. Business logic in SQL (DB table schemas is 80% of business logic anyway...); Java or Python or whatever is just some glue to connect user interface or API to database. (I think "glue code" in parent referred to "what is not the UI or DB" BTW, i.e. the "Java app".) When the database is the most fundamental thing, then trying to abstract away the database can be a bit like trying to abstract away the programming language you are using. I have code where I would much rather rewrite the "backend" in another language than swap out the database. I think it was a very good analogy to say that abstracting over database is a bit like abstracting over what programming language you are using.

sametmax 9y ago

Congratulation, you just rewrote the base for an orm.

kodablah 9y ago
Sorry, put in my no-ORM edit as you may have been writing this. No, I mean higher level than that. ORMs suck as solving your problems specifically. Think more like an API than an ORM. I just started a side project and here's the beginning of an example of the iface I mean [0] and here's the beginning of an example of the sqlite impl I mean [1] (all untested and really early stage of course, but the idea is there). 0 - https://github.com/cretz/yukup/blob/eecb3b24b33f40e6a1eefaa5a3a6ebcd1d2cb084/yukup/storage/storage.go https://github.com/cretz/yukup/blob/eecb3b24b33f40e6a1eefaa5... 1 - https://github.com/cretz/yukup/tree/eecb3b24b33f40e6a1eefaa5a3a6ebcd1d2cb084/yukup/storage/sqlite https://github.com/cretz/yukup/tree/eecb3b24b33f40e6a1eefaa5...
- sametmax 9y ago
  But they abstract the underlying db very well and have an uniform api that you can rely on. It's also tested, documented and battletested. It has drawbacks but if accepting several db is an important goal, you don't want to handcraft this unless you can measure inaceptable perfs.
  - kodablah 9y ago
    > they abstract the underlying db very well I am admittedly unfamiliar with this specific case, but even a cursory glance showed me some sqlite-specific code embedded in the main library portion [0]. 0 - https://github.com/beetbox/beets/blob/4f7c1c9beda6c2a5b0705e868fb61e9b0af0e16d/beets/library.py#L107 https://github.com/beetbox/beets/blob/4f7c1c9beda6c2a5b0705e...
rekwah 9y ago
More of a DAO than an ORM.
- sametmax 9y ago
  Hence "base". Good orm like sqlalchemy have several layers, each of them building on each others, and you can use each if them independantly depending how much abstraction you want.

jpitz 9y ago

But what is the value add in this specific situation?

kodablah 9y ago
I am admittedly unfamiliar with this specific situation, but if this is a common request it can alleviate the requests by allowing others to write their own impls without decoupling what exists. Again though, maybe that's already present here and I am ignorant of the details.

Analemma_ 9y ago

Why? I know this is kind of a snarky response, but seriously... why? This would take developer resources away from real feature work, introduce boatloads of complexity, add more points of failure, make testing and validation harder and make it harder to reason about the entire system end-to-end. Not that that isn't sometimes necessary, but you need a compelling reason, not just jumping into it because it sounds like a good idea. Programmers have this strange compulsion to design swappable interfaces even when they're totally unnecessary and more often than not just break everything while– ironically– only one implementation is ever used. YAGNI.

kodablah 9y ago
> Why? To allow others to impl the data store they want and to avoid having to write blog posts like these. It's not that much real work to quickly pull out an iface on a stable system. Assuming it would be just mostly pass-through to your main impl anyways and this is a common approach in refactorization. You don't need some huge system, just drop it back a layer and abstract it. Often, what you end up learning about your system when you do this minimal work (again, nothing big) is that you screwed up and buried SQL and string concatenation and a bunch of other hard-to-review pieces all throughout your app because you keep a stranglehold on terms like YAGNI when separation of concerns has real value. Sometimes you also end up learning that your tests don't cover what you thought they did. Nobody wants bloat, but this adds very little and helps devs clamoring for alternative stores.
- jimktrains2 9y ago
  > To allow others to impl the data store they want and to avoid having to write blog posts like these. But why? For the longest time MySQL didn't support window function or CTEs. What do you do if you're using those? (I'm not sure of SQLite's window support, but I'm pretty sure it has CTEs.) If you're using postgres, should you not use pg_trgm (trigram indexer to use an index for regex searches on text), postgis (GIS tools), pg-routing (routing tools), ossp_uuid (uuid support), better types (boolean, range, inet, geometric, &c), or features such as RETURNING just because someone doesn't like my choice of database? No, I'm going to do what makes my life easier and my application faster and more featureful.

mamcx 9y ago

This was for long time the "correct" way to think about this. Let me show you why is insane. Imagine you work on Java. Do you build a DSL/Transpiler to Java so you MAYBE could change later to C#? A RDBMS is even more important that the glue code. Why hide it, why think is something "easy" to trow away? Why think is nut to code most or all the code on it?

kodablah 9y ago
> Imagine you work on Java. Do you build a DSL/Transpiler to Java so you MAYBE could change later to C#? Why would I do such a thing? I rarely ever promote abstraction for abstraction's sake and will in almost all cases disagree with premature abstraction. There are levels and for simple CRUD things, a thin contract over your DB is a reasonable tradeoff. > A RDBMS is even more important that the glue code. Why hide it, why think is something "easy" to trow away? I don't think you even need glue code. You just need a well-specified contract. You don't want to throw away anything of course. > Why think is nut to code most or all the code on it? I don't think that. Often it can be prudent to use plsql/triggers for things. Doesn't mean the frontend of your web app needs to build the SQL in the JS though, right? Just a high level "DoThing" contract with a description of what it should do is fine.
- dagss 9y ago
  In many projects the database is way more important than the "backend" or what have you. Business logic in SQL (DB table schemas is 80% of business logic anyway...); Java or Python or whatever is just some glue to connect user interface or API to database. (I think "glue code" in parent referred to "what is not the UI or DB" BTW, i.e. the "Java app".) When the database is the most fundamental thing, then trying to abstract away the database can be a bit like trying to abstract away the programming language you are using. I have code where I would much rather rewrite the "backend" in another language than swap out the database. I think it was a very good analogy to say that abstracting over database is a bit like abstracting over what programming language you are using.