8 ms·

Show HN: Storex – A modular and portable database abstraction for JavaScript

52 points by Blahah 8y ago

We'd like to show you Storex, a database abstraction layer that allows you to move your database interaction code between client- and server-side. By surrounding it with modular, recombinable packages for common problems such as schema migrations, it's possible to re-use logic touching the database across a wide variety of databases, allowing you to develop your code entirely in-browser in your daily development workflow, and then move your database to PostgreSQL/MySQL back-end once you're ready. Storex was designed modularly enough to easily be adapted to mBaaS like Firebase, which is coming up soon. Right now it's being used in WorldBrain's Memex (worldbrain.io) to provide you with a client-side full-text searchable database of every page you've seen online and your annotations on them (up to 5GB of data for some of our users.) We'd appreciate your thoughts very much, and don't hesitate to get involved! More info: https://medium.com/worldbrain/storex-a-modular-and-portable-database-abstraction-ecosystem-for-javascript-c38194335c66 https://medium.com/worldbrain/storex-a-modular-and-portable-...

eterps 8y ago
I like this strategy, are there similar efforts?
- ShishKabab 8y ago
  There is one DAL which doesn't take relations into account, nor has a plug-in based approach, but I don't remember it's name. Also didn't have the goal of being a collection of packages that solve problems 'around' your data, like migrations and throwing it back and forth between client and server.
publicfig 8y ago
I was starting to work on my own project just this week to deal with the exact same issues and use cases. Excited to check this out and see how it evolves! This could absolutely be my own misunderstanding, but you mention that it's explicitly not an ORM, but reading through the docs, it seems like the collections and querying aspects of this project are just that. It actually seems like most of what this project's scope seems to be is to be used as an ORM (which I personally am fine with). Can you elaborate on that at all?
- ShishKabab 8y ago
  Hey, Vincent (author of article) here! Indeed it's a fine line, but ORMs give you objects with methods that allow you to modify or further query the database, like User.save() or User.objects.find(). This is for me the distinction, and in Storex you get data objects back, but all the manipulation happens through one object, the StorageManager.
  - joshribakoff 8y ago
    So its a data mapper instead of active record, but still an ORM.
    - ShishKabab 8y ago
      Ah, thanks for pointing that out. Will read up on my terminology ;)
elliotlarson 8y ago
This is great. I definitely feel like the front-end could use a more ORM-like, feature rich data layer. I generally use an ORM on the backend. When I get the data to the front-end, it's still relational in nature, and I still want similar tools for working with it (e.g. schema definition, validations, and an api that lets me query and manipulate related data). It seems like Ember Data is closest thing. I wish it could more easily be used outside of the context of Ember.
- ShishKabab 8y ago
  You're free to implement an ORM on top if you feel that style of access suits you more. My mail is in the article, will endorse your package and help you implement it in a clean way if you want :)
  - elliotlarson 8y ago
    Right... sorry. I didn't make my point very well. What I was trying to say was that this is sort of an ORM, which makes sense to me. I see a need for that on the front-end. I see your other comments about how this isn't exactly an ORM, which I see as a semantic discussion. It's ORM enough for my purposes. I bring up Ember Data because it's an example of battle tested and feature rich front-end ORM. I'm thinking it would be more popular if it wasn't tied to Ember so closely.
    - ShishKabab 8y ago
      Thank you for your feedback, I got a bit confused there :) Indeed I think it's nice to be able to mix and match pieces of existing stuff, which I think more software should do. The multi-device sync I'm designing right now will also not be coupled tightly to Storex, so you can use it easily inside existing applications.
ghego1 8y ago
Cool, we have developed something very similar in the past months and we're thinking of releasing it in open source, I'll keep an eye on your library as well!
ghego1 8y ago
Cool! We have developed something very similar in the last months and we are thinking of releasing it in open source as well, I'll keep an eye on your library as well! :-)
- ShishKabab 8y ago
  Very nice, curious to see how we could learn from each other :)
ralusek 8y ago
How is this not an ORM? Just because it's technically separate modules which are being combined to form an ORM? How does this meaningfully differ from Sequelize, Waterline, Objection, or Loopback's ORM?
- ShishKabab 8y ago
  Sequelize is only for SQL databases. This is also for the front end and in the future also for Firebase and others like it. And there's more in the article about adjacent functionality planned, which is outside of the scope of an ORM. And yes, got my terminology a bit wrong: meant to say it's not ActiveRecord.
graphememes 8y ago
This is an ORM.
- ShishKabab 8y ago
  See comments below.
TheAceOfHearts 8y ago
I have mixed feelings about this. Sharing database abstractions between the client and the server might make sense for some applications, but I worry it might often be a code smell. I used to believe in the dream of sharing models between the client and server, but that illusion was shattered on first contact with reality. With that said, I'm still open to having my mind changed. I think meteor [0] set out with a similar goal, if not more ambitious, and it doesn't appear to have done very well. The limitations are pretty severe, since you're basically targeting the lowest common denominator. You cannot reduce the inherent complexity of the problem which these databases set out to solve without sacrificing functionality. My thoughts on this have been all over the place, but lately I've been leaning towards adopting fewer abstraction layers. At some point you'll be forced to actually understand each underlying system anyway, and by designing your data models with a specific database in mind you can leverage all of its features to maximum effect in order to achieve optimal performance. It might be useful to review Django's ORM design for ideas and inspiration, as they also implement the data mapper pattern. You should consider copying Django's migration commands [1] as well. The following refers mostly to server databases, but some of it can apply to client databases as well. You probably shouldn't be running migrations automatically, especially in production. We're all human and mistakes happen, but there are steps you can take to reduce their potential impact, if not eliminate them entirely within certain processes. Running migrations outside of peak hours reduces the chance you'll bump into performance problems and that it'll affect users. Always create a snapshot or backup before running migrations, data loss is never acceptable! When a user trusts you with their data it's your responsibility to handle it with immense care. This tool doesn't help with the more challenging problems related to schema management. Zero-downtime migrations are the industry standard. There's a large set of common migration operations which require a full table lock unless they're written with deliberate care so as to avoid those pitfalls. Sever-side migrations must always be done in lock-step, with fully backwards-compatible code capable of handling both versions during a transition. Client-side applications are usually expected to remain largely fully compatible with both backwards and future changes. [0] https://www.meteor.com/ https://www.meteor.com/ [1] https://docs.djangoproject.com/en/2.1/topics/migrations/ https://docs.djangoproject.com/en/2.1/topics/migrations/
- ShishKabab 8y ago
ramon 8y ago
I think this can make it easier for hackers to understand your data model right from the beginning. Even with Firebase I don't manipulate data in the client because anyone can just open a f12 and start changing the data as they wish, it's a good idea but it's a hacking world and we need to make sure that data manipulation stays in the backend only with CORS, SSL and requests protection against injections either in SQL or anything else can even be a header. I understand were you're coming from, I used to think why not. Well the security is still the barrier and even back in the old ages of Java EE the beans stayed safe guarded in the backend from the Web Project.
- ShishKabab 8y ago
  If that's your concern, part of the plan for later would be to make it easier to generate a REST/GraphQL API that you can easily consume on the client side, while not having any schema data in the front-end. Say you'd have a TodoStorage which has high level methods like markAsDone(), you could have that either write directly to IndexedDB in your dev workflow, or move that class to the backend, and replace the front-end version of TodoStorage talk to a GraphQL API instead. The point is to have a choice, and allow the user of the library to understand the risks of technologies involved and use and mix them according to their needs with minimal efforts.

52 points by Blahah 8y ago

eterps 8y ago

I like this strategy, are there similar efforts?

ShishKabab 8y ago
There is one DAL which doesn't take relations into account, nor has a plug-in based approach, but I don't remember it's name. Also didn't have the goal of being a collection of packages that solve problems 'around' your data, like migrations and throwing it back and forth between client and server.

publicfig 8y ago

I was starting to work on my own project just this week to deal with the exact same issues and use cases. Excited to check this out and see how it evolves! This could absolutely be my own misunderstanding, but you mention that it's explicitly not an ORM, but reading through the docs, it seems like the collections and querying aspects of this project are just that. It actually seems like most of what this project's scope seems to be is to be used as an ORM (which I personally am fine with). Can you elaborate on that at all?

ShishKabab 8y ago
Hey, Vincent (author of article) here! Indeed it's a fine line, but ORMs give you objects with methods that allow you to modify or further query the database, like User.save() or User.objects.find(). This is for me the distinction, and in Storex you get data objects back, but all the manipulation happens through one object, the StorageManager.
- joshribakoff 8y ago
  So its a data mapper instead of active record, but still an ORM.
  - ShishKabab 8y ago
    Ah, thanks for pointing that out. Will read up on my terminology ;)

elliotlarson 8y ago

This is great. I definitely feel like the front-end could use a more ORM-like, feature rich data layer. I generally use an ORM on the backend. When I get the data to the front-end, it's still relational in nature, and I still want similar tools for working with it (e.g. schema definition, validations, and an api that lets me query and manipulate related data). It seems like Ember Data is closest thing. I wish it could more easily be used outside of the context of Ember.

ShishKabab 8y ago
You're free to implement an ORM on top if you feel that style of access suits you more. My mail is in the article, will endorse your package and help you implement it in a clean way if you want :)
- elliotlarson 8y ago
  Right... sorry. I didn't make my point very well. What I was trying to say was that this is sort of an ORM, which makes sense to me. I see a need for that on the front-end. I see your other comments about how this isn't exactly an ORM, which I see as a semantic discussion. It's ORM enough for my purposes. I bring up Ember Data because it's an example of battle tested and feature rich front-end ORM. I'm thinking it would be more popular if it wasn't tied to Ember so closely.
  - ShishKabab 8y ago
    Thank you for your feedback, I got a bit confused there :) Indeed I think it's nice to be able to mix and match pieces of existing stuff, which I think more software should do. The multi-device sync I'm designing right now will also not be coupled tightly to Storex, so you can use it easily inside existing applications.

ghego1 8y ago

Cool, we have developed something very similar in the past months and we're thinking of releasing it in open source, I'll keep an eye on your library as well!

ghego1 8y ago

Cool! We have developed something very similar in the last months and we are thinking of releasing it in open source as well, I'll keep an eye on your library as well! :-)

ShishKabab 8y ago
Very nice, curious to see how we could learn from each other :)

ralusek 8y ago

How is this not an ORM? Just because it's technically separate modules which are being combined to form an ORM? How does this meaningfully differ from Sequelize, Waterline, Objection, or Loopback's ORM?

ShishKabab 8y ago
Sequelize is only for SQL databases. This is also for the front end and in the future also for Firebase and others like it. And there's more in the article about adjacent functionality planned, which is outside of the scope of an ORM. And yes, got my terminology a bit wrong: meant to say it's not ActiveRecord.

graphememes 8y ago

This is an ORM.

ShishKabab 8y ago
See comments below.

TheAceOfHearts 8y ago

I have mixed feelings about this. Sharing database abstractions between the client and the server might make sense for some applications, but I worry it might often be a code smell. I used to believe in the dream of sharing models between the client and server, but that illusion was shattered on first contact with reality. With that said, I'm still open to having my mind changed. I think meteor [0] set out with a similar goal, if not more ambitious, and it doesn't appear to have done very well. The limitations are pretty severe, since you're basically targeting the lowest common denominator. You cannot reduce the inherent complexity of the problem which these databases set out to solve without sacrificing functionality. My thoughts on this have been all over the place, but lately I've been leaning towards adopting fewer abstraction layers. At some point you'll be forced to actually understand each underlying system anyway, and by designing your data models with a specific database in mind you can leverage all of its features to maximum effect in order to achieve optimal performance. It might be useful to review Django's ORM design for ideas and inspiration, as they also implement the data mapper pattern. You should consider copying Django's migration commands [1] as well. The following refers mostly to server databases, but some of it can apply to client databases as well. You probably shouldn't be running migrations automatically, especially in production. We're all human and mistakes happen, but there are steps you can take to reduce their potential impact, if not eliminate them entirely within certain processes. Running migrations outside of peak hours reduces the chance you'll bump into performance problems and that it'll affect users. Always create a snapshot or backup before running migrations, data loss is never acceptable! When a user trusts you with their data it's your responsibility to handle it with immense care. This tool doesn't help with the more challenging problems related to schema management. Zero-downtime migrations are the industry standard. There's a large set of common migration operations which require a full table lock unless they're written with deliberate care so as to avoid those pitfalls. Sever-side migrations must always be done in lock-step, with fully backwards-compatible code capable of handling both versions during a transition. Client-side applications are usually expected to remain largely fully compatible with both backwards and future changes. [0] https://www.meteor.com/ https://www.meteor.com/ [1] https://docs.djangoproject.com/en/2.1/topics/migrations/ https://docs.djangoproject.com/en/2.1/topics/migrations/

ShishKabab 8y ago

ramon 8y ago

I think this can make it easier for hackers to understand your data model right from the beginning. Even with Firebase I don't manipulate data in the client because anyone can just open a f12 and start changing the data as they wish, it's a good idea but it's a hacking world and we need to make sure that data manipulation stays in the backend only with CORS, SSL and requests protection against injections either in SQL or anything else can even be a header. I understand were you're coming from, I used to think why not. Well the security is still the barrier and even back in the old ages of Java EE the beans stayed safe guarded in the backend from the Web Project.

ShishKabab 8y ago
If that's your concern, part of the plan for later would be to make it easier to generate a REST/GraphQL API that you can easily consume on the client side, while not having any schema data in the front-end. Say you'd have a TodoStorage which has high level methods like markAsDone(), you could have that either write directly to IndexedDB in your dev workflow, or move that class to the backend, and replace the front-end version of TodoStorage talk to a GraphQL API instead. The point is to have a choice, and allow the user of the library to understand the risks of technologies involved and use and mix them according to their needs with minimal efforts.