3 ms·

OT: What happened in the database space and why? In 2000 you would have had a hard time naming enough database systems to run out of fingers for counting. But i

by danbruc 10y ago

OT: What happened in the database space and why? In 2000 you would have had a hard time naming enough database systems to run out of fingers for counting. But in recent years database system seem to pop out of the ground like JavaScript frameworks. So what changed? Did the idea of a general purpose database system break because the usage scenarios became so diverse that we can no longer pretend that one size fits almost all? Is this just a perception thing because everybody started uploading his weekend hack toy database to GitHub and blogs about it? And what really irritates me the most, database systems are really complex pieces of software requiring years and years of development time, where does this time come from and who pays for it? Especially having more and more different databases system to choose from does not really promise that developing yet another one will be an easy commercial success.

jinjin2 10y ago
Realm has been available for many years and is actually really different from what else is out there by the fact that it is a real object database. It has just been flying under the radar a bit given that it has only been available for mobile devices. I for one think that it is awesome that it is now available for node.js as well. Especially with the new replication functionality that allows you to share live data between node instances running on different machines.
- lobster_johnson 10y ago
  "Real" object databases — "OODBMS" — uses to be a thing, too. They still survive in some niches, like finance and health, though largely as legacy systems. Products like Objectivity, ObjectStore, POET, Caché (which is built on MUMPS), GemStone (Smalltalk) etc. were once big, expensive products that seemed poised to take over from old-hat SQL databases. Those databases failed not necessarily because the technology was wrong, although they certainly made some choices that didn't help; those databases were often tightly coupled with opaque binary object serialization formats (with no way of looking into a database without going the binary route) that were in turn tightly coupled with the app's class hiearchy. For example, several of these actually post-processed your code (e.g. your Java bytecode) to insert serialization code, so that when you followed a relationship (e.g. "library.books[0].author"), it would automatically query under the hood and return the author as an object. They ended up being clunky to work with as a result; for example, in the beginning, very few of them had a query language, so the only way to query the data was by following relationships through your graph of objects. Realm looks interesting as a new approach to the object database design, but at the same time seems a bit limited. It doesn't seem to have schemas or any kind of role-based permissions, and seems to make the same mistake of not including a query language (so you can't build a REPL for it, and every language binding needs its own query builder). The transaction semantics also seem to be entirely undefined (is it read-committed?). From what I can tell, it's also "offline first", in the sense that you have a local database that can sync to/from a server, but you're technically always working on an offline copy of the master data; while nice to have for a mobile app, it does limit what you can use it for. I also don't see any support for fine-grained updates. Every write you do seems to overwrite, which I assume means the last write always wins; how do you deal with conflicts, or implement things like counters, maps or arrays that need to be incrementally updated?
  - brmunk 10y ago
    Realm certainly has a schema - anything that made you think it doesn't? It's ACID and Serializable. It also has a query language. But as one of the core design principles is to be as native to the language and platform as possible, the query language takes different forms. In Java it's a fluent API, in Cocoa it's NSPredicates string based. And the JS string based is roughly identical to that as well. It's absolutely designed to be offline first. I'm curious as to which limitations you see with that? Operational Transform is used to resolve conflicts at the property level, so I would say it's much more fine-grained than most other systems that would overwrite entire objects. The specific resolution rules are very simple and intuitive and how to deal with custom needs are described here: https://realm.io/docs/realm-object-server/#conflict-resolution https://realm.io/docs/realm-object-server/#conflict-resoluti.... There is ordered lists already and explicit counters are being exposed in the SDKs at the moment. You might find this article useful as well: https://realm.io/news/eventually-consistent-making-a-mobile-first-distributed-system/ https://realm.io/news/eventually-consistent-making-a-mobile-...
    - lobster_johnson 10y ago
      I couldn't, and still can't, find any documentation or "list of features" that mention schema support, query language or ACID semantics. The documentation is pretty sparse and disorganized. (By query language I don't mean an opaque builder based on NSPredicates or whatever, I mean a query language, written as text, which is portable across clients.) OT is great, but it doesn't have that until it's actually exposed. My pet use case is string edits, which absolutely need to be cleanly conflict-resolvable. The inconvenience with offline-first is that by definition you're never operating on current data: You have to sync to a central server before the data is visible to others, and you're editing potentially stale data. The latency of inbound syncing is particularly dependent on the volume of changes being sent from upstream, which might require that a client, so as not to be overwhelmed with updates, subscribe to a very small subset of the entire dataset. For example, if you open up a UI that edits a single document, you might want to subscribe to changes to only that document, in order to update the UI in real time. I don't know if that's even possible with Realm, or if you can in fact run it in "classic client/server" (i.e. "online first") mode.
HodGreeley 10y ago
NoSQL databases arose in response to the growth and change in data use, driven largely by the web. Typical needs cited include scalability, availability, performance, and flexibility. Development has been going on for a long time. I think most would agree there is no longer a one size fits all (or even most). Realm (and I know someone will correct me if I'm wrong) started out as an on-device only replacement for relational-style dbs focused on ease-of-use and performance. It's not the only one. Couchbase Mobile is a NoSQL stack that has an on-device db that's also easier to use, and has focused for many years on getting sync right (which is a hard problem).

by danbruc 10y ago

jinjin2 10y ago

Realm has been available for many years and is actually really different from what else is out there by the fact that it is a real object database. It has just been flying under the radar a bit given that it has only been available for mobile devices. I for one think that it is awesome that it is now available for node.js as well. Especially with the new replication functionality that allows you to share live data between node instances running on different machines.

lobster_johnson 10y ago
"Real" object databases — "OODBMS" — uses to be a thing, too. They still survive in some niches, like finance and health, though largely as legacy systems. Products like Objectivity, ObjectStore, POET, Caché (which is built on MUMPS), GemStone (Smalltalk) etc. were once big, expensive products that seemed poised to take over from old-hat SQL databases. Those databases failed not necessarily because the technology was wrong, although they certainly made some choices that didn't help; those databases were often tightly coupled with opaque binary object serialization formats (with no way of looking into a database without going the binary route) that were in turn tightly coupled with the app's class hiearchy. For example, several of these actually post-processed your code (e.g. your Java bytecode) to insert serialization code, so that when you followed a relationship (e.g. "library.books[0].author"), it would automatically query under the hood and return the author as an object. They ended up being clunky to work with as a result; for example, in the beginning, very few of them had a query language, so the only way to query the data was by following relationships through your graph of objects. Realm looks interesting as a new approach to the object database design, but at the same time seems a bit limited. It doesn't seem to have schemas or any kind of role-based permissions, and seems to make the same mistake of not including a query language (so you can't build a REPL for it, and every language binding needs its own query builder). The transaction semantics also seem to be entirely undefined (is it read-committed?). From what I can tell, it's also "offline first", in the sense that you have a local database that can sync to/from a server, but you're technically always working on an offline copy of the master data; while nice to have for a mobile app, it does limit what you can use it for. I also don't see any support for fine-grained updates. Every write you do seems to overwrite, which I assume means the last write always wins; how do you deal with conflicts, or implement things like counters, maps or arrays that need to be incrementally updated?
- brmunk 10y ago
  Realm certainly has a schema - anything that made you think it doesn't? It's ACID and Serializable. It also has a query language. But as one of the core design principles is to be as native to the language and platform as possible, the query language takes different forms. In Java it's a fluent API, in Cocoa it's NSPredicates string based. And the JS string based is roughly identical to that as well. It's absolutely designed to be offline first. I'm curious as to which limitations you see with that? Operational Transform is used to resolve conflicts at the property level, so I would say it's much more fine-grained than most other systems that would overwrite entire objects. The specific resolution rules are very simple and intuitive and how to deal with custom needs are described here: https://realm.io/docs/realm-object-server/#conflict-resolution https://realm.io/docs/realm-object-server/#conflict-resoluti.... There is ordered lists already and explicit counters are being exposed in the SDKs at the moment. You might find this article useful as well: https://realm.io/news/eventually-consistent-making-a-mobile-first-distributed-system/ https://realm.io/news/eventually-consistent-making-a-mobile-...
  - lobster_johnson 10y ago
    I couldn't, and still can't, find any documentation or "list of features" that mention schema support, query language or ACID semantics. The documentation is pretty sparse and disorganized. (By query language I don't mean an opaque builder based on NSPredicates or whatever, I mean a query language, written as text, which is portable across clients.) OT is great, but it doesn't have that until it's actually exposed. My pet use case is string edits, which absolutely need to be cleanly conflict-resolvable. The inconvenience with offline-first is that by definition you're never operating on current data: You have to sync to a central server before the data is visible to others, and you're editing potentially stale data. The latency of inbound syncing is particularly dependent on the volume of changes being sent from upstream, which might require that a client, so as not to be overwhelmed with updates, subscribe to a very small subset of the entire dataset. For example, if you open up a UI that edits a single document, you might want to subscribe to changes to only that document, in order to update the UI in real time. I don't know if that's even possible with Realm, or if you can in fact run it in "classic client/server" (i.e. "online first") mode.

HodGreeley 10y ago

NoSQL databases arose in response to the growth and change in data use, driven largely by the web. Typical needs cited include scalability, availability, performance, and flexibility. Development has been going on for a long time. I think most would agree there is no longer a one size fits all (or even most). Realm (and I know someone will correct me if I'm wrong) started out as an on-device only replacement for relational-style dbs focused on ease-of-use and performance. It's not the only one. Couchbase Mobile is a NoSQL stack that has an on-device db that's also easier to use, and has focused for many years on getting sync right (which is a hard problem).