5 ms·

Its usually only after a while you realize almost every piece of meaningful data is relational. It just didn't look that way when the project started. But now y

Its usually only after a while you realize almost every piece of meaningful data is relational. It just didn't look that way when the project started. But now you're committed on the wrong database and its very costly to switch back to SQL. Literally every project I saw using MongoDB ended up going back to SQL within the first 2 years after realizing the data is indeed very much relational and theres no clean way to model it using documents. You always end up with either tons of duplication across documents, which is hell to maintain, or tons of multi-document queries with hacks to look ACID, which is also hell to maintain. Sure Mongo makes it easy to prototype applications, but it makes it very complex to build robust and maintainable software. Its especially bad if you think your data isn't relational, because it almost certainly is. Disclaimer: I believe Datomic to be the game changing database; because it values simplicity and composition and these attributes drive the entire design.

fnayr 9y ago
Video games storing player data are a great example of nonrelational data. I'm intending on writing a blog post after I finish my game detailing the structure of data I store and why it was so perfect for mongodb.
- javiermaestro 9y ago
  Please do, because I'm interested in seeing how non-relational it is... I truly believe it has to be otherwise, so I'd love to read your findings!
  - imtringued 9y ago
    You still need ACID transactions over multiple entries even if your data is nonrelational otherwise there is the potential for item and money duping bugs. A simple example would be a marketplace. Player buys item X with Y gold from another player. 1. Server checks that item X exists. 2. Server checks that the player has at least Y gold. 3. Server removes the gold from the player 4. Server gives gold to the seller. 5. Server removes item from marketplace. 6. Server adds item to inventory. What if someone maliciously crafts two requests in a way that step 2 of the second request happens before step 3 of the first request? The money is deducted properly but the account can now have a negative balance and there are now two instances of the item.
- mustardo 9y ago
  On the surface it sounds like you might have a case for Mongo, lookout for scenarios like... * Trading in game items between two users (needs multi document atomic locks if you don't want duplicate or lost items) assuming your "schema" is a document per user * You want to rename or restructure an attribute in the future, with no schema it's not possible change migrate data easily without writing ad hoc code (maybe you can use third party tools) or changing queries to expect data in multiple "schemas" which quickly gets painful Good Luck!
  - fnayr 9y ago
    Yes players can sell items to other players, so that's the one place so far I've needed to worry about atomicity, but even mongodb docs give examples with how to deal with something like that: https://docs.mongodb.com/manual/tutorial/perform-two-phase-commits/ https://docs.mongodb.com/manual/tutorial/perform-two-phase-c... So yes, it's annoying for a very small % of what I'm doing, but 99% of my updates/writes are within a single document, so I find it very nice for development.
  - drmirror 9y ago
    About the first use case, well it turns out MongoDB just introduced multi-document transactions today.
    - mustardo 9y ago
      If only someone had posted that information to HN
  - mygo 9y ago
    > You want to rename or restructure an attribute in the future, with no schema it's not possible change migrate data easily without writing ad hoc code (maybe you can use third party tools) or changing queries to expect data in multiple "schemas" which quickly gets painful You can have schemas with MongoDB. There are various libraries to facilitate database design by schema specification. Also renaming or restructuring your data is not necessarily an easy task with SQL. The nature of a database dictates that how good it works for your application depends on up to how well thought-out your schema is. Having to change your schema around is tasking. One of the reported advantages of document stores when they were becoming trendy was that it was easy to change your schema since your schema is essentially determined and regulated at the application layer. Also MongoDB has ACIDic transactions now (freaking finally) so if it’s as-advertised then I feel like half of your argument is not really a strong one any more.
nemo44x 9y ago
Most meaningful data is in fact not relational. When you consider machine generated data, logs, metrics, network event data and all other types of data like this you find there are not relations in it. This data is meaningful because it allows to analyze what's going on over massive systems, detect when problems will happen, find bottle necks in applications and infrastructure, among many other use cases. Application domain data tends to be relational I'd agree. But in general, this makes up a very small percentage of meaningful data in the world.
- aeorgnoieang 9y ago
  > When you consider machine generated data, logs, metrics, network event data and all other types of data like this you find there are not relations in it. I find that hard to believe. Maybe not that the raw data isn't already relational, but that there are no relations real or implied. If logs contain info about 'things' and any of those things can be considered to be the 'same thing' for multiple entries, then there's a relation right there – entry to thing. And even metrics and network event data I'd expect to be full of cryptic IDs that reference some 'thing', i.e. a typical 'code' for which it's really nice to have a table with at least a friendly description. Admittedly some of this data – or maybe even most of this data – isn't very 'deeply relational', but it definitely seems that claiming that "there are [no] relations in it" isn't strictly true.
  - nemo44x 9y ago
    Well, it's all a point of view really. The data is in the form of an "event". An occurrence of a fact which occurred at a particular time and has data associated with it. So therefore, a "relation" as constructed in a relational database isn't appropriate. You aren't truly denormalizing the data when repeating ID's or tags or labels in this type of data. This is because, at that time that was in the fact the associated ID, tag, label, etc. It would make the stored event false if a field associated with it were to be changed as at the time of the event, that field did not have that value. But it's all semantics really at that point. Anyways, a relational database is a poor solution for this type of data. The stored data gains little to nothing, and may even negatively affect it's integrity (at time t, the event DID have this ID; it DID have this label), when stored relationally. Each event is discrete and there will be many of them which optimizes better for scale than relational organization. I guess my point was there is vastly more useful data appropriate for a non-relational database than there is for relational databases. You might say it still has a "relation" in an abstract sense but this data does not need relational semantics within the database it resides in.
ZenoArrow 9y ago
> "I believe Datomic to be the game changing database" Have you used it? If so, what was your use case?

fnayr 9y ago

Video games storing player data are a great example of nonrelational data. I'm intending on writing a blog post after I finish my game detailing the structure of data I store and why it was so perfect for mongodb.

javiermaestro 9y ago
Please do, because I'm interested in seeing how non-relational it is... I truly believe it has to be otherwise, so I'd love to read your findings!
- imtringued 9y ago
  You still need ACID transactions over multiple entries even if your data is nonrelational otherwise there is the potential for item and money duping bugs. A simple example would be a marketplace. Player buys item X with Y gold from another player. 1. Server checks that item X exists. 2. Server checks that the player has at least Y gold. 3. Server removes the gold from the player 4. Server gives gold to the seller. 5. Server removes item from marketplace. 6. Server adds item to inventory. What if someone maliciously crafts two requests in a way that step 2 of the second request happens before step 3 of the first request? The money is deducted properly but the account can now have a negative balance and there are now two instances of the item.
mustardo 9y ago
On the surface it sounds like you might have a case for Mongo, lookout for scenarios like... * Trading in game items between two users (needs multi document atomic locks if you don't want duplicate or lost items) assuming your "schema" is a document per user * You want to rename or restructure an attribute in the future, with no schema it's not possible change migrate data easily without writing ad hoc code (maybe you can use third party tools) or changing queries to expect data in multiple "schemas" which quickly gets painful Good Luck!
- fnayr 9y ago
  Yes players can sell items to other players, so that's the one place so far I've needed to worry about atomicity, but even mongodb docs give examples with how to deal with something like that: https://docs.mongodb.com/manual/tutorial/perform-two-phase-commits/ https://docs.mongodb.com/manual/tutorial/perform-two-phase-c... So yes, it's annoying for a very small % of what I'm doing, but 99% of my updates/writes are within a single document, so I find it very nice for development.
- drmirror 9y ago
  About the first use case, well it turns out MongoDB just introduced multi-document transactions today.
  - mustardo 9y ago
    If only someone had posted that information to HN
- mygo 9y ago
  > You want to rename or restructure an attribute in the future, with no schema it's not possible change migrate data easily without writing ad hoc code (maybe you can use third party tools) or changing queries to expect data in multiple "schemas" which quickly gets painful You can have schemas with MongoDB. There are various libraries to facilitate database design by schema specification. Also renaming or restructuring your data is not necessarily an easy task with SQL. The nature of a database dictates that how good it works for your application depends on up to how well thought-out your schema is. Having to change your schema around is tasking. One of the reported advantages of document stores when they were becoming trendy was that it was easy to change your schema since your schema is essentially determined and regulated at the application layer. Also MongoDB has ACIDic transactions now (freaking finally) so if it’s as-advertised then I feel like half of your argument is not really a strong one any more.

nemo44x 9y ago

Most meaningful data is in fact not relational. When you consider machine generated data, logs, metrics, network event data and all other types of data like this you find there are not relations in it. This data is meaningful because it allows to analyze what's going on over massive systems, detect when problems will happen, find bottle necks in applications and infrastructure, among many other use cases. Application domain data tends to be relational I'd agree. But in general, this makes up a very small percentage of meaningful data in the world.

aeorgnoieang 9y ago
> When you consider machine generated data, logs, metrics, network event data and all other types of data like this you find there are not relations in it. I find that hard to believe. Maybe not that the raw data isn't already relational, but that there are no relations real or implied. If logs contain info about 'things' and any of those things can be considered to be the 'same thing' for multiple entries, then there's a relation right there – entry to thing. And even metrics and network event data I'd expect to be full of cryptic IDs that reference some 'thing', i.e. a typical 'code' for which it's really nice to have a table with at least a friendly description. Admittedly some of this data – or maybe even most of this data – isn't very 'deeply relational', but it definitely seems that claiming that "there are [no] relations in it" isn't strictly true.
- nemo44x 9y ago
  Well, it's all a point of view really. The data is in the form of an "event". An occurrence of a fact which occurred at a particular time and has data associated with it. So therefore, a "relation" as constructed in a relational database isn't appropriate. You aren't truly denormalizing the data when repeating ID's or tags or labels in this type of data. This is because, at that time that was in the fact the associated ID, tag, label, etc. It would make the stored event false if a field associated with it were to be changed as at the time of the event, that field did not have that value. But it's all semantics really at that point. Anyways, a relational database is a poor solution for this type of data. The stored data gains little to nothing, and may even negatively affect it's integrity (at time t, the event DID have this ID; it DID have this label), when stored relationally. Each event is discrete and there will be many of them which optimizes better for scale than relational organization. I guess my point was there is vastly more useful data appropriate for a non-relational database than there is for relational databases. You might say it still has a "relation" in an abstract sense but this data does not need relational semantics within the database it resides in.

ZenoArrow 9y ago

> "I believe Datomic to be the game changing database" Have you used it? If so, what was your use case?