7 ms·

> Because some people can't stand having to work with SQL,migrations,schema and constraints The thing is, if you actually try and write an app using MongoDB, y

by Lazare 10y ago

> Because some people can't stand having to work with SQL,migrations,schema and constraints The thing is, if you actually try and write an app using MongoDB, you will rapidly find that you: 1) Have migrations (except they're going to be some scary ad hoc nodejs script that loop through your document store and modify fields on the fly). 2) Have schemas (except they'll be implicit and undocumented) 3) Constraints (except they'll be hidden inside your app logic, and violating them will cause data corruption). The biggest lie about NoSQL databases is that they're schemaless. If you're EVER going to read the data back and do anything with it, it has a schema.

catnaroek 10y ago
Exactly. While the process of designing the structure of your data can make you feel like “you're not getting real work done”, in the long run, it actually prevents headaches caused by inconsistent data. Data always has a structure, it's just that some people are too lazy or mentally feeble to figure out what it is.
- collyw 10y ago
  For me that is the most important aspect of starting / designing an application. If the data model is accurate, then the code falls into place easily. If its not quite right, more and more code ends up in the application trying to make up for the poor data model.
  - varunom 10y ago
    I second this. My first task in any project is to design the whole data model based on current requirements and while designing it I think of the interfaces and how would they read and write data (to refine requirements). Writing views and actions/APIs on top of well-formed data model then becomes a breeze.
    - collyw 10y ago
      Its so un-agile (but it works).
      - nathan_long 10y ago
        Agile doesn't mean "don't gather requirements or plan anything." It just means that you evaluate your results frequently and maybe change course, instead of waiting until the end when you're "done".
red_hare 10y ago
This. My company used mongo for years before we got our shit together. Schmemas were always implicit (until we got our shit together and started defining and enforcing them with Python Schematics). Migrations were crazy scripts you run in prod or hacks you stick into your code to "transition". And yes, surprise constrains left and right causing awful anti-patterns. One-character key names to save disk. Hashed values for indexed keys to save memory. Awkward structuring to improve query performance. The worst part is, we now have tons of important data in these databases and almost no one understands the legacy crazy app logic that makes them tick.
- xapata 10y ago
  > used mongo for years before we got our shit together. That's actually a legit use case. Use MongoDB while you get your shit together. I use global variables while I'm noodling around in code. Eventually I refactor.
  - arctor 10y ago
    The problem is, in a great portion of real world projects "eventually" never comes and there's just no time for any major refactoring or replacing technologies since you are too busy implementing the feature that was needed two weeks ago.
    - derefr 10y ago
      I've often dreamed of a specific type of software built and released as "prototypeware", where any app created using it will have certain built-in scaling limits—and going past them will irrevocably force the app into a read-only mode. It would warn anyone monitoring it well in advance of hitting such a limit, of course. But there'd be no way to just slide the limit upward or otherwise tarry. It'd force the migration to something better just as if it were a Big Customer with Enterprise Compliance Demands. If an enforceable mechanism like that existed, I'd be a lot more confident in mocking things up. Stick SQLite in for the database, munge HTML and Javascript together, whatever—it's literally going to slap away the hand of anyone who tries to use it on a production workload, so why not? (Going further, it'd be interesting to create some sort of quagmire of a software license, specifically for prototypeware, such that you'd be forced to rewrite all the prototype code instead of reusing even a hair of it in production. Maybe something like reassigning the IP to a trust, with the trust having an obligation to sue anyone and everyone who tries to create derivative works of the code they've been handed?)
michel-slm 10y ago
Schema on read, as opposed to schema on write, as it says in the excellent "Designing Data Intensive Applications" book ... nightmarish to deal with.
nevi-me 10y ago
Mongo has its weaknesses, yes its main strength is cited as its simplicity, or that its quick to get something out the door. I agree with your last comment. I can't help but to laugh at people who think they would get away with designing a database with no schema. Schemaless for me meant that unless you enforce constraints, there won't be any. There's a reason why there are ORMs even though Mongo drivers are sufficient for most cases. 1) I've always designed my data with future changes in mind. I often spend up to an hour thinking of possibilities of data that I want to store in a collection, before writing the schema. The flexibility i have with Mongo is that if I think I need a field but am unsure of the exact data type to store, i.e. is it a string or array of strings, or array of objects with strings? In that case I just leave the field as an object and change it later. The plus being that as long as I haven't stored anything with that field, I can always change its type without a 'migration script'. I've only needed to 'migrate' by updating documents 4-5 time. When GeoJSON landed, and a few other times when I needed small changes to my data. 3) The only way I can think of enforcing constraints on < 3.2 is through indices, which is insufficient. Most ORMs do the enforcing. I've never needed to enforce them at an app level. I've used MongoDB primarily for its Geo support, and JSON enabling me to get things done quicker relative to maintaining SQL tables. I've got a small but interesting use case, public transit. https://movinggauteng.co.za https://movinggauteng.co.za and https://rwt.to https://rwt.to. When I started with the projects, PostgreSQL + PostGIS felt like a black box, and I wanted something that would give me ease and flexibility. At the time hstore was the talk of the day, but seemed to not meet my needs. It would now with JSON, but I'll stick with Mongo for now.
nulagrithom 10y ago
This list makes me want to cry a little. It rings too true. > Have migrations (except they're going to be some scary ad hoc nodejs script that loop through your document store and modify fields on the fly). I literally just spent the better part of tonight AND yesterday evening dealing with one of these scripts. I had pulled down the production table to locally test the script (gross), but when I later ran it in the production environment, we'd somehow had an array sneak in to what was an object field. The whole thing just felt like a mess.
- Lazare 10y ago
  Oh god, I'm getting flashbacks. Because you can't just test it on one document and see if it works; you have no guarantee that all the documents will be identical. And if the migration script crashes halfway through... oh man.
  - krylon 10y ago
    > And if the migration script crashes halfway through... oh man. Schema-issues and typing aside, I looked at MongoDB just long enough to find out there are no transactions, then ran away, quickly. For a lot of tasks, I guess I would find MongoDB very useful, but lack of transactions is a complete deal breaker for me. Not having a real schema, referential integrity and all that makes them even more important, IMHO. At work, I have had more than one quickly-hacked-together Perl script crash on me in the middle of a run. Having proper transactions has saved my butt repeatedly.
  - nathan_long 10y ago
    By contrast, with Postgres you'd have an explicit schema which all records obey, and even migrations can be done in transactions.
nrinaudo 10y ago
I often liken NoSQL databases to dynamically typed languages. With a NoSQL database, you have an implicit schema, but it will only be enforced and fail at runtime - when your code expects a field but failed to find it, for instance. With a dynamically typed language, you have implicit types, but only enforced at runtime - when your code expects a value to be an int but finds a boolean, for instance. And both are fine, there is a need for both. I can see how the flexibility of being able to change, well, everything by just flipping a switch in your head ("this is an int now") might be helpful for, say, data exploration problems. It's just that in a production environment, these features of NoSQL databases and dynamically typed languages turn into massive sources of problems and oh god, just don't.
- eloff 10y ago
  You and Lazare are right on the money. And the thing with the database is that the code that inserts/updates it has to agree with all the querying code about what the implicit schema should be - but it's implicit and scattered around your code - so on a large team it's very hard for everyone to understand that implicit contract and it's going to be a constant source of production bugs. Schemas don't change that much compared to code, having a strict schema enforced by the database saves you so much time and pain and downtime in the long run.
mgkimsal 10y ago
#2 - it's basically "schema on read" vs "schema on write". You always have a schema; where and how it's defined is the only question.
matwood 10y ago
To add to your point about schemas. The new generation has not learned that the data almost always outlives whatever throwaway front end was written to work with said data. Tying the data to some sort of flavor of the month framework is setting up for all sorts of pain later. I despise mysql, but even it is better than mongo. At least with it I can easily transition the data to many different uses. Also, and a point I find amusing is that many users of nosql claim schemaless and then go and write a layer on top of the datastore to enforce a schema. It would have been so much simpler to use a RDMS out the gate instead of badly implementing one.
Idontreddit 10y ago
> If you're EVER going to read the data back and do anything with it, it has a schema. You are giving the NoSQL crowd too much credit. Some abominations have no recognizable schema at all. The data store will just contain arbitrary dump of data which different developers decided their "schema" should be. The number of "columns" will vary, the "columns" will have arbitrary formats, so on and so forth. If one developer decided to separate name into "first: John", "last: Doe", you will have that. If another decided to have "name: John Doe". That's what will be there. If one developer decided social security should be "SSN: 123-45-6789" and another decided it should "SSN: 123456780", well you are going to have fun cleaning up the data at the business or even application layer. But that's not even the big issue with MongoDB. It's their lack of ACID compliance!

by Lazare 10y ago

catnaroek 10y ago

Exactly. While the process of designing the structure of your data can make you feel like “you're not getting real work done”, in the long run, it actually prevents headaches caused by inconsistent data. Data always has a structure, it's just that some people are too lazy or mentally feeble to figure out what it is.

collyw 10y ago
For me that is the most important aspect of starting / designing an application. If the data model is accurate, then the code falls into place easily. If its not quite right, more and more code ends up in the application trying to make up for the poor data model.
- varunom 10y ago
  I second this. My first task in any project is to design the whole data model based on current requirements and while designing it I think of the interfaces and how would they read and write data (to refine requirements). Writing views and actions/APIs on top of well-formed data model then becomes a breeze.
  - collyw 10y ago
    Its so un-agile (but it works).
    - nathan_long 10y ago
      Agile doesn't mean "don't gather requirements or plan anything." It just means that you evaluate your results frequently and maybe change course, instead of waiting until the end when you're "done".

red_hare 10y ago

This. My company used mongo for years before we got our shit together. Schmemas were always implicit (until we got our shit together and started defining and enforcing them with Python Schematics). Migrations were crazy scripts you run in prod or hacks you stick into your code to "transition". And yes, surprise constrains left and right causing awful anti-patterns. One-character key names to save disk. Hashed values for indexed keys to save memory. Awkward structuring to improve query performance. The worst part is, we now have tons of important data in these databases and almost no one understands the legacy crazy app logic that makes them tick.

xapata 10y ago
> used mongo for years before we got our shit together. That's actually a legit use case. Use MongoDB while you get your shit together. I use global variables while I'm noodling around in code. Eventually I refactor.
- arctor 10y ago
  The problem is, in a great portion of real world projects "eventually" never comes and there's just no time for any major refactoring or replacing technologies since you are too busy implementing the feature that was needed two weeks ago.
  - derefr 10y ago
    I've often dreamed of a specific type of software built and released as "prototypeware", where any app created using it will have certain built-in scaling limits—and going past them will irrevocably force the app into a read-only mode. It would warn anyone monitoring it well in advance of hitting such a limit, of course. But there'd be no way to just slide the limit upward or otherwise tarry. It'd force the migration to something better just as if it were a Big Customer with Enterprise Compliance Demands. If an enforceable mechanism like that existed, I'd be a lot more confident in mocking things up. Stick SQLite in for the database, munge HTML and Javascript together, whatever—it's literally going to slap away the hand of anyone who tries to use it on a production workload, so why not? (Going further, it'd be interesting to create some sort of quagmire of a software license, specifically for prototypeware, such that you'd be forced to rewrite all the prototype code instead of reusing even a hair of it in production. Maybe something like reassigning the IP to a trust, with the trust having an obligation to sue anyone and everyone who tries to create derivative works of the code they've been handed?)

michel-slm 10y ago

Schema on read, as opposed to schema on write, as it says in the excellent "Designing Data Intensive Applications" book ... nightmarish to deal with.

nevi-me 10y ago

Mongo has its weaknesses, yes its main strength is cited as its simplicity, or that its quick to get something out the door. I agree with your last comment. I can't help but to laugh at people who think they would get away with designing a database with no schema. Schemaless for me meant that unless you enforce constraints, there won't be any. There's a reason why there are ORMs even though Mongo drivers are sufficient for most cases. 1) I've always designed my data with future changes in mind. I often spend up to an hour thinking of possibilities of data that I want to store in a collection, before writing the schema. The flexibility i have with Mongo is that if I think I need a field but am unsure of the exact data type to store, i.e. is it a string or array of strings, or array of objects with strings? In that case I just leave the field as an object and change it later. The plus being that as long as I haven't stored anything with that field, I can always change its type without a 'migration script'. I've only needed to 'migrate' by updating documents 4-5 time. When GeoJSON landed, and a few other times when I needed small changes to my data. 3) The only way I can think of enforcing constraints on < 3.2 is through indices, which is insufficient. Most ORMs do the enforcing. I've never needed to enforce them at an app level. I've used MongoDB primarily for its Geo support, and JSON enabling me to get things done quicker relative to maintaining SQL tables. I've got a small but interesting use case, public transit. https://movinggauteng.co.za https://movinggauteng.co.za and https://rwt.to https://rwt.to. When I started with the projects, PostgreSQL + PostGIS felt like a black box, and I wanted something that would give me ease and flexibility. At the time hstore was the talk of the day, but seemed to not meet my needs. It would now with JSON, but I'll stick with Mongo for now.

nulagrithom 10y ago

This list makes me want to cry a little. It rings too true. > Have migrations (except they're going to be some scary ad hoc nodejs script that loop through your document store and modify fields on the fly). I literally just spent the better part of tonight AND yesterday evening dealing with one of these scripts. I had pulled down the production table to locally test the script (gross), but when I later ran it in the production environment, we'd somehow had an array sneak in to what was an object field. The whole thing just felt like a mess.

Lazare 10y ago
Oh god, I'm getting flashbacks. Because you can't just test it on one document and see if it works; you have no guarantee that all the documents will be identical. And if the migration script crashes halfway through... oh man.
- krylon 10y ago
  > And if the migration script crashes halfway through... oh man. Schema-issues and typing aside, I looked at MongoDB just long enough to find out there are no transactions, then ran away, quickly. For a lot of tasks, I guess I would find MongoDB very useful, but lack of transactions is a complete deal breaker for me. Not having a real schema, referential integrity and all that makes them even more important, IMHO. At work, I have had more than one quickly-hacked-together Perl script crash on me in the middle of a run. Having proper transactions has saved my butt repeatedly.
- nathan_long 10y ago
  By contrast, with Postgres you'd have an explicit schema which all records obey, and even migrations can be done in transactions.

nrinaudo 10y ago

I often liken NoSQL databases to dynamically typed languages. With a NoSQL database, you have an implicit schema, but it will only be enforced and fail at runtime - when your code expects a field but failed to find it, for instance. With a dynamically typed language, you have implicit types, but only enforced at runtime - when your code expects a value to be an int but finds a boolean, for instance. And both are fine, there is a need for both. I can see how the flexibility of being able to change, well, everything by just flipping a switch in your head ("this is an int now") might be helpful for, say, data exploration problems. It's just that in a production environment, these features of NoSQL databases and dynamically typed languages turn into massive sources of problems and oh god, just don't.

eloff 10y ago
You and Lazare are right on the money. And the thing with the database is that the code that inserts/updates it has to agree with all the querying code about what the implicit schema should be - but it's implicit and scattered around your code - so on a large team it's very hard for everyone to understand that implicit contract and it's going to be a constant source of production bugs. Schemas don't change that much compared to code, having a strict schema enforced by the database saves you so much time and pain and downtime in the long run.

mgkimsal 10y ago

#2 - it's basically "schema on read" vs "schema on write". You always have a schema; where and how it's defined is the only question.

matwood 10y ago

To add to your point about schemas. The new generation has not learned that the data almost always outlives whatever throwaway front end was written to work with said data. Tying the data to some sort of flavor of the month framework is setting up for all sorts of pain later. I despise mysql, but even it is better than mongo. At least with it I can easily transition the data to many different uses. Also, and a point I find amusing is that many users of nosql claim schemaless and then go and write a layer on top of the datastore to enforce a schema. It would have been so much simpler to use a RDMS out the gate instead of badly implementing one.

Idontreddit 10y ago

> If you're EVER going to read the data back and do anything with it, it has a schema. You are giving the NoSQL crowd too much credit. Some abominations have no recognizable schema at all. The data store will just contain arbitrary dump of data which different developers decided their "schema" should be. The number of "columns" will vary, the "columns" will have arbitrary formats, so on and so forth. If one developer decided to separate name into "first: John", "last: Doe", you will have that. If another decided to have "name: John Doe". That's what will be there. If one developer decided social security should be "SSN: 123-45-6789" and another decided it should "SSN: 123456780", well you are going to have fun cleaning up the data at the business or even application layer. But that's not even the big issue with MongoDB. It's their lack of ACID compliance!