10 ms·

100% of my friends who have used Mongo/similar NoSQL have given up and had a nasty rewrite back to pgSQL. This seems to be the journey: 1. Lack of migrations

100% of my friends who have used Mongo/similar NoSQL have given up and had a nasty rewrite back to pgSQL. This seems to be the journey: 1. Lack of migrations is awesome! We can iterate so quickly for MVP 2. Get users 3. Add features, still enjoying the speed of iteration 4. Get more users 5. Start building reporting features for enterprise/customer support/product metrics (ie: when the real potential success starts) 6. Realise you desperately need joins, transactions and other SQL features 7. Pause product dev for 1-3+ months to migrate back to SQL, or do some weird parallel development process to move it piecemeal back. I think the most interesting question though is would they be able to get MVP and initial customers that set off this if they were moving (slightly) slower due to SQL and slight overhead that comes with? My thought is definitely yes.

brandur 9y ago
> I think the most interesting question though is would they be able to get MVP and initial customers that set off this if they were moving (slightly) slower due to SQL and slight overhead that comes with? I've used Postgres and Mongo pretty extensively, and for any reasonably seasoned developer, the startup overhead of an SQL system is a myth. There may upfront cost to learning how an RDMS and SQL work in the first place, but once you're familiar with them, they'll be faster than Mongo on any new project. The schemaless concept of a document database seems to be the major selling factor in velocity of movement, but once you've got a good handle on a migration framework in the vein of ActiveRecord or other popular software, that's negated completely. It also really doesn't take long before schemaless starts to cause big problems for you in terms of data consistency -- it's not just the big players that get bitten by this. The simplified query language is another one. SQL is a little bit obtuse, but it's not that bad once you have a handle on it, and a lot of people are familiar with it. Once you add in an ORM layer, the lazy-style access of a framework like Sequel or SQLAlchemy makes the developer experience quite a bit better than any Mongo APIs that I've seen. Also, after you get beyond trivial usage, SQL's flexibility so wildly outstrips Mongo's query documents that it's not even worth talking about. Postgres on the other hand ships with a great management CLI, a very powerful REPL (psql), and features like data types/constraints/transactions that guarantee you correctness with zero effort on your part. I can only speak for myself, but I'd take Postgres to the hackathon any day of the week.
- martinald 9y ago
  I totally agree with you, and started writing something about how understanding a good ORM takes nearly all the headache away. I think the thing people do find slow is a lot of 'documents within documents' in SQL. It turns out this is usually a bad development pattern long term but it is super fast being able to just add docs inside docs with no configuration. It feels very slow writing foreign keys, navigation props and schemas for this in SQL vs JSON, where you can just dump your object in and you're done. Basically; I think with noSQL you get some very short term gain for a lot of long term pain, and you're right, ORMs and other tooling solves this mostly. I myself fell for this trap, and while it was a nightmare it actually matured me more as a professional more than anything I've ever done recently. Regardless of crazy hype, I don't think I'll ever fall for a solution so easily without evaluating it properly. I think I assumed the "crowd" had done the tech due diligence on this stuff and it definitely wasn't the case.
  - macintux 9y ago
    I agree that Postgres trumps Mongo for most use cases, but if ORM is the answer, you might be asking the wrong question. I love that Fowler discusses[1] how to avoid ORMs and offers only two possibilities: > Either you use the relational model in memory, or you don't use it in the database. He clearly has a blind spot. The correct answer, at least in some cases, is don't use objects in the first place. [1]: https://www.martinfowler.com/bliki/OrmHate.html https://www.martinfowler.com/bliki/OrmHate.html
    - anentropic 9y ago
      I've been quite skeptical of the anti-ORM sentiments, since I always found the ORM handy, not a problem etc. Two things recently started to change my mind One is, after profiling some views in our Django app I found that a significant portion of the time in list views (i.e. returning many objects) was spent instantiating ORM model objects. Not on fetching them from the db, but turning them into objects whose imminent destiny is to get turned back into dicts for serializing to JSON. So we switched to values() queries which return plain dicts, but then the ORM is just a query engine. The second was, in a large app, finding that many of the tables start to require a degree of care to only perform 'safe' queries, i.e. ones supported by appropriate indexes. The integration of the ORM into the framework provides many conveniences, but this tends to involve a lack of awareness/control over exactly which queries get made. Most of the patterns that are 'idiomatic' to the framework also tend to involve a lot of ORM querying logic pushed down in the view layer... while our need to perform only a restricted set of 'safe' queries pushed us in the other direction, towards a repository layer with carefully curated query code. I'm seriously considering exploring ORM-less development in future
wolco 9y ago
I'm doing the same for another project. Reporting with $wind can be limiting Schemaless db is great feature. Joins can be done through an orm like Laravel's eloquent which allows you to join mysql to mongo using a cross-join table on the mysql side. Adding columns in an mpv product is trivial. Getting a stream of unstructured data and being able to save the different data to a document that is part of the same collection is an amazing feature that is needed still.
- DaiPlusPlus 9y ago
  Adding a non-NULL column to an existing table with millions of rows, and then updating the entity/model objects upwards towards the UI is non-trivial - I find myself "queuing up" planned database schema changes and applying them in a batch. What I'd love is a traditional database system with the concept of "cheap columns" that store "just data" scalar values, which can be easily created, dropped and manipulated - they're addressable so they can be used in queries and statistics/analytics, but the values themselves won't need indexes or referential-integrity - and for an ORM and Scaffolding system to support them - that way I won't pull out my hair when the client says one day "let's go from a single column for the customer's name to having separate title/first/middle/last/suffix columns"... and back again the week after.
  - jdmichal 9y ago
    So something like PostgreSQL `json` and `jsonb` types? You can select them out into columns, so they should play nicely with ORMs. https://www.postgresql.org/docs/current/static/datatype-json.html https://www.postgresql.org/docs/current/static/datatype-json...
  - solidsnack9000 9y ago
    It seems like the "column families" of Cassandra would appeal to you.
jwilliams 9y ago
I largely agree with your journey and your premise. However, I would add that I've seen just as many beast-of-a-thing SQL databases as NoSQL ones. In fact, anything I've seen at sufficient scale has lots of wrangling with their database. While your step 7 usually ends up with a better tech result, it's done at a time when the organization knows the problem domain much (much) better. They have real users. You'd probably get a similar result going from one SQL database to another.
athenot 9y ago
In the words of one of my former DBAs: "MySQL makes a pretty good noSQL database". That is to say you can do a quick and dirty key-value store in MySQL and it will be plenty fast. When you're ready to start normalizing things, just start pulling values out into their own columns and/or tables. Conversely, when starting a product, don't worry about deeply normalized databases: use a relational DB as a glorified key-value store, then normalize things as you go. You'll get the best of both worlds.
- sushisource 9y ago
  Yep. I've done the same thing by using JSONB in postgres.
- tracker1 9y ago
  I've done similar for most SQL databases having either an xml, or json column to store ancillary information that didn't need to be accessed or indexed in ways that aren't programmatic. For example, if you support Amazon Payments, Paypal and a payment gateway, the shape of the data for the transaction is different, and rarely looked at, why not shove it into a bigtext/blob field as xml/json.
rgo 9y ago
Here's my personal version of the story: With SQL DBs (Oracle, SQLServer and MySQL): 1. SQL database migrations where killing us. Going back and forward in a dev environment was impossible. No hot deploy in production. 2. Could not work well with application user-defined fields: adding columns adhoc to the database, indexing them, normalizing and denormalizing, performance issues, everything was a problem. 3. Blobs holding logging data got unmanageable quickly. 4. Joins where very hard to optimize even though the team had a lot of DBA experience fine tuning databases. 5. Had to build a very complex architecture around the database for a product that was not that complex: cache, search, database, blob store, distributed, etc. And with all our 1990s and 2000s previous experiences in data warehousing, business intelligence and DB optimization tools, we were still wasting valuable time with SQL design, indexing, query planning and parameter optimization. So we gave MongoDB a try. First as a cache. Later as the only DB. Our journey: 1. Heard about Mongo. Tried the DB. The driver worked great. To me that's the number one "marketing antics behind MongoDB": their strategy creating drivers and supporting the programmer community. 2. Understood what NoSQL meant and forgot about joins altogether. 3. Understood what NoSQL meant and built transactions into atomic documents. 4. Understood what NoSQL meant and stopped relying on the database for type, primary and foreign key constraints, default values, triggers (argh!), stored procedures (2x argh!), etc. 5. Simplified the architecture with integrated search, queue and cache. Less moving parts = joy. 6. Result: very low maintenance, easy install, configuration, replication and migrations. 99.999% availability. 7. Bonus: we even implemented a very high frequency, atomic distributed semaphore system with a FIFO queue that reaps zombies using Mongo built-in networking features. So we've reduced DB-related issues by an order of magnitude. How? I think because NoSQL is a way of saying the DB should not be magically answering random queries. A database should be a data store, period -- just store and retrieve data the way the app needs it. By focusing our energies on getting the data right as documents for a document store meant data flows as objects from code in and out of Mongo. I believe people underestimate how important (and productive) it is to keep the same data structures flowing between the UI (JSON), server (Object/Hash/Dictionary) and DB (document). It makes code easier to read and more resillient to errors. But SQL DBs come with a convenience layer bolted on to run random user queries with things like OUTER joins and GROUP BYs. For that we need to flatten data into tables, which clashes with typically how data flows in an app. SQL DBs however are great as the single source of truth for data: a schema can be laid out and enforced independently of code, so it's safely guarded from programmers breaking it. Business sets up a SQL DB so that their reporting people can query data on demand while consultants with zero knowledge of the business can write code limited by constraints managed by DBAs. SQL is even taught at business schools, which is revealing of who its target audience actually is. Bottom-line: SQL and schema enforcing are end-user features we did not need to build our tool. On the other hand, every single MongoDB feature is something we need and use profusely.
randomdata 9y ago
0. Finally, an API that actually attempts to fit modern programming paradigms! I'm not sure the unstructured nature of many NoSQL databases was even the selling feature so much as the NoSQL part. People were growing tired of dealing with lacklustre ORMs and poor integration with their programming tools and were desperate, even in light of the shortcomings, to try something new. It is funny that we have a thousand and one different general purpose programming languages that try to solve the problems with general purpose computing, but SQL dialects (with slight variation here and there), have very little competition, despite all the problems with the SQL language. A NoSQL relational database could have potentially hit the sweet spot, but Mongo and similar databases came first and caused enough trouble, as you have described, that I feel like they've killed the momentum in looking for better solutions to access database data (relational ones included). Nowadays many of those people are just happy to have a relational database again, even that means accepting SQL.
- jdmichal 9y ago
  > It is funny that we have a thousand and one different general purpose programming languages that try to solve the problems with general purpose computing, but SQL dialects (with slight variation here and there), have very little competition, despite all the problems with the SQL language. This is kinda like complaining that there's only one implementation of lambda calculus. Yes, SQL is not directly a relational algebra, but it's pretty close. (Also, tables are not relations and SQL deals with that.)
  - randomdata 9y ago
    > This is kinda like complaining that there's only one implementation of lambda calculus. Yes, SQL is not directly a relational algebra, but it's pretty close. And yet someone proficient in SQL will not necessarily be able to read a traditionally-written lambda calculus expression as the languages differ significantly (and not just because of tables), so I think this proves my point that there are different, and hopefully better, ways to express the interaction with the database. There is no reason why the protocol has to be exactly as SQL is. There are theoretically an infinite number of ways to communicate the same intent to a RDBMS. For instance, SQL does not do composability well. There is no fundamental reason why you should not be able to write query fragments, simplifying the way many queries are constructed and avoiding the huge mess that monolithic, hard to debug, queries often become. It is simply that SQL does not allow for it. A better API could improve on this significantly, without losing any features a relational database provides. You don't have to throw all database theory out the window just to change the API away from one that was designed 40 years ago.
rmrfrmrf 9y ago
This is kind of a ridiculous statement given the massive popularity of NoSQL DBs like Cassandra, Redis, and Lucene. Not to mention what you described seems like a perfectly reasonable architectural evolution. Are we all really expecting to make a single decision on a DB for the entirety of an app's life? It makes a lot of sense to use a flexible db schema while the data takes shape in an app's infancy, then solidify things with a more structured database schema further down the line. If you're abstracting things away in ORMs anyway, it might not even be that painful if you're just switching out DB drivers.
- jimktrains2 9y ago
  Most people don't use Redis and Lucene as general-purpose data stores; they're normally used for specific purposes.
  - thehardsphere 9y ago
    I would go so far to say that nobody should ever use Lucene as a data store. It's very easy for it to get corrupted because it was explictly designed to support fast full-text searches at the expense of persistence (because your data is supposed to be persisted elsewhere).
- camus2 9y ago
  Nobody uses Redis like a database and expects persistence of data. Redis is usually used in the context of a message queue. Lucene isn't a database, it's a full text search engine. Cassandra has a query language which is very close to SQL, so it's not really NoSQL.
  - jimktrains2 9y ago
    > Redis is usually used in the context of a message queue. Or caches, or simple data stores for things like counts.
  - manigandham 9y ago
    > Nobody uses Redis like a database and expects persistence of data. Plenty of people do, it's got just as good persistence and backups as anything else. It's very versatile and used for lots of situations, with message queues just being one option (although not a great fit until it gets a stream data type).
konschubert 9y ago
My feeling is that the people who flock to NoSQL have never used a good ORM such as Django to abstract away the raw SQL queries. A good ORM gives you the simplicity of NoSQL plus all the power of a relational database.
- thehardsphere 9y ago
  I think that is only true of a subset of NoSQL users. And even if it is true that Django's ORM is very nice, there are tradeoffs to using an ORM that mean you don't get "all the power" of whatever database you are using. Maybe all the power you care about; for instance, power in expressiveness and easy cross platform support, at the expense of performance optimizations that aren't important to you. But it's still a tradeoff.
  - konschubert 9y ago
    You're right.
moeamaya 9y ago
JSONB in pgSQL fills in so nicely for steps 1-4. Once you need columns and relationships, the relational store is already there!
manigandham 9y ago
Yes, relational databases are easy deal with. A single machine with proper backups can handle data needs for most companies. The explosion of managed service providers makes this even cheaper, faster and easier to start with. ORMs can do 95% of most project data access needs and a little bit of SQL knowledge allows for easily creating and editing tables - or use the numerous GUI tools out there. If anything, SQL actually lets you be more agile by allowing for quick queries to fix data or do a complicated 1-time report without having to write up and deploy code. This probably has more to do with the SQL language itself which is why it's surprising that MongoDB never tried to implement any of it.
- tracker1 9y ago
  But the cost of a server big enough to handle a million simultaneous users can be cost prohibitive if you can get away with a few smaller nodes and something that scales horizontally... not to mention failover/redundancy which can be downright expensive and/or painful in many SQL variants.
  - qaq 9y ago
    The cost of dev time having to reimplement some of the features available in RDBMS on top of NoSQL stores can be NxTimes the cost of of that box.
  - manigandham 9y ago
    That's the whole point - worry about that when you actually have a million simultaneous users, which is a huge number. There are plenty of options like reading from replicas, using a caching layer, or moving specific data to another datastore - all work that's better done when you actually have success and the resources to understand your data access patterns and implement a smart solution when it's needed. Basic replication has been part of every major relational database for decades. Where is it expensive or painful? Do you really need 100% uptime? There's a vast ecosystem for relational databases that can help with all of this and now you have CitusDB, MemSQL and CockroachDB if you need built-in next-gen scaling and availability.
vram22 9y ago
>I think the most interesting question though is would they be able to get MVP and initial customers that set off this if they were moving (slightly) slower due to SQL and slight overhead that comes with? Why the heck not? People (startups and others) were using RDBMSs and SQL for years before NoSQL was a thing. Also, if slight delay is a thing, how about the impact of your point 7 on customers?
mosselman 9y ago
What nonsense. I have never considered using postgres a hindrance due to migrations or something of the like. If anything, migrations and having a consistent database makes it easier to reason about what you code should do without having to take care of missing pieces of information in noSQL objects that have become stale because of 'speed of iteration'.
- ibejoeb 9y ago
  I really wonder what all of these claimants do. Do they change their data models and corresponding procedures, and then just leave all of the stale, nonconforming data in there? How is that working out?
TruffleMuffin 9y ago
I have been using Mongo for several years now on a fresh build that follows basically the outline you have here. The fresh build was triggered in part because of the cost (development time and infrastructure) in using Microsoft SQL Server. The only difference is that I got to step 5, realised I had two very different reporting requirements. One that Mongo works well with (real time aggregated data), and one which SQL works well with (high granularity, long time period, historical stuff). So, I use Mongo for the first requirement, and SQL for the later. I see many posts like this, and indeed many replies like it also (your thread currently). Curious, why do they always focus on have only one database to fit all their needs? Is my situation special, can't you just make a decision about what tool is best for the job and use it? Is there some eventuality where having databases specific to needs results in disaster that I am just not aware of?
hacktothefuture 9y ago
This! 100% this. I loved phases 1-4. Then 5 hit and horror show. I realized quickly that I was building kludgy extraction routines to get the reports/metrics I needed. Once the horrible process became unmanageable I would have gladly paid the longer development/iteration time on phases 1-4 in order to have a robust and resilient system which allows proper reporting.

brandur 9y ago

> I think the most interesting question though is would they be able to get MVP and initial customers that set off this if they were moving (slightly) slower due to SQL and slight overhead that comes with? I've used Postgres and Mongo pretty extensively, and for any reasonably seasoned developer, the startup overhead of an SQL system is a myth. There may upfront cost to learning how an RDMS and SQL work in the first place, but once you're familiar with them, they'll be faster than Mongo on any new project. The schemaless concept of a document database seems to be the major selling factor in velocity of movement, but once you've got a good handle on a migration framework in the vein of ActiveRecord or other popular software, that's negated completely. It also really doesn't take long before schemaless starts to cause big problems for you in terms of data consistency -- it's not just the big players that get bitten by this. The simplified query language is another one. SQL is a little bit obtuse, but it's not that bad once you have a handle on it, and a lot of people are familiar with it. Once you add in an ORM layer, the lazy-style access of a framework like Sequel or SQLAlchemy makes the developer experience quite a bit better than any Mongo APIs that I've seen. Also, after you get beyond trivial usage, SQL's flexibility so wildly outstrips Mongo's query documents that it's not even worth talking about. Postgres on the other hand ships with a great management CLI, a very powerful REPL (psql), and features like data types/constraints/transactions that guarantee you correctness with zero effort on your part. I can only speak for myself, but I'd take Postgres to the hackathon any day of the week.

martinald 9y ago
I totally agree with you, and started writing something about how understanding a good ORM takes nearly all the headache away. I think the thing people do find slow is a lot of 'documents within documents' in SQL. It turns out this is usually a bad development pattern long term but it is super fast being able to just add docs inside docs with no configuration. It feels very slow writing foreign keys, navigation props and schemas for this in SQL vs JSON, where you can just dump your object in and you're done. Basically; I think with noSQL you get some very short term gain for a lot of long term pain, and you're right, ORMs and other tooling solves this mostly. I myself fell for this trap, and while it was a nightmare it actually matured me more as a professional more than anything I've ever done recently. Regardless of crazy hype, I don't think I'll ever fall for a solution so easily without evaluating it properly. I think I assumed the "crowd" had done the tech due diligence on this stuff and it definitely wasn't the case.
- macintux 9y ago
  I agree that Postgres trumps Mongo for most use cases, but if ORM is the answer, you might be asking the wrong question. I love that Fowler discusses[1] how to avoid ORMs and offers only two possibilities: > Either you use the relational model in memory, or you don't use it in the database. He clearly has a blind spot. The correct answer, at least in some cases, is don't use objects in the first place. [1]: https://www.martinfowler.com/bliki/OrmHate.html https://www.martinfowler.com/bliki/OrmHate.html
  - anentropic 9y ago
    I've been quite skeptical of the anti-ORM sentiments, since I always found the ORM handy, not a problem etc. Two things recently started to change my mind One is, after profiling some views in our Django app I found that a significant portion of the time in list views (i.e. returning many objects) was spent instantiating ORM model objects. Not on fetching them from the db, but turning them into objects whose imminent destiny is to get turned back into dicts for serializing to JSON. So we switched to values() queries which return plain dicts, but then the ORM is just a query engine. The second was, in a large app, finding that many of the tables start to require a degree of care to only perform 'safe' queries, i.e. ones supported by appropriate indexes. The integration of the ORM into the framework provides many conveniences, but this tends to involve a lack of awareness/control over exactly which queries get made. Most of the patterns that are 'idiomatic' to the framework also tend to involve a lot of ORM querying logic pushed down in the view layer... while our need to perform only a restricted set of 'safe' queries pushed us in the other direction, towards a repository layer with carefully curated query code. I'm seriously considering exploring ORM-less development in future

wolco 9y ago

I'm doing the same for another project. Reporting with $wind can be limiting Schemaless db is great feature. Joins can be done through an orm like Laravel's eloquent which allows you to join mysql to mongo using a cross-join table on the mysql side. Adding columns in an mpv product is trivial. Getting a stream of unstructured data and being able to save the different data to a document that is part of the same collection is an amazing feature that is needed still.

DaiPlusPlus 9y ago
Adding a non-NULL column to an existing table with millions of rows, and then updating the entity/model objects upwards towards the UI is non-trivial - I find myself "queuing up" planned database schema changes and applying them in a batch. What I'd love is a traditional database system with the concept of "cheap columns" that store "just data" scalar values, which can be easily created, dropped and manipulated - they're addressable so they can be used in queries and statistics/analytics, but the values themselves won't need indexes or referential-integrity - and for an ORM and Scaffolding system to support them - that way I won't pull out my hair when the client says one day "let's go from a single column for the customer's name to having separate title/first/middle/last/suffix columns"... and back again the week after.
- jdmichal 9y ago
  So something like PostgreSQL `json` and `jsonb` types? You can select them out into columns, so they should play nicely with ORMs. https://www.postgresql.org/docs/current/static/datatype-json.html https://www.postgresql.org/docs/current/static/datatype-json...
- solidsnack9000 9y ago
  It seems like the "column families" of Cassandra would appeal to you.

jwilliams 9y ago

I largely agree with your journey and your premise. However, I would add that I've seen just as many beast-of-a-thing SQL databases as NoSQL ones. In fact, anything I've seen at sufficient scale has lots of wrangling with their database. While your step 7 usually ends up with a better tech result, it's done at a time when the organization knows the problem domain much (much) better. They have real users. You'd probably get a similar result going from one SQL database to another.

athenot 9y ago

In the words of one of my former DBAs: "MySQL makes a pretty good noSQL database". That is to say you can do a quick and dirty key-value store in MySQL and it will be plenty fast. When you're ready to start normalizing things, just start pulling values out into their own columns and/or tables. Conversely, when starting a product, don't worry about deeply normalized databases: use a relational DB as a glorified key-value store, then normalize things as you go. You'll get the best of both worlds.

sushisource 9y ago
Yep. I've done the same thing by using JSONB in postgres.
tracker1 9y ago
I've done similar for most SQL databases having either an xml, or json column to store ancillary information that didn't need to be accessed or indexed in ways that aren't programmatic. For example, if you support Amazon Payments, Paypal and a payment gateway, the shape of the data for the transaction is different, and rarely looked at, why not shove it into a bigtext/blob field as xml/json.

rgo 9y ago

Here's my personal version of the story: With SQL DBs (Oracle, SQLServer and MySQL): 1. SQL database migrations where killing us. Going back and forward in a dev environment was impossible. No hot deploy in production. 2. Could not work well with application user-defined fields: adding columns adhoc to the database, indexing them, normalizing and denormalizing, performance issues, everything was a problem. 3. Blobs holding logging data got unmanageable quickly. 4. Joins where very hard to optimize even though the team had a lot of DBA experience fine tuning databases. 5. Had to build a very complex architecture around the database for a product that was not that complex: cache, search, database, blob store, distributed, etc. And with all our 1990s and 2000s previous experiences in data warehousing, business intelligence and DB optimization tools, we were still wasting valuable time with SQL design, indexing, query planning and parameter optimization. So we gave MongoDB a try. First as a cache. Later as the only DB. Our journey: 1. Heard about Mongo. Tried the DB. The driver worked great. To me that's the number one "marketing antics behind MongoDB": their strategy creating drivers and supporting the programmer community. 2. Understood what NoSQL meant and forgot about joins altogether. 3. Understood what NoSQL meant and built transactions into atomic documents. 4. Understood what NoSQL meant and stopped relying on the database for type, primary and foreign key constraints, default values, triggers (argh!), stored procedures (2x argh!), etc. 5. Simplified the architecture with integrated search, queue and cache. Less moving parts = joy. 6. Result: very low maintenance, easy install, configuration, replication and migrations. 99.999% availability. 7. Bonus: we even implemented a very high frequency, atomic distributed semaphore system with a FIFO queue that reaps zombies using Mongo built-in networking features. So we've reduced DB-related issues by an order of magnitude. How? I think because NoSQL is a way of saying the DB should not be magically answering random queries. A database should be a data store, period -- just store and retrieve data the way the app needs it. By focusing our energies on getting the data right as documents for a document store meant data flows as objects from code in and out of Mongo. I believe people underestimate how important (and productive) it is to keep the same data structures flowing between the UI (JSON), server (Object/Hash/Dictionary) and DB (document). It makes code easier to read and more resillient to errors. But SQL DBs come with a convenience layer bolted on to run random user queries with things like OUTER joins and GROUP BYs. For that we need to flatten data into tables, which clashes with typically how data flows in an app. SQL DBs however are great as the single source of truth for data: a schema can be laid out and enforced independently of code, so it's safely guarded from programmers breaking it. Business sets up a SQL DB so that their reporting people can query data on demand while consultants with zero knowledge of the business can write code limited by constraints managed by DBAs. SQL is even taught at business schools, which is revealing of who its target audience actually is. Bottom-line: SQL and schema enforcing are end-user features we did not need to build our tool. On the other hand, every single MongoDB feature is something we need and use profusely.

randomdata 9y ago

0. Finally, an API that actually attempts to fit modern programming paradigms! I'm not sure the unstructured nature of many NoSQL databases was even the selling feature so much as the NoSQL part. People were growing tired of dealing with lacklustre ORMs and poor integration with their programming tools and were desperate, even in light of the shortcomings, to try something new. It is funny that we have a thousand and one different general purpose programming languages that try to solve the problems with general purpose computing, but SQL dialects (with slight variation here and there), have very little competition, despite all the problems with the SQL language. A NoSQL relational database could have potentially hit the sweet spot, but Mongo and similar databases came first and caused enough trouble, as you have described, that I feel like they've killed the momentum in looking for better solutions to access database data (relational ones included). Nowadays many of those people are just happy to have a relational database again, even that means accepting SQL.

jdmichal 9y ago
> It is funny that we have a thousand and one different general purpose programming languages that try to solve the problems with general purpose computing, but SQL dialects (with slight variation here and there), have very little competition, despite all the problems with the SQL language. This is kinda like complaining that there's only one implementation of lambda calculus. Yes, SQL is not directly a relational algebra, but it's pretty close. (Also, tables are not relations and SQL deals with that.)
- randomdata 9y ago
  > This is kinda like complaining that there's only one implementation of lambda calculus. Yes, SQL is not directly a relational algebra, but it's pretty close. And yet someone proficient in SQL will not necessarily be able to read a traditionally-written lambda calculus expression as the languages differ significantly (and not just because of tables), so I think this proves my point that there are different, and hopefully better, ways to express the interaction with the database. There is no reason why the protocol has to be exactly as SQL is. There are theoretically an infinite number of ways to communicate the same intent to a RDBMS. For instance, SQL does not do composability well. There is no fundamental reason why you should not be able to write query fragments, simplifying the way many queries are constructed and avoiding the huge mess that monolithic, hard to debug, queries often become. It is simply that SQL does not allow for it. A better API could improve on this significantly, without losing any features a relational database provides. You don't have to throw all database theory out the window just to change the API away from one that was designed 40 years ago.

rmrfrmrf 9y ago

This is kind of a ridiculous statement given the massive popularity of NoSQL DBs like Cassandra, Redis, and Lucene. Not to mention what you described seems like a perfectly reasonable architectural evolution. Are we all really expecting to make a single decision on a DB for the entirety of an app's life? It makes a lot of sense to use a flexible db schema while the data takes shape in an app's infancy, then solidify things with a more structured database schema further down the line. If you're abstracting things away in ORMs anyway, it might not even be that painful if you're just switching out DB drivers.

jimktrains2 9y ago
Most people don't use Redis and Lucene as general-purpose data stores; they're normally used for specific purposes.
- thehardsphere 9y ago
  I would go so far to say that nobody should ever use Lucene as a data store. It's very easy for it to get corrupted because it was explictly designed to support fast full-text searches at the expense of persistence (because your data is supposed to be persisted elsewhere).
camus2 9y ago
Nobody uses Redis like a database and expects persistence of data. Redis is usually used in the context of a message queue. Lucene isn't a database, it's a full text search engine. Cassandra has a query language which is very close to SQL, so it's not really NoSQL.
- jimktrains2 9y ago
  > Redis is usually used in the context of a message queue. Or caches, or simple data stores for things like counts.
- manigandham 9y ago
  > Nobody uses Redis like a database and expects persistence of data. Plenty of people do, it's got just as good persistence and backups as anything else. It's very versatile and used for lots of situations, with message queues just being one option (although not a great fit until it gets a stream data type).

konschubert 9y ago

My feeling is that the people who flock to NoSQL have never used a good ORM such as Django to abstract away the raw SQL queries. A good ORM gives you the simplicity of NoSQL plus all the power of a relational database.

thehardsphere 9y ago
I think that is only true of a subset of NoSQL users. And even if it is true that Django's ORM is very nice, there are tradeoffs to using an ORM that mean you don't get "all the power" of whatever database you are using. Maybe all the power you care about; for instance, power in expressiveness and easy cross platform support, at the expense of performance optimizations that aren't important to you. But it's still a tradeoff.
- konschubert 9y ago
  You're right.

moeamaya 9y ago

JSONB in pgSQL fills in so nicely for steps 1-4. Once you need columns and relationships, the relational store is already there!

manigandham 9y ago

Yes, relational databases are easy deal with. A single machine with proper backups can handle data needs for most companies. The explosion of managed service providers makes this even cheaper, faster and easier to start with. ORMs can do 95% of most project data access needs and a little bit of SQL knowledge allows for easily creating and editing tables - or use the numerous GUI tools out there. If anything, SQL actually lets you be more agile by allowing for quick queries to fix data or do a complicated 1-time report without having to write up and deploy code. This probably has more to do with the SQL language itself which is why it's surprising that MongoDB never tried to implement any of it.

tracker1 9y ago
But the cost of a server big enough to handle a million simultaneous users can be cost prohibitive if you can get away with a few smaller nodes and something that scales horizontally... not to mention failover/redundancy which can be downright expensive and/or painful in many SQL variants.
- qaq 9y ago
  The cost of dev time having to reimplement some of the features available in RDBMS on top of NoSQL stores can be NxTimes the cost of of that box.
- manigandham 9y ago
  That's the whole point - worry about that when you actually have a million simultaneous users, which is a huge number. There are plenty of options like reading from replicas, using a caching layer, or moving specific data to another datastore - all work that's better done when you actually have success and the resources to understand your data access patterns and implement a smart solution when it's needed. Basic replication has been part of every major relational database for decades. Where is it expensive or painful? Do you really need 100% uptime? There's a vast ecosystem for relational databases that can help with all of this and now you have CitusDB, MemSQL and CockroachDB if you need built-in next-gen scaling and availability.

vram22 9y ago

>I think the most interesting question though is would they be able to get MVP and initial customers that set off this if they were moving (slightly) slower due to SQL and slight overhead that comes with? Why the heck not? People (startups and others) were using RDBMSs and SQL for years before NoSQL was a thing. Also, if slight delay is a thing, how about the impact of your point 7 on customers?

mosselman 9y ago

What nonsense. I have never considered using postgres a hindrance due to migrations or something of the like. If anything, migrations and having a consistent database makes it easier to reason about what you code should do without having to take care of missing pieces of information in noSQL objects that have become stale because of 'speed of iteration'.

ibejoeb 9y ago
I really wonder what all of these claimants do. Do they change their data models and corresponding procedures, and then just leave all of the stale, nonconforming data in there? How is that working out?

TruffleMuffin 9y ago

I have been using Mongo for several years now on a fresh build that follows basically the outline you have here. The fresh build was triggered in part because of the cost (development time and infrastructure) in using Microsoft SQL Server. The only difference is that I got to step 5, realised I had two very different reporting requirements. One that Mongo works well with (real time aggregated data), and one which SQL works well with (high granularity, long time period, historical stuff). So, I use Mongo for the first requirement, and SQL for the later. I see many posts like this, and indeed many replies like it also (your thread currently). Curious, why do they always focus on have only one database to fit all their needs? Is my situation special, can't you just make a decision about what tool is best for the job and use it? Is there some eventuality where having databases specific to needs results in disaster that I am just not aware of?

hacktothefuture 9y ago

This! 100% this. I loved phases 1-4. Then 5 hit and horror show. I realized quickly that I was building kludgy extraction routines to get the reports/metrics I needed. Once the horrible process became unmanageable I would have gladly paid the longer development/iteration time on phases 1-4 in order to have a robust and resilient system which allows proper reporting.