8 ms·

You were fortunate to recognize that MongoDB was the wrong tool for your job, and lucky to be able to move to Postgres instead of continuing to throw your time

by gregjor 14y ago

You were fortunate to recognize that MongoDB was the wrong tool for your job, and lucky to be able to move to Postgres instead of continuing to throw your time and effort away. I see the ad hominem "you're an ignorant idiot" attacks already started, along with advice like using regexes to do case-insensitive searches. Watching the NoSQL "movement" encounter the problems RDBMSs fixed 20 years ago and then hand-wave and kludge them away is frustrating. I wrote about some of this in http://typicalprogrammer.com/?p=14 http://typicalprogrammer.com/?p=14. Look at the bright side: programmers who are writing NoSQL-backed apps are creating the fossil fuel that will keep programmers who know RDBMs working into our retirement years. I already have more work than I can do fixing web apps that were built around crap data management tools that failed to scale beyond a few thousand users. Your Postgres expertise will still be a money-making skill long after MongoDB is forgotten.

gaius 14y ago
The thing with the NoSQL guys is that many of them seem not to be in a position to make an educated comparison. For example, an, uhh, enthusiastic MongoDB advocate recently informed me that MongoDB was superior to Oracle because in Oracle you had to poll a table to see if it changed. Except, no, that isn't actually true: http://docs.oracle.com/cd/B19306_01/appdev.102/b14251/adfns_dcn.htm#BGBBHGAH http://docs.oracle.com/cd/B19306_01/appdev.102/b14251/adfns_... - and that document is from 2005. And you could do a trigger and an AQ message/callback 5 years before that (at least). You haven't needed to poll an Oracle database for changes in a loooong time. Basically, every evangelism point, you have to double-check and cross-reference, because as you say, the NoSQL guys are encountering issues the RDBMS community addressed years ago (7 in my example, but the sharding stuff, 20+ years) - except they think they are discovering it for the first time!
- gregjor 14y ago
  Programming is more like fashion than science in this regard. Every decade or so something truly new happens in the software world. All the rest is mostly sound and fury, signifying nothing. If you're young or new to programming it's easy to mistake the buzz around things like NoSQL for innovation when they are usually re-discoveries of old (and often discarded or obsolete) ideas dressed up in new clothes. There's also the tendency to favor new shiny things and reject old crufty (but proven) things, to want to be part of what seems like the leading edge, to be that guy in the cube farm who is playing with the cool new stuff. I have been programming longer than RDBMSs have been available, so I know from experience what it's like to manage large databases in application code, and how hard it can be to maintain consistency or do accurate queries and aggregation with half-baked tools. It's frustrating to see a new generation of programmers go through this, but it's human nature to ignore the past. My fourteen year old son wears his pants pulled down below his waist, Vans shoes, hoodies, lots of hair. He looks pretty much like I did when I was fourteen back in the 1970s. The underlying technologies are the same: pants, shirt, shoes, sweater, hair. The only differences are superficial. To him that style is edgy and contemporary and something his parents don't get. NoSQL is the gangster fashion of programming right now.
  - daleharvey 14y ago
    I cannot remember a more aggravating discussion on Hacker News. A lot of people who are working with / building and using 'NoSQL' databases are the very same people building the RDBMS tools that you are so eager to defend. You have stated elsewhere that you do not know anything about 'NoSQL' stores, so instead of insulting a huge number of people far more experienced than yourself, why dont you attempt to learn? (or at the very least, avoid disregarding the thoughts of a huge sector of the industry)
    - gregjor 14y ago
      I cannot remember a more aggravating discussion on Hacker News. I suggest taking exchanges of opinion less personally. You have stated elsewhere that you do not know anything about 'NoSQL' stores No, I didn't. I have read about them, gone to conferences, and gone through tutorials for a few NoSQL products, but haven't had a use for them in my own work. Since I have 35+ years of programming and database experience I am not viewing these things through the eyes of a newb. And I have lots of experience with database management pre-RDBMS. I did say I haven't found any use for the current batch of NoSQL tools in my own projects, but obviously lots of other people have. And I have clearly indicated that everything I write is my opinion based on my experience, not word from on high. Again, get a grip. instead of insulting a huge number of people far more experienced than yourself No insult intended. My opinions, your mileage may vary. Ad hominem attacks are, on the other hand, deliberate insults. why dont you attempt to learn? (or at the very least, avoid disregarding the thoughts of a huge sector of the industry) Thanks for the career advice. My advice to you and some of the other commenters frothing at the mouth is to disconnect your ego from your preferred tools.
    - einhverfr 14y ago
      In fact I was watching a presentation somewhere (I forget where), where folks were using NoSQL databases as a sort of pre-processing layer for the RDBMS. In this regard it's very useful on the high-end. Where I am less convinced is outside the high-end, and outside the idea of a transition tier between input and RDBMS (or between RDBMS and output even). As a persistence layer, NoSQL is applicable largely to the subset of cases where Object Oriented Databases were. Here the GP is right that the industry is keen on relearning the same lessons every couple of decades.... However, as pre-processing and post-processing, there seems to me to be a much larger set of use cases out there but again only where the RDBMS is no longer really able to handle everything you want it to do.
- zemo 14y ago
  MongoDB is disproportionately liked by the inexperienced. There are things to like about it, and being able to have sparse secondary b-tree indexes on arbitrary data is, overall, pretty rad. For prototyping, being able to just toss some shit in there, especially when your data comes from an external service and you're not in control of your incoming data (very common these days), it works great for simple use cases and CRUD apps (which is a lion's share of new projects in the industry of the Internet). Being able to predict all the incoming data ahead of time isn't always doable, because a lot of services have underdocumented APIs, and every now and then you'll get data that's just ever-so-slightly different than what you expected. With Mongo, you can at least guarantee you're holding onto that, so that you can use this existing data in the future. That's way less contrived than it sounds. E.g., a third-party changes one of the data formats, adds a field or changes a datatype on one, for instance. If you have a strict schema, you might fail to write some of that data until you update everything. With Mongo, you can at least capture that data and get it working later. I'm using it on one of my projects, and as I said, there are things to like about it, but graph traversals remain problematic. Overall, I'd say I'm pretty unhappy with MongoDB, and I wish I hadn't chosen it; it doesn't work well for my project. A lot of things I'm doing boil down to performing some kind of graph traversal, which is painful to do at runtime in Mongo (the potential solution space is too large to be precomputed). From what I've seen, MongoDB hasn't been working for me very well with a highly-connected data set.
  - pnathan 14y ago
    What I like about Mongo is that it's very free-form; I can splat some data in there and not be too concerned about it until later on. I don't really like dealing with file systems for holding data. Of course Mongo ain't the best at relational data... :o
    - einhverfr 14y ago
      But this is the fundamental trade-off isn't it? If you aren't concerned about your data on input then you can't reliably transform it on output, right? In this regard fixed schemas slow down one side of development (handling input) but speed up another side (handling output, and related tasks like reporting).
- taligent 14y ago
  This kind of rubbish really needs to stop. Just because you don't agree with or understand their choices does not mean that the majority of "NoSQL guys" are ignorant or uneducated. Some of the biggest companies e.g. Twitter, Foursquare, Google, Amazon all rely on NoSQL. The real issue I see is that by dismissing NoSQL as only for fools RDBMS developers are failing to see why they are popular to begin with. Take PostgreSQL for example and how difficult it is to shard/replicate compared to CouchDB or MongoDB. This is an area PostgreSQL should see as an opportunity for improvement. But dismissing huge groups of people as uneducated just makes you seem uneducated.
  - gaius 14y ago
    I think we did "sharding" with relational databases... Back in the 80s. Then we got fast hash joins and partitioning and it turns out that the disadvantages of sharding just aren't worth it. The NoSQL crew will figure this out too around 2030 :-p
    - taligent 14y ago
      I guess so will the likes of Facebook, Google, Twitter, eBay etc. Must be amazing to know more about databases than all of the world's biggest companies put together.
      - gaius 14y ago
        Why? The guys at FB or Google are just guys like you and me, don't believe the hype, they are not superheroes or gurus despite their much talked about interview process (the Google interviews I did were a walk in the park compared to GS, btw). Some will know more than me sure, but some'll know less.
        taligent 14y ago
        Those were only a few examples. I can keep listing companies that use sharding. The idea that it is some 1980's approach to scaling databases is laughable.
blaines 14y ago
So do you believe there is a use case for a document-oriented database? I feel like your comment writes off a huge swath of useful technology. I said this below, but I'll say it again. Data is malleable, and writing apps to fit around any datastore seems wrong. I write applications to fulfill their use case. When the needs of the application change, so could it's database (or other dependencies). http://gigaom.com/cloud/mongodb-or-mysql-why-not-both/ http://gigaom.com/cloud/mongodb-or-mysql-why-not-both/
- gregjor 14y ago
  I don't know if there's a use case. I have never had one come up in my own projects. I do sometimes need to manage a large number of documents (unstructured data) but I have found the Unix file system sufficient for those, maybe augmented by metadata in the database. I've worked on big applications with big databases where DB operations and aggregations started to get too slow but so far I or someone smarter on the team has always been able to figure out a solution without throwing the RDBMS out. One big advantage of working with RDBMSs is the roads are paved and well-mapped. By contrast you still need a machete and good luck in the NoSQL world, and even then you can get sick or lost in the jungle. I haven't personally worked on a web app with the kinds of database scaling issues I've dealt with in logistics applications involving tens of millions of movements every day. RDBMSs were invented to allow data malleability. Go back to the old textbooks and look at what preceded the relational model. Remember that data is more valuable and permanent than application code. When you have multiple applications accessing the same database, each with their own implementation of consistency rules and their own internal idea of the data schema, come back and explain how that is more fun than using an RDBMS and being able to sleep nights.
  - blaines 14y ago
    Respectfully, I believe your comment is not very open minded. I believe the obvious answer is "Of course there's a use case!", just like there's obviously a use case for RDBMS, NoSQL (Mongo, Cassandra, BigTable, Redis, etc) serves a purpose. I'll give you a couple examples. Google stores a huge amount of data in BigTable [1] which is fundamental for the operation of Gmail at scale. Another, Facebook initially used Cassandra [2] for it's Inbox search, then migrated to HBase when their needs changed for their new messaging feature. Do we know if the the underlying data is stored in RDBMS? I'm guessing not at Google, but yes for Facebook/Inbox search. But my point is I'm not ruling it out, I'm open to different options. [1] http://en.wikipedia.org/wiki/BigTable http://en.wikipedia.org/wiki/BigTable [2] http://en.wikipedia.org/wiki/Apache_Cassandra#History http://en.wikipedia.org/wiki/Apache_Cassandra#History
wamatt 14y ago
> Watching the NoSQL "movement" encounter the problems RDBMSs fixed 20 years ago and then hand-wave and kludge them away is frustrating. But but but.... NoSQL is cool ;)

by gregjor 14y ago

gaius 14y ago

The thing with the NoSQL guys is that many of them seem not to be in a position to make an educated comparison. For example, an, uhh, enthusiastic MongoDB advocate recently informed me that MongoDB was superior to Oracle because in Oracle you had to poll a table to see if it changed. Except, no, that isn't actually true: http://docs.oracle.com/cd/B19306_01/appdev.102/b14251/adfns_dcn.htm#BGBBHGAH http://docs.oracle.com/cd/B19306_01/appdev.102/b14251/adfns_... - and that document is from 2005. And you could do a trigger and an AQ message/callback 5 years before that (at least). You haven't needed to poll an Oracle database for changes in a loooong time. Basically, every evangelism point, you have to double-check and cross-reference, because as you say, the NoSQL guys are encountering issues the RDBMS community addressed years ago (7 in my example, but the sharding stuff, 20+ years) - except they think they are discovering it for the first time!

gregjor 14y ago
Programming is more like fashion than science in this regard. Every decade or so something truly new happens in the software world. All the rest is mostly sound and fury, signifying nothing. If you're young or new to programming it's easy to mistake the buzz around things like NoSQL for innovation when they are usually re-discoveries of old (and often discarded or obsolete) ideas dressed up in new clothes. There's also the tendency to favor new shiny things and reject old crufty (but proven) things, to want to be part of what seems like the leading edge, to be that guy in the cube farm who is playing with the cool new stuff. I have been programming longer than RDBMSs have been available, so I know from experience what it's like to manage large databases in application code, and how hard it can be to maintain consistency or do accurate queries and aggregation with half-baked tools. It's frustrating to see a new generation of programmers go through this, but it's human nature to ignore the past. My fourteen year old son wears his pants pulled down below his waist, Vans shoes, hoodies, lots of hair. He looks pretty much like I did when I was fourteen back in the 1970s. The underlying technologies are the same: pants, shirt, shoes, sweater, hair. The only differences are superficial. To him that style is edgy and contemporary and something his parents don't get. NoSQL is the gangster fashion of programming right now.
- daleharvey 14y ago
  I cannot remember a more aggravating discussion on Hacker News. A lot of people who are working with / building and using 'NoSQL' databases are the very same people building the RDBMS tools that you are so eager to defend. You have stated elsewhere that you do not know anything about 'NoSQL' stores, so instead of insulting a huge number of people far more experienced than yourself, why dont you attempt to learn? (or at the very least, avoid disregarding the thoughts of a huge sector of the industry)
  - gregjor 14y ago
    I cannot remember a more aggravating discussion on Hacker News. I suggest taking exchanges of opinion less personally. You have stated elsewhere that you do not know anything about 'NoSQL' stores No, I didn't. I have read about them, gone to conferences, and gone through tutorials for a few NoSQL products, but haven't had a use for them in my own work. Since I have 35+ years of programming and database experience I am not viewing these things through the eyes of a newb. And I have lots of experience with database management pre-RDBMS. I did say I haven't found any use for the current batch of NoSQL tools in my own projects, but obviously lots of other people have. And I have clearly indicated that everything I write is my opinion based on my experience, not word from on high. Again, get a grip. instead of insulting a huge number of people far more experienced than yourself No insult intended. My opinions, your mileage may vary. Ad hominem attacks are, on the other hand, deliberate insults. why dont you attempt to learn? (or at the very least, avoid disregarding the thoughts of a huge sector of the industry) Thanks for the career advice. My advice to you and some of the other commenters frothing at the mouth is to disconnect your ego from your preferred tools.
  - einhverfr 14y ago
    In fact I was watching a presentation somewhere (I forget where), where folks were using NoSQL databases as a sort of pre-processing layer for the RDBMS. In this regard it's very useful on the high-end. Where I am less convinced is outside the high-end, and outside the idea of a transition tier between input and RDBMS (or between RDBMS and output even). As a persistence layer, NoSQL is applicable largely to the subset of cases where Object Oriented Databases were. Here the GP is right that the industry is keen on relearning the same lessons every couple of decades.... However, as pre-processing and post-processing, there seems to me to be a much larger set of use cases out there but again only where the RDBMS is no longer really able to handle everything you want it to do.
zemo 14y ago
MongoDB is disproportionately liked by the inexperienced. There are things to like about it, and being able to have sparse secondary b-tree indexes on arbitrary data is, overall, pretty rad. For prototyping, being able to just toss some shit in there, especially when your data comes from an external service and you're not in control of your incoming data (very common these days), it works great for simple use cases and CRUD apps (which is a lion's share of new projects in the industry of the Internet). Being able to predict all the incoming data ahead of time isn't always doable, because a lot of services have underdocumented APIs, and every now and then you'll get data that's just ever-so-slightly different than what you expected. With Mongo, you can at least guarantee you're holding onto that, so that you can use this existing data in the future. That's way less contrived than it sounds. E.g., a third-party changes one of the data formats, adds a field or changes a datatype on one, for instance. If you have a strict schema, you might fail to write some of that data until you update everything. With Mongo, you can at least capture that data and get it working later. I'm using it on one of my projects, and as I said, there are things to like about it, but graph traversals remain problematic. Overall, I'd say I'm pretty unhappy with MongoDB, and I wish I hadn't chosen it; it doesn't work well for my project. A lot of things I'm doing boil down to performing some kind of graph traversal, which is painful to do at runtime in Mongo (the potential solution space is too large to be precomputed). From what I've seen, MongoDB hasn't been working for me very well with a highly-connected data set.
- pnathan 14y ago
  What I like about Mongo is that it's very free-form; I can splat some data in there and not be too concerned about it until later on. I don't really like dealing with file systems for holding data. Of course Mongo ain't the best at relational data... :o
  - einhverfr 14y ago
    But this is the fundamental trade-off isn't it? If you aren't concerned about your data on input then you can't reliably transform it on output, right? In this regard fixed schemas slow down one side of development (handling input) but speed up another side (handling output, and related tasks like reporting).
taligent 14y ago
This kind of rubbish really needs to stop. Just because you don't agree with or understand their choices does not mean that the majority of "NoSQL guys" are ignorant or uneducated. Some of the biggest companies e.g. Twitter, Foursquare, Google, Amazon all rely on NoSQL. The real issue I see is that by dismissing NoSQL as only for fools RDBMS developers are failing to see why they are popular to begin with. Take PostgreSQL for example and how difficult it is to shard/replicate compared to CouchDB or MongoDB. This is an area PostgreSQL should see as an opportunity for improvement. But dismissing huge groups of people as uneducated just makes you seem uneducated.
- gaius 14y ago
  I think we did "sharding" with relational databases... Back in the 80s. Then we got fast hash joins and partitioning and it turns out that the disadvantages of sharding just aren't worth it. The NoSQL crew will figure this out too around 2030 :-p
  - taligent 14y ago
    I guess so will the likes of Facebook, Google, Twitter, eBay etc. Must be amazing to know more about databases than all of the world's biggest companies put together.
    - gaius 14y ago
      Why? The guys at FB or Google are just guys like you and me, don't believe the hype, they are not superheroes or gurus despite their much talked about interview process (the Google interviews I did were a walk in the park compared to GS, btw). Some will know more than me sure, but some'll know less.
      - taligent 14y ago
        Those were only a few examples. I can keep listing companies that use sharding. The idea that it is some 1980's approach to scaling databases is laughable.

blaines 14y ago

So do you believe there is a use case for a document-oriented database? I feel like your comment writes off a huge swath of useful technology. I said this below, but I'll say it again. Data is malleable, and writing apps to fit around any datastore seems wrong. I write applications to fulfill their use case. When the needs of the application change, so could it's database (or other dependencies). http://gigaom.com/cloud/mongodb-or-mysql-why-not-both/ http://gigaom.com/cloud/mongodb-or-mysql-why-not-both/

gregjor 14y ago
I don't know if there's a use case. I have never had one come up in my own projects. I do sometimes need to manage a large number of documents (unstructured data) but I have found the Unix file system sufficient for those, maybe augmented by metadata in the database. I've worked on big applications with big databases where DB operations and aggregations started to get too slow but so far I or someone smarter on the team has always been able to figure out a solution without throwing the RDBMS out. One big advantage of working with RDBMSs is the roads are paved and well-mapped. By contrast you still need a machete and good luck in the NoSQL world, and even then you can get sick or lost in the jungle. I haven't personally worked on a web app with the kinds of database scaling issues I've dealt with in logistics applications involving tens of millions of movements every day. RDBMSs were invented to allow data malleability. Go back to the old textbooks and look at what preceded the relational model. Remember that data is more valuable and permanent than application code. When you have multiple applications accessing the same database, each with their own implementation of consistency rules and their own internal idea of the data schema, come back and explain how that is more fun than using an RDBMS and being able to sleep nights.
- blaines 14y ago
  Respectfully, I believe your comment is not very open minded. I believe the obvious answer is "Of course there's a use case!", just like there's obviously a use case for RDBMS, NoSQL (Mongo, Cassandra, BigTable, Redis, etc) serves a purpose. I'll give you a couple examples. Google stores a huge amount of data in BigTable [1] which is fundamental for the operation of Gmail at scale. Another, Facebook initially used Cassandra [2] for it's Inbox search, then migrated to HBase when their needs changed for their new messaging feature. Do we know if the the underlying data is stored in RDBMS? I'm guessing not at Google, but yes for Facebook/Inbox search. But my point is I'm not ruling it out, I'm open to different options. [1] http://en.wikipedia.org/wiki/BigTable http://en.wikipedia.org/wiki/BigTable [2] http://en.wikipedia.org/wiki/Apache_Cassandra#History http://en.wikipedia.org/wiki/Apache_Cassandra#History

wamatt 14y ago

> Watching the NoSQL "movement" encounter the problems RDBMSs fixed 20 years ago and then hand-wave and kludge them away is frustrating. But but but.... NoSQL is cool ;)