8 ms·

MongoDB Days

70 points by creamyhorror 13y ago

creamyhorror 13y ago
I'm most interested in the features that commercial DBs like Oracle have that free/open-source DBs like Postgres, MySQL, & NoSQL DBs don't. Are things like "a materialized view (1996!), a continuous query, the result cache" available in any free DBs nowadays? There's more of this sort of criticism in the following old thread, "SQL Databases Don't Scale": https://news.ycombinator.com/item?id=690656 https://news.ycombinator.com/item?id=690656 where a few commenters say (somewhat unpleasant) things like: - I find that this type of FUD comes about from people that aren't good at designing and implementing large databases, or can't afford the technology that can pull it off, so they slam the technology rather than accept that they, themselves, are the ones lacking. Most of them tend to come from the typical LAMP/SlashDot crowd that only have experience with the minor technologies. - For me, thousands of transactions per second and 10s of terabytes of data on a single database is normal. It's unremarkable, it's everyday, it's what we do, we have done it for years. And I know of installations handling 10x that. It's only people who's only "experience" is websites that whinge about how RDBMS can't handle their tiny datasets. - Mr. Wiggins article would be better titled something like "ACID databases have scalability problems, especially cheap ones startups use" How true are these criticisms nowadays? Is open-source still far behind, or is it (as I think) more than good-enough for 98% of use-cases? edit: Thanks for the responses, sounds like I'll be trying out Postgres for my upcoming personal project.
- latch 13y ago
  There are a lot free DBs options, so I'd bet a lot against anything but the most esoteric feature being only available in a commercial solution. The foundation of CouchDB queries are materialized view / continuous query. PgPool provides result caching.
- __alexs 13y ago
  > Are things like "a materialized view (1996!), a continuous query, the result cache" available in any free DBs nowadays? Getting there... http://www.postgresql.org/docs/9.3/static/rules-materializedviews.html http://www.postgresql.org/docs/9.3/static/rules-materialized... (The rest of his examples are complicated Oracleisms and you'd probably get pretty far with MVs.)
  - omarqureshi 13y ago
    9.3 matviews are a step forward, yet you could probably roll your own matviews with triggers.
    - inflagranti 13y ago
      The nice thing with materialized views in DB2 for instance (don't know about oracle) is that they are automatically picked by the optimizer to replace a join. So your logical query is - as it should be - completely exasperated from the physical storage underneath. The DBA just puts in a materialized view and the application will speed up magically :)
      - noelherrick 13y ago
        I think you mean "completely abstracted from the physical storage underneath" :) Although, I'm sure queries exasperate the storage all the time. Regardless, I had no idea DB2 was so smart. I guess you get what you pay for.
- gizzlon 13y ago
  > I'm most interested in the features that commercial DBs like Oracle have that free/open-source DBs [..] don't. Are things like "a materialized view (1996!), a continuous query, the result cache" available in any free DBs nowadays? I cant answer your question, but looking for feature X in other products is the wrong approach IMHO. You're often better served by looking at what problem feature X is solving, and how that problem is other places. Sometimes the problem doesn't even exist.
  - gizzlon 13y ago
    You're often better served by looking at what problem feature X is solving, and how that problem is solved other places
- raverbashing 13y ago
  " so they slam the technology rather than accept that they, themselves, are the ones lacking. Most of them tend to come from the typical LAMP/SlashDot crowd that only have experience with the minor technologies." Behind Facebook and Twitter there's a big usage of MySQL. Not even PostgreSQL, MySQL. Of course it's a pain for them, but it works and they're profiting from it. The question is, how much they are saving by not using Oracle or other proprietary DB? They priced themselves out of the web, now they can't "cry to mummy" about it. "For me, thousands of transactions per second and 10s of terabytes of data on a single database is normal" It helps when you have dedicated top notch hardware (especially true some years ago) and the startups have to work with EC2 and EBS (ok, there are some better choices, still)
  - astral303 13y ago
    I just looked at the Oracle price list: http://www.oracle.com/us/corporate/pricing/technology-price-list-070617.pdf http://www.oracle.com/us/corporate/pricing/technology-price-... Processor License cost (excludes support): * Standard Edition: $17,500 * Enterprise Edition: $47,500 (that's the one with Materialized View Query Rewrite) * Partitioning option: $11,500 * Advanced compression: $11,500 (basic compression is apparently slow?) Per processor. This is why all these other databases exist. Very few businesses, certainly not many startups, have the kinds of value of the data stored to warrant that kind of cost.
    - raverbashing 13y ago
      Yes, very expensive. However, if you're a bank, you may not need too many servers and 50k per server is change. I believe in the beginning of the web, Oracle wanted to push a per-user pricing model. Yes, if your website has 10 users, it pays 10x$PRICE, 100, 100x$PRICE Nobody could work with that (with Oracle)
- mathnode 13y ago
  It all comes down to cost barrier to entry, it's a shame IBM mainframes are still so damn expensive. Perhaps if somebody could offer {DB2|Oracle|Informix|Sybase} As A Service, like what many providers do with PostgreSQL and MySQL, it would be a different story for startups. My profression involves moving MSSQL and Sybase databases to MySQL and MariaDB, then I go and read some of the features from the article in Oracle and DB2 documentation and think to myself "I ain't helping fucking nobody", the higher-ups in enterprise and startups usually only see upfront costs, not long term benefits. As per your first question; they are getting there. PostgreSQL now has a built-in function for managing materialised views, in MySQL/MariaDB that is still hand-rolled. MariaDB however does have Virtual Columns and Dynamic columns (to tackle nested tables, as per the article). Look to MariaDB for the MySQL developments, as Oracle are bringing very little to the table in terms of matching MySQL features to Oracle (not surprising). But they are resolving important security, and infrastructure issues and improving InnoDB heavily. Also PostgreSQL does have HSTORE, for storing JSON data types, very tasty.
jacques_chester 13y ago
A better title might be: "MongoDB solves problems MySQL didn't solve at the time."
area51org 13y ago
Perhaps better-put: MongoDB solves problems OP doesn't understand because he sees them in terms of purely relational SQL databases.
- iand675 13y ago
  Genuine question: What are these use cases that MongoDB solves better? I'm a fan of several other non-relational databases, but I haven't really encountered any use-cases for MongoDB that make it seem like a good fit.
  - a5huynh 13y ago
    MongoDB has a nice page which explains the core MongoDB use cases: http://docs.mongodb.org/manual/use-cases/ http://docs.mongodb.org/manual/use-cases/ and also here: https://www.mongodb.com/solutions https://www.mongodb.com/solutions
  - bdcravens 13y ago
    For arbitrarily structured data, document databases are great. Of course, this isn't specific to MongoDB.
  - gaius 13y ago
    Mongo is a great way to cache and distribute intermediate results in large computations, for example, if you have an efficient way to serialize and deserialize the actual in-memory representation of your data (e.g. if all your machines are the same architecture, you can just shovel blobs of memory around this way). I wouldn't use Oracle for that, the overheads are too great.
  - raverbashing 13y ago
    What's great specifically about MongoDB (this may apply to others, and/or not be good for your needs, but here goes) - Easy to set up (including replication) - Fast - Data in "JSON" format (it's BSON internally) - Javascript (used for map/reduce mainly) - Good availability of libraries Theoretically, CouchDB would have been better, but I've heard some bad stories (remember UbuntuOne)? I don't have much experience with other NOSQL DBs (except for Redis, it's great, but I like to call it a "DB Toolkit" - I'm not dissing it, I love it)
jwilliams 13y ago
Shrug. Oracle suffers from problems that you don't get on your z/OS running IMS on top of VSAM files (which by the way is hierarchical, not relational - which is what your big ole bank will be using). Talking about the relational model and "sound mathematical underpinnings" is fine, but I've seen dozens.. hundreds?.. of production relational databases and they've all been monsters. Most have been partially or significantly denormalized for good/bad reasons. All have their own warts, usually significant. If people end up normalizing their MongoDB as the author suggests - Well I'd expect that. It's pretty rare to get your DB model right first time. Any thought that you can is probably tinged with madness. If you can have a good stab at it and iterate you're ahead. Plus if you've ever worked with a TPS (or TM/2PC) you'll know they're a nightmare. Any volume systems I've worked on dispensed with them and use a reconcile/compensate mechanism. MongoDB has gone down a replica/sharded model that focuses on Map/Reduce. How successful this is, well that's a different argument - but comparing it to an old-school big-iron mentality is a waste of time.
- jacques_chester 13y ago
  I think the main revelation in databases was the division of the world into OLTP and OLAP. So you keep your normalised schema for production and have a methodologically denormalised schema for reporting. Mind you, computer science courses generally don't teach OLAP -- stuff like dimensional modelling. My university didn't when I was there, though they've rolled it into a larger "Data Mining" course alongside MapReduce and friends.
  - jwilliams 13y ago
    Agree. & this makes sense from an architectural perspective, since the demands of each domain are different. I was arguing from practical terms and my own experience. Even in established, DB-heavy, "we have hundreds of DBAs" organisations, you can't have a porcelain schema. It evolves over time, breaks, get optimised and de-optimised. The Oracle shops I've worked in don't have one schema, they'll have dozens. Usually because they can't shoehorn emerging requirements into the databases they have. Or because of project timelines. Or they need to be performance tuned in some special way, which in the case of Oracle could be considered a dark art. These orgs may have just one big data warehouse, but that becomes a dumping ground. It's possible to do your dimensional modelling (I worked on star schemas in the past, but that's about it) -- but it's a big ball of mud to try and weave together. Even then your OLAP is under-stress because it's got a lot of ground to cover. Usually they start out as marketing engines, but get co-opted into all sorts of things. The most common and troublesome is regulatory reporting. Once your data warehouse gets used for something like regulatory reporting you're a bit stuck as nobody wants to touch it. Then, because point to point integrations between the databases is brittle and cost-prohibitive -- they break the golden rule and take data out of the OLAP and put it back into the OLTP. I won't name names, but this is commonplace. SalesForce is one of the biggest Oracle DB users (I read biggest at one stage, but can't find the reference). Even then the SalesForce data model is so denormalised you could call it one big table[1]. I use MongoDB day-to-day and it's fit for purpose for what I do. That said, I'm not a rabid fan and an alternative/hybrid approach is always on the horizon. What I don't buy is the "the DB world has already solved all these problems", particularly when things like "relational integrity" and "two phase commit" get thrown in the mix - truth is there is plenty to be solved. [1] http://www.dbms2.com/2011/09/15/database-architecture-salesforce-com-force-com-and-database/ http://www.dbms2.com/2011/09/15/database-architecture-salesf...
raverbashing 13y ago
"We don’t work the way we do because tables are a limitation of the technology, we use the relational model because it has sound mathematical underpinnings, and the technology reflects that†. Where’s the rigour in MongoDB’s model?" It seems all anti-NoSQL rants are the same whining and refusal to understand. "Sound mathematical base"? Really? So, if I can't describe something mathematically (I can, by the way) it's not worth it? A computer program is a mathematical description, there you have it. The relational model breaks for very common use cases nowadays. Yes, maybe you think it's fun to do a query across how many tables to get the information you want, but if your website has a non-trivial traffic then the solution is usually to add more cache. That's (one of the reasons) why PostgreSQL has hstore. Beyond the fanboy insistence that you can do everything with relational DBs, Postgres have accepted the reality that you need a more flexible data structure. Edit: yes, please continue showing your contempt while I have to code around the limitation of relational databases.
- Nodex 13y ago
  +1 - nice retort ;)
kokey 13y ago
Last year I wrote a tool for a bank to suck in MongoDB data from 5 big nodes on physical hardware, into an Oracle database running on a virtual machine. The idea was to make it easier for others to write their reporting queries against an SQL database that they understand. It turns out with the right tweaks the Oracle database also performed a lot better, on a lot less hardware. It was one of the things that really improved my impression of the Oracle database product. The article also reminds me of how a father and son went to a Microsoft presentation in 2000, where Microsoft showed their solution to the tricky problem of integrating multile backend servers. Their solution was to have front end tiers close to the client, and the client getting thinner. The son was very impressed. The father said 'that's what IBM did before the 70s!'
Nodex 13y ago
Use the best tool for the job - it's that simple. RDBMS is NOT a silver bullet and nor is mongodb
callumjones 13y ago
I'm not so sure about the criticism of applications being exposed as web services. If you have a system setup where different parts run on different services you don't want to have to co-ordinate and resync all the applications when their underlying data structures change. Web services solve this by having a agreed contract of communicating between systems, service A knows of a better way of accessing its own data that service B knows of accessing system A's data.
- gaius 13y ago
  What if you need to do something that touches both A and B?
  - callumjones 13y ago
    Then you would go through both A and B's web services?
don_draper 13y ago
He has a DBA mindsight in a FullStack world. There are many projects now that only require one or two FullStack engineers. Not one dba, one QA, two developers, one deployment/operations, one requirements analyst, etc. In this new FullStack world MongoDB is a great fit: changes are easy (no alter table add new_thing varchar(200); It's interface is JavaScript, the same language we use in other places; It's super easy to get started with (definitely not the case with Oracle); Oh and if we need to scale later, it might scale better too.
- deleted 13y ago
  [deleted]
- charliesome 13y ago
  Changes are easy? How do you migrate documents created before you changed your schema to documents created after? Or do you deal with an entire collection of documents all wildly varying in shape?
- dodyg 13y ago
  FullStack? seriously?
- smacktoward 13y ago
  Yes, in our brave new FullStack™ world we've gone from having three or four people who are each insanely good at something to having one person who is not insanely good at anything. Progress!
  - walshemj 13y ago
    yes id still til take "dave" our database guru at BT - whose first boss was Edsger Dijkstra.
  - nasalgoat 13y ago
    Sounds like the 3-in-1 printer model.
astral303 13y ago
Data locality is not about seek times on disk, it's about network transfer times between different nodes! Linear horizontal scalability often needs sharding of data, and with a document store like MongoDB, you can easily shard a complex denormalized document. Now, if I have this complex document normalized into several tables, how am I going to easily shard several tables, all such that I can execute successful joins that only need to execute on one leaf node? What if I start reusing a small piece of data in one of these normalized tables? I might be forced to go between network nodes to get this data. Normalization is like premature optimization. I can take any program and modify it such that every piece executes optimally fast, but I am likely to compromise on clarity or to add complexity while doing such a refactoring. In the end, it probably got me no real-world performance boost that mattered. 80/20% rule and all. Same thing with normalization: automatically making all my data fully normalized from the start is a like a bad premature optimization habit that we are forced into with relational databases out of A) sheer habit and school teachings, B) lack of easy support for nested structured data.
- gaius 13y ago
  That is of course what I was referring to when I talked about logical vs physical data models. You define your LDM then selectively denormalize it when you create the PDM. The MySQL guys are very into sharding too but of course without hash joins, you can't join big tables anyway, so this "optimization" is easy because it costs you nothing.
- wheaties 13y ago
  You can do all of that with a SQL DB. In fact, Postgres works as an amazing key-value store if you want it to. They've even got some damned good performance in hstore compared to many use cases of Mongo. I use MongoDB and SQL DBs every day.
  - astral303 13y ago
    Mongo is much more than a key-value store. Document sub-components and subarrays can be indexed and aggregated against. I don't see how you can do that with hstore. Yes, I can totally see how Postgres and hstore can run circles around key-value storage. And by "all that", I would love to know how to use the open-source versions of Postgres or MySQL to have transparently-sharded tables with joins AND have that be in a replicated environment where I can do real-time failover.
olegp 13y ago
One of the comments in the original article said that developers use MongoDB because they're too lazy to use RDBMS. I don't see anything wrong with that - laziness is a virtue among developers! Seriously though, we are using MongoDB with great success at StartHQ (https://starthq.com https://starthq.com) having done a lot of work with relational databases before. It's a great fit for startups where the schema is constantly evolving & the amount of data stored can be quite small. Also, by talking to the DB directly, without an ORM, we can keep things really simple. I dread to think of what the same code would look like if we were to use a relational database, either with or without an ORM.
Pxtl 13y ago
Relational databases are a kernel of mathematical relational beauty buried under a million tons of ugly hacks you have to do to keep them performant.

70 points by creamyhorror 13y ago

I'm most interested in the features that commercial DBs like Oracle have that free/open-source DBs like Postgres, MySQL, & NoSQL DBs don't. Are things like "a materialized view (1996!), a continuous query, the result cache" available in any free DBs nowadays? There's more of this sort of criticism in the following old thread, "SQL Databases Don't Scale": https://news.ycombinator.com/item?id=690656 https://news.ycombinator.com/item?id=690656 where a few commenters say (somewhat unpleasant) things like: - I find that this type of FUD comes about from people that aren't good at designing and implementing large databases, or can't afford the technology that can pull it off, so they slam the technology rather than accept that they, themselves, are the ones lacking. Most of them tend to come from the typical LAMP/SlashDot crowd that only have experience with the minor technologies. - For me, thousands of transactions per second and 10s of terabytes of data on a single database is normal. It's unremarkable, it's everyday, it's what we do, we have done it for years. And I know of installations handling 10x that. It's only people who's only "experience" is websites that whinge about how RDBMS can't handle their tiny datasets. - Mr. Wiggins article would be better titled something like "ACID databases have scalability problems, especially cheap ones startups use" How true are these criticisms nowadays? Is open-source still far behind, or is it (as I think) more than good-enough for 98% of use-cases? edit: Thanks for the responses, sounds like I'll be trying out Postgres for my upcoming personal project.

latch 13y ago
There are a lot free DBs options, so I'd bet a lot against anything but the most esoteric feature being only available in a commercial solution. The foundation of CouchDB queries are materialized view / continuous query. PgPool provides result caching.
__alexs 13y ago
> Are things like "a materialized view (1996!), a continuous query, the result cache" available in any free DBs nowadays? Getting there... http://www.postgresql.org/docs/9.3/static/rules-materializedviews.html http://www.postgresql.org/docs/9.3/static/rules-materialized... (The rest of his examples are complicated Oracleisms and you'd probably get pretty far with MVs.)
- omarqureshi 13y ago
  9.3 matviews are a step forward, yet you could probably roll your own matviews with triggers.
  - inflagranti 13y ago
    The nice thing with materialized views in DB2 for instance (don't know about oracle) is that they are automatically picked by the optimizer to replace a join. So your logical query is - as it should be - completely exasperated from the physical storage underneath. The DBA just puts in a materialized view and the application will speed up magically :)
    - noelherrick 13y ago
      I think you mean "completely abstracted from the physical storage underneath" :) Although, I'm sure queries exasperate the storage all the time. Regardless, I had no idea DB2 was so smart. I guess you get what you pay for.
gizzlon 13y ago
> I'm most interested in the features that commercial DBs like Oracle have that free/open-source DBs [..] don't. Are things like "a materialized view (1996!), a continuous query, the result cache" available in any free DBs nowadays? I cant answer your question, but looking for feature X in other products is the wrong approach IMHO. You're often better served by looking at what problem feature X is solving, and how that problem is other places. Sometimes the problem doesn't even exist.
- gizzlon 13y ago
  You're often better served by looking at what problem feature X is solving, and how that problem is solved other places
raverbashing 13y ago
" so they slam the technology rather than accept that they, themselves, are the ones lacking. Most of them tend to come from the typical LAMP/SlashDot crowd that only have experience with the minor technologies." Behind Facebook and Twitter there's a big usage of MySQL. Not even PostgreSQL, MySQL. Of course it's a pain for them, but it works and they're profiting from it. The question is, how much they are saving by not using Oracle or other proprietary DB? They priced themselves out of the web, now they can't "cry to mummy" about it. "For me, thousands of transactions per second and 10s of terabytes of data on a single database is normal" It helps when you have dedicated top notch hardware (especially true some years ago) and the startups have to work with EC2 and EBS (ok, there are some better choices, still)
- astral303 13y ago
  I just looked at the Oracle price list: http://www.oracle.com/us/corporate/pricing/technology-price-list-070617.pdf http://www.oracle.com/us/corporate/pricing/technology-price-... Processor License cost (excludes support): * Standard Edition: $17,500 * Enterprise Edition: $47,500 (that's the one with Materialized View Query Rewrite) * Partitioning option: $11,500 * Advanced compression: $11,500 (basic compression is apparently slow?) Per processor. This is why all these other databases exist. Very few businesses, certainly not many startups, have the kinds of value of the data stored to warrant that kind of cost.
  - raverbashing 13y ago
    Yes, very expensive. However, if you're a bank, you may not need too many servers and 50k per server is change. I believe in the beginning of the web, Oracle wanted to push a per-user pricing model. Yes, if your website has 10 users, it pays 10x$PRICE, 100, 100x$PRICE Nobody could work with that (with Oracle)
mathnode 13y ago
It all comes down to cost barrier to entry, it's a shame IBM mainframes are still so damn expensive. Perhaps if somebody could offer {DB2|Oracle|Informix|Sybase} As A Service, like what many providers do with PostgreSQL and MySQL, it would be a different story for startups. My profression involves moving MSSQL and Sybase databases to MySQL and MariaDB, then I go and read some of the features from the article in Oracle and DB2 documentation and think to myself "I ain't helping fucking nobody", the higher-ups in enterprise and startups usually only see upfront costs, not long term benefits. As per your first question; they are getting there. PostgreSQL now has a built-in function for managing materialised views, in MySQL/MariaDB that is still hand-rolled. MariaDB however does have Virtual Columns and Dynamic columns (to tackle nested tables, as per the article). Look to MariaDB for the MySQL developments, as Oracle are bringing very little to the table in terms of matching MySQL features to Oracle (not surprising). But they are resolving important security, and infrastructure issues and improving InnoDB heavily. Also PostgreSQL does have HSTORE, for storing JSON data types, very tasty.

jacques_chester 13y ago

A better title might be: "MongoDB solves problems MySQL didn't solve at the time."

area51org 13y ago

Perhaps better-put: MongoDB solves problems OP doesn't understand because he sees them in terms of purely relational SQL databases.

iand675 13y ago
Genuine question: What are these use cases that MongoDB solves better? I'm a fan of several other non-relational databases, but I haven't really encountered any use-cases for MongoDB that make it seem like a good fit.
- a5huynh 13y ago
  MongoDB has a nice page which explains the core MongoDB use cases: http://docs.mongodb.org/manual/use-cases/ http://docs.mongodb.org/manual/use-cases/ and also here: https://www.mongodb.com/solutions https://www.mongodb.com/solutions
- bdcravens 13y ago
  For arbitrarily structured data, document databases are great. Of course, this isn't specific to MongoDB.
- gaius 13y ago
  Mongo is a great way to cache and distribute intermediate results in large computations, for example, if you have an efficient way to serialize and deserialize the actual in-memory representation of your data (e.g. if all your machines are the same architecture, you can just shovel blobs of memory around this way). I wouldn't use Oracle for that, the overheads are too great.
- raverbashing 13y ago
  What's great specifically about MongoDB (this may apply to others, and/or not be good for your needs, but here goes) - Easy to set up (including replication) - Fast - Data in "JSON" format (it's BSON internally) - Javascript (used for map/reduce mainly) - Good availability of libraries Theoretically, CouchDB would have been better, but I've heard some bad stories (remember UbuntuOne)? I don't have much experience with other NOSQL DBs (except for Redis, it's great, but I like to call it a "DB Toolkit" - I'm not dissing it, I love it)

jwilliams 13y ago

Shrug. Oracle suffers from problems that you don't get on your z/OS running IMS on top of VSAM files (which by the way is hierarchical, not relational - which is what your big ole bank will be using). Talking about the relational model and "sound mathematical underpinnings" is fine, but I've seen dozens.. hundreds?.. of production relational databases and they've all been monsters. Most have been partially or significantly denormalized for good/bad reasons. All have their own warts, usually significant. If people end up normalizing their MongoDB as the author suggests - Well I'd expect that. It's pretty rare to get your DB model right first time. Any thought that you can is probably tinged with madness. If you can have a good stab at it and iterate you're ahead. Plus if you've ever worked with a TPS (or TM/2PC) you'll know they're a nightmare. Any volume systems I've worked on dispensed with them and use a reconcile/compensate mechanism. MongoDB has gone down a replica/sharded model that focuses on Map/Reduce. How successful this is, well that's a different argument - but comparing it to an old-school big-iron mentality is a waste of time.

jacques_chester 13y ago
I think the main revelation in databases was the division of the world into OLTP and OLAP. So you keep your normalised schema for production and have a methodologically denormalised schema for reporting. Mind you, computer science courses generally don't teach OLAP -- stuff like dimensional modelling. My university didn't when I was there, though they've rolled it into a larger "Data Mining" course alongside MapReduce and friends.
- jwilliams 13y ago
  Agree. & this makes sense from an architectural perspective, since the demands of each domain are different. I was arguing from practical terms and my own experience. Even in established, DB-heavy, "we have hundreds of DBAs" organisations, you can't have a porcelain schema. It evolves over time, breaks, get optimised and de-optimised. The Oracle shops I've worked in don't have one schema, they'll have dozens. Usually because they can't shoehorn emerging requirements into the databases they have. Or because of project timelines. Or they need to be performance tuned in some special way, which in the case of Oracle could be considered a dark art. These orgs may have just one big data warehouse, but that becomes a dumping ground. It's possible to do your dimensional modelling (I worked on star schemas in the past, but that's about it) -- but it's a big ball of mud to try and weave together. Even then your OLAP is under-stress because it's got a lot of ground to cover. Usually they start out as marketing engines, but get co-opted into all sorts of things. The most common and troublesome is regulatory reporting. Once your data warehouse gets used for something like regulatory reporting you're a bit stuck as nobody wants to touch it. Then, because point to point integrations between the databases is brittle and cost-prohibitive -- they break the golden rule and take data out of the OLAP and put it back into the OLTP. I won't name names, but this is commonplace. SalesForce is one of the biggest Oracle DB users (I read biggest at one stage, but can't find the reference). Even then the SalesForce data model is so denormalised you could call it one big table[1]. I use MongoDB day-to-day and it's fit for purpose for what I do. That said, I'm not a rabid fan and an alternative/hybrid approach is always on the horizon. What I don't buy is the "the DB world has already solved all these problems", particularly when things like "relational integrity" and "two phase commit" get thrown in the mix - truth is there is plenty to be solved. [1] http://www.dbms2.com/2011/09/15/database-architecture-salesforce-com-force-com-and-database/ http://www.dbms2.com/2011/09/15/database-architecture-salesf...

raverbashing 13y ago

"We don’t work the way we do because tables are a limitation of the technology, we use the relational model because it has sound mathematical underpinnings, and the technology reflects that†. Where’s the rigour in MongoDB’s model?" It seems all anti-NoSQL rants are the same whining and refusal to understand. "Sound mathematical base"? Really? So, if I can't describe something mathematically (I can, by the way) it's not worth it? A computer program is a mathematical description, there you have it. The relational model breaks for very common use cases nowadays. Yes, maybe you think it's fun to do a query across how many tables to get the information you want, but if your website has a non-trivial traffic then the solution is usually to add more cache. That's (one of the reasons) why PostgreSQL has hstore. Beyond the fanboy insistence that you can do everything with relational DBs, Postgres have accepted the reality that you need a more flexible data structure. Edit: yes, please continue showing your contempt while I have to code around the limitation of relational databases.

Nodex 13y ago
+1 - nice retort ;)

kokey 13y ago

Last year I wrote a tool for a bank to suck in MongoDB data from 5 big nodes on physical hardware, into an Oracle database running on a virtual machine. The idea was to make it easier for others to write their reporting queries against an SQL database that they understand. It turns out with the right tweaks the Oracle database also performed a lot better, on a lot less hardware. It was one of the things that really improved my impression of the Oracle database product. The article also reminds me of how a father and son went to a Microsoft presentation in 2000, where Microsoft showed their solution to the tricky problem of integrating multile backend servers. Their solution was to have front end tiers close to the client, and the client getting thinner. The son was very impressed. The father said 'that's what IBM did before the 70s!'

Nodex 13y ago

Use the best tool for the job - it's that simple. RDBMS is NOT a silver bullet and nor is mongodb

callumjones 13y ago

I'm not so sure about the criticism of applications being exposed as web services. If you have a system setup where different parts run on different services you don't want to have to co-ordinate and resync all the applications when their underlying data structures change. Web services solve this by having a agreed contract of communicating between systems, service A knows of a better way of accessing its own data that service B knows of accessing system A's data.

gaius 13y ago
What if you need to do something that touches both A and B?
- callumjones 13y ago
  Then you would go through both A and B's web services?

don_draper 13y ago

He has a DBA mindsight in a FullStack world. There are many projects now that only require one or two FullStack engineers. Not one dba, one QA, two developers, one deployment/operations, one requirements analyst, etc. In this new FullStack world MongoDB is a great fit: changes are easy (no alter table add new_thing varchar(200); It's interface is JavaScript, the same language we use in other places; It's super easy to get started with (definitely not the case with Oracle); Oh and if we need to scale later, it might scale better too.

deleted 13y ago
[deleted]
charliesome 13y ago
Changes are easy? How do you migrate documents created before you changed your schema to documents created after? Or do you deal with an entire collection of documents all wildly varying in shape?
dodyg 13y ago
FullStack? seriously?
smacktoward 13y ago
Yes, in our brave new FullStack™ world we've gone from having three or four people who are each insanely good at something to having one person who is not insanely good at anything. Progress!
- walshemj 13y ago
  yes id still til take "dave" our database guru at BT - whose first boss was Edsger Dijkstra.
- nasalgoat 13y ago
  Sounds like the 3-in-1 printer model.

astral303 13y ago

Data locality is not about seek times on disk, it's about network transfer times between different nodes! Linear horizontal scalability often needs sharding of data, and with a document store like MongoDB, you can easily shard a complex denormalized document. Now, if I have this complex document normalized into several tables, how am I going to easily shard several tables, all such that I can execute successful joins that only need to execute on one leaf node? What if I start reusing a small piece of data in one of these normalized tables? I might be forced to go between network nodes to get this data. Normalization is like premature optimization. I can take any program and modify it such that every piece executes optimally fast, but I am likely to compromise on clarity or to add complexity while doing such a refactoring. In the end, it probably got me no real-world performance boost that mattered. 80/20% rule and all. Same thing with normalization: automatically making all my data fully normalized from the start is a like a bad premature optimization habit that we are forced into with relational databases out of A) sheer habit and school teachings, B) lack of easy support for nested structured data.

gaius 13y ago
That is of course what I was referring to when I talked about logical vs physical data models. You define your LDM then selectively denormalize it when you create the PDM. The MySQL guys are very into sharding too but of course without hash joins, you can't join big tables anyway, so this "optimization" is easy because it costs you nothing.
wheaties 13y ago
You can do all of that with a SQL DB. In fact, Postgres works as an amazing key-value store if you want it to. They've even got some damned good performance in hstore compared to many use cases of Mongo. I use MongoDB and SQL DBs every day.
- astral303 13y ago
  Mongo is much more than a key-value store. Document sub-components and subarrays can be indexed and aggregated against. I don't see how you can do that with hstore. Yes, I can totally see how Postgres and hstore can run circles around key-value storage. And by "all that", I would love to know how to use the open-source versions of Postgres or MySQL to have transparently-sharded tables with joins AND have that be in a replicated environment where I can do real-time failover.

olegp 13y ago

One of the comments in the original article said that developers use MongoDB because they're too lazy to use RDBMS. I don't see anything wrong with that - laziness is a virtue among developers! Seriously though, we are using MongoDB with great success at StartHQ (https://starthq.com https://starthq.com) having done a lot of work with relational databases before. It's a great fit for startups where the schema is constantly evolving & the amount of data stored can be quite small. Also, by talking to the DB directly, without an ORM, we can keep things really simple. I dread to think of what the same code would look like if we were to use a relational database, either with or without an ORM.

Pxtl 13y ago

Relational databases are a kernel of mathematical relational beauty buried under a million tons of ugly hacks you have to do to keep them performant.