10 ms·

Why MongoDB is a bad choice for storing our scraped data

86 points by reinhardt 13y ago

thehodge 13y ago
I read the whole post waiting to see what they ended up using as we are having similar issues, only to find that it's another post I have to wait for..
- m4tthumphrey 13y ago
  Me too, except I skipped straight to the end :)
- DoubleCluster 13y ago
  Just a plain old RDMS?
- rubinelli 13y ago
  Look into Cassandra. Very fast writes and scales linearly (although there is some elbow grease involved in keyspace distribution). Our analytics platform guys are very happy with it, after flirting with several other options (including big, beefy RDBMS). Great for large, flat, denormalized tables.
- monstrado 13y ago
  Based on their use case, I'd expect they went with either HBase or Cassandra. I'm quite partial to HBase, it's insanely scalable and has a lot of pretty amazing features, but at the cost of knowing exactly what you want to do with your application beforehand. I'm not too familiar with Cassandra, but the scalability of an HBase table is almost entirely dependent on your key design. Judging from their use case and requirements, they would likely use a incremental key design which would allow for super fast range scans, of course, this leads to region server hotspotting, which may or not may not be a big deal to them.
- pablohoffman 13y ago
  We went with HBase. Cassandra would have been suitable too, but we already use Hadoop for data processing so it was a natural choice within the infrastructure ecosystem. We will write a followup about that.
  - monstrado 13y ago
    Clouderan here! Glad to hear you guys went with HBase, I'm looking forward to your follow up post. Will you detail your key design / architectural setup? Did you guys roll your own HBase environment or did you go with the CDH? If you're using the CDH version and have any questions, feel free to shoot an email to cdh-user.
    - pablohoffman 13y ago
      We are using CDH4.2 and have had a very positive experience so far. Cloudera has in fact been an inspiration for us to follow, you guys have really struck the right balance between open source and commercial support. We follow the same philosophy with Scrapy (an open source web crawling framework), as you do with Hadoop and its ecosystem.
      - monstrado 13y ago
        That's really awesome to hear, thanks for your kind words. I'm looking forward to the follow up blog, depending on your key design you may be able to take advantage of Impala for ad-hoc queries using SQL.
dkhenry 13y ago
you lost me here """ Ordered data Some data (e.g. crawl logs) needs to be returned in the order it was written. Retrieving data in order requires sorting which is impractical when the number of records gets large. "" it requires _indexing_ and is quite feasable as I do it every day with stock ticker logs ( also required to be retrieved incrementially ) There are a few other flags that make me wonder about the exact limitations you found, but I will be anticipating your follow up post to see what your fix was since some of those issues are very common.
- deleted 13y ago
  [deleted]
- mayank 13y ago
  No kidding. Without details, it really sounds like the author is a bit clueless. He mentions the lack of joins, but doesn't say a word about Mapreduce. "MongoDB needs to walk the index from the beginning to the offset..." You don't "walk an index". It's an index. "Too many databases" sounds a little suspicious. Why not add an indexed field to partition records? Complaining about a lack of schema, transactions and triggers? Really? Did you read the docs at all before starting? MongoDB is not without its problems, but friend, I think you wanted either Postgres or Hadoop.
  - PavlovsCat 13y ago
    You don't "walk an index". It's an index If you have an address book, you don't have to walk through the city to find an address, but you do have to look through your address book in some way or other. Of course, you can have an index of the index ("C starts at page 7"), but then you have to look through the index of the index.
    - shane42 13y ago
      The problems with pagination are explained better on this SO post: http://stackoverflow.com/questions/7228169/slow-pagination-over-tons-of-records-in-mongo http://stackoverflow.com/questions/7228169/slow-pagination-o... Mongo docs used the "walk" terminology
JulianMorrison 13y ago
I really don't understand why people use MongoDB. It seems like it's a elegant technological metaphor (lets use mmap, the OS is our cache and we can overwrite in place in RAM) that in practise turns out to be a terrible idea. Overwrite/mmap cannot be made reliable, requires blocking write-locks, wastes disk, and causes problems shuffling data around as it grows. Add other bad decisions (keys aren't interned, seriously?)and it's just a terrible limping monster. Abandon it, walk away.
- deleted 13y ago
  [deleted]
- aroman 13y ago
  Because when you're not operating at significant scale, or have certain specific use cases, it's a fantastically elegant solution and one that's very quick and easy to set up. I have used mongodb for a number of smaller projects, and I have had an excellent experience. It's not "a terrible idea in practice". It might be a terrible fit for what you want, but that doesn't mean it's bad technology.
  - JulianMorrison 13y ago
    Are you dead certain those projects aren't ever eating data, and nor are they about to crash tomorrow with an unrecoverably hosed DB? I still say it's bad technology. Use plain old SQL instead.
    - lmm 13y ago
      "plain old SQL" can require a lot of mangling one's data to fit its constraints. I refuse to believe that there isn't a better key-value store for the case where the values are json documents, even if mongodb isn't it.
      - gaius 13y ago
        Except it isn't "SQL" it's first-order predicate calculus - a provably sound way to store and query your data.
eknkc 13y ago
It's like people started complaining about MongoDB just for the sake of it. I guess it's the new trend? - Ordered data and skip / limit: These would run just fine on any database system. Given that you have appropriate indexes. It does not matter if there are a trillion items total, as long as you are seeking over an index and the result set is in reasonable size. - Restrictions: A lot of software has restrictions. Filesystems has file name limitations. RDBMSs have table / column name limitations. It's a fact of life. Why is this a con for MongoDB? - Impossible to keep working set in memory: It is a fair argument that MongoDB has shitty memory management because it just delegates the responsibility to OS. However, this is a concern with any DBMS. Also, given that there are appropriate indexes, you don't need to keep the entire database on memory. This comes back to indexing problem. - No transactions / lack of schema / no joins...: I don't remember mongoDB claiming to have such features. My car can't fly. I'm not complaining. (Well, sometimes) - Locking: Fair point. Better I/O performance might come handy (like an SSD) or eventually sharding. - Poor space efficiency: Fair point about fragmentation and field names. Compression can be achieved on the filesystem level. There was an article about that a couple of days ago. I'm not sure about pefroamnce though. - Too many databases: This should not be a big issue. Mongo does not go ahead and allocate a couple gigagbytes for each db, it uses incremental file sizes. - Silent failures: Yep.. There it fails miserably. Recent versions are better though.
- drtse4 13y ago
  It isn't new, i remember people complaining about MongoDB since its first releases.
- kalleboo 13y ago
  Why are you taking this personally? They're just listing reasons why it's not a good fit for them. Useful information to others who are trying to pick a database for similar applications.
  - eknkc 13y ago
    I'm sorry if it looks like I'm attacking the criticism. Nope, I would not use MongoDB ever again, after a year and a half with it. I have my reasons for this decision. I just don't like people bashing something without valid reasons. It might just be a perfect solution for similar applications, this is not a good way to evaluate.
leothekim 13y ago
I'm interested in hearing what the author's new storage system is. What would be compelling is to hear if the same hardware and storage with the new storage system performed better than mongo with some semblance of concrete metrics. There are a lot of complaints here about mongo -- all of them not new -- but no hard numbers.
latchkey 13y ago
I use Mongo for storing a fairly large amount of scraped data and it works great. Some of the data I store is results from bike races that I want to display on my website in a better way than it is displayed elsewhere. The columns change which makes Mongo a great fit, but the data is pretty static. The real issue here is that it feels like the author has just 'discovered' these problems as if Mongo was hiding them all along and after a long time using the system he just found them. The reality is that all of the things he brings up are well documented. It is fascinating to me how people pick a buzzword database and don't bother to think about how their application might run poorly on it over time.
- kernel_sanders 13y ago
  Got a link? I can't get enough bike.
- calpaterson 13y ago
  Every time there is a mongodb retrospective or experience report posted to HN, the top comment is one along the lines of "Well, these issues are all well documented." Firstly, the fact that some drawback is well documented does not excuse the fact that it is a drawback. Second, while some drawbacks are documented some implications of these drawbacks are nuanced and only become obvious with experience. A good example of this is the implications of "schemaless" databases (more accurately: databases that do not check data against a schema). Not having to migrate tables is a boon for lots of development. It's also a giant pain if it turns out that bugs cause data integrity issues. Third, this experience report is really useful since poorly structured scrape data is one of the areas that I would have considered to be ideal for mongodb. Most people don't have perfect foresight. I don't fault the author on his lack of omniscience with respect to how mongodb would turn out for them. His original reasoning (given in paragraph 1, sentence 1) does not seem stupid.
craigching 13y ago
Whenever I see the "You don't need Mongo DB, use an SQL database" and then in the flaming back and forth, I never see my key problem mentioned: MongoDB makes it easy to scale out (replica sets and sharding), where is the "easy to setup replicated and sharded open source SQL database?" I mean, I know that Postgres has replication (via Slony? honestly, it's been awhile since I looked at their solutions) but I don't recall it being as dead simple to set up. For me, setting up replication needs to be easy because we redistribute the store as part of our product and we need scalability (both replication for redundancy and sharding for scaling). So I'm honestly asking here, where is the easy to use sharded and replicated open source SQL store that I've been missing?
- AlisdairO 13y ago
  Postgres has hot standby built in nowadays, and it works well. Sharding certainly isn't as easy - the technical compromises that mongo makes make it pretty trivial to implement, whereas it's relatively hard to make it work in an RDBMS while maintaining all the expected capabilities. It generally requires some application-level work on open source dbs. With that said, I really think many people grossly underestimate the effectiveness of scale-up. It's worth remembering that Stack Overflow (for example) is still running on a single pair of master/hot standby database machines.
  - threeseed 13y ago
    Standby is a pretty poor solution compared to replica set let alone what Cassandra has to offer. Sharding is trivial on MongoDB/Cassandra and it is open source. So let's be accurate here. It is a problem inherent with the SQL databases. And I think you underestimate the benefits of scaling out. If I want to ensure close to 100% uptime or have a server closer to my users than Cassandra or even MongoDB would be infinitely easier to setup and manage than Postgres. These are "very nice to haves" for even the tiniest startup.
    - craigching 13y ago
      Thanks, that's very much what I'm talking about. Cassandra would be my ideal store, I absolutely love it except for the ability to index across nodes. My understanding, when last I looked at it, was that indexes were only local and didn't span Cassandra nodes. Does Cassandra now have properly distributed indexes? When I was looking at implementing Cassandra instead of Mongo DB, it seemed like we had to create reverse column family (IIRC, been away from Cassandra for a bit now). Is that still the case?
wiremine 13y ago
Quote from the original author in the comments. TLDR: it was human error that got us into this situation: "The lack of joins & transactions of course did factor into the original decision. My point (which perhaps could be clearer) was that MongoDB ended up being used outside of the area in which we originally intended to use it. There was some reluctance to add another technology when we could get by with what we had for what was (initially) only a small use. Additionally, some limitations were not always well understood by web developers (who were new to mongo and enthusiastic to try it). I see this as our mistake. With hindsight, it’s clear we should have introduced an RDBMS immediately and kept MongoDB for managing the crawl data."
- jeffdavis 13y ago
  Databases almost always grow outside of their original scope, much more so than applications. And there is a good reason: data is more valuable when it's combined with other data. So I think it's reasonable to be cautious about using a system that can't effectively grow outside of its initial special purpose.
- foobar1988 13y ago
  Proof that the OP was just looking for more Hacker News cred so they wrote about why MongoDB sucks but in reality wanted to discuss their "human error" :p Nice marketing guys http://blog.serverdensity.com/does-everyone-hate-mongodb/ http://blog.serverdensity.com/does-everyone-hate-mongodb/
nosespray 13y ago
People aren't complaining just to complain. When you can't even ctrl-c out of the shell there is a huge issue. It's a known bug, around since v1.8, 'minor' priority. Yeah, thanks for trapping me in the shell.
- mratzloff 13y ago
  Apparently this is the ticket being referred to: https://jira.mongodb.org/browse/SERVER-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel https://jira.mongodb.org/browse/SERVER-2986?page=com.atlassi...
- nasalgoat 13y ago
  Ahh yes, the fun "Control-Z-kill-pid" trick. I do enjoy it so.
nramirezuy 13y ago
Why you people hate MongoDB? It is great to start with NoSQL, and for new little projects.
KaiserPro 13y ago
Now the interesting thing about this post is this, I can see why they wanted to use mongodb, and I can see why it bit them in the arse. What interests me is why they would want to keep everything in the database? I'd assume that they need to aggregate and curate the scraped data. After the initial scrape the majority of actions surely are going to be on the metadata of the scraped content? (where is said data, when was it scraped, how big, relationship to other data, etc) This data is much smaller and can be stored in relational database, as its proper structured data with relationships. This allows the nasty unstructured data to be kept on a plain boring filesystem. After all filesystems are exceptionally mature, universal, multilevel key-value stores. Now people will say that filesystems don't scale, well that's not really true. ext4/ntfs on a single system won't scale, but something like lustre/gluster(although not as neat)/gpfs scales linearly with the amount of nodes you apply to it.
- shane42 13y ago
  This approach (scraped data on FS + metadata in DB) works well for storing scraped data. It was the first thing I prototyped when we started the project to move away from MongoDB. We've worked on similar designs in the past where the data is in S3, it's a common pattern. We'd need to code the searching, filtering, paginating, (distributed?) job management ourselves while being careful to keep the DB & metadata consistent. It works best if each file is a reasonable 'chunk' of data (not too big, not tiny). None of this is a problem, and it scales very well as you said. In the end, we went with HBase for crawl data in the new system. Of course, you can look at this as files on a filesystem (HDFS or others) :) It does a lot of what we would otherwise have to code ourselves and it's a good fit for applications we want to build on that data in future (e.g. storing other crawl datastructures, processing with hadoop). I'll provide more details on that in the next post.

86 points by reinhardt 13y ago

I read the whole post waiting to see what they ended up using as we are having similar issues, only to find that it's another post I have to wait for..

m4tthumphrey 13y ago
Me too, except I skipped straight to the end :)
DoubleCluster 13y ago
Just a plain old RDMS?
rubinelli 13y ago
Look into Cassandra. Very fast writes and scales linearly (although there is some elbow grease involved in keyspace distribution). Our analytics platform guys are very happy with it, after flirting with several other options (including big, beefy RDBMS). Great for large, flat, denormalized tables.
monstrado 13y ago
Based on their use case, I'd expect they went with either HBase or Cassandra. I'm quite partial to HBase, it's insanely scalable and has a lot of pretty amazing features, but at the cost of knowing exactly what you want to do with your application beforehand. I'm not too familiar with Cassandra, but the scalability of an HBase table is almost entirely dependent on your key design. Judging from their use case and requirements, they would likely use a incremental key design which would allow for super fast range scans, of course, this leads to region server hotspotting, which may or not may not be a big deal to them.
pablohoffman 13y ago
We went with HBase. Cassandra would have been suitable too, but we already use Hadoop for data processing so it was a natural choice within the infrastructure ecosystem. We will write a followup about that.
- monstrado 13y ago
  Clouderan here! Glad to hear you guys went with HBase, I'm looking forward to your follow up post. Will you detail your key design / architectural setup? Did you guys roll your own HBase environment or did you go with the CDH? If you're using the CDH version and have any questions, feel free to shoot an email to cdh-user.
  - pablohoffman 13y ago
    We are using CDH4.2 and have had a very positive experience so far. Cloudera has in fact been an inspiration for us to follow, you guys have really struck the right balance between open source and commercial support. We follow the same philosophy with Scrapy (an open source web crawling framework), as you do with Hadoop and its ecosystem.
    - monstrado 13y ago
      That's really awesome to hear, thanks for your kind words. I'm looking forward to the follow up blog, depending on your key design you may be able to take advantage of Impala for ad-hoc queries using SQL.

dkhenry 13y ago

you lost me here """ Ordered data Some data (e.g. crawl logs) needs to be returned in the order it was written. Retrieving data in order requires sorting which is impractical when the number of records gets large. "" it requires _indexing_ and is quite feasable as I do it every day with stock ticker logs ( also required to be retrieved incrementially ) There are a few other flags that make me wonder about the exact limitations you found, but I will be anticipating your follow up post to see what your fix was since some of those issues are very common.

deleted 13y ago
[deleted]
mayank 13y ago
No kidding. Without details, it really sounds like the author is a bit clueless. He mentions the lack of joins, but doesn't say a word about Mapreduce. "MongoDB needs to walk the index from the beginning to the offset..." You don't "walk an index". It's an index. "Too many databases" sounds a little suspicious. Why not add an indexed field to partition records? Complaining about a lack of schema, transactions and triggers? Really? Did you read the docs at all before starting? MongoDB is not without its problems, but friend, I think you wanted either Postgres or Hadoop.
- PavlovsCat 13y ago
  You don't "walk an index". It's an index If you have an address book, you don't have to walk through the city to find an address, but you do have to look through your address book in some way or other. Of course, you can have an index of the index ("C starts at page 7"), but then you have to look through the index of the index.
  - shane42 13y ago
    The problems with pagination are explained better on this SO post: http://stackoverflow.com/questions/7228169/slow-pagination-over-tons-of-records-in-mongo http://stackoverflow.com/questions/7228169/slow-pagination-o... Mongo docs used the "walk" terminology

JulianMorrison 13y ago

I really don't understand why people use MongoDB. It seems like it's a elegant technological metaphor (lets use mmap, the OS is our cache and we can overwrite in place in RAM) that in practise turns out to be a terrible idea. Overwrite/mmap cannot be made reliable, requires blocking write-locks, wastes disk, and causes problems shuffling data around as it grows. Add other bad decisions (keys aren't interned, seriously?)and it's just a terrible limping monster. Abandon it, walk away.

deleted 13y ago
[deleted]
aroman 13y ago
Because when you're not operating at significant scale, or have certain specific use cases, it's a fantastically elegant solution and one that's very quick and easy to set up. I have used mongodb for a number of smaller projects, and I have had an excellent experience. It's not "a terrible idea in practice". It might be a terrible fit for what you want, but that doesn't mean it's bad technology.
- JulianMorrison 13y ago
  Are you dead certain those projects aren't ever eating data, and nor are they about to crash tomorrow with an unrecoverably hosed DB? I still say it's bad technology. Use plain old SQL instead.
  - lmm 13y ago
    "plain old SQL" can require a lot of mangling one's data to fit its constraints. I refuse to believe that there isn't a better key-value store for the case where the values are json documents, even if mongodb isn't it.
    - gaius 13y ago
      Except it isn't "SQL" it's first-order predicate calculus - a provably sound way to store and query your data.

eknkc 13y ago

It's like people started complaining about MongoDB just for the sake of it. I guess it's the new trend? - Ordered data and skip / limit: These would run just fine on any database system. Given that you have appropriate indexes. It does not matter if there are a trillion items total, as long as you are seeking over an index and the result set is in reasonable size. - Restrictions: A lot of software has restrictions. Filesystems has file name limitations. RDBMSs have table / column name limitations. It's a fact of life. Why is this a con for MongoDB? - Impossible to keep working set in memory: It is a fair argument that MongoDB has shitty memory management because it just delegates the responsibility to OS. However, this is a concern with any DBMS. Also, given that there are appropriate indexes, you don't need to keep the entire database on memory. This comes back to indexing problem. - No transactions / lack of schema / no joins...: I don't remember mongoDB claiming to have such features. My car can't fly. I'm not complaining. (Well, sometimes) - Locking: Fair point. Better I/O performance might come handy (like an SSD) or eventually sharding. - Poor space efficiency: Fair point about fragmentation and field names. Compression can be achieved on the filesystem level. There was an article about that a couple of days ago. I'm not sure about pefroamnce though. - Too many databases: This should not be a big issue. Mongo does not go ahead and allocate a couple gigagbytes for each db, it uses incremental file sizes. - Silent failures: Yep.. There it fails miserably. Recent versions are better though.

drtse4 13y ago
It isn't new, i remember people complaining about MongoDB since its first releases.
kalleboo 13y ago
Why are you taking this personally? They're just listing reasons why it's not a good fit for them. Useful information to others who are trying to pick a database for similar applications.
- eknkc 13y ago
  I'm sorry if it looks like I'm attacking the criticism. Nope, I would not use MongoDB ever again, after a year and a half with it. I have my reasons for this decision. I just don't like people bashing something without valid reasons. It might just be a perfect solution for similar applications, this is not a good way to evaluate.

leothekim 13y ago

I'm interested in hearing what the author's new storage system is. What would be compelling is to hear if the same hardware and storage with the new storage system performed better than mongo with some semblance of concrete metrics. There are a lot of complaints here about mongo -- all of them not new -- but no hard numbers.

latchkey 13y ago

I use Mongo for storing a fairly large amount of scraped data and it works great. Some of the data I store is results from bike races that I want to display on my website in a better way than it is displayed elsewhere. The columns change which makes Mongo a great fit, but the data is pretty static. The real issue here is that it feels like the author has just 'discovered' these problems as if Mongo was hiding them all along and after a long time using the system he just found them. The reality is that all of the things he brings up are well documented. It is fascinating to me how people pick a buzzword database and don't bother to think about how their application might run poorly on it over time.

kernel_sanders 13y ago
Got a link? I can't get enough bike.
calpaterson 13y ago
Every time there is a mongodb retrospective or experience report posted to HN, the top comment is one along the lines of "Well, these issues are all well documented." Firstly, the fact that some drawback is well documented does not excuse the fact that it is a drawback. Second, while some drawbacks are documented some implications of these drawbacks are nuanced and only become obvious with experience. A good example of this is the implications of "schemaless" databases (more accurately: databases that do not check data against a schema). Not having to migrate tables is a boon for lots of development. It's also a giant pain if it turns out that bugs cause data integrity issues. Third, this experience report is really useful since poorly structured scrape data is one of the areas that I would have considered to be ideal for mongodb. Most people don't have perfect foresight. I don't fault the author on his lack of omniscience with respect to how mongodb would turn out for them. His original reasoning (given in paragraph 1, sentence 1) does not seem stupid.

craigching 13y ago

Whenever I see the "You don't need Mongo DB, use an SQL database" and then in the flaming back and forth, I never see my key problem mentioned: MongoDB makes it easy to scale out (replica sets and sharding), where is the "easy to setup replicated and sharded open source SQL database?" I mean, I know that Postgres has replication (via Slony? honestly, it's been awhile since I looked at their solutions) but I don't recall it being as dead simple to set up. For me, setting up replication needs to be easy because we redistribute the store as part of our product and we need scalability (both replication for redundancy and sharding for scaling). So I'm honestly asking here, where is the easy to use sharded and replicated open source SQL store that I've been missing?

AlisdairO 13y ago
Postgres has hot standby built in nowadays, and it works well. Sharding certainly isn't as easy - the technical compromises that mongo makes make it pretty trivial to implement, whereas it's relatively hard to make it work in an RDBMS while maintaining all the expected capabilities. It generally requires some application-level work on open source dbs. With that said, I really think many people grossly underestimate the effectiveness of scale-up. It's worth remembering that Stack Overflow (for example) is still running on a single pair of master/hot standby database machines.
- threeseed 13y ago
  Standby is a pretty poor solution compared to replica set let alone what Cassandra has to offer. Sharding is trivial on MongoDB/Cassandra and it is open source. So let's be accurate here. It is a problem inherent with the SQL databases. And I think you underestimate the benefits of scaling out. If I want to ensure close to 100% uptime or have a server closer to my users than Cassandra or even MongoDB would be infinitely easier to setup and manage than Postgres. These are "very nice to haves" for even the tiniest startup.
  - craigching 13y ago
    Thanks, that's very much what I'm talking about. Cassandra would be my ideal store, I absolutely love it except for the ability to index across nodes. My understanding, when last I looked at it, was that indexes were only local and didn't span Cassandra nodes. Does Cassandra now have properly distributed indexes? When I was looking at implementing Cassandra instead of Mongo DB, it seemed like we had to create reverse column family (IIRC, been away from Cassandra for a bit now). Is that still the case?

wiremine 13y ago

Quote from the original author in the comments. TLDR: it was human error that got us into this situation: "The lack of joins & transactions of course did factor into the original decision. My point (which perhaps could be clearer) was that MongoDB ended up being used outside of the area in which we originally intended to use it. There was some reluctance to add another technology when we could get by with what we had for what was (initially) only a small use. Additionally, some limitations were not always well understood by web developers (who were new to mongo and enthusiastic to try it). I see this as our mistake. With hindsight, it’s clear we should have introduced an RDBMS immediately and kept MongoDB for managing the crawl data."

jeffdavis 13y ago
Databases almost always grow outside of their original scope, much more so than applications. And there is a good reason: data is more valuable when it's combined with other data. So I think it's reasonable to be cautious about using a system that can't effectively grow outside of its initial special purpose.
foobar1988 13y ago
Proof that the OP was just looking for more Hacker News cred so they wrote about why MongoDB sucks but in reality wanted to discuss their "human error" :p Nice marketing guys http://blog.serverdensity.com/does-everyone-hate-mongodb/ http://blog.serverdensity.com/does-everyone-hate-mongodb/

nosespray 13y ago

People aren't complaining just to complain. When you can't even ctrl-c out of the shell there is a huge issue. It's a known bug, around since v1.8, 'minor' priority. Yeah, thanks for trapping me in the shell.

mratzloff 13y ago
Apparently this is the ticket being referred to: https://jira.mongodb.org/browse/SERVER-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel https://jira.mongodb.org/browse/SERVER-2986?page=com.atlassi...
nasalgoat 13y ago
Ahh yes, the fun "Control-Z-kill-pid" trick. I do enjoy it so.

nramirezuy 13y ago

Why you people hate MongoDB? It is great to start with NoSQL, and for new little projects.

KaiserPro 13y ago

Now the interesting thing about this post is this, I can see why they wanted to use mongodb, and I can see why it bit them in the arse. What interests me is why they would want to keep everything in the database? I'd assume that they need to aggregate and curate the scraped data. After the initial scrape the majority of actions surely are going to be on the metadata of the scraped content? (where is said data, when was it scraped, how big, relationship to other data, etc) This data is much smaller and can be stored in relational database, as its proper structured data with relationships. This allows the nasty unstructured data to be kept on a plain boring filesystem. After all filesystems are exceptionally mature, universal, multilevel key-value stores. Now people will say that filesystems don't scale, well that's not really true. ext4/ntfs on a single system won't scale, but something like lustre/gluster(although not as neat)/gpfs scales linearly with the amount of nodes you apply to it.

shane42 13y ago
This approach (scraped data on FS + metadata in DB) works well for storing scraped data. It was the first thing I prototyped when we started the project to move away from MongoDB. We've worked on similar designs in the past where the data is in S3, it's a common pattern. We'd need to code the searching, filtering, paginating, (distributed?) job management ourselves while being careful to keep the DB & metadata consistent. It works best if each file is a reasonable 'chunk' of data (not too big, not tiny). None of this is a problem, and it scales very well as you said. In the end, we went with HBase for crawl data in the new system. Of course, you can look at this as files on a filesystem (HDFS or others) :) It does a lot of what we would otherwise have to code ourselves and it's a good fit for applications we want to build on that data in future (e.g. storing other crawl datastructures, processing with hadoop). I'll provide more details on that in the next post.