4 ms·

you lost me here """ Ordered data Some data (e.g. crawl logs) needs to be returned in the order it was written. Retrieving data in order requires sorting whic

by dkhenry 13y ago

you lost me here """ Ordered data Some data (e.g. crawl logs) needs to be returned in the order it was written. Retrieving data in order requires sorting which is impractical when the number of records gets large. "" it requires _indexing_ and is quite feasable as I do it every day with stock ticker logs ( also required to be retrieved incrementially ) There are a few other flags that make me wonder about the exact limitations you found, but I will be anticipating your follow up post to see what your fix was since some of those issues are very common.

deleted 13y ago
[deleted]
mayank 13y ago
No kidding. Without details, it really sounds like the author is a bit clueless. He mentions the lack of joins, but doesn't say a word about Mapreduce. "MongoDB needs to walk the index from the beginning to the offset..." You don't "walk an index". It's an index. "Too many databases" sounds a little suspicious. Why not add an indexed field to partition records? Complaining about a lack of schema, transactions and triggers? Really? Did you read the docs at all before starting? MongoDB is not without its problems, but friend, I think you wanted either Postgres or Hadoop.
- PavlovsCat 13y ago
  You don't "walk an index". It's an index If you have an address book, you don't have to walk through the city to find an address, but you do have to look through your address book in some way or other. Of course, you can have an index of the index ("C starts at page 7"), but then you have to look through the index of the index.
  - shane42 13y ago
    The problems with pagination are explained better on this SO post: http://stackoverflow.com/questions/7228169/slow-pagination-over-tons-of-records-in-mongo http://stackoverflow.com/questions/7228169/slow-pagination-o... Mongo docs used the "walk" terminology
  - mayank 13y ago
    Not sure what you're getting at. "Walk" usually means a sequential scan. An index is sorted, so you can binary search.
    - PavlovsCat 13y ago
      See other reply. If the docs say MongoDB "walks", it's hilarious to complain that someone obviously didn't read the docs for saying "walk", too.
  - monstrado 13y ago
    It looks like they went with HBase for the replacement which means they can scan an index range, keys are in lexicographical order, so it makes it pretty easy to scan over a series of data with a single RPC call.
monstrado 13y ago
With a database like HBase, it's already ordered lexicographically which makes it easy to grab a range of data in the order it was written in. You could have a key design like <reverse_domain>-<epoch> which would allow quick scans over large amounts of data. ie..Scan from <object_id>-1368536860 to <object_id>-1368540450 HBase is multidimensional though, which allows you to keep N numbers of versions of a cell. By default you will get the latest version of the cell back, but you could also opt to receive N versions back, which is useful for time series use cases.
- swah 13y ago
  The last time I looked there weren't many resources explaining how to design NoSQL databases (how to compose your keys, when to avoid normalization, etc). Has this improved?
  - monstrado 13y ago
    Well, NoSQL databases is a pretty broad term. Not all NoSQL databases are created alike, for example, MongoDB is a "document orientated database" where as HBase is a "column-oriented store" based on the Google BigTable whitepaper. As far as I know, key design is not an important aspect with MongoDB but I could be mistaken. HBase has a pretty awesome book (http://www.hbasebook.com/ http://www.hbasebook.com/), which has an entire chapter dedicated to key design. Lars (the author) also has a pretty in depth 1 hour video on key design (http://www.youtube.com/watch?v=_HLoH_PgrLk http://www.youtube.com/watch?v=_HLoH_PgrLk). HBase is pretty widely used, I've seen 1200+ node clusters running production tables.
leothekim 13y ago
Also, mongo has natural ordering which would do what the author wants without sorting.
- berito 13y ago
  No mongo natural order is just order on the disk. It is not always in reverse insertion order
- berito 13y ago
  No mongo natural order is just order on the disk. It is not always in reverse insertion order
shane42 13y ago
Take the example of crawl logs. Each log entry has a log level, timestamp and message. Typical use would be to view all ERROR (or higher) log levels, show all entries with a specific text in the message, or download the entire log. All of these should be in timestamp order. It's a shame that natural order is not insert order for non-capped collections. It's a good point that some of this can be achieved with indexing, I should have given more details in the blog post.
- dkhenry 13y ago
  Right for that example ,not knowing the specifics, I would index on timestamp and then I could sort by timestamp which once indexed should be a relativly fast operation. I could even go one step better and make the _id a construct of { <timestamp>,<loglevel>,<fuzz> } Then Inserting would be done in order and I would get a magic index for free (depending on if I often query by loglevel I might leave it out) This gives locality of timestamps and helps keep "Hot" sections in memory.
deleted 13y ago
[deleted]

by dkhenry 13y ago

deleted 13y ago

[deleted]

mayank 13y ago

No kidding. Without details, it really sounds like the author is a bit clueless. He mentions the lack of joins, but doesn't say a word about Mapreduce. "MongoDB needs to walk the index from the beginning to the offset..." You don't "walk an index". It's an index. "Too many databases" sounds a little suspicious. Why not add an indexed field to partition records? Complaining about a lack of schema, transactions and triggers? Really? Did you read the docs at all before starting? MongoDB is not without its problems, but friend, I think you wanted either Postgres or Hadoop.

PavlovsCat 13y ago
You don't "walk an index". It's an index If you have an address book, you don't have to walk through the city to find an address, but you do have to look through your address book in some way or other. Of course, you can have an index of the index ("C starts at page 7"), but then you have to look through the index of the index.
- shane42 13y ago
  The problems with pagination are explained better on this SO post: http://stackoverflow.com/questions/7228169/slow-pagination-over-tons-of-records-in-mongo http://stackoverflow.com/questions/7228169/slow-pagination-o... Mongo docs used the "walk" terminology
- mayank 13y ago
  Not sure what you're getting at. "Walk" usually means a sequential scan. An index is sorted, so you can binary search.
  - PavlovsCat 13y ago
    See other reply. If the docs say MongoDB "walks", it's hilarious to complain that someone obviously didn't read the docs for saying "walk", too.
- monstrado 13y ago
  It looks like they went with HBase for the replacement which means they can scan an index range, keys are in lexicographical order, so it makes it pretty easy to scan over a series of data with a single RPC call.

monstrado 13y ago

With a database like HBase, it's already ordered lexicographically which makes it easy to grab a range of data in the order it was written in. You could have a key design like <reverse_domain>-<epoch> which would allow quick scans over large amounts of data. ie..Scan from <object_id>-1368536860 to <object_id>-1368540450 HBase is multidimensional though, which allows you to keep N numbers of versions of a cell. By default you will get the latest version of the cell back, but you could also opt to receive N versions back, which is useful for time series use cases.

swah 13y ago
The last time I looked there weren't many resources explaining how to design NoSQL databases (how to compose your keys, when to avoid normalization, etc). Has this improved?
- monstrado 13y ago
  Well, NoSQL databases is a pretty broad term. Not all NoSQL databases are created alike, for example, MongoDB is a "document orientated database" where as HBase is a "column-oriented store" based on the Google BigTable whitepaper. As far as I know, key design is not an important aspect with MongoDB but I could be mistaken. HBase has a pretty awesome book (http://www.hbasebook.com/ http://www.hbasebook.com/), which has an entire chapter dedicated to key design. Lars (the author) also has a pretty in depth 1 hour video on key design (http://www.youtube.com/watch?v=_HLoH_PgrLk http://www.youtube.com/watch?v=_HLoH_PgrLk). HBase is pretty widely used, I've seen 1200+ node clusters running production tables.

leothekim 13y ago

Also, mongo has natural ordering which would do what the author wants without sorting.

berito 13y ago
No mongo natural order is just order on the disk. It is not always in reverse insertion order
berito 13y ago
No mongo natural order is just order on the disk. It is not always in reverse insertion order

shane42 13y ago

Take the example of crawl logs. Each log entry has a log level, timestamp and message. Typical use would be to view all ERROR (or higher) log levels, show all entries with a specific text in the message, or download the entire log. All of these should be in timestamp order. It's a shame that natural order is not insert order for non-capped collections. It's a good point that some of this can be achieved with indexing, I should have given more details in the blog post.

dkhenry 13y ago
Right for that example ,not knowing the specifics, I would index on timestamp and then I could sort by timestamp which once indexed should be a relativly fast operation. I could even go one step better and make the _id a construct of { <timestamp>,<loglevel>,<fuzz> } Then Inserting would be done in order and I would get a magic index for free (depending on if I often query by loglevel I might leave it out) This gives locality of timestamps and helps keep "Hot" sections in memory.

deleted 13y ago

[deleted]