3 ms·

DynamoDB Local Secondary Indexes

38 points by jeffbarr 14y ago

socialist_coder 14y ago
This still doesn't prevent you from having a bunch of lookup tables if ever want to query your dynamo db data on anything but the primary key =( I wish it was more like the Google App Engine datastore where you could have indexes on any column and then fetch objects based on that.
- tom_b 14y ago
  This exact issue continues to confuse me. I don't want to be in the business of maintaining some weird, non-intuitive structure to simplify my queries. I've pondered just concatenating (or hashing a concatenation) of fields (a compound primary key) to use as a primary key on DynamoDB tables. Makes me nuts - this turns into analysis paralysis for me with DynamoDB . . .
  - socialist_coder 14y ago
    Yeah, it's kind of bogus. While I would say, "just go nuts with your lookup tables", you have to manage your quota for those additional lookup tables, so then instead of just figuring out how much quota to buy for 1 table you now have to do it for 1+X tables. And since you never want to run out of quota you end up paying for a lot more quota than you are using. Luckily in my app I only have 1 lookup I ever want to do (find your account based on your social network id, so you can restore your account after reinstalling the app + connecting to facebook), so just maintaining this single lookup table is not that bad. If you really need to query on a bunch of other fields then maybe you should maintain a single table in RDS (MySql) and just use it for looking up the DynamoDb primary key. I think that is what I would do.
- bkirkbri 14y ago
  I wouldn't be surprised if they roll out hash-key-less lookup at some point. This announcement shows they realize that the inclusion of a feature which _could_ be built on the initial primitives can reduce barriers to adoption. IMHO the challenge is to communicate the increased cost when making use of these enhanced features. One of the amazing things about DynamoDB is that AWS have learned the pain points of DBaaS from SimpleDB and structured the features and pricing of DDB to be more "true" to the limitations of distributed datastores. So, while they can likely introduce hash-key-less lookup, they have to balance the increased adoption with the downside of unhappy/confused customers that expected it to be free. As it stands now, customers need to build that capability themselves, which is instructive in terms of the costs to implement it.
  - socialist_coder 14y ago
    100% agree. I'm really looking forward to it. They have a great service so far and it seems like they are improving it all the time.
Kartificial 14y ago
Ah, nice feature addition. I recently started using DynamoDB and was wondering why the range was restricted to only one field.
tom_b 14y ago
The ability to "project some or all of the table's other attributes into a secondary index" smells a bit like a materialized view in the description. I am often thinking in a star-schema direction (too much time data warehousing) rather than normalized designs. For DynamoDB, I stumble around the primary key for such setups given that a rather large subset of the tuple values in a star schema row (plus a timestamp) actually represent the primary (compound) key for the metric measurement. Probably need to re-think all that sometime.
tav 14y ago
For any AWS folk on here, it'd be great to know how this fits with the consistency guarantees. Can we do strongly consistent index queries? Also, any chance we could have strongly consistent auto-expiring keys in DynamoDB? Would make DynamoDB a very useful tool for synchronisation/lease management and other funky uses. I also sent Werner an email asking if it would be suitable to use DynamoDB as a block device — you could then build a filesystem on top and benefit from DynamoDB features like replication, consistency, etc. Unfortunately, I fear the email must have slipped through his no-doubt-busy-inbox. If any of DynamoDB developers could shed any light on its suitability as a block device, that would be awesome! Thanks.
- bkirkbri 14y ago
  > I also sent Werner an email asking if it would be suitable to use DynamoDB as a block device — you could then build a filesystem on top and benefit from DynamoDB features like replication, consistency, etc. This. 100 times. A filesystem was my first thought when DDB was announced last year. I can't imagine it would be harder to build than GridFS was. [1] That said, I'm not talking about a FUSE-like filesystem. More like HDFS or GridFS -- a blobstore+ if you will. [1] http://docs.mongodb.org/manual/core/gridfs/ http://docs.mongodb.org/manual/core/gridfs/
  - cpleppert 14y ago
    DynamoDB is optimized for storing relatively small records on redundant fast SSDs. You would blow through all your read and write units instantly using it as a BLOB store.
    - bkirkbri 14y ago
      Well, that depends on your BLOBs, how you use them and what it's worth to you, doesn't it?
- jturow 14y ago
  My name is Jon, and I’m a member of the DynamoDB team. To your specific question about consistency guarantees, local secondary indexes support both strong and eventual consistency. The indexes are updated automatically when the primary index is updated. For more information, have a look at our FAQ guide and Developer Guides. http://aws.amazon.com/dynamodb/faqs/ http://aws.amazon.com/dynamodb/faqs/ http://aws.amazon.com/documentation/dynamodb/ http://aws.amazon.com/documentation/dynamodb/
deleted 14y ago
[deleted]

38 points by jeffbarr 14y ago

This still doesn't prevent you from having a bunch of lookup tables if ever want to query your dynamo db data on anything but the primary key =( I wish it was more like the Google App Engine datastore where you could have indexes on any column and then fetch objects based on that.

tom_b 14y ago
This exact issue continues to confuse me. I don't want to be in the business of maintaining some weird, non-intuitive structure to simplify my queries. I've pondered just concatenating (or hashing a concatenation) of fields (a compound primary key) to use as a primary key on DynamoDB tables. Makes me nuts - this turns into analysis paralysis for me with DynamoDB . . .
- socialist_coder 14y ago
  Yeah, it's kind of bogus. While I would say, "just go nuts with your lookup tables", you have to manage your quota for those additional lookup tables, so then instead of just figuring out how much quota to buy for 1 table you now have to do it for 1+X tables. And since you never want to run out of quota you end up paying for a lot more quota than you are using. Luckily in my app I only have 1 lookup I ever want to do (find your account based on your social network id, so you can restore your account after reinstalling the app + connecting to facebook), so just maintaining this single lookup table is not that bad. If you really need to query on a bunch of other fields then maybe you should maintain a single table in RDS (MySql) and just use it for looking up the DynamoDb primary key. I think that is what I would do.
bkirkbri 14y ago
I wouldn't be surprised if they roll out hash-key-less lookup at some point. This announcement shows they realize that the inclusion of a feature which _could_ be built on the initial primitives can reduce barriers to adoption. IMHO the challenge is to communicate the increased cost when making use of these enhanced features. One of the amazing things about DynamoDB is that AWS have learned the pain points of DBaaS from SimpleDB and structured the features and pricing of DDB to be more "true" to the limitations of distributed datastores. So, while they can likely introduce hash-key-less lookup, they have to balance the increased adoption with the downside of unhappy/confused customers that expected it to be free. As it stands now, customers need to build that capability themselves, which is instructive in terms of the costs to implement it.
- socialist_coder 14y ago
  100% agree. I'm really looking forward to it. They have a great service so far and it seems like they are improving it all the time.

Kartificial 14y ago

Ah, nice feature addition. I recently started using DynamoDB and was wondering why the range was restricted to only one field.

tom_b 14y ago

The ability to "project some or all of the table's other attributes into a secondary index" smells a bit like a materialized view in the description. I am often thinking in a star-schema direction (too much time data warehousing) rather than normalized designs. For DynamoDB, I stumble around the primary key for such setups given that a rather large subset of the tuple values in a star schema row (plus a timestamp) actually represent the primary (compound) key for the metric measurement. Probably need to re-think all that sometime.

tav 14y ago

For any AWS folk on here, it'd be great to know how this fits with the consistency guarantees. Can we do strongly consistent index queries? Also, any chance we could have strongly consistent auto-expiring keys in DynamoDB? Would make DynamoDB a very useful tool for synchronisation/lease management and other funky uses. I also sent Werner an email asking if it would be suitable to use DynamoDB as a block device — you could then build a filesystem on top and benefit from DynamoDB features like replication, consistency, etc. Unfortunately, I fear the email must have slipped through his no-doubt-busy-inbox. If any of DynamoDB developers could shed any light on its suitability as a block device, that would be awesome! Thanks.

bkirkbri 14y ago
> I also sent Werner an email asking if it would be suitable to use DynamoDB as a block device — you could then build a filesystem on top and benefit from DynamoDB features like replication, consistency, etc. This. 100 times. A filesystem was my first thought when DDB was announced last year. I can't imagine it would be harder to build than GridFS was. [1] That said, I'm not talking about a FUSE-like filesystem. More like HDFS or GridFS -- a blobstore+ if you will. [1] http://docs.mongodb.org/manual/core/gridfs/ http://docs.mongodb.org/manual/core/gridfs/
- cpleppert 14y ago
  DynamoDB is optimized for storing relatively small records on redundant fast SSDs. You would blow through all your read and write units instantly using it as a BLOB store.
  - bkirkbri 14y ago
    Well, that depends on your BLOBs, how you use them and what it's worth to you, doesn't it?
jturow 14y ago
My name is Jon, and I’m a member of the DynamoDB team. To your specific question about consistency guarantees, local secondary indexes support both strong and eventual consistency. The indexes are updated automatically when the primary index is updated. For more information, have a look at our FAQ guide and Developer Guides. http://aws.amazon.com/dynamodb/faqs/ http://aws.amazon.com/dynamodb/faqs/ http://aws.amazon.com/documentation/dynamodb/ http://aws.amazon.com/documentation/dynamodb/

deleted 14y ago

[deleted]