6 ms·

Very informative read … however, I am left not really understanding what the Entity Manager is … perhaps a migration tool to move data from NoSQL to RDBMS, perh

Very informative read … however, I am left not really understanding what the Entity Manager is … perhaps a migration tool to move data from NoSQL to RDBMS, perhaps a front end helper service? Is it FOSS, a PAYG service? How can I deploy it? Perhaps this article long enough already and is intended to whet our appetite.

karmaniverous 2y ago
I'm the author, and thanks for asking! The article is really just background, submitted by a friend. Still building the demo & documentation, so my apologies for the confusion. Entity Manager is a framework for defining, managing, and most importantly QUERYING an entity model with DynamoDB. It's actually platform-generic, so the DynamoDB-specific machinery is implemented at https://github.com/karmaniverous/entity-client-dynamodb https://github.com/karmaniverous/entity-client-dynamodb Entity Manager's most important feature is that it permits a simple, scheduled partition sharding configuration and then transparent, multi-index querying of data across shards with a very compact, fluent query API. This resolves the biggest challenge of using DynamoDB at scale, which is that very large data sets MUST be sharded, and a given query can ONLY operate against a single shard. If you're querying on the basis of a related record, you won't know which shard your results will be on so you must query ALL shards. Entity Manager reduces this to an effortless operation: once you've defined your sharding strategy for a given entity, you can forget sharding is even a thing. For some more color on Entity Manager within the context of SQL vs NoSQL databases, please review this (much shorter!) article: https://karmanivero.us/projects/entity-manager/sql-vs-nosql https://karmanivero.us/projects/entity-manager/sql-vs-nosql
- librasteve 2y ago
  thanks … that’s really helpful
- vdvsvwvwvwvwv 2y ago
  That makes me want to learn more. I will read this. Like most been burned and frustrated by this sort of sharded DB and unperformant queries if you dont or cant use the partition key. Usually you need a second table amirite :) And seen others burned. Especially when your intro is a Jira ticket learning on the job!
  - karmaniverous 2y ago
    Very cool of you to say so! I've actually been using the JS version of EM in production for over a year. It's been working flawlessly. The TS version is a complete rewrite that factors in a BUNCH of lessons learned and is completely--maybe obsessively lol--type-safe. The query builder got a LOT of attention, and the fluent API reduces even complex queries to a super-compact, declarative coding experience. I'm pushing a big update tonight and will then resume my focus on the demo & docs, basically the companion stuff to this one. Should be ready for use in a couple of weeks. Thanks for the interest, it really means a lot to me!
mustime 2y ago
[dead]

karmaniverous 2y ago

I'm the author, and thanks for asking! The article is really just background, submitted by a friend. Still building the demo & documentation, so my apologies for the confusion. Entity Manager is a framework for defining, managing, and most importantly QUERYING an entity model with DynamoDB. It's actually platform-generic, so the DynamoDB-specific machinery is implemented at https://github.com/karmaniverous/entity-client-dynamodb https://github.com/karmaniverous/entity-client-dynamodb Entity Manager's most important feature is that it permits a simple, scheduled partition sharding configuration and then transparent, multi-index querying of data across shards with a very compact, fluent query API. This resolves the biggest challenge of using DynamoDB at scale, which is that very large data sets MUST be sharded, and a given query can ONLY operate against a single shard. If you're querying on the basis of a related record, you won't know which shard your results will be on so you must query ALL shards. Entity Manager reduces this to an effortless operation: once you've defined your sharding strategy for a given entity, you can forget sharding is even a thing. For some more color on Entity Manager within the context of SQL vs NoSQL databases, please review this (much shorter!) article: https://karmanivero.us/projects/entity-manager/sql-vs-nosql https://karmanivero.us/projects/entity-manager/sql-vs-nosql

librasteve 2y ago
thanks … that’s really helpful
vdvsvwvwvwvwv 2y ago
That makes me want to learn more. I will read this. Like most been burned and frustrated by this sort of sharded DB and unperformant queries if you dont or cant use the partition key. Usually you need a second table amirite :) And seen others burned. Especially when your intro is a Jira ticket learning on the job!
- karmaniverous 2y ago
  Very cool of you to say so! I've actually been using the JS version of EM in production for over a year. It's been working flawlessly. The TS version is a complete rewrite that factors in a BUNCH of lessons learned and is completely--maybe obsessively lol--type-safe. The query builder got a LOT of attention, and the fluent API reduces even complex queries to a super-compact, declarative coding experience. I'm pushing a big update tonight and will then resume my focus on the demo & docs, basically the companion stuff to this one. Should be ready for use in a couple of weeks. Thanks for the interest, it really means a lot to me!

mustime 2y ago

[dead]