4 ms·

I'm sure Fauna is a great database and probably cheaper in many cases. I just have some issues with the "Complex Example". I just don't feel it is realistic tha

I'm sure Fauna is a great database and probably cheaper in many cases. I just have some issues with the "Complex Example". I just don't feel it is realistic that anyone familiar with DynamoDB would create such a schema. It comes across like a good schema for Fauna is being forced onto DynamoDB, without an evaluation of what would be the recommended "DynamoDB way" of solving the customer's needs. > We have an accounts table with 20 secondary indexes defined for all the possible sort fields (DynamoDB’s maximum—Fauna has no limit). The usecase of having 20 secondary indexes in DDB is an extremely rare case. Arguably should be considered an anti-pattern, only used for an application transitioning query patterns in some way. If this is the norm for an application, I'd argue the product managers/developers do not understand their customer's needs well enough. I'd assume that at this stage in the product's life, a basic Postgress installation is likely a better choice. Additionally, if the query pattern really needs to be "super flexible" for the long term, you'll find that eventually you'll need more and more of ElasticSearch's tech (or similar tech). A very common pattern is to use DDB Streams to ElasticSearch connector (obviously sacrificing query-after-write consistency). > Viewing just the default account screen queries 7 indexes and 25 documents. A typical activity update transactionally updates 3 documents at a time with 10 dependency checks and modifies all 35 indexes. This is such a red flag. If your application requires this from DDB, you should change your schema (probably more de-normalization). However, the example doesn't have enough information for me to suggest a better schema to meet the customer's needs. Disclaimer: I work at Amazon, but not in AWS. My opinions are my own.

evanweaver 6y ago
We are in agreement, the difference in experience as you move between denormalized key/value style modeling and normalized relational modeling is the core of the post. DynamoDB has added relational-like features, but using them in a traditional relational way goes against its architectural grain. Is it necessary that data modeling flexibility must decline as an application matures and scales, though? This was one of the larger millstones around our neck at Twitter and what we are building Fauna to avoid.
- yazaddaruvala 6y ago
  > Is it necessary that data modeling flexibility must decline as an application matures and scales, though? Yeah, this is a valuable question, and I agree its not an obvious answer. The tricky part is that at enough scale, humans are really the bottleneck. Teams step on each others toes, abuse schemas, add data to a schema "because", etc, etc. Its possible that to best manage the humans, de-normalization and relying on replicated data stores with slightly different views of the data is simplest. Add again if "overly flexible" is a long term product requirement, I'd argue you're going to eventually need full text search with all the power of Lucene (I'm betting its on your roadmap). If Fauna perfectly addresses this problem domain, its likely quite helpful, but this article did not convince me it'll always be cheaper / better than DynamoDB + ElasticSearch for the complex usecases. That said, I look forward to the day I'm proved wrong :)
  - databrecht 6y ago
    Easy searching is definitely in our roadmap but for people who might pass by I did want to point out that you can already get some form of text search due to the way we index arrays. You can easily write a sort of 'inverted index' yourself if you place ngrams in an array within a document. The concept of bindings makes this particularly easy. This is option 2 as explained here: https://stackoverflow.com/questions/62109035/how-to-get-documents-that-contain-sub-string-in-faunadb/62131536#62131536 https://stackoverflow.com/questions/62109035/how-to-get-docu... We realize it's not the perfect solution and doesn't deliver the best Dev XP at this point though :)
arpinum 6y ago
> I'd argue the product managers/developers do not understand their customer's needs well enough The limited sorting options in AWS services seem to be optimising AWS’ costs rather than understanding customer needs. I’m often frustrated by the experience when I think I can click on a column and can’t. DynamoDB doesn’t handle the use case of diverse user groups exploring data and slicing through it. That’s ok, every database has its strengths. But don’t dismiss the idea that unconstrained access patterns can be the solution to a customer need.
- redis_mlc 6y ago
  > DynamoDB doesn’t handle the use case of diverse user groups exploring data and slicing through it. Key value stores like Dynamodb and HBase deliberately have limited indexes. So your expectation is not rooted in reality. MySQL is the most tolerant at many indexes, so use MySQL EDS.
pier25 6y ago
> I'd argue the product managers/developers do not understand their customer's needs well enough. I'd assume that at this stage in the product's life, a basic Postgress installation is likely a better choice. You mean compared to DynamoDB? Because Fauna is just as flexible as Postgres.
- yazaddaruvala 6y ago
  Yeah, sorry. I was just comparing to DynamoDB, I have no comparison between Fauna and Postgres.

evanweaver 6y ago

We are in agreement, the difference in experience as you move between denormalized key/value style modeling and normalized relational modeling is the core of the post. DynamoDB has added relational-like features, but using them in a traditional relational way goes against its architectural grain. Is it necessary that data modeling flexibility must decline as an application matures and scales, though? This was one of the larger millstones around our neck at Twitter and what we are building Fauna to avoid.

yazaddaruvala 6y ago
> Is it necessary that data modeling flexibility must decline as an application matures and scales, though? Yeah, this is a valuable question, and I agree its not an obvious answer. The tricky part is that at enough scale, humans are really the bottleneck. Teams step on each others toes, abuse schemas, add data to a schema "because", etc, etc. Its possible that to best manage the humans, de-normalization and relying on replicated data stores with slightly different views of the data is simplest. Add again if "overly flexible" is a long term product requirement, I'd argue you're going to eventually need full text search with all the power of Lucene (I'm betting its on your roadmap). If Fauna perfectly addresses this problem domain, its likely quite helpful, but this article did not convince me it'll always be cheaper / better than DynamoDB + ElasticSearch for the complex usecases. That said, I look forward to the day I'm proved wrong :)
- databrecht 6y ago
  Easy searching is definitely in our roadmap but for people who might pass by I did want to point out that you can already get some form of text search due to the way we index arrays. You can easily write a sort of 'inverted index' yourself if you place ngrams in an array within a document. The concept of bindings makes this particularly easy. This is option 2 as explained here: https://stackoverflow.com/questions/62109035/how-to-get-documents-that-contain-sub-string-in-faunadb/62131536#62131536 https://stackoverflow.com/questions/62109035/how-to-get-docu... We realize it's not the perfect solution and doesn't deliver the best Dev XP at this point though :)

arpinum 6y ago

> I'd argue the product managers/developers do not understand their customer's needs well enough The limited sorting options in AWS services seem to be optimising AWS’ costs rather than understanding customer needs. I’m often frustrated by the experience when I think I can click on a column and can’t. DynamoDB doesn’t handle the use case of diverse user groups exploring data and slicing through it. That’s ok, every database has its strengths. But don’t dismiss the idea that unconstrained access patterns can be the solution to a customer need.

redis_mlc 6y ago
> DynamoDB doesn’t handle the use case of diverse user groups exploring data and slicing through it. Key value stores like Dynamodb and HBase deliberately have limited indexes. So your expectation is not rooted in reality. MySQL is the most tolerant at many indexes, so use MySQL EDS.

pier25 6y ago

> I'd argue the product managers/developers do not understand their customer's needs well enough. I'd assume that at this stage in the product's life, a basic Postgress installation is likely a better choice. You mean compared to DynamoDB? Because Fauna is just as flexible as Postgres.

yazaddaruvala 6y ago
Yeah, sorry. I was just comparing to DynamoDB, I have no comparison between Fauna and Postgres.