6 ms·

You're not the only one. Some time ago I worked with a bunch of people who staked their careers on moving everything to AWS and converting it to serverless arc

You're not the only one. Some time ago I worked with a bunch of people who staked their careers on moving everything to AWS and converting it to serverless architecture. Here is just one example of how fishy it was. They used Dynamo. During presentations to the outgroup, they constatly praised Dynamo as being flexible, scalable and fast. However, internally they constantly struggled with various limitations of the database, from limitations of indexing to not being able to get certain types of metadata. My point is, the architecture they were designing was heavily bent around Dynamo's strength and weaknesses. Considering how extreme and peculiar those strength and weaknesses are, switching to another database would require heavy re-engineering of the rest of their architecture, much more so than, say, switching between Oracle, MS SQL and Postgress.

scarface74 8y ago
If you are actually taking advantage of all of the features of sql server or Oracle, you’re not just going to switch overnight. I’m no fan of DynamoDB and a multi certified AWS fan, but, I know it’s strengths and weaknesses and so should anyone else.
- vhold 8y ago
  A product that actually uses all the features of Oracle would be naturally hilarious. Oracle's documentation is over 500 megabytes compressed. It's an alternative computing reality. Here is a 5559 page book of error codes. Imagine having a physical copy of this on your desk. https://docs.oracle.com/en/database/oracle/oracle-database/18/errmg/error-messages.pdf https://docs.oracle.com/en/database/oracle/oracle-database/1...
  - seibelj 8y ago
    Holy shit
    - StavrosK 8y ago
      This is exactly right, wow. I am almost speechless.
  - nickstefan12 8y ago
    > alternative computing reality This such the perfect description. I was working on a Django feature last year that had to work on all DBs and testing the oracle stuff was like entering a new reality lol.
  - FireBeyond 8y ago
    And full of such amazing utility, such as: " CLSRSC-00009: No value passed as OLR locations Cause: No value was passed as OLR locations. Action: None "
- colllectorof 8y ago
  SQL Server and Oracle have different features and idiosyncrasies, but use the same fundamental models of data storage that is a product of decades of research and development and isn't controlled or promoted by a single company. You're not going to suddenly run into some fundamental limitation that will suddenly fuck you over on architectural level. This happens all the time with databases like Dynamo. All of them feel amazing when you use them in the "intended" way, but they also have extreme limitations and those limitations are different for every product. Why? Because they aren't based on some fundamental data storage model, but rather on an aggregation of specific use cases, which are different across different products. Here is a mild practical example: https://blog.codebarrel.io/why-we-switched-from-dynamodb-back-to-rds-before-we-even-released-3c2ee092120c https://blog.codebarrel.io/why-we-switched-from-dynamodb-bac...
  - scarface74 8y ago
    I’m very well aware of that. If they thought about using DynamoDB and they did even cursory research, they would know that it’s basically a glorified key value store with the chance of using multiple key value types - ie global and local indexes. There are certain (limited) use cases where it is good. Every technology choice takes you down a certain road and limits your implementation choices in certain ways. It’s not the fault of the tool if they choose one that doesn’t meet their needs. Also, the beauty of something like AWS, is that you can do “polyglot persistence”. You can have different services in your infrastructure use different types of storage depending on the use case.
  - mypalmike 8y ago
    Dynamo, like many NoSQL dbs, is (partition key, subkey, value). Partition key gets you to a storage node. Subkey gets you range queries on that storage node. That's the "fundamental data storage model" for most of NoSQL. And it's extremely useful for some use cases. You could have mapped your use case to this storage model and gotten ridiculously fast queries on vast mountains of data, which is largely the point of NoSQL. But this would have required data duplication, home-made management tools for dealing with said data duplication, and other work that was clearly better spent implementing the much simpler SQL solution. A benefit of the SQL approach is that if your queries start getting bogged down from growth in data size and/or request volume (not entirely unexpected behavior when you have a couple joins), you can move to a different model where you treat the SQL db as a slow-access source of truth and periodically generate key/value into NoSQL or Redis/ES/etc for actually running queries.
- dragonwriter 8y ago
  > If you are actually taking advantage of all of the features of sql server or Oracle I doubt any real world application comes anywhere close to doing that. Heck, I’d be mildly surprised if any real world enterprise, considering all their apps together, does that for either SQL Server or Oracle.
ravedave5 8y ago
This is any system that uses a database. It's just worse because only aws has Dynamo.
- acdha 8y ago
  I think you’re underestimating the degrees of difference. There are many apps which support multiple SQL databases but far fewer support multiple NoSQL systems because the differences are more significant than an ORM can cover up.
pavelrub 8y ago
Not sure I understand this. Why did they choose Dynamo? And what does serverless has to do with this? This seems like a simple case of people choosing the wrong tool for the job, without really understanding its limitations or capabilities. > say, switching between Oracle, MS SQL and Postgress. That's because those are all relational DBs that use SQL. If they chose some other NoSQL DB they would likely have encountered similar "peculiar" strengths and weaknesses. And I'm still not sure what this has to do with serverless - they could've have easily implemented a serverless architecture with MySQL or Postgress on RDS instead of Dynamo. But they chose not to.
- davidjnelson 8y ago
  Aws might have fixed this by now with severless aurora, but as of earlier this year the issue was connection pools didn’t work on lambda functions due to their ephemeral nature not being able to keep the tcp sockets open for more than a few minutes before the functions froze or were terminated.
  - pavelrub 8y ago
    Connection pools always worked quite well with Lambdas, as long as the connection pool was initiated in the global scope of the running code, and not within the handler function. See [1] and [2] [1] - https://docs.aws.amazon.com/lambda/latest/dg/running-lambda-code.html https://docs.aws.amazon.com/lambda/latest/dg/running-lambda-... [2] - https://blog.spotinst.com/2017/11/19/best-practices-serverless-connection-pooling-database/ https://blog.spotinst.com/2017/11/19/best-practices-serverle...
  - dlhavema 8y ago
    We use pools in lambdas to Amazon Aurora and it works out fine. Only limit we hit was when the db server was configured way too small.
    - zedpm 8y ago
      Are you running your Lambdas inside the VPC (and incurring huge cold-start latency), or are you running your database exposed to the public internet? I've avoided using Lambda for anything that requires relational database connectivity because those appear to be the two options, and neither is viable for me.
matwood 8y ago
This sounds more like an argument to not use a NoSQL database without a specific use case. That's an argument I completely agree with. I'm not sure it's a specific argument against Dynamo or AWS in general though.

scarface74 8y ago

If you are actually taking advantage of all of the features of sql server or Oracle, you’re not just going to switch overnight. I’m no fan of DynamoDB and a multi certified AWS fan, but, I know it’s strengths and weaknesses and so should anyone else.

vhold 8y ago
A product that actually uses all the features of Oracle would be naturally hilarious. Oracle's documentation is over 500 megabytes compressed. It's an alternative computing reality. Here is a 5559 page book of error codes. Imagine having a physical copy of this on your desk. https://docs.oracle.com/en/database/oracle/oracle-database/18/errmg/error-messages.pdf https://docs.oracle.com/en/database/oracle/oracle-database/1...
- seibelj 8y ago
  Holy shit
  - StavrosK 8y ago
    This is exactly right, wow. I am almost speechless.
- nickstefan12 8y ago
  > alternative computing reality This such the perfect description. I was working on a Django feature last year that had to work on all DBs and testing the oracle stuff was like entering a new reality lol.
- FireBeyond 8y ago
  And full of such amazing utility, such as: " CLSRSC-00009: No value passed as OLR locations Cause: No value was passed as OLR locations. Action: None "
colllectorof 8y ago
SQL Server and Oracle have different features and idiosyncrasies, but use the same fundamental models of data storage that is a product of decades of research and development and isn't controlled or promoted by a single company. You're not going to suddenly run into some fundamental limitation that will suddenly fuck you over on architectural level. This happens all the time with databases like Dynamo. All of them feel amazing when you use them in the "intended" way, but they also have extreme limitations and those limitations are different for every product. Why? Because they aren't based on some fundamental data storage model, but rather on an aggregation of specific use cases, which are different across different products. Here is a mild practical example: https://blog.codebarrel.io/why-we-switched-from-dynamodb-back-to-rds-before-we-even-released-3c2ee092120c https://blog.codebarrel.io/why-we-switched-from-dynamodb-bac...
- scarface74 8y ago
  I’m very well aware of that. If they thought about using DynamoDB and they did even cursory research, they would know that it’s basically a glorified key value store with the chance of using multiple key value types - ie global and local indexes. There are certain (limited) use cases where it is good. Every technology choice takes you down a certain road and limits your implementation choices in certain ways. It’s not the fault of the tool if they choose one that doesn’t meet their needs. Also, the beauty of something like AWS, is that you can do “polyglot persistence”. You can have different services in your infrastructure use different types of storage depending on the use case.
- mypalmike 8y ago
  Dynamo, like many NoSQL dbs, is (partition key, subkey, value). Partition key gets you to a storage node. Subkey gets you range queries on that storage node. That's the "fundamental data storage model" for most of NoSQL. And it's extremely useful for some use cases. You could have mapped your use case to this storage model and gotten ridiculously fast queries on vast mountains of data, which is largely the point of NoSQL. But this would have required data duplication, home-made management tools for dealing with said data duplication, and other work that was clearly better spent implementing the much simpler SQL solution. A benefit of the SQL approach is that if your queries start getting bogged down from growth in data size and/or request volume (not entirely unexpected behavior when you have a couple joins), you can move to a different model where you treat the SQL db as a slow-access source of truth and periodically generate key/value into NoSQL or Redis/ES/etc for actually running queries.
dragonwriter 8y ago
> If you are actually taking advantage of all of the features of sql server or Oracle I doubt any real world application comes anywhere close to doing that. Heck, I’d be mildly surprised if any real world enterprise, considering all their apps together, does that for either SQL Server or Oracle.

ravedave5 8y ago

This is any system that uses a database. It's just worse because only aws has Dynamo.

acdha 8y ago
I think you’re underestimating the degrees of difference. There are many apps which support multiple SQL databases but far fewer support multiple NoSQL systems because the differences are more significant than an ORM can cover up.

pavelrub 8y ago

Not sure I understand this. Why did they choose Dynamo? And what does serverless has to do with this? This seems like a simple case of people choosing the wrong tool for the job, without really understanding its limitations or capabilities. > say, switching between Oracle, MS SQL and Postgress. That's because those are all relational DBs that use SQL. If they chose some other NoSQL DB they would likely have encountered similar "peculiar" strengths and weaknesses. And I'm still not sure what this has to do with serverless - they could've have easily implemented a serverless architecture with MySQL or Postgress on RDS instead of Dynamo. But they chose not to.

davidjnelson 8y ago
Aws might have fixed this by now with severless aurora, but as of earlier this year the issue was connection pools didn’t work on lambda functions due to their ephemeral nature not being able to keep the tcp sockets open for more than a few minutes before the functions froze or were terminated.
- pavelrub 8y ago
  Connection pools always worked quite well with Lambdas, as long as the connection pool was initiated in the global scope of the running code, and not within the handler function. See [1] and [2] [1] - https://docs.aws.amazon.com/lambda/latest/dg/running-lambda-code.html https://docs.aws.amazon.com/lambda/latest/dg/running-lambda-... [2] - https://blog.spotinst.com/2017/11/19/best-practices-serverless-connection-pooling-database/ https://blog.spotinst.com/2017/11/19/best-practices-serverle...
- dlhavema 8y ago
  We use pools in lambdas to Amazon Aurora and it works out fine. Only limit we hit was when the db server was configured way too small.
  - zedpm 8y ago
    Are you running your Lambdas inside the VPC (and incurring huge cold-start latency), or are you running your database exposed to the public internet? I've avoided using Lambda for anything that requires relational database connectivity because those appear to be the two options, and neither is viable for me.

matwood 8y ago

This sounds more like an argument to not use a NoSQL database without a specific use case. That's an argument I completely agree with. I'm not sure it's a specific argument against Dynamo or AWS in general though.