7 ms·

Speaking of limitations with scan in DynamoDb, I've recently been working with DynamoDB and found that our implementation was solely using scan operations. I wr

Speaking of limitations with scan in DynamoDb, I've recently been working with DynamoDB and found that our implementation was solely using scan operations. I wrote a couple lines of code to generate a secondary index, used a query operation instead for our endpoint serving page responses and that's going to save our company ~$15k this year (more as we scale) and decreased our page load times by 10x (3s to .3s) for all of our users. In total that's going to save over a year of page load time for all of our users over the next year. It's just odd to me that I had to make this change in the first place. I wonder how many millions of dollars AWS is raking in just because devs are unaware of things like that. Also makes me wonder how much energy is being wasted on inefficient DB operations..

camhart 4y ago
Their documentation is very clear about scans being expensive. Amazon isn't to blame for the poor engineering decision.
- lofatdairy 4y ago
  That was my initial impression reading the OP as well, if your implementation is slow because you're using scans, well, it's because you're using scans. That said, I honestly don't know enough about the details regarding OP's implementation and why they couldn't refactor to properly Query the PK, but it sounds similar to how you'd authenticate reads on Firebase so maybe it's habits from a different platform kicking in. That said, Amplify allowing users to just get into GraphQL whether or not they actually need it might be a legitimate noob trap.
  - nslog 4y ago
    Eh their documentation is pretty clear about all this stuff and all the new directives released a couple years back (like the PK and index stuff) clearly guide people towards using indexed queries and not scans: https://docs.amplify.aws/cli/graphql/data-modeling/ https://docs.amplify.aws/cli/graphql/data-modeling/ A lot of this is on the author IMO
wnolens 4y ago
"I wonder how many devs don't know that hash tables are bad for range scans". Fixed that for you.
ctvo 4y ago
> It's just odd to me that I had to make this change in the first place. I wonder how many millions of dollars AWS is raking in just because devs are unaware of things like that. AWS documentation very clearly outlines the behavior of a query and a scan. The internet is full of guides [1][2] on how to model your data so you utilize queries vs. scans. If the developers at your company misuse a product, and aren't aware of these features when does personal responsibility come in? How is AWS as a platform suppose to discern that the trade-off with a GSI is what you want, and how is this scenario not generalized to every product? People misuse SQL databases all the time, for example. 1 - https://www.alexdebrie.com/posts/dynamodb-single-table/ https://www.alexdebrie.com/posts/dynamodb-single-table/ 2 - https://www.sensedeep.com/blog/posts/2021/dynamodb-singletable-design.html https://www.sensedeep.com/blog/posts/2021/dynamodb-singletab...
- badrequest 4y ago
  There are two sides to this arrangment: devs who mostly don't have a choice over the tools they use vs. the trillion-dollar company that is building and documenting them. The notion that the devs have just as much culpability in how things end up as AWS is absurd.
  - ctvo 4y ago
    > devs who mostly don't have a choice over the tools they use I'm not going to assume that DynamoDB was forced upon them. I don't know. What I'm more sure about: Developers have a choice in if they want to invest the time in understanding the tools they use, and learning about the tool's capabilities. In this specific case, we're not talking about a lot of time. 2-3 hours to read the documentation for a new database you're going to be building your business on top of. What is your argument exactly?
  - icedchai 4y ago
    I've personally seen developers "forced" into using DynamoDB because some AWS professional services consultant suggested it. You could argue the developers should've known better, but they had relatively little real world experience. The costs of Dynamo are relatively small compared to the development costs of using it when it's not a good fit.
JimmyAustin 4y ago
The equivalent in a relational database would be writing a query without a where statement, and then implementing the where in application code. AWS has a lot of unimpressive bits, but DynamoDB is not one of them, you just have to know how to use it properly (like any database!), and design your schema for the database (which isn't always easy). In general, using scan means you aren't using Dynamo correctly. Caveat: I used to work at Amazon (not in AWS), but don't anymore.
- nkozyra 4y ago
  > The equivalent in a relational database would be writing a query without a where statement, Sort of, though I'd say scan operations are typically more expensive in any RDMS. The probably becomes when you want/need something more than the secondary index, then you gotta get more creative.
lightbendover 4y ago
Years ago after joining a new team, I was shocked that I just had a GSI to solve a performance problem that was somehow eluding a dozen engineers making around 200k each.
geodel 4y ago
Well what are you doing is ideally lot many engineers should do. But in my experience devs, IT departments, companies are drunk on Cloud Mezcal. So any effort which need effort is "oops thats not our business lets increase instances in Cloud to improve performance" And I am not making this up, A service we are running on ten datacenter VMs, an equivalent service is running on 150 instances of same size in AWS. The cloud service is crappy architected, developed Java application but instead of fixing it , solution is to double/triple or 10 times the instance using Kubernetes Autoscaler. One thing they have going is ever larger budget since it is next gen cloud service, whereas my serivce being plain old efficient Java app is legacy to be deprecated and replaced with new cloud version.
- re-thc 4y ago
  This has less to do with the cloud and more to do with the common mantra of "premature optimization". To a lot of engineers there's always bigger fish to fry.
silisili 4y ago
I just learned DynamoDB this year, after I was forced into it...not a NoSQL fan. In any event, while I find the structure and docs pretty confounding overall, they do make that behavior pretty clear. The libraries even separate out scans from queries, with the latter taking only a key. I'd put the blame on the implementer in this case, not AWS.
- willio58 4y ago
  I actually agree. This was probably only an issue because some dev I work with didn't read the docs or didn't think it mattered at the time because our userbase was so small the difference between a scan and a query was negligible at the time. Part of me does wonder if there would be a benefit in DynamoDB providing a function that under the hood creates secondary indices and uses queries where that would be the most efficient choice automatically. But maybe that would be a better fit for an ORM of some sort.
  - gregplaysguitar 4y ago
    That sounds like a query planner… but it’s fundamentally not really what dynamodb is about. Query planners are amazing but the tradeoff is unpredictable performance - dependent on traffic, data size etc. Dynamodb offers “predictable performance at any scale” precisely because it doesn’t offer this sort of abstraction.
  - silisili 4y ago
    That would have been a killer feature for me. Instead of reading about GSIs, LSIs, and even going so far as having to create a dummy field to do sorting properly[1], it would be pretty nice if you could somehow input the queries you want to do, and have Dynamo make suggestions. 1 - In a collection, I could find no way to easily do 'give me the latest record.', which is super easy in rdb. I ended up creating a key something static like 'x', with date as the range key to accomplish this. If anyone has a cleaner way, please share.
yamtaddle 4y ago
Our entire industry is plagued by people rushing forward while having no idea how the tools at their disposal work. It's immensely wasteful. I can't even claim to be immune from this, though I do try to know what's available and how to use it. I'm damn near 40 but it's not unusual for me to still, today, stumble on some tool older than me that does exactly what some New Shiny crap does—but, not infrequently, better, at least for some common use-cases, plus it'll have all the kinks ironed out, or at least documented. Good luck promoting any of that in the workplace, though. For all we go on and on about the importance of re-use and avoiding NIH and all that, we sure are quick to disregard solutions that we can't npm-install. No clue what the fix for this is, or if it even makes sense, all things considered, to try to fix it.
- Swizec 4y ago
  Symptom of a rapidly growing industry. With N(developers) doubling every 5 years or so, chasing newness at all times and not knowing about good old things is the norm. I'd rather have that than an industry where any tech newer than 10 years is radical and scary. Us old farts can remind the kids that old solutions exist when appropriate. It is much harder to try introducing something new into an org that doesn't want it, but needs it.
- sleepybrett 4y ago
  Yeah I liken it to someone who picks up a blacksmiths hammer and then goes and drives some nails with it. It's like 'sure I guess it works, but man there are way better options that will work better'. Happens with all our industry tools, but perhaps there is no better example than database choice. Engineers often see a database simply as a place to store things, it's just a box.. they work on a product for awhile that's using mongo, they are more likely to choose it later when they are developing a solution because they've used it before and know it 'works' and more importantly they know something about it. Databases as a topic of study are complicated, there are tons of tradeoffs happening all over the place and until you hit any given pothole you may not really understand those tradeoffs. I guess what I'm saying is that in my life I'm lucky enough to have an old friend that is REALLY into databses. You should find this friend or become this friend. Honestly at the end of the day this is all human nature. The hammer worked for you before why do the research to find out that there is a better hammer for you to use that would get your job done quicker and easier. You've gotta get these boards nailed down RIGHT FUCKING NOW or your bosses are going to be pissed.
vanjajaja1 4y ago
It's funny to read something like this contrast against all the statements about how leetcode interviews do not help and ds/a is something you can learn on the job
simplotek 4y ago
> It's just odd to me that I had to make this change in the first place. I wonder how many millions of dollars AWS is raking in just because devs are unaware of things like that. What "things like that"? Do you mean the basics of querying data with DynamoDB? https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-query-scan.html https://docs.aws.amazon.com/amazondynamodb/latest/developerg... This is literally the second topic in any intro to DynamoDB course, that comes right after creating your first DynamoDB table. Even that covers explicitly the basic requirements of running queries.
lozenge 4y ago
That's funny because reading SQL doesn't tell you whether the DB will scan or use indexes. With Dynamo DB it's a separate api request, it couldn't get more explicit.

camhart 4y ago

Their documentation is very clear about scans being expensive. Amazon isn't to blame for the poor engineering decision.

lofatdairy 4y ago
That was my initial impression reading the OP as well, if your implementation is slow because you're using scans, well, it's because you're using scans. That said, I honestly don't know enough about the details regarding OP's implementation and why they couldn't refactor to properly Query the PK, but it sounds similar to how you'd authenticate reads on Firebase so maybe it's habits from a different platform kicking in. That said, Amplify allowing users to just get into GraphQL whether or not they actually need it might be a legitimate noob trap.
- nslog 4y ago
  Eh their documentation is pretty clear about all this stuff and all the new directives released a couple years back (like the PK and index stuff) clearly guide people towards using indexed queries and not scans: https://docs.amplify.aws/cli/graphql/data-modeling/ https://docs.amplify.aws/cli/graphql/data-modeling/ A lot of this is on the author IMO

wnolens 4y ago

"I wonder how many devs don't know that hash tables are bad for range scans". Fixed that for you.

ctvo 4y ago

> It's just odd to me that I had to make this change in the first place. I wonder how many millions of dollars AWS is raking in just because devs are unaware of things like that. AWS documentation very clearly outlines the behavior of a query and a scan. The internet is full of guides [1][2] on how to model your data so you utilize queries vs. scans. If the developers at your company misuse a product, and aren't aware of these features when does personal responsibility come in? How is AWS as a platform suppose to discern that the trade-off with a GSI is what you want, and how is this scenario not generalized to every product? People misuse SQL databases all the time, for example. 1 - https://www.alexdebrie.com/posts/dynamodb-single-table/ https://www.alexdebrie.com/posts/dynamodb-single-table/ 2 - https://www.sensedeep.com/blog/posts/2021/dynamodb-singletable-design.html https://www.sensedeep.com/blog/posts/2021/dynamodb-singletab...

badrequest 4y ago
There are two sides to this arrangment: devs who mostly don't have a choice over the tools they use vs. the trillion-dollar company that is building and documenting them. The notion that the devs have just as much culpability in how things end up as AWS is absurd.
- ctvo 4y ago
  > devs who mostly don't have a choice over the tools they use I'm not going to assume that DynamoDB was forced upon them. I don't know. What I'm more sure about: Developers have a choice in if they want to invest the time in understanding the tools they use, and learning about the tool's capabilities. In this specific case, we're not talking about a lot of time. 2-3 hours to read the documentation for a new database you're going to be building your business on top of. What is your argument exactly?
- icedchai 4y ago
  I've personally seen developers "forced" into using DynamoDB because some AWS professional services consultant suggested it. You could argue the developers should've known better, but they had relatively little real world experience. The costs of Dynamo are relatively small compared to the development costs of using it when it's not a good fit.

JimmyAustin 4y ago

The equivalent in a relational database would be writing a query without a where statement, and then implementing the where in application code. AWS has a lot of unimpressive bits, but DynamoDB is not one of them, you just have to know how to use it properly (like any database!), and design your schema for the database (which isn't always easy). In general, using scan means you aren't using Dynamo correctly. Caveat: I used to work at Amazon (not in AWS), but don't anymore.

nkozyra 4y ago
> The equivalent in a relational database would be writing a query without a where statement, Sort of, though I'd say scan operations are typically more expensive in any RDMS. The probably becomes when you want/need something more than the secondary index, then you gotta get more creative.

lightbendover 4y ago

Years ago after joining a new team, I was shocked that I just had a GSI to solve a performance problem that was somehow eluding a dozen engineers making around 200k each.

geodel 4y ago

Well what are you doing is ideally lot many engineers should do. But in my experience devs, IT departments, companies are drunk on Cloud Mezcal. So any effort which need effort is "oops thats not our business lets increase instances in Cloud to improve performance" And I am not making this up, A service we are running on ten datacenter VMs, an equivalent service is running on 150 instances of same size in AWS. The cloud service is crappy architected, developed Java application but instead of fixing it , solution is to double/triple or 10 times the instance using Kubernetes Autoscaler. One thing they have going is ever larger budget since it is next gen cloud service, whereas my serivce being plain old efficient Java app is legacy to be deprecated and replaced with new cloud version.

re-thc 4y ago
This has less to do with the cloud and more to do with the common mantra of "premature optimization". To a lot of engineers there's always bigger fish to fry.

silisili 4y ago

I just learned DynamoDB this year, after I was forced into it...not a NoSQL fan. In any event, while I find the structure and docs pretty confounding overall, they do make that behavior pretty clear. The libraries even separate out scans from queries, with the latter taking only a key. I'd put the blame on the implementer in this case, not AWS.

willio58 4y ago
I actually agree. This was probably only an issue because some dev I work with didn't read the docs or didn't think it mattered at the time because our userbase was so small the difference between a scan and a query was negligible at the time. Part of me does wonder if there would be a benefit in DynamoDB providing a function that under the hood creates secondary indices and uses queries where that would be the most efficient choice automatically. But maybe that would be a better fit for an ORM of some sort.
- gregplaysguitar 4y ago
  That sounds like a query planner… but it’s fundamentally not really what dynamodb is about. Query planners are amazing but the tradeoff is unpredictable performance - dependent on traffic, data size etc. Dynamodb offers “predictable performance at any scale” precisely because it doesn’t offer this sort of abstraction.
- silisili 4y ago
  That would have been a killer feature for me. Instead of reading about GSIs, LSIs, and even going so far as having to create a dummy field to do sorting properly[1], it would be pretty nice if you could somehow input the queries you want to do, and have Dynamo make suggestions. 1 - In a collection, I could find no way to easily do 'give me the latest record.', which is super easy in rdb. I ended up creating a key something static like 'x', with date as the range key to accomplish this. If anyone has a cleaner way, please share.

yamtaddle 4y ago

Our entire industry is plagued by people rushing forward while having no idea how the tools at their disposal work. It's immensely wasteful. I can't even claim to be immune from this, though I do try to know what's available and how to use it. I'm damn near 40 but it's not unusual for me to still, today, stumble on some tool older than me that does exactly what some New Shiny crap does—but, not infrequently, better, at least for some common use-cases, plus it'll have all the kinks ironed out, or at least documented. Good luck promoting any of that in the workplace, though. For all we go on and on about the importance of re-use and avoiding NIH and all that, we sure are quick to disregard solutions that we can't npm-install. No clue what the fix for this is, or if it even makes sense, all things considered, to try to fix it.

Swizec 4y ago
Symptom of a rapidly growing industry. With N(developers) doubling every 5 years or so, chasing newness at all times and not knowing about good old things is the norm. I'd rather have that than an industry where any tech newer than 10 years is radical and scary. Us old farts can remind the kids that old solutions exist when appropriate. It is much harder to try introducing something new into an org that doesn't want it, but needs it.
sleepybrett 4y ago
Yeah I liken it to someone who picks up a blacksmiths hammer and then goes and drives some nails with it. It's like 'sure I guess it works, but man there are way better options that will work better'. Happens with all our industry tools, but perhaps there is no better example than database choice. Engineers often see a database simply as a place to store things, it's just a box.. they work on a product for awhile that's using mongo, they are more likely to choose it later when they are developing a solution because they've used it before and know it 'works' and more importantly they know something about it. Databases as a topic of study are complicated, there are tons of tradeoffs happening all over the place and until you hit any given pothole you may not really understand those tradeoffs. I guess what I'm saying is that in my life I'm lucky enough to have an old friend that is REALLY into databses. You should find this friend or become this friend. Honestly at the end of the day this is all human nature. The hammer worked for you before why do the research to find out that there is a better hammer for you to use that would get your job done quicker and easier. You've gotta get these boards nailed down RIGHT FUCKING NOW or your bosses are going to be pissed.

vanjajaja1 4y ago

It's funny to read something like this contrast against all the statements about how leetcode interviews do not help and ds/a is something you can learn on the job

simplotek 4y ago

> It's just odd to me that I had to make this change in the first place. I wonder how many millions of dollars AWS is raking in just because devs are unaware of things like that. What "things like that"? Do you mean the basics of querying data with DynamoDB? https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-query-scan.html https://docs.aws.amazon.com/amazondynamodb/latest/developerg... This is literally the second topic in any intro to DynamoDB course, that comes right after creating your first DynamoDB table. Even that covers explicitly the basic requirements of running queries.

lozenge 4y ago

That's funny because reading SQL doesn't tell you whether the DB will scan or use indexes. With Dynamo DB it's a separate api request, it couldn't get more explicit.