4 ms·

We can now achieve pretty high scalability from day 1 with a tiny bit of "engineering cost" up front. Serverless on AWS is pretty cheap and can scale quickly.

by huzaif 7y ago

ludamad 7y ago
I hated DynamoDB. What good is there about it other than convenience?
- Scarbutt 7y ago
  If you don't mind watching a video: https://www.youtube.com/watch?v=6yqfmXiZTlM https://www.youtube.com/watch?v=6yqfmXiZTlM
- ignoramous 7y ago
  I've found that KV stores like DynamoDB make for a good control-plane configuration repository. For instance, say, you need to know if a client, X, is allowed to access a resource, Y. And, say, you've clients in order of millions and resources in order of 100s, and you've got very specific queries to execute on such denormalized data and need consistently low latency and high throughput across key-combinations. Another good use-case is to store checkpointing information. Say, you've processed some task and would like to check-in the result. Either the information fits the 400KB DynamoDB limit or you use DynamoDB as a index to a S3 file. You could do those things with managed or self-hosted RDBMS, but DynamoDB takes away the need to manage the hardware, the backups, the scale-ups, and the scale-outs, reduces ceremony whilst dealing with locks, schemas, misbehaving clients, and myraid other configuration knobs whilst also fitting your queries patterns to a tee. KV stores typically give you consistent performance on reads and writes, if you avoid cascading relationships between two or more keys, and make just the right amount of trade-offs in terms of both cross-cluster data-consistency and cross-table data-consistency. Besides, in terms of features, one can add a write-through cache in front of a DynamoDB table, can point-in-time-restore data up to a minute granularity, can create on-demand tables that scale with load (not worry about provisioned capacity anymore), can auto-stream updates to Elasticsearch for materialised views or consume the updates in real-time themselves, can replicate tables world-wide with lax consistency guarantees and so on...with very little fuss, if any. Running databases is hard. I pretty much exclusively favour a managed solution over self-hosted one, at this point. And for denormalized data, a managed KV store makes for a viable solution, imo.
  - danenania 7y ago
    All good points, but one thing people should look at very closely before choosing DynamoDB as a primary db is the transaction limits. Most apps are going to have some operations that should be atomic and involve more than 25 items. With DynamoDB, your only option currently is to break these up into multiple transactions and hope none of them fail. But as you scale, eventually some will fail, while others in the same request succeed, leaving your data in an inconsistent state. While this could be ok for some apps, I think for most use cases it's really bad and ends up being more trouble than what you save on ops in the long run, especially considering options like Aurora that, while not as hands-off as Dynamo, are still pretty low-maintenance and don't limit transactions at all.
aratakareigen 7y ago
Great, but this reads like a particularly blunt Amazon ad. Is there a way to achieve "high scalability" without selling my soul to Amazon?
- dumbfoundded 7y ago
  I think if you use something like serverless, you can abstract the cloud layer. I've never used it for anything more than a toy project though. https://serverless.com/ https://serverless.com/
- huzaif 7y ago
  Yes, it does read like that. In the context of a start-up, cost is a big factor and then perhaps (hopefully) handling growth. You could start small and refactor apps/infrastructure as you grow but I am unsure how one could afford to do that efficiently while also managing a growing startup. On the selling soul to cloud provider, I don't see it like that. I have a start-up to bootstrap and I want to see it grow before making altruistic decisions that would sustain the business model. Once you are past the initial growth stage, there are many options for serverless, gateway, caches, proxies that can be orchestrated in K8 on commodity VMs in the datacenter. Though this is where you would need some decent financial backing. (I am not associated with Amazon, Google or Azure. I do run my start-up on Azure.)
  - ignoramous 7y ago
    I'm down a similar route, but I must point out that beyond a certain number of users / scale, Serverless becomes cost-prohibitive. For instance, per back-of-the-napkin calculation, the Serverless load I run right now, though very cost-effective for the smaller userbase I've got, would quickly spiral out of control once I cross a threshold (which is at 40k users). At 5M users, I'd be paying an astonishing 100x the cost than if I hosted the services on a VPS. That said, Serverless does reduce DevOps to an extent but introduces different but fewer other complications. As patio11 would like to remind us all, we've got a revenue problem, not a cost problem. [0] [0] https://news.ycombinator.com/item?id=22202301 https://news.ycombinator.com/item?id=22202301
- 7y ago
papito 7y ago
I bet that DNC Iowa primaries app was serverless. Problem solved! [dusts off hands].
marriedWpt 7y ago
AWS seems like it would be expensive long term. Between my issues with AWS currently and the exterior look of Amazon, I'm skeptical AWS is a good solution.
- lbriner 7y ago
  Like most providers, it does depend. Some products are priced very competitively while others seem over-the-top. For smaller companies, the cloud is a cheaper starting point for many systems but even for larger organisations, there are savings to be made by outsourcing your servers. Do you know how much it costs to install and maintain a decent air-con system for your server room? One of the other major advantages of cloud is that you can save a lot in support staff. Compare the wages of even 1 decent sysadmin looking after your own hardware compared to several thousand dollars of AWS and it's still loads cheaper. Hardware upgrades, OS updates etc. are often automatic or hidden.
dkarras 7y ago
It will cost an arm and a leg when you eventually grow though.

by huzaif 7y ago

ludamad 7y ago

I hated DynamoDB. What good is there about it other than convenience?

Scarbutt 7y ago
If you don't mind watching a video: https://www.youtube.com/watch?v=6yqfmXiZTlM https://www.youtube.com/watch?v=6yqfmXiZTlM
ignoramous 7y ago
I've found that KV stores like DynamoDB make for a good control-plane configuration repository. For instance, say, you need to know if a client, X, is allowed to access a resource, Y. And, say, you've clients in order of millions and resources in order of 100s, and you've got very specific queries to execute on such denormalized data and need consistently low latency and high throughput across key-combinations. Another good use-case is to store checkpointing information. Say, you've processed some task and would like to check-in the result. Either the information fits the 400KB DynamoDB limit or you use DynamoDB as a index to a S3 file. You could do those things with managed or self-hosted RDBMS, but DynamoDB takes away the need to manage the hardware, the backups, the scale-ups, and the scale-outs, reduces ceremony whilst dealing with locks, schemas, misbehaving clients, and myraid other configuration knobs whilst also fitting your queries patterns to a tee. KV stores typically give you consistent performance on reads and writes, if you avoid cascading relationships between two or more keys, and make just the right amount of trade-offs in terms of both cross-cluster data-consistency and cross-table data-consistency. Besides, in terms of features, one can add a write-through cache in front of a DynamoDB table, can point-in-time-restore data up to a minute granularity, can create on-demand tables that scale with load (not worry about provisioned capacity anymore), can auto-stream updates to Elasticsearch for materialised views or consume the updates in real-time themselves, can replicate tables world-wide with lax consistency guarantees and so on...with very little fuss, if any. Running databases is hard. I pretty much exclusively favour a managed solution over self-hosted one, at this point. And for denormalized data, a managed KV store makes for a viable solution, imo.
- danenania 7y ago
  All good points, but one thing people should look at very closely before choosing DynamoDB as a primary db is the transaction limits. Most apps are going to have some operations that should be atomic and involve more than 25 items. With DynamoDB, your only option currently is to break these up into multiple transactions and hope none of them fail. But as you scale, eventually some will fail, while others in the same request succeed, leaving your data in an inconsistent state. While this could be ok for some apps, I think for most use cases it's really bad and ends up being more trouble than what you save on ops in the long run, especially considering options like Aurora that, while not as hands-off as Dynamo, are still pretty low-maintenance and don't limit transactions at all.

aratakareigen 7y ago

Great, but this reads like a particularly blunt Amazon ad. Is there a way to achieve "high scalability" without selling my soul to Amazon?

dumbfoundded 7y ago
I think if you use something like serverless, you can abstract the cloud layer. I've never used it for anything more than a toy project though. https://serverless.com/ https://serverless.com/
huzaif 7y ago
Yes, it does read like that. In the context of a start-up, cost is a big factor and then perhaps (hopefully) handling growth. You could start small and refactor apps/infrastructure as you grow but I am unsure how one could afford to do that efficiently while also managing a growing startup. On the selling soul to cloud provider, I don't see it like that. I have a start-up to bootstrap and I want to see it grow before making altruistic decisions that would sustain the business model. Once you are past the initial growth stage, there are many options for serverless, gateway, caches, proxies that can be orchestrated in K8 on commodity VMs in the datacenter. Though this is where you would need some decent financial backing. (I am not associated with Amazon, Google or Azure. I do run my start-up on Azure.)
- ignoramous 7y ago
  I'm down a similar route, but I must point out that beyond a certain number of users / scale, Serverless becomes cost-prohibitive. For instance, per back-of-the-napkin calculation, the Serverless load I run right now, though very cost-effective for the smaller userbase I've got, would quickly spiral out of control once I cross a threshold (which is at 40k users). At 5M users, I'd be paying an astonishing 100x the cost than if I hosted the services on a VPS. That said, Serverless does reduce DevOps to an extent but introduces different but fewer other complications. As patio11 would like to remind us all, we've got a revenue problem, not a cost problem. [0] [0] https://news.ycombinator.com/item?id=22202301 https://news.ycombinator.com/item?id=22202301
7y ago

papito 7y ago

I bet that DNC Iowa primaries app was serverless. Problem solved! [dusts off hands].

marriedWpt 7y ago

AWS seems like it would be expensive long term. Between my issues with AWS currently and the exterior look of Amazon, I'm skeptical AWS is a good solution.

lbriner 7y ago
Like most providers, it does depend. Some products are priced very competitively while others seem over-the-top. For smaller companies, the cloud is a cheaper starting point for many systems but even for larger organisations, there are savings to be made by outsourcing your servers. Do you know how much it costs to install and maintain a decent air-con system for your server room? One of the other major advantages of cloud is that you can save a lot in support staff. Compare the wages of even 1 decent sysadmin looking after your own hardware compared to several thousand dollars of AWS and it's still loads cheaper. Hardware upgrades, OS updates etc. are often automatic or hidden.

dkarras 7y ago

It will cost an arm and a leg when you eventually grow though.