4 ms·
So, I'm actually intensely curios about how this thing works, even though its probably a trade secret or something. How do you prevent oscillations in resource
by pm90 12y ago
So, I'm actually intensely curios about how this thing works, even though its probably a trade secret or something. How do you prevent oscillations in resource allocations when the traffic is fluctuating? How do you know if e.g. an adversary is just DDoS'ing your application to make it more expensive to run your application? I guess there must be some kind of threshold that you can set? But then this would cut off the website for the real customers...
- bhouston 12y ago> an adversary is just DDoS'ing your application to make it more expensive to run your application? It isn't the compute engines job to determine if workload is malicious. That is a separate but real problem.
- ajessup 12y agoPM for GCE Autoscaling here. On protection from malicious traffic spikes - you can set a max-instances threshold to prevent over-spend which you can change at any time. Ultimately though, from an instance's perspective, there's not a lot of difference between a malicious DDoS and a genuine traffic spike. You can mitigate this (a) by setting low max-instances thresholds when you aren't expecting high traffic, and (b) upstream filtering or QOS management to filter our DDoS traffic from legitimate traffic.
- _dark_matter_ 12y agoDo you have models for traffic pattern differences between spikes and DDoS attacks? It seems that a DDoS attack would have different patterns (quicker spike, no buildup, etc.). Just speculation here.
- ajessup 12y agoAutoscaler will respond differently to different traffic patterns, but won't go as far as identifying a particular traffic pattern as malicious and refusing to grow resources to accommodate it (most architectures filter malicious requests upstream of the instance itself, eg. at the load balancer or via a proxy). Even if you could identify malicious usage, it's still a matter of policy on how you want to respond to it - given that if it can reach your VM it will likely be affecting legitimate traffic too.
- jbkkd 12y agoWould it be possible in the future to set the thresholds based on time? Say I expect high traffic only during mornings, and not want to handle it during the evening if it occurs. Any chance this is in the making?
- rev_bird 12y ago> its probably a trade secret or something You can do auto-scaling without Amazon's secret sauce (or Google's), it just relies on triggers to spin up new boxes. I'm not very familiar with the Google setup, but you could use the Amazon API to pull metrics about boxes, then send a request to add another box to the auto-scaling group manually if you wanted. > How do you prevent oscillations in resource allocations when the traffic is fluctuating? Oscillation is kind of the point -- if you need x servers to serve y hits per hour, you'll have x servers when you need it. But when you're down to y/2 hits per hour, you don't need to be paying for x servers anymore. It just depends on load -- for example, you can set the group to add a box when average CPU rises above 70 percent, and then set it to remove a box when average CPU falls below 30 percent. > How do you know if e.g. an adversary is just DDoS'ing your application to make it more expensive to run your application? I guess there must be some kind of threshold that you can set? But then this would cut off the website for the real customers... This isn't a problem for auto-scaling, it's a problem for web stacks – the whole point of the "auto" part of it is that increase traffic means increased resources. Going counter to that gets complicated quickly. I'm sure there are folks who know more about DDoS mitigation than I do, but putting protection in front of your app with something like Akamai should help, as can keeping an eye out for bizarre spikes from certain IP ranges.
- ajessup 12y agoIt's worth noting that the GCE Autoscaler is configured quite differently to the AWS one. For the latter you define explicit scale conditions and triggers. On the former you define a utilisation metric and a target.
- cgio 12y agoResource scaling is not that tough a problem computationally. The tough aspect of it is the practice of defining the driver of scale needs. Past that it is just calculating based on triggers a continuous or discrete scale. I have done it for FTEs in a huge mining operation and it worked like a charm (of course the requirement for scaling was not as real-time as in this case.) Use cases I had to also cater for in that case were: - reallocation from pools before scaling out, which is not very complex either, but it adds a bit of complexity, especially if you optimise for cost along the way. - linked scaling requirements, e.g. a team grows but there is a bottom and top limit to team size and when you get a new team you also need a supervisor On oscillations, you can weather them out by the way you treat your driver sampling window (static or variable) and a cost function. Normally oscillation is not bad for the resource allocation scenario, it is the intention of it, but for FTEs the cost can be more difficult to manage than for computational resources.