3 ms·

I've been bitten by Parameter Store's rate limiting (non-increasable) from just launching a new environment of a regular application (we were storing a lot of v

by Cpoll 8y ago

I've been bitten by Parameter Store's rate limiting (non-increasable) from just launching a new environment of a regular application (we were storing a lot of values, and there's no real bulk GET as far as I recall). There are countless threads about this problem online. I imagine that with a serveless workflow, this might be an even bigger problem. I haven't explored Secrets Manager yet though.

dmlittle 8y ago
Fetching parameters by path is an easy way to get bulk parameters and avoiding the rate limits.
timclark 8y ago
It has caused us failures as lambdas have scaled up as it is quite easy to hit a rate limit.
- sudosteph 8y ago
  For me, I was able to reduce the calls from lambda pretty significantly by doing the parameter pull step outside of the handler, so it is available as part of the execution context between runs (similar to how many people do DB connection with lambda). I still do the decrypt step within the handler though because I don't like the idea of having decrypted values lying around, but the limit for KMS decrypt operations is independent of SSM, and is a 10k/s shared limit in the regions I use, so I haven't had any issues yet.
scarface74 8y ago
I’ve been bitten by it doing something as simple as creating parameters using CloudFormation. The only way around it is judicious use of DependsOn to make them single threaded.
- Cpoll 8y ago
  Thanks for this, I was considering bringing params into CF, the main blocker being that it doesn't support secure params (which is obvious in retrospect, they wouldn't be secure in the template).
  - scarface74 8y ago
    For secure strings in cloud formation, I use a cloud formation parameter ( - meaning I have to enter the parameter manually and set NoEcho to true. And then set the CF parameter value to the SSM Parameter value. This is with the help of a lambda backed custom resource https://svdgraaf.nl/2018/04/13/CloudFormation-ssm-secure-string-support-boto3-custom-resource.html https://svdgraaf.nl/2018/04/13/CloudFormation-ssm-secure-str... Obviously , this can’t be a part of your CI/CD Pipeline. I run the CloudFormation template manually in the console to enter the parameters and my CI Pipeline can then update the stack when needed.

by Cpoll 8y ago

dmlittle 8y ago

Fetching parameters by path is an easy way to get bulk parameters and avoiding the rate limits.

timclark 8y ago

It has caused us failures as lambdas have scaled up as it is quite easy to hit a rate limit.

sudosteph 8y ago
For me, I was able to reduce the calls from lambda pretty significantly by doing the parameter pull step outside of the handler, so it is available as part of the execution context between runs (similar to how many people do DB connection with lambda). I still do the decrypt step within the handler though because I don't like the idea of having decrypted values lying around, but the limit for KMS decrypt operations is independent of SSM, and is a 10k/s shared limit in the regions I use, so I haven't had any issues yet.

scarface74 8y ago

I’ve been bitten by it doing something as simple as creating parameters using CloudFormation. The only way around it is judicious use of DependsOn to make them single threaded.

Cpoll 8y ago
Thanks for this, I was considering bringing params into CF, the main blocker being that it doesn't support secure params (which is obvious in retrospect, they wouldn't be secure in the template).
- scarface74 8y ago
  For secure strings in cloud formation, I use a cloud formation parameter ( - meaning I have to enter the parameter manually and set NoEcho to true. And then set the CF parameter value to the SSM Parameter value. This is with the help of a lambda backed custom resource https://svdgraaf.nl/2018/04/13/CloudFormation-ssm-secure-string-support-boto3-custom-resource.html https://svdgraaf.nl/2018/04/13/CloudFormation-ssm-secure-str... Obviously , this can’t be a part of your CI/CD Pipeline. I run the CloudFormation template manually in the console to enter the parameters and my CI Pipeline can then update the stack when needed.