5 ms·

The “duration of invocation” payment model (discussed in #4) is ultimately what turned me off of lambda for anything that involves an external API. You could ha

by paulgb 5y ago

The “duration of invocation” payment model (discussed in #4) is ultimately what turned me off of lambda for anything that involves an external API. You could have a lambda function running smoothly, and then one day get a surprisingly high bill because a downstream service that used to respond in 100ms encountered issues and started taking 3000ms to respond so you’re suddenly paying ~30x per invocation through no fault of your own.

eecc 5y ago
Wouldn’t you implement that via an outbound SQS? (If that’s even possible)
fredley 5y ago
I wonder how long it will be before async lambdas are a thing, as in you can yield while awaiting a response - avoiding this issue.
- red2awn 5y ago
  I think what you are describing is already here, in the form of Cloudflare Workers. See https://developers.cloudflare.com/workers/platform/limits#bundled-usage-model https://developers.cloudflare.com/workers/platform/limits#bu...
  - paulgb 5y ago
    That's pretty cool. If I were building the same service today, I'd probably use Workers.
dragonfax 5y ago
Any external API that could reasonably take a long time to return (because of the work its doing) should have an asynchronous API. I.e. you submit the request to start the work and can later do another request to check if its complete and get any results. Rather than waiting on a live request. Good practice for using external APIs is to NOT use any default http client settings and always provide your own timeouts to what you consider reasonable for connections and responses, as well as using a context system with deadlines so you can time out any requests that are taking more than a reasonable time to complete. Making your described surprise long expensive requests into nothing more than short errors (which hopefully you'll pick up on after a while, as long as you've got your alerting system setup right).
- paulgb 5y ago
  > Any external API that could reasonably take a long time to return should have an asynchronous API. I agree, but that doesn’t solve the problem here. The remote API was asynchronous, I was just waiting for it to ack my instruction. Because of issues out of my control (network congestion, maybe?) the time to get a 200 OK from the server shot up. > always provide your own timeouts to what you consider reasonable for connections and responses Agreed here as well, and I was providing my own timeout. The problem is that (cost-wise) it’s fine for 1% of requests to hit a 5-second timeout, but gets expensive when 100% of requests do. And lowering the timeout means that during normal times, requests that have latency in the tail of the distribution but ultimately go through would fail, which is undesirable.
  - riquito 5y ago
    I don't think that's what they meant by asynchronous api. More likely something along the lines of "schedule a job - 1 fast request" "call me back at my endpoint when the job is finished" (or variations, the gist is you are not stuck waiting for the server response)
    - paulgb 5y ago
      Right, but even in schedule a job - 1 fast request, there's still some waiting for the server to ack that it got the message. That's the request that became slow. I don't see how there's any getting around that, async API or not.
    - colinmhayes 5y ago
      How can you schedule a job within 100ms if it takes 3000ms to establish a connection with the server running the other api due to network congestion? Maybe you can send the request in UDP, but you'll get tons of situations where the secondary api never ran.
vp8989 5y ago
Are you actually likely to have a "high bill" on Lambda but also be "surprised" that all your requests are 30x slower? Presumably anything that can generate a "high bill" also has some reasonable level of on-call, alerting, SLAs etc ... such that if your Lambda invocations were all taking 30x longer people would notice.
- paulgb 5y ago
  I really did get a “surprisingly high bill” because of it. It wasn't high in the absolute sense, but it was something like 10x my usual monthly cost in a single day, and would have been higher if I didn't get an alert. The reason I went for serverless over a server in the first place was so that I wouldn't have to wake up in the middle of the night because of server issues, so alerting is not a satisfying solution. Especially where the problem is more an artifact of how the service is billed than how much it consumes actual server resources.
  - qvrjuec 5y ago
    You're making a tradeoff. You no longer have to alarm for things like your server failing due to disk failure or other stuff that lambda abstracts away from you, but you do have to alarm on things that will increase cost. Constant cost(if you aren't dynamically scaling) and more headache vs variable cost and less headache
    - paulgb 5y ago
      It would be a trade-off I'd consider if alternatives didn't exist, but they do. Someone down-thread mentioned Cloudflare Workers, which have a sensible pricing model. These days I use Google Cloud Run, which gives me everything I wanted from Lambda but has model where you pay by CPU time rather than invocation time, and allows a CPU to handle multiple requests.
andrew_ 5y ago
why aren't you using a mechanism to timeout downstream dependencies?
- paulgb 5y ago
  I am, see here for why it's not enough: https://news.ycombinator.com/item?id=29118774 https://news.ycombinator.com/item?id=29118774

by paulgb 5y ago

eecc 5y ago

Wouldn’t you implement that via an outbound SQS? (If that’s even possible)

fredley 5y ago

I wonder how long it will be before async lambdas are a thing, as in you can yield while awaiting a response - avoiding this issue.

red2awn 5y ago
I think what you are describing is already here, in the form of Cloudflare Workers. See https://developers.cloudflare.com/workers/platform/limits#bundled-usage-model https://developers.cloudflare.com/workers/platform/limits#bu...
- paulgb 5y ago
  That's pretty cool. If I were building the same service today, I'd probably use Workers.

dragonfax 5y ago

Any external API that could reasonably take a long time to return (because of the work its doing) should have an asynchronous API. I.e. you submit the request to start the work and can later do another request to check if its complete and get any results. Rather than waiting on a live request. Good practice for using external APIs is to NOT use any default http client settings and always provide your own timeouts to what you consider reasonable for connections and responses, as well as using a context system with deadlines so you can time out any requests that are taking more than a reasonable time to complete. Making your described surprise long expensive requests into nothing more than short errors (which hopefully you'll pick up on after a while, as long as you've got your alerting system setup right).

paulgb 5y ago
> Any external API that could reasonably take a long time to return should have an asynchronous API. I agree, but that doesn’t solve the problem here. The remote API was asynchronous, I was just waiting for it to ack my instruction. Because of issues out of my control (network congestion, maybe?) the time to get a 200 OK from the server shot up. > always provide your own timeouts to what you consider reasonable for connections and responses Agreed here as well, and I was providing my own timeout. The problem is that (cost-wise) it’s fine for 1% of requests to hit a 5-second timeout, but gets expensive when 100% of requests do. And lowering the timeout means that during normal times, requests that have latency in the tail of the distribution but ultimately go through would fail, which is undesirable.
- riquito 5y ago
  I don't think that's what they meant by asynchronous api. More likely something along the lines of "schedule a job - 1 fast request" "call me back at my endpoint when the job is finished" (or variations, the gist is you are not stuck waiting for the server response)
  - paulgb 5y ago
    Right, but even in schedule a job - 1 fast request, there's still some waiting for the server to ack that it got the message. That's the request that became slow. I don't see how there's any getting around that, async API or not.
  - colinmhayes 5y ago
    How can you schedule a job within 100ms if it takes 3000ms to establish a connection with the server running the other api due to network congestion? Maybe you can send the request in UDP, but you'll get tons of situations where the secondary api never ran.

vp8989 5y ago

Are you actually likely to have a "high bill" on Lambda but also be "surprised" that all your requests are 30x slower? Presumably anything that can generate a "high bill" also has some reasonable level of on-call, alerting, SLAs etc ... such that if your Lambda invocations were all taking 30x longer people would notice.

paulgb 5y ago
I really did get a “surprisingly high bill” because of it. It wasn't high in the absolute sense, but it was something like 10x my usual monthly cost in a single day, and would have been higher if I didn't get an alert. The reason I went for serverless over a server in the first place was so that I wouldn't have to wake up in the middle of the night because of server issues, so alerting is not a satisfying solution. Especially where the problem is more an artifact of how the service is billed than how much it consumes actual server resources.
- qvrjuec 5y ago
  You're making a tradeoff. You no longer have to alarm for things like your server failing due to disk failure or other stuff that lambda abstracts away from you, but you do have to alarm on things that will increase cost. Constant cost(if you aren't dynamically scaling) and more headache vs variable cost and less headache
  - paulgb 5y ago
    It would be a trade-off I'd consider if alternatives didn't exist, but they do. Someone down-thread mentioned Cloudflare Workers, which have a sensible pricing model. These days I use Google Cloud Run, which gives me everything I wanted from Lambda but has model where you pay by CPU time rather than invocation time, and allows a CPU to handle multiple requests.

andrew_ 5y ago

why aren't you using a mechanism to timeout downstream dependencies?

paulgb 5y ago
I am, see here for why it's not enough: https://news.ycombinator.com/item?id=29118774 https://news.ycombinator.com/item?id=29118774