4 ms·

Serverless Benchmark – An independent benchmark of serverless providers

40 points by gitgud 7y ago

aloknnikhil 7y ago
AWS' p99 is very impressive for hot functions. Pleasantly surprised by IBM's performance here. Google seems to perform the worst of the lot. It's also odd that Google's median cold start is faster than a hot function. Maybe the concurrency is triggering some sort of back-off? PS: Adding cost per request as a comparison would be nice.
sdan 7y ago
Interestingly I’ve gotten much better results with workers than I have with lambda
- tyingq 7y ago
  It doesn't say where the metrics are collected from, and whether they include the network latency from whatever point that is.
  - sdan 7y ago
    I did some testing a while back from servers globally (I used Postman w/ global monitoring) to test latency. Pretty sure Postman uses some big cloud service (AWS or GCP) so you'd think Lambda would've had an advantage. Interestingly, that didn't seem to be the case (workers had 3-4x less latency than Lambda).
    - tyingq 7y ago
      That is interesting. I was trying to roughly say that Cloudflare stats might be artificially high/slow if measured from the center vs the "edge". Sounds like perhaps not?
      - sdan 7y ago
        Yeah I kinda understood what you were saying, but as I also surprisingly found out, it seems like they're fast in general regardless of where you're located. I'll try to do more testing soon.
        tyingq 7y ago
        I made a url shortener with Cloudflare workers and their edge KV store. I did notice that it worked very fast in the US, but not so fast in Europe. The KV store seems to require a US->EU trip if it's not a "hot key". Or basically, it seems "single master" with the center in the US. No such issue with the worker itself though...just the KV store. If I hardcoded the data, or used the regular page cache, it was fast everywhere.
        steveklabnik 7y ago
        Hi, PM of KV here. You should be seeing speeds improve in Europe within the next few weeks. Your intuition about the single master is currently correct, we're adding a second master in Europe soon, for exactly this reason.
Normal_gaussian 7y ago
When graphs are shown next to each other like this it is important to use the same scale across them. The Percentile graphs make IBM cloud functions look significantly better than the competition, despite being middle of the pack or worse. Aside from that nit (which I see everywhere), this is awesome.

40 points by gitgud 7y ago

AWS' p99 is very impressive for hot functions. Pleasantly surprised by IBM's performance here. Google seems to perform the worst of the lot. It's also odd that Google's median cold start is faster than a hot function. Maybe the concurrency is triggering some sort of back-off? PS: Adding cost per request as a comparison would be nice.

sdan 7y ago

Interestingly I’ve gotten much better results with workers than I have with lambda

tyingq 7y ago
It doesn't say where the metrics are collected from, and whether they include the network latency from whatever point that is.
- sdan 7y ago
  I did some testing a while back from servers globally (I used Postman w/ global monitoring) to test latency. Pretty sure Postman uses some big cloud service (AWS or GCP) so you'd think Lambda would've had an advantage. Interestingly, that didn't seem to be the case (workers had 3-4x less latency than Lambda).
  - tyingq 7y ago
    That is interesting. I was trying to roughly say that Cloudflare stats might be artificially high/slow if measured from the center vs the "edge". Sounds like perhaps not?
    - sdan 7y ago
      Yeah I kinda understood what you were saying, but as I also surprisingly found out, it seems like they're fast in general regardless of where you're located. I'll try to do more testing soon.
      - tyingq 7y ago
        I made a url shortener with Cloudflare workers and their edge KV store. I did notice that it worked very fast in the US, but not so fast in Europe. The KV store seems to require a US->EU trip if it's not a "hot key". Or basically, it seems "single master" with the center in the US. No such issue with the worker itself though...just the KV store. If I hardcoded the data, or used the regular page cache, it was fast everywhere.
        steveklabnik 7y ago
        Hi, PM of KV here. You should be seeing speeds improve in Europe within the next few weeks. Your intuition about the single master is currently correct, we're adding a second master in Europe soon, for exactly this reason.

Normal_gaussian 7y ago

When graphs are shown next to each other like this it is important to use the same scale across them. The Percentile graphs make IBM cloud functions look significantly better than the competition, despite being middle of the pack or worse. Aside from that nit (which I see everywhere), this is awesome.