5 ms·

How we processed €5M in donations in 2 hours using Cloudflare and Stripe

59 points by pvhee 6y ago

superkuh 6y ago
The write up seems to be: Don't, pay some giant corps to do it.
atlbeer 6y ago
https://news.ycombinator.com/item?id=25264506 https://news.ycombinator.com/item?id=25264506
darsoli 6y ago
This post seems to go into more detail on the technical architecture. I've read a lot of these style posts, and oftentimes the results don't end up being that interesting - but I have to say this post is different. Nice job and good overview of all the different layers in the stack. I didn't realize there was a meaningful difference between Cloudflare Workers and Lambda, but now will have to check it out.
- nicoburns 6y ago
  Indeed Cloudflare Workers have much lower latency, and much much quicker cold starts (to the point that you probably won't notice them). IMO this makes them much more useful for typical web APIs than lambda.
  - jariel 6y ago
    My understanding is that 'when hot' AWS Lambdas are pretty quick, and so long as there is basically any traffic, the remain hot? Do you have any details?
    - nicoburns 6y ago
      There are also cold starts every time the number of concurrent requests increases. Also: lambdas are only really useful for the low-to-no traffic situation. If you have constant traffic you're better off using a small VM.
      - jariel 6y ago
        "Also: lambdas are only really useful for the low-to-no traffic situation. If you have constant traffic you're better off using a small VM." I don't understand this. The entire point of Lambda's is to offload the overhead and complexity of managing severs - and - to allow for large occasional spikes in traffic. Even with the limitation of '5 second warmup per concurrent' Lambda - that's not a big deal. It means maybe 100-200 users have to wait a few seconds while 100-200 Lambdas warm up - and with 1.5 user spike ... you really don't know what you're going to get, but with Lambda's at least you're pretty much guaranteed it will work. For cost efficiency, with stable tech, stable/predictable traffic, and enough scale that you have the right team to be able to manage your EC2's properly - yes that makes sense. But you need some scale to get to that point wherein that cost efficiency is worth it given that Lambda's 'just work' fairly well fairly easily. But I definitely could be missing something.
        vp8989 6y ago
        I believe what is being implied is that because Cloudflare Workers operate inside of a much lighter construct, they are able to burst to higher concurrency inside of shorter time windows. Lambda can burst to 3000 and then an additional 500 more per minute after that. https://docs.aws.amazon.com/lambda/latest/dg/invocation-scaling.html https://docs.aws.amazon.com/lambda/latest/dg/invocation-scal... Based on the scale mentioned in the article (hundreds of RPS) it's likely Lambda would also have been able to handle it just fine. On another note, using non-provisioned infrastructure (aka "serverless") for an expected bursty load (TV campaign) is bordering on negligence. It sounds like lots of potential donations were missed because the Stripe account was not set up to cope with the load. It turned out to be a wash because Stripe donated 100k but if you change the business context of this system from "receiving donations" to "taking credit card payments" ... this outcome would not be considered acceptable.
  - ignoramous 6y ago
    Workers and cold-starts? If you count 0ms as coldstarts, then sure. Also, Workers supports HTTP3 (QUIC) and ESNI already. https://twitter.com/eastdakota/status/1288855462931177472 https://twitter.com/eastdakota/status/1288855462931177472
reacharavindh 6y ago
Any chance we could know the total infrastructure spend of this setup?
- pvhee 6y ago
  Total infrastructure cost is less than 100 USD, but this is an estimate. As this was directly deployed onto RTE's Cloudflare account, we don't have the breakdown for this. However if we count around 4 million requests at 0.5 USD / million requests, you can see that costs from the serverless infrastructure itself are negligible at a few dollars only. The major "infrastructure" cost stems from Sentry, which we used for application monitoring & crash reporting. We're on the team plan at 29USD / month with added on-demand spend of 100 USD, which we've nearly exhausted considering a large number of events were sent to Sentry. We had also enabled spike protection, which means we're paying less as certain events are rate limited in any given minute to avoid massive overspend (which we would have certainly run into).
jariel 6y ago
Small point: what really is the advantage in pushing those services 'to the edge'? AWS lambdas, when they are warm, are fast. Supposing there is a 100ms extra response time due to non-edge and AWS Lambda for whatever reason - is 100ms really the issue when you're facing a blast of 1.5M users? Issn't it the sheer onslaught of scale that is the concern? It'd seem to be Static hosting some content on S3 + Lambda+Stripe basically would handle the task no problem. Or am I missing something?
- faitswulff 6y ago
  Are lambdas limited by region? They noted in the post that they would be expecting the Irish diaspora from 130+ countries to be participating as well.
- pvhee 6y ago
  If you can push services to the edge without adding complexity to your setup, why wouldn't you? The simplicity (and pricing model) of running code on Cloudflare Workers should be reason enough to do this. I agree that AWS Lambda for this project would have been more than sufficient, together with S3 for static hosting. However, the advantage of CF workers over is that you combine both the CDN, static hosting, the API layer and the back-end logic running in one and the same worker, therefore significantly simplifying your setup. With AWS (and without using Lambda@Edge), we'd probably have to use a combination of Cloudfront, S3, API Gateway & Lambda. All of these could easily be defined in code with something like the Serverless framework or AWS SAM, but there are a lot more moving parts here when compared to CF Workers.
jjeaff 6y ago
The real feat is to not get your stripe account shutdown and funds held while receiving millions of donations in a single hour. Not a dig on Stripe in particular, but I always run into trouble with banks and parent processors before the servers start getting overloaded.

59 points by pvhee 6y ago

superkuh 6y ago

The write up seems to be: Don't, pay some giant corps to do it.

atlbeer 6y ago

https://news.ycombinator.com/item?id=25264506 https://news.ycombinator.com/item?id=25264506

darsoli 6y ago

This post seems to go into more detail on the technical architecture. I've read a lot of these style posts, and oftentimes the results don't end up being that interesting - but I have to say this post is different. Nice job and good overview of all the different layers in the stack. I didn't realize there was a meaningful difference between Cloudflare Workers and Lambda, but now will have to check it out.

nicoburns 6y ago
Indeed Cloudflare Workers have much lower latency, and much much quicker cold starts (to the point that you probably won't notice them). IMO this makes them much more useful for typical web APIs than lambda.
- jariel 6y ago
  My understanding is that 'when hot' AWS Lambdas are pretty quick, and so long as there is basically any traffic, the remain hot? Do you have any details?
  - nicoburns 6y ago
    There are also cold starts every time the number of concurrent requests increases. Also: lambdas are only really useful for the low-to-no traffic situation. If you have constant traffic you're better off using a small VM.
    - jariel 6y ago
      "Also: lambdas are only really useful for the low-to-no traffic situation. If you have constant traffic you're better off using a small VM." I don't understand this. The entire point of Lambda's is to offload the overhead and complexity of managing severs - and - to allow for large occasional spikes in traffic. Even with the limitation of '5 second warmup per concurrent' Lambda - that's not a big deal. It means maybe 100-200 users have to wait a few seconds while 100-200 Lambdas warm up - and with 1.5 user spike ... you really don't know what you're going to get, but with Lambda's at least you're pretty much guaranteed it will work. For cost efficiency, with stable tech, stable/predictable traffic, and enough scale that you have the right team to be able to manage your EC2's properly - yes that makes sense. But you need some scale to get to that point wherein that cost efficiency is worth it given that Lambda's 'just work' fairly well fairly easily. But I definitely could be missing something.
      - vp8989 6y ago
        I believe what is being implied is that because Cloudflare Workers operate inside of a much lighter construct, they are able to burst to higher concurrency inside of shorter time windows. Lambda can burst to 3000 and then an additional 500 more per minute after that. https://docs.aws.amazon.com/lambda/latest/dg/invocation-scaling.html https://docs.aws.amazon.com/lambda/latest/dg/invocation-scal... Based on the scale mentioned in the article (hundreds of RPS) it's likely Lambda would also have been able to handle it just fine. On another note, using non-provisioned infrastructure (aka "serverless") for an expected bursty load (TV campaign) is bordering on negligence. It sounds like lots of potential donations were missed because the Stripe account was not set up to cope with the load. It turned out to be a wash because Stripe donated 100k but if you change the business context of this system from "receiving donations" to "taking credit card payments" ... this outcome would not be considered acceptable.
- ignoramous 6y ago
  Workers and cold-starts? If you count 0ms as coldstarts, then sure. Also, Workers supports HTTP3 (QUIC) and ESNI already. https://twitter.com/eastdakota/status/1288855462931177472 https://twitter.com/eastdakota/status/1288855462931177472

reacharavindh 6y ago

Any chance we could know the total infrastructure spend of this setup?

pvhee 6y ago
Total infrastructure cost is less than 100 USD, but this is an estimate. As this was directly deployed onto RTE's Cloudflare account, we don't have the breakdown for this. However if we count around 4 million requests at 0.5 USD / million requests, you can see that costs from the serverless infrastructure itself are negligible at a few dollars only. The major "infrastructure" cost stems from Sentry, which we used for application monitoring & crash reporting. We're on the team plan at 29USD / month with added on-demand spend of 100 USD, which we've nearly exhausted considering a large number of events were sent to Sentry. We had also enabled spike protection, which means we're paying less as certain events are rate limited in any given minute to avoid massive overspend (which we would have certainly run into).

jariel 6y ago

Small point: what really is the advantage in pushing those services 'to the edge'? AWS lambdas, when they are warm, are fast. Supposing there is a 100ms extra response time due to non-edge and AWS Lambda for whatever reason - is 100ms really the issue when you're facing a blast of 1.5M users? Issn't it the sheer onslaught of scale that is the concern? It'd seem to be Static hosting some content on S3 + Lambda+Stripe basically would handle the task no problem. Or am I missing something?

faitswulff 6y ago
Are lambdas limited by region? They noted in the post that they would be expecting the Irish diaspora from 130+ countries to be participating as well.
pvhee 6y ago
If you can push services to the edge without adding complexity to your setup, why wouldn't you? The simplicity (and pricing model) of running code on Cloudflare Workers should be reason enough to do this. I agree that AWS Lambda for this project would have been more than sufficient, together with S3 for static hosting. However, the advantage of CF workers over is that you combine both the CDN, static hosting, the API layer and the back-end logic running in one and the same worker, therefore significantly simplifying your setup. With AWS (and without using Lambda@Edge), we'd probably have to use a combination of Cloudfront, S3, API Gateway & Lambda. All of these could easily be defined in code with something like the Serverless framework or AWS SAM, but there are a lot more moving parts here when compared to CF Workers.

jjeaff 6y ago

The real feat is to not get your stripe account shutdown and funds held while receiving millions of donations in a single hour. Not a dig on Stripe in particular, but I always run into trouble with banks and parent processors before the servers start getting overloaded.