4 ms·
Hey as a (way smaller) competitor of yours, I honestly feel you. I understand you may not want my advice. But anyways, here it goes :) I have dealt with simil
by amzans 6y ago
Hey as a (way smaller) competitor of yours, I honestly feel you.
I understand you may not want my advice. But anyways, here it goes :)
I have dealt with similar issues too at my full-time job. We ingest data in the billions range too, and our costs are a lot lower than what you describe.
* On the public endpoint, can you move the first point of contact closer to the edge? You want to block requests before they even reach your ingestion pipeline. If not, add a few geographical Load Balancers with the sole job of accepting requests before forwarding them to your Lambda. Divert traffic at the DNS level using a geo routing policy on Route53.
* Are you also hitting DynamoDB for every request as part of your spam system? I found it quite expensive on-demand, unless you provision capacity. We used to pay tens of thousands per month for DynamoDB in the past, when a single Redis instance would have done it. Specially if it's temporary, non-critical data such as spam detection.
* Do you batch the incoming events before triggering SQS?
* How are your networking costs going? They tend to creep up in the AWS bills too.
It just sounds to me you clearly need something in-between the public endpoint and those Lambdas. But maybe I'm missing some info. Otherwise you're going to keep paying for capacity that a "traditional" server could handle, without costing you more for each request.
If you can't fight back, and don't want real requests to be lost, put a server at the battle-front, and buffer all raw HTTP requests into a queue made for this (like Kinesis). You can then process events and recover the data at your own pace. I did this in the past, and could handle 60k+ req/s with 2 instances on AWS. I used a simple Go tool to capture the traffic https://github.com/buger/goreplay https://github.com/buger/goreplay
Also, at my full-time job we use Kinesis to buffer all analytics events, it's cheaper than SQS and handles billions of data points per month. This also keeps the ingest rate constant, but you seem to already do that with your workers.
Full disclosure: I'm Anthony from http://panelbear.com/ http://panelbear.com/ , and I just want to offer honest help.
- amzans 6y agoAdditionally, be sure to check how your script behaves with SPAs and the browser history API. You seem to be binding to the browser history as-is, and might be firing lots of non-page view events. It's not just a theoretical case, I saw this for many customers with Panelbear. Many JS frameworks use the history API in ways you may not be considering. That's why I would recommend de-duplicating events on the client-side before sending them to your ingestion pipeline. But that is DDoS matters aside.