3 ms·
I wrote a framework that does regexes of the Common Crawl. The Common Crawl is a series of captures of billions of pages of the WWW. Each member in the series h
by cldellow 5y ago
I wrote a framework that does regexes of the Common Crawl. The Common Crawl is a series of captures of billions of pages of the WWW. Each member in the series has ~60 TB captures.
My framework runs a fairly simple pipeline: for each 1GB file, stream it from S3, decompress it, search it with a regex library, stream results back to S3.
A1 spot instances have consistently been cheaper for this task when measured on a $/GB scanned basis. Part of this might be that the framework doesn't need lots of RAM, so paying for more than 2GB per core on the non-A1 instances is just wasted expense for me.
- orf 5y agoWouldn’t a lambda be far cheaper?
- cldellow 5y agoIn my experience, no. Quite the opposite. Lambdas can be cost effective for bursty loads, but if you can push sustained CPU utilization, as this example does, EC2 is cheaper. For example, an ARM-based Lambda with 2 GB of RAM running for 1 hour is $0.09. An a1.medium is $0.025/hr on-demand or $0.0084/hr spot. Part of this is that Lambda scales your CPU performance with your memory usage. So even if your code can run in 128 MB of RAM, there's no cost benefit to you if it's CPU-constrained, because it will run 16x slower than if you allocated it 2048 MB of RAM.