4 ms·
Would this crazy-ass strategy work? 1. Spin up 50 AWS Lightsail instances[0] for parallelism 2. In each instance download from S3 and upload to B2. S3 to any
by desdiv 7y ago
Would this crazy-ass strategy work?
1. Spin up 50 AWS Lightsail instances[0] for parallelism
2. In each instance download from S3 and upload to B2.
S3 to any AWS service in the same region is free[1]. $5 Lightsail instances come with 2TB of data transfers each, so 50 of them can easily handle 23TB. The whole transfer can be done within a few hours so the total computing cost is less than $10 ($5 / 30 * 50 = $8.3). Total data retrieval cost for S3 is ($0.0007 per GB) * 23,000GB = $16.1.
[0] https://aws.amazon.com/lightsail https://aws.amazon.com/lightsail
[1] "Transfers between S3 buckets or from Amazon S3 to any service(s) within the same AWS Region are free." according to https://aws.amazon.com/s3/pricing/ https://aws.amazon.com/s3/pricing/
- manigandham 7y agoLightsail instances have terrible bandwidth throughput. It has high limits because it's all overbooked low-priority traffic shunted off their network as soon as possible.
- tgtweak 7y agoThis is absolutely sneaky and would work. I believe S3 also has file access cost but it's minimal. If you had a partitioned list of what you needed to move per node, so that coordination was minimal between them, this would be pretty viable.
- coolgeek 7y agoLightSail has (or at least had) a hard limit of 20 instances. They also have a soft limit of 2 instances, after which you must request an upgrade to a higher limit. I had to submit a support request explaining my intended usage. It took a week to get approved. The stated reason for these limits is to avoid unexpectedly large bills. But I suspect that it's also to prevent crazy-ass strategies for getting around bandwidth costs.