4 ms·
Checkout Apache Iceberg which makes it fairly trivial to get high throughput from S3 without much fine-tuning. Bursts from 0 to 50Gbps should be possible from S
by bushbaba 4y ago
Checkout Apache Iceberg which makes it fairly trivial to get high throughput from S3 without much fine-tuning. Bursts from 0 to 50Gbps should be possible from S3 without much effort, just have object sizes that are in the NN+ MiB range. Personally, Lustre is a mess, it's expensive and even more pain to fine-tune.
- KaiserPro 4y ago> Lustre is a mess, it's expensive and even more pain to fine-tune. Its a huge raid-0, so long as your entire team understands that, you'll be ok. Its a lot better than in 2008, but now that AWS have a managed service, I'd just use that. (my heart is always in GPFS land...)
- awiesenhofer 4y agoFrom https://iceberg.apache.org https://iceberg.apache.org > Iceberg is a high-performance format for huge analytic tables. How would that help speedup S3? Genuine question?
- bushbaba 4y agoIceberg doesn’t use the directory structure to represent what objects belong to which table. The change allows objects to have significant entropy in the object keys prefix. This avoids hot spotting s3 and allows for greater bursts to s3 QPs.
- awiesenhofer 4y agoOh wow, thats neat! Thanks!