4 ms·

Its much more complex than described. The author is making a brilliant argument for getting a secondhand workstation and shoving under their desk. If you are

Its much more complex than described. The author is making a brilliant argument for getting a secondhand workstation and shoving under their desk. If you are doing multi machine batch style processing, then you won't be using ondemand, you'd use the spot pricing. The missing argument in that part is storage costs. Managing a high speed, highly available synchronous file system that can do a sustained 50gb/sec is hard bloody work (no S3 isnt a good fit, too much management overhead) Don't get me wrong AWS _is_ expensive if you are using a machine for more than a month or two. however if you are doing highly parallel stuff, Batch and lustre on demand is pretty ace. If you are doing a multi-year project, then real steel is where its at. Assuming you have factored in hosting, storage and admin costs.

lostmsu 4y ago
Isn't 50GB sec like 5 NVMe Gen 5 SSDs + 1 or 2 for redundancy? Actually, you are right. Consumer SSDs I've seen only do about 1.5GB/s sustained.
- pclmulqdq 4y ago
  Even (high-end) consumer SSDs can saturate a PCIe gen 4 x4 link if you are doing sequential reads. Non-sequential hurts on even enterprise SSDs.
- davidmr 4y ago
  Not in the context the person you responded to meant it. Yes, you can very easily get 50GB/s from a few NVMe devices on a single box. Getting 50GB/s on a POSIX-ish filesystem exported to 1000 servers is very possible and common, but orders of magnitude more complicated. 500GB/s is tougher still. 5TB/s is real tough, but real fun.
bushbaba 4y ago
Checkout Apache Iceberg which makes it fairly trivial to get high throughput from S3 without much fine-tuning. Bursts from 0 to 50Gbps should be possible from S3 without much effort, just have object sizes that are in the NN+ MiB range. Personally, Lustre is a mess, it's expensive and even more pain to fine-tune.
- KaiserPro 4y ago
  > Lustre is a mess, it's expensive and even more pain to fine-tune. Its a huge raid-0, so long as your entire team understands that, you'll be ok. Its a lot better than in 2008, but now that AWS have a managed service, I'd just use that. (my heart is always in GPFS land...)
- awiesenhofer 4y ago
  From https://iceberg.apache.org https://iceberg.apache.org > Iceberg is a high-performance format for huge analytic tables. How would that help speedup S3? Genuine question?
  - bushbaba 4y ago
    Iceberg doesn’t use the directory structure to represent what objects belong to which table. The change allows objects to have significant entropy in the object keys prefix. This avoids hot spotting s3 and allows for greater bursts to s3 QPs.
    - awiesenhofer 4y ago
      Oh wow, thats neat! Thanks!
gautamdivgi 4y ago
Even for multi-year, if you factor in everything does it still come out cheaper and AWS? Would you be running everything 24x7 on an HPC? I don’t think so. You need scale at some points and there are probably times where research is done on your desktop. You could invest in an HPC - but I think the human cost of maintaining one especially if you’re in a high cost of living area (e.g. Bay Area, NYC, etc.) is going to be pretty high. Admin cost, UPS, cable wiring, heat/cooling etc. can all be pretty expensive. Maintenance of these can be pretty pricey too. Are there any companies that remotely manage data centers and rent out bare metal infra?

lostmsu 4y ago

Isn't 50GB sec like 5 NVMe Gen 5 SSDs + 1 or 2 for redundancy? Actually, you are right. Consumer SSDs I've seen only do about 1.5GB/s sustained.

pclmulqdq 4y ago
Even (high-end) consumer SSDs can saturate a PCIe gen 4 x4 link if you are doing sequential reads. Non-sequential hurts on even enterprise SSDs.
davidmr 4y ago
Not in the context the person you responded to meant it. Yes, you can very easily get 50GB/s from a few NVMe devices on a single box. Getting 50GB/s on a POSIX-ish filesystem exported to 1000 servers is very possible and common, but orders of magnitude more complicated. 500GB/s is tougher still. 5TB/s is real tough, but real fun.

bushbaba 4y ago

Checkout Apache Iceberg which makes it fairly trivial to get high throughput from S3 without much fine-tuning. Bursts from 0 to 50Gbps should be possible from S3 without much effort, just have object sizes that are in the NN+ MiB range. Personally, Lustre is a mess, it's expensive and even more pain to fine-tune.

KaiserPro 4y ago
> Lustre is a mess, it's expensive and even more pain to fine-tune. Its a huge raid-0, so long as your entire team understands that, you'll be ok. Its a lot better than in 2008, but now that AWS have a managed service, I'd just use that. (my heart is always in GPFS land...)
awiesenhofer 4y ago
From https://iceberg.apache.org https://iceberg.apache.org > Iceberg is a high-performance format for huge analytic tables. How would that help speedup S3? Genuine question?
- bushbaba 4y ago
  Iceberg doesn’t use the directory structure to represent what objects belong to which table. The change allows objects to have significant entropy in the object keys prefix. This avoids hot spotting s3 and allows for greater bursts to s3 QPs.
  - awiesenhofer 4y ago
    Oh wow, thats neat! Thanks!

gautamdivgi 4y ago

Even for multi-year, if you factor in everything does it still come out cheaper and AWS? Would you be running everything 24x7 on an HPC? I don’t think so. You need scale at some points and there are probably times where research is done on your desktop. You could invest in an HPC - but I think the human cost of maintaining one especially if you’re in a high cost of living area (e.g. Bay Area, NYC, etc.) is going to be pretty high. Admin cost, UPS, cable wiring, heat/cooling etc. can all be pretty expensive. Maintenance of these can be pretty pricey too. Are there any companies that remotely manage data centers and rent out bare metal infra?