4 ms·

That would have to be the worst job in the world - keeping Lustre going as an Amazon service, with management that utterly lacks understanding and sympathy.

davidmr 8y ago
They’d have to pay me a lot of money to do it, that’s for sure. I’d love to see the disaster recovery plans. Every major Lustre site I’m aware of has had a data loss “incident” at some point in their history. It’s possible AWS has it all figured out with background backups and block device replication and whatnot, but I’m skeptical.
- mbreese 8y ago
  Given that they call the non-S3 linked version 'ephemeral', I'm not sure there is a plan. I think S3 is the plan.
  - pinewurst 8y ago
    'Ephemeral' was/is the original Lustre design model. It was intended for high performance swap/scratch at Livermore with a short data lifespan - your higher priority bomb sim forces mine to roll out to disk and back in later, and that's it. Lustre, even today, isn't long term stable. The longer you leave data on it, the greater the probability of corruption - even silently.
    - mbreese 8y ago
      I've seen Lustre backed with ZFS listed a few places. Is the idea here to help mitigate the possibility of corruption?
      - agapon 8y ago
        LLNL is the core force behind ZoL and it's primarily them who use ZFS-backed Lustre.
        pinewurst 8y ago
        I think ZoL is LLNL’s attempt to make up for inflicting Lustre on us.
- pinewurst 8y ago
  I have doubts based on my experience with Lustre and a certain understanding of AWS operations. I'm guessing they're going for the Iranian minefield clearing technique - get a mob of kids, hand them plastic keys to heaven (or RSUs), and march them through the field.
notacoward 8y ago
Same thought here. I spent two-plus years debugging Lustre issues for a very small set of customers. It was an absolute beast. Build process was a compatibility-killing license-violating nightmare. Fell over at the slightest provocation, with little info to help figure out why. Provided no metrics to speak of, and the thicket of inter-related settings (especially timeouts) made effective tuning almost impossible. I'd guess that Amazon spent many engineer-years removing or rewriting significant pieces, and even more establishing the safe configuration envelope for what remained. Even then, it's probably a nightmare for the SREs (or whatever Amazon calls them) who have to keep it running.

davidmr 8y ago

They’d have to pay me a lot of money to do it, that’s for sure. I’d love to see the disaster recovery plans. Every major Lustre site I’m aware of has had a data loss “incident” at some point in their history. It’s possible AWS has it all figured out with background backups and block device replication and whatnot, but I’m skeptical.

mbreese 8y ago
Given that they call the non-S3 linked version 'ephemeral', I'm not sure there is a plan. I think S3 is the plan.
- pinewurst 8y ago
  'Ephemeral' was/is the original Lustre design model. It was intended for high performance swap/scratch at Livermore with a short data lifespan - your higher priority bomb sim forces mine to roll out to disk and back in later, and that's it. Lustre, even today, isn't long term stable. The longer you leave data on it, the greater the probability of corruption - even silently.
  - mbreese 8y ago
    I've seen Lustre backed with ZFS listed a few places. Is the idea here to help mitigate the possibility of corruption?
    - agapon 8y ago
      LLNL is the core force behind ZoL and it's primarily them who use ZFS-backed Lustre.
      - pinewurst 8y ago
        I think ZoL is LLNL’s attempt to make up for inflicting Lustre on us.
pinewurst 8y ago
I have doubts based on my experience with Lustre and a certain understanding of AWS operations. I'm guessing they're going for the Iranian minefield clearing technique - get a mob of kids, hand them plastic keys to heaven (or RSUs), and march them through the field.

notacoward 8y ago

Same thought here. I spent two-plus years debugging Lustre issues for a very small set of customers. It was an absolute beast. Build process was a compatibility-killing license-violating nightmare. Fell over at the slightest provocation, with little info to help figure out why. Provided no metrics to speak of, and the thicket of inter-related settings (especially timeouts) made effective tuning almost impossible. I'd guess that Amazon spent many engineer-years removing or rewriting significant pieces, and even more establishing the safe configuration envelope for what remained. Even then, it's probably a nightmare for the SREs (or whatever Amazon calls them) who have to keep it running.