3 ms·

This sounds kind of interesting. What happened?

• Spooky23 8y ago
  The constraints moved! We had a write intensive workload for a user community that would generate short (~5s) periods of high traffic (6-9Gb) against a file server — this would impact everyone because sessions would be interrupted. It was devilishly difficult to troubleshoot because it was lost in the statistics, aggregated in 30-100s chunks. Fixing that revealed another constraint where we hit a constraint with the file server process itself. It’s one of these things that was interesting because the storage with fancy SSD can handle the workload no sweat. In the old days, monitoring would catch the io constraint first.
  • exikyut 8y ago
    Hmm. This highlights the importance of statistics AND insane threshold edge detection. Good one to file away, thanks.
    • Spooky23 8y ago
      Yeah, we’re working on a way to grab real-time stats during peak load conditions to help correct for outliers lost in averages. Another key thing is to be aware where heavy write load generators are on SMB, and scope the SMB shares around them to scope failure domains. There are limits, and the vendors (Microsoft or 3rd party NAS) have a hard time finding them too.