3 ms·
The latter. We have a lot of metrics and data, but I need something consistent beyond "this box looks like its using a lot of CPU/memory/IO." Our workloads ar
by chucky_z 6y ago
The latter. We have a lot of metrics and data, but I need something consistent beyond "this box looks like its using a lot of CPU/memory/IO."
Our workloads are dynamic enough that a box using 20% or 80% are both within range of normal, with some single threaded workloads so per-core stats are also not great.
I'm also not looking to fix anything, just have the ability to say, with a high degree of confidence, "hey, our system detected this box looks funky, can you just pull it out and replace it?"
- verdverm 6y agoThat's the hard part of the problem, extracting signals from the noise. Have a look at the Google Autopilot paper (cloud scheduling, not self driving). There are a few interesting ideas that could be adapted. They generally have some good ideas if you peruse their research page, specifically the data center and related topics.
- chucky_z 6y agoAwesome! For anyone who finds this through search, here's that whitepaper: https://dl.acm.org/doi/pdf/10.1145/3342195.3387524 https://dl.acm.org/doi/pdf/10.1145/3342195.3387524 I don't think it helps at a server level, but it does provide me some ideas, especially looking at some of the algorithms they're using. My problems are mostly CPU related, whereas this paper seems to focus on Memory. I still can definitely get something useful from here. :)
- verdverm 6y agoYes, most cloud applications / systems struggle with memory and optimizations around the system behaviors / configuration. It is rare to find something CPU bound, and when you do, it's usually isolated so it doesn't step on other programs. I thought the algos and strategy would be most useful to you. Glad you found it interesting!