24 ms·
What were the stories?
by trendy0 4y ago
What were the stories?
- karlney 4y agoOne time, a few years ago a particularly nasty query was executed over and over again and it took a few hours to find it and then block it. And during that time so many nodes had became slow and unresponsive that another (for us) previously unseen memory leak started to occur. Nodes kept building up queues of unanswered ping requests on them. And the requests contained our 100Mb large cluster state, so the heaps filled up and evenmore nodes became unresponsive. And from then on the whole thing turned into a death spiral of doom. After trying, and failing to get it under control for 48 hours we gave up and rebuilt the whole cluster from scratch, using the snapshots we store on S3. The recovery took another 90 hours or so. That was not a fun week.
- thatwasunusual 4y agoNon-technically, it was a horrow-show. I worked at the company from 2005/06 to 2012, when I quit after witnessing shameful behaviour towards women, a party-culture that literally lead to rape allegations, and a CEO who looted the company for money and shipped it to tax-havens. One of the area managers - Kaveh, IIRC - also had a double morale on line with Trump. He was very "don't put your pen in the company ink", and proceeded to get one of his subordinates pregnant. I remember there was an "anti Meltwater blog" at some point, but I can't find it now. I don't remember the URL either, so I can't look it up on archive.org. However, this site[0] seems to contain copy-pasted stuff from it. As said, my experiences are 10+ years old, and hopefully things have improved. [0] https://meltwater66.rssing.com/chan-11517474/latest.php https://meltwater66.rssing.com/chan-11517474/latest.php