2 ms·

I've heard horror stories from friends about working at meltwater. Setting that aside for a moment, this is an amazing software engineering achievement. Pulli

I've heard horror stories from friends about working at meltwater. Setting that aside for a moment, this is an amazing software engineering achievement. Pulling off this level of scale with Elasticsearch is no easy feat and very impressive from a technical perspective. When you're running ES with petabytes of mission critical data as a core service powering the universe of a business, cluster rebuilds aren't an option (or maybe they are, as a last resort, but absolutely will not be acceptable on an ongoing basis). Relying on Elasticsearch mega-clusters in this manner is akin to running an ultra-marathon with a really sharp pair of scissors glued in each hand. Or maybe even more extreme than my (admittedly lame) analogy. Running nodes with such high shard counts is an appreciably precarious proposition, because there is a fair amount of overhead in the Elasticsearch management protocol. I wonder what the performance testing strategy entailed. I have a lot of respect for the engineers working to make this project and service a success story. When it comes to Elasticsearch at scale, such outcomes are the exception.

karlney 4y ago
Thank you for those kind words. And yes we have had our fair share of pain with the old cluster for sure. the new version (7.17) is still behaving a lot better so far and feels a lot more predictable.
trendy0 4y ago
What were the stories?
- karlney 4y ago
  One time, a few years ago a particularly nasty query was executed over and over again and it took a few hours to find it and then block it. And during that time so many nodes had became slow and unresponsive that another (for us) previously unseen memory leak started to occur. Nodes kept building up queues of unanswered ping requests on them. And the requests contained our 100Mb large cluster state, so the heaps filled up and evenmore nodes became unresponsive. And from then on the whole thing turned into a death spiral of doom. After trying, and failing to get it under control for 48 hours we gave up and rebuilt the whole cluster from scratch, using the snapshots we store on S3. The recovery took another 90 hours or so. That was not a fun week.
- thatwasunusual 4y ago
  Non-technically, it was a horrow-show. I worked at the company from 2005/06 to 2012, when I quit after witnessing shameful behaviour towards women, a party-culture that literally lead to rape allegations, and a CEO who looted the company for money and shipped it to tax-havens. One of the area managers - Kaveh, IIRC - also had a double morale on line with Trump. He was very "don't put your pen in the company ink", and proceeded to get one of his subordinates pregnant. I remember there was an "anti Meltwater blog" at some point, but I can't find it now. I don't remember the URL either, so I can't look it up on archive.org. However, this site[0] seems to contain copy-pasted stuff from it. As said, my experiences are 10+ years old, and hopefully things have improved. [0] https://meltwater66.rssing.com/chan-11517474/latest.php https://meltwater66.rssing.com/chan-11517474/latest.php
andrelaszlo 4y ago
I usually don't explain my downvotes, but I thought that your comment was good overall, but the "horror stories from friends about working at meltwater" without explaining what they are just makes it a bit unfair. As criticism, it's very vague, and as someone who doesn't work at Meltwater (for the last 5 years or so at least) it doesn't give me any information either. Well except that there are rumors about Meltwater, but that would be true about any large corporation. Maybe I misunderstood and the horror stories were about ES, but I got it as being about the company itself. Could you expand? What type of stories? :)

karlney 4y ago

Thank you for those kind words. And yes we have had our fair share of pain with the old cluster for sure. the new version (7.17) is still behaving a lot better so far and feels a lot more predictable.

trendy0 4y ago

What were the stories?

karlney 4y ago
One time, a few years ago a particularly nasty query was executed over and over again and it took a few hours to find it and then block it. And during that time so many nodes had became slow and unresponsive that another (for us) previously unseen memory leak started to occur. Nodes kept building up queues of unanswered ping requests on them. And the requests contained our 100Mb large cluster state, so the heaps filled up and evenmore nodes became unresponsive. And from then on the whole thing turned into a death spiral of doom. After trying, and failing to get it under control for 48 hours we gave up and rebuilt the whole cluster from scratch, using the snapshots we store on S3. The recovery took another 90 hours or so. That was not a fun week.
thatwasunusual 4y ago
Non-technically, it was a horrow-show. I worked at the company from 2005/06 to 2012, when I quit after witnessing shameful behaviour towards women, a party-culture that literally lead to rape allegations, and a CEO who looted the company for money and shipped it to tax-havens. One of the area managers - Kaveh, IIRC - also had a double morale on line with Trump. He was very "don't put your pen in the company ink", and proceeded to get one of his subordinates pregnant. I remember there was an "anti Meltwater blog" at some point, but I can't find it now. I don't remember the URL either, so I can't look it up on archive.org. However, this site[0] seems to contain copy-pasted stuff from it. As said, my experiences are 10+ years old, and hopefully things have improved. [0] https://meltwater66.rssing.com/chan-11517474/latest.php https://meltwater66.rssing.com/chan-11517474/latest.php

andrelaszlo 4y ago

I usually don't explain my downvotes, but I thought that your comment was good overall, but the "horror stories from friends about working at meltwater" without explaining what they are just makes it a bit unfair. As criticism, it's very vague, and as someone who doesn't work at Meltwater (for the last 5 years or so at least) it doesn't give me any information either. Well except that there are rumors about Meltwater, but that would be true about any large corporation. Maybe I misunderstood and the horror stories were about ES, but I got it as being about the company itself. Could you expand? What type of stories? :)