mikljohansson

5 ms·

1.
by mikljohansson 5mo ago
Wow, looks so nice! I'm going to use the widget for my model cards
2.
by mikljohansson 8y ago
Thanks! We have a fair bit of a punny humor in our Gothenburg-Sweden engineering office where the information retrieval and big data magic happens. One of our plugins for Elasticsearch that significantly improves GC pressure and performance
3.
by mikljohansson 8y ago
Our query complexity and query response times follow a somewhat exponential pattern, with a lot of quick queries and a long tail of more monstrous queries. We do have a large number of very complex queries coming in. What we deem to be towa
4.
by mikljohansson 8y ago
Our whole dataset exported as compressed JSON is likely not much more than 100TB. The petabytes come from all the index datastructures Elasticsearch/Lucene builds, as well as the high replication factor needed to keep up with the query
5.
by mikljohansson 8y ago
A hot/cold tier architecture might work for setups that have a predictable and sharp cutoff from hot to cold. E.g. a logging setup, where the current days index gets all the indexing and queries. And there's a low concurrency occa
6.
by mikljohansson 8y ago
We at Meltwater have some custom plugins for Elasticsearch that make a number of modifications to how queries are executed, and completely replace some low level Elasticsearch query implementations. We're also running a custom ES 1.7 v
7.
by mikljohansson 8y ago
In our case at Meltwater we also have new documents as well as updates to old documents coming in. Meaning older indexes are actively being modified with low latency requirements on visibility and consistency of those modifications. This ma
8.
Optimal Shard Placement in a Petabyte Scale Elasticsearch Cluster(underthehood.meltwater.com)
11 points by mikljohansson 8y ago|0 comments
9.
Using Machine Learning to Load Balance Elasticsearch Queries(underthehood.meltwater.com)
10 points by mikljohansson 8y ago|1 comments
10.
by mikljohansson 11y ago
We ended up creating https://github.com/meltwater/secretary to allow storing encrypted secrets in config files checked into Git by devteams. The encrypted secrets are passed as env vars through the continuous delivery