4 ms·
I recently wrote a fun blog post (https://pncnmnp.github.io/blogs/odd-sketches.html https://pncnmnp.github.io/blogs/odd-sketches.html) about how to estimate Jac
by pncnmnp 4y ago
I recently wrote a fun blog post (https://pncnmnp.github.io/blogs/odd-sketches.html https://pncnmnp.github.io/blogs/odd-sketches.html) about how to estimate Jaccard Similarity using min hashing, what b-bit min hashing is, and how to improve upon its limitations using a 2014 data structure called odd sketches.
Jaccard Similarity's history is also quite interesting. From my blog:
> In the late 19th century, the United States and several European nations were focused on developing strategies for weather forecasting, particularly for storm warnings. In 1884, Sergeant John Finley of the U.S. Army Signal Corps conducted experiments aimed at creating a tornado forecasting program for 18 regions in the United States east of the Rockies. To the surprise of many, Finley claimed his programs were 95.6% to 98.6% accurate, with some areas even achieving a 100% accuracy rate. Upon publishing his findings, Finley's methods were criticized by contemporaries who pointed out flaws in his verification strategies and proposed their solutions. This sparked a renewed interest in weather prediction, which is now referred to as the "Finley Affair."
> One of these contemporaries was Grove Karl Gilbert. Just two months after Finley's publication, Gilbert pointed out that, based on Finley's strategy, a 98.2% accuracy rate could be achieved simply by forecasting no tornado warning. Gilbert then introduced an alternative strategy, which is now known as Jaccard Similarity.
> So why is it named Jaccard Similarity? As it turns out, nearly three decades after Sergeant John Finley's tornado forecasting program in the 1880s, Paul Jaccard independently developed the same concept while studying the distribution of alpine flora.
- B0073D 4y agoRecently I wanted to try and use this to filter out logs I don't care about, but it seemed a lot more involved than I initially thought. I essentially wanted to use this as a way to flexibly filter out items without having to come up with a regex for every line item. I wonder if anyone has done this before...
- dopidopHN 4y agoI often feel modern tools should offer that
- ta988 4y agoBayesian filters like for emails? You mark them as important or noise and over time it will learn. These are extremely easy to put in place and you don't have to preannotate as it learns as you go.
- nl 4y agoYes Bayesian filters work well for this. I had an idea Splunk had them built in? But it's about 5 lines of Python anyway.
- zamadatix 4y agoCould always rely on the Levenshtein distance. You have to be careful with similarity approaches though as you may end up filtering important messages because they are structurally similar to the unimportant message.
- voldacar 4y agoMaybe you could use a language model embedding to define some kind of semantic distance.
- funkylisp 4y agoJust filter out logs by the file, line that generated it (i.e. that had the log statement). Even if the actual log entry changes (e.g. because of a formatted str with vars) they will always have the same source.