4 ms·
We have a similar setup, but I've been generating UUIDs for each event at logging time to avoid any computational overhead for deduping.
by wwright 6y ago
We have a similar setup, but I've been generating UUIDs for each event at logging time to avoid any computational overhead for deduping.
- cbsmith 6y agoUnless they are type1 uuid's, there's probably just as much computational overhead going on in the uuid creation. ;-)
- wwright 6y agoWell, in our case, the event generators are almost never CPU bound but the logstash nodes are. Generating a UUID (which is mostly just waiting for the kernel to hand you some bytes, which allows other threads time to do useful work) before sending has no opportunity cost and frees up the more-valuable logstash resources. (These are micro-optimizations anyway.)
- cbsmith 6y agoYeah... I'd definitely remove logstash from that pipeline for that reason.
- fluential 6y agoThe way to approach that in logstash is to use fingerprint plugin which can do various formats (including uuids), MURMUR3 seems to be the fastest here. https://www.elastic.co/guide/en/logstash/current/plugins-filters-fingerprint.html https://www.elastic.co/guide/en/logstash/current/plugins-fil.... Edit: typo