3 ms·
Not quite. Many of our extractors are stateful - ie. They need to see previous packets in the session to extract keys, state, etc. This is done by having fron
by londons_explore 6y ago
Not quite.
Many of our extractors are stateful - ie. They need to see previous packets in the session to extract keys, state, etc.
This is done by having front ends which direct raw packets to backends based on which session they are a part of - the obvious one being the tcp 4-tuple.
The backends then don't each have many sessions to look after - perhaps tens of thousands each. That means they can keep many kilobytes of state. Each time they get a packet, they process it by updating the state. Certain state transitions (eg. The completion of a credit card transaction in a finance feed session) will trigger another event to be emitted, which goes through the same system again, and might go into another extractor or be persisted for analysts to view.
- DAlperin 6y agoVery interesting. Is this written up anywhere?