4 ms·

unified2 is not a great format and does not sound very flexible for further development. The JSON marshaling is expensive but is not the first of my worries whe

by syoc 6y ago

unified2 is not a great format and does not sound very flexible for further development. The JSON marshaling is expensive but is not the first of my worries when doing line rate inspection on traffic at those rates. I think most shops have a million integrations that they want to set up for their IDS alerts and JSON is pretty close to a lingua franca.

_wldu 6y ago
JSON is widely used. There is no arguing that. And for small to medium networks, JSON is fine. However, when you are monitoring multiple 100 Gb links and you produce hundreds of GB of logs a day, JSON is the wrong format. Especially if you want to search in a reasonable period of time. Zeek can output JSON as well (in addition to plaintext). I've done comparisons. Using jq to search zeek JSON is five times slower than using the C++ simdjson library (fastest JSON parser known to humankind). And simdjson is three times slower than bro-cut on the plaintext logs. This may not sound like a lot, but it is extremely important when monitoring multiple 100Gb links and you have 40 GB JSON conn logs every hour and you need queries to run in a reasonable period of time.
- Godel_unicode 6y ago
  If you're generating that much logging though, you're definitely not using brocut/jq/etc, you're going to be well into Kafka/Hadoop/elk territory. See also, Apache metron.
- justinsaccount 6y ago
  jq is just slow.. using github.com/buger/jsonparser to slice out some fields from a large json log is about 10x faster than using jq. The most common mistake I see is people composing pipelines and putting jq before grep. Or worse, using select() in jq. The grep should always come first. Even if you need to do something like .. | fgrep value | jq ... | fgrep value to pre-filter, and then filter again to rule out any false positives. I have some tooling built around jsonparser and fastjson that I need to split out and open source. jq just makes me sad.. very capable, but overkill for what most people use it for and very slow for the common case. source: wrote the C version of bro-cut that you like so much :)

by syoc 6y ago

_wldu 6y ago

JSON is widely used. There is no arguing that. And for small to medium networks, JSON is fine. However, when you are monitoring multiple 100 Gb links and you produce hundreds of GB of logs a day, JSON is the wrong format. Especially if you want to search in a reasonable period of time. Zeek can output JSON as well (in addition to plaintext). I've done comparisons. Using jq to search zeek JSON is five times slower than using the C++ simdjson library (fastest JSON parser known to humankind). And simdjson is three times slower than bro-cut on the plaintext logs. This may not sound like a lot, but it is extremely important when monitoring multiple 100Gb links and you have 40 GB JSON conn logs every hour and you need queries to run in a reasonable period of time.

Godel_unicode 6y ago
If you're generating that much logging though, you're definitely not using brocut/jq/etc, you're going to be well into Kafka/Hadoop/elk territory. See also, Apache metron.
justinsaccount 6y ago
jq is just slow.. using github.com/buger/jsonparser to slice out some fields from a large json log is about 10x faster than using jq. The most common mistake I see is people composing pipelines and putting jq before grep. Or worse, using select() in jq. The grep should always come first. Even if you need to do something like .. | fgrep value | jq ... | fgrep value to pre-filter, and then filter again to rule out any false positives. I have some tooling built around jsonparser and fastjson that I need to split out and open source. jq just makes me sad.. very capable, but overkill for what most people use it for and very slow for the common case. source: wrote the C version of bro-cut that you like so much :)