36 ms·

What does 20 GB/sec mean in the context of a search?

nomel 10y ago
The article [1] was about a log search/parsing service, so 20 GB of log files/second. Log files are usually text. [1] http://blog.scalyr.com/2014/05/searching-20-gbsec-systems-engineering-before-algorithms/ http://blog.scalyr.com/2014/05/searching-20-gbsec-systems-en...
- gravypod 10y ago
  Every few years I stay in CS I see something crazy. I hope that the next crazy thing I see is a log file that isn't text. I can't imagine who would think that would have been useful to program IF it does exist.
  - emcq 10y ago
    Some folks force every line to be Jason serializable. That can be nice for not needing to rewrite a parser for structured log lines. Another case is if you primarily send data via messages between processes, you could save those streams as log data. For example tcp streams or protobuf. The binary formats will be faster and smaller than converting all log output to text. While these scenarios may not apply to the common case, they certainly exist within the field.
  - kryptiskt 10y ago
    Doesn't systemd do that with journald?
    - duncan_bayne 10y ago
      Yes and it's one of the many reasons I'm migrating to FreeBSD once 11.0 is out (chipset support). Not saying it's wrong for all cases, but it's not for me.
  - to3m 10y ago
    Why would you want your log files purely as text? It tends to take up a lot more space when you're storing a useful amount of readable information. Category prefixes and time stamps alone take up enough space that you could benefit from having them stored concisely. Indentation actually takes up space. The lines are delimited, not counted (you can have a trailing count). And so on. Space will always be a bottleneck, and generating the data will always take time; no reason to make that worse. The average log byte will be written once and then never looked at by a human. Producing ASCII text also requires a lot more code. Entire articles have been written on the correct way to print a float as decimal digits... or, you could memcpy 4 bytes into a buffer, while knowing ahead of time exactly how much space you'll need (and the value will round trip perfectly). Something similar goes for integers (using something like LEB128 encoding). In both cases the encoded data will likely take up less space than most useful text representations, even taking into account any control bytes, while being a lot quicker to produce. "But what if there's a bug?"... well, you'll probably find it in one of your first tests, when you write the first thing that sprang to mind. There's not much space for bugs to sneak in. People don't like this because we have crappy tools. If `cat' supported this kind of encoding as-is, nobody would complain about the idea. You can tell by the popularity of gzip... who complains about gzip? But good luck trying to read a gzip file without the right tools.
    - falcolas 10y ago
      The right tools for a gzipped log file is: less (courtesy of lesspipe). And/or zgrep. In the worst case scenario, you pipe the output of a gunzip through whatever tools you really want to use. The same tools work on compressed and uncompressed logs, and all of that "wasted space" compresses really well. Nothing like watching gigs of log files compress down to a few megs. It's even supported automatically in most Linux operating systems via logrotate.

nomel 10y ago

The article [1] was about a log search/parsing service, so 20 GB of log files/second. Log files are usually text. [1] http://blog.scalyr.com/2014/05/searching-20-gbsec-systems-engineering-before-algorithms/ http://blog.scalyr.com/2014/05/searching-20-gbsec-systems-en...

gravypod 10y ago
Every few years I stay in CS I see something crazy. I hope that the next crazy thing I see is a log file that isn't text. I can't imagine who would think that would have been useful to program IF it does exist.
- emcq 10y ago
  Some folks force every line to be Jason serializable. That can be nice for not needing to rewrite a parser for structured log lines. Another case is if you primarily send data via messages between processes, you could save those streams as log data. For example tcp streams or protobuf. The binary formats will be faster and smaller than converting all log output to text. While these scenarios may not apply to the common case, they certainly exist within the field.
- kryptiskt 10y ago
  Doesn't systemd do that with journald?
  - duncan_bayne 10y ago
    Yes and it's one of the many reasons I'm migrating to FreeBSD once 11.0 is out (chipset support). Not saying it's wrong for all cases, but it's not for me.
- to3m 10y ago
  Why would you want your log files purely as text? It tends to take up a lot more space when you're storing a useful amount of readable information. Category prefixes and time stamps alone take up enough space that you could benefit from having them stored concisely. Indentation actually takes up space. The lines are delimited, not counted (you can have a trailing count). And so on. Space will always be a bottleneck, and generating the data will always take time; no reason to make that worse. The average log byte will be written once and then never looked at by a human. Producing ASCII text also requires a lot more code. Entire articles have been written on the correct way to print a float as decimal digits... or, you could memcpy 4 bytes into a buffer, while knowing ahead of time exactly how much space you'll need (and the value will round trip perfectly). Something similar goes for integers (using something like LEB128 encoding). In both cases the encoded data will likely take up less space than most useful text representations, even taking into account any control bytes, while being a lot quicker to produce. "But what if there's a bug?"... well, you'll probably find it in one of your first tests, when you write the first thing that sprang to mind. There's not much space for bugs to sneak in. People don't like this because we have crappy tools. If `cat' supported this kind of encoding as-is, nobody would complain about the idea. You can tell by the popularity of gzip... who complains about gzip? But good luck trying to read a gzip file without the right tools.
  - falcolas 10y ago
    The right tools for a gzipped log file is: less (courtesy of lesspipe). And/or zgrep. In the worst case scenario, you pipe the output of a gunzip through whatever tools you really want to use. The same tools work on compressed and uncompressed logs, and all of that "wasted space" compresses really well. Nothing like watching gigs of log files compress down to a few megs. It's even supported automatically in most Linux operating systems via logrotate.