3 ms·

I believe pipes for logs were invented by SumoLogic 10+ years ago. Or maybe someone before that.

• valyala 2y ago
  Canonical pipes were invented ~50 years ago by Unix creators. They are successfully used for logs' analysis to this day. Unix pipes have a simple idea - connecting the output of one program to the input of another program. This allows building arbitrary complex data processing pipelines by combining simple programs like grep, cat, awk, cut, head, etc., via pipes. These pipelines have nice properties: - The data is processed in a streaming manner. This allows processing unlimited data streams without excess resource usage. - The data processing speed is limited by the slowest program in the pipeline. This allows pausing and resuming data processing at any time. For example, by putting 'less' to the end of the pipeline. - The data processing pipeline is terminated instantly as soon as some program in the pipeline terminates. For example, if you put 'tail -100' to the end of pipeline, then only the first 100 lines of data will be processed. VictoriaLogs adapts these properties into logs processing with LogsQL pipes. The difference is that the data, which is flown between pipes, represents structured logs [1]. [1] https://docs.victoriametrics.com/victorialogs/querying/#command-line https://docs.victoriametrics.com/victorialogs/querying/#comm...