6 ms·

The most scalable, fastest reaction time and simplest log processing tool is awk.

Godel_unicode 8y ago
Ok, I'll bite. Most scalable, you say. I have 250TB of Django logs, how do you recommend I use awk to process them to determine the 99th percentile response time that's faster than using SparkSQL?
- heavenlyblue 8y ago
  SparkSQL doesn’t support gz. Are your logs splittable on a file-by-file basis or are they in gz format? Where are your logs stored? Is that a distributed storage? Will SparkSQL not eat all of the bandwidth of it’s ethernet interfaces? Yeah; sure.
  - Godel_unicode 8y ago
    It's not so much that SparkSQL doesn't support gz as that gz is slow because you can't parallelize the reads. Regardless, parquet format in hdfs so yarn can allocate containers local to the chunk to be processed. Scales nicely.
    - heavenlyblue 8y ago
      Yeah, but you are making an assumption I’ve got these 350Gb of logs in HDFS format. Which takes time to set up.
      - Godel_unicode 8y ago
        Not really, you said SparkSQL doesn't support gz, which is incorrect and the thrust of my comment. The anecdote about parquet is orthogonal to gz support. pedantic sidebar: hdfs isn't a file format, it's a distributed file system layered over a traditional on-disk filesystem. For example you might have: json logs, in a gz-formatted file, tracked in the hdfs filesystem, stored on disk in an ext4-formatted filesystem.
  - EdwardDiego 8y ago
    > SparkSQL doesn’t support gz Yes it does. Source - use Spark SQL routinely. You're right that multiple small Gzipped files are not an ideal input source as it'll create a bunch of small tasks, but Spark definitely does support GZ.
- firebacon 8y ago
  Off-topic, but do you actually run ad-hoc SparkSQL queries on the whole dataset sometimes? Are the logs actually stored as text files on disk? How long does such a query take and/or how many racks of machines do you need for that? Should be more than 1 million drive-seconds just to read the data from disk, right?
- nwmcsween 8y ago
  You don't, you process the logs right away with ask as in piping to awk store the statistics in a file.

Ok, I'll bite. Most scalable, you say. I have 250TB of Django logs, how do you recommend I use awk to process them to determine the 99th percentile response time that's faster than using SparkSQL?

heavenlyblue 8y ago
SparkSQL doesn’t support gz. Are your logs splittable on a file-by-file basis or are they in gz format? Where are your logs stored? Is that a distributed storage? Will SparkSQL not eat all of the bandwidth of it’s ethernet interfaces? Yeah; sure.
- Godel_unicode 8y ago
  It's not so much that SparkSQL doesn't support gz as that gz is slow because you can't parallelize the reads. Regardless, parquet format in hdfs so yarn can allocate containers local to the chunk to be processed. Scales nicely.
  - heavenlyblue 8y ago
    Yeah, but you are making an assumption I’ve got these 350Gb of logs in HDFS format. Which takes time to set up.
    - Godel_unicode 8y ago
      Not really, you said SparkSQL doesn't support gz, which is incorrect and the thrust of my comment. The anecdote about parquet is orthogonal to gz support. pedantic sidebar: hdfs isn't a file format, it's a distributed file system layered over a traditional on-disk filesystem. For example you might have: json logs, in a gz-formatted file, tracked in the hdfs filesystem, stored on disk in an ext4-formatted filesystem.
- EdwardDiego 8y ago
  > SparkSQL doesn’t support gz Yes it does. Source - use Spark SQL routinely. You're right that multiple small Gzipped files are not an ideal input source as it'll create a bunch of small tasks, but Spark definitely does support GZ.
firebacon 8y ago
Off-topic, but do you actually run ad-hoc SparkSQL queries on the whole dataset sometimes? Are the logs actually stored as text files on disk? How long does such a query take and/or how many racks of machines do you need for that? Should be more than 1 million drive-seconds just to read the data from disk, right?
nwmcsween 8y ago
You don't, you process the logs right away with ask as in piping to awk store the statistics in a file.