4 ms·

I had a task the other day to aggregate some logs. So I wrote a one liner, which did most of what I wanted. I took about 4 minutes to run. Then I decided to ru

by Svoka 8y ago

I had a task the other day to aggregate some logs. So I wrote a one liner, which did most of what I wanted. I took about 4 minutes to run. Then I decided to run it on larger dataset (because I needed too). Like week of logs, not a day of logs. While it was running, I wrote rust CLI, which was working like `cat /*.log | logparser` and did one day in 12 seconds, and a week in a two minutes. And I gave up waiting on awk, btw. It is not always better to use command line. If you have gigabytes or tens of gigabytes of data, it would be easier to write some cli tool to help you out. Also, it was much easier to put significantly more complex logic into it because of type checking, and, you know, being actual high level programming language, not hack&slash awk script. EDIT: Looking back on my "one liner" vs "rust cli" I would not be able to make meaningful adjustments to one liner comprehension. It is, to my sorrow, write-only thing.

draw_down 8y ago
Yeah. These tools are very convenient when they work, but there are lots of situations where they don’t quite cut it.
unhammer 8y ago
If your awk script gets too long / unreadable you just put it in a file and use some whitespace and longer variable names. AWK scripts tend to be very readable (much more so than e.g. sed) as long as they stick to the "stateful filters" use-case as https://news.ycombinator.com/item?id=19294195 https://news.ycombinator.com/item?id=19294195 calls it, but yes they have their limits. If speed is a concern, you may want to try using mawk instead of GNU awk/gawk. I've had 4x speedups with mawk.
- DenseComet 8y ago
  Based on his timings, rust achieved a 20x speedup vs a 4x speedup if he used mawk.
  - hedora 8y ago
    Yep, and adding gnu parallel or xargs to the mix would give a # of cores speedup, which would make mawk faster than the single-threaded rust on a > 6 core machine (roughly).
- Svoka 8y ago
  I also find that programming with languages which has strong type system is very helpful, if you trying to make anything with logic you can't fit in one simple sentence. For example, I was parsing logs. They had entries urlencoded JSON, one document per line, each could be invalid, I had to extract 'id' field, and count number of entries with same IDs, and number of entries with same IDs and with special marker. Then take only entries with 10000+ results. You can totally write cat/grep/awk/sort/head. But then I wanted to add a field, and it was hard to edit. Rust solved my problem, and I had pleasant time writing code, not editing foot long line of untyped code.
redsavagefiero 8y ago
Would you mind posting your awk write-only monstrosity?
- Svoka 8y ago
  It wasn't all awk, there was a several commands connected by pipe, extracting data from JSON entries, some grep filtering, awk to aggregate data by key, etc. I tried to look it up and couldn't find, sorry. It was more than half a year ago.
  - redsavagefiero 8y ago
    Doesn't sound like performance was your end goal unless it was to demonstrate that your understanding of the unix tools could produce a solution slower than a custom built solution in the language of champions.

by Svoka 8y ago

draw_down 8y ago

Yeah. These tools are very convenient when they work, but there are lots of situations where they don’t quite cut it.

unhammer 8y ago

If your awk script gets too long / unreadable you just put it in a file and use some whitespace and longer variable names. AWK scripts tend to be very readable (much more so than e.g. sed) as long as they stick to the "stateful filters" use-case as https://news.ycombinator.com/item?id=19294195 https://news.ycombinator.com/item?id=19294195 calls it, but yes they have their limits. If speed is a concern, you may want to try using mawk instead of GNU awk/gawk. I've had 4x speedups with mawk.

DenseComet 8y ago
Based on his timings, rust achieved a 20x speedup vs a 4x speedup if he used mawk.
- hedora 8y ago
  Yep, and adding gnu parallel or xargs to the mix would give a # of cores speedup, which would make mawk faster than the single-threaded rust on a > 6 core machine (roughly).
Svoka 8y ago
I also find that programming with languages which has strong type system is very helpful, if you trying to make anything with logic you can't fit in one simple sentence. For example, I was parsing logs. They had entries urlencoded JSON, one document per line, each could be invalid, I had to extract 'id' field, and count number of entries with same IDs, and number of entries with same IDs and with special marker. Then take only entries with 10000+ results. You can totally write cat/grep/awk/sort/head. But then I wanted to add a field, and it was hard to edit. Rust solved my problem, and I had pleasant time writing code, not editing foot long line of untyped code.

redsavagefiero 8y ago

Would you mind posting your awk write-only monstrosity?

Svoka 8y ago
It wasn't all awk, there was a several commands connected by pipe, extracting data from JSON entries, some grep filtering, awk to aggregate data by key, etc. I tried to look it up and couldn't find, sorry. It was more than half a year ago.
- redsavagefiero 8y ago
  Doesn't sound like performance was your end goal unless it was to demonstrate that your understanding of the unix tools could produce a solution slower than a custom built solution in the language of champions.