4 ms·

More accurate title: Local Linux user discovers awk is a great and portable tool for handling structured text (it is)!

by juped 7y ago

Arnavion 7y ago
I did discover one minor bit of non-portability. To show the 10 most recent firewall log entries, the dashboard runs `clog filter.log | tail -10r` (`tail -10r` is the equivalent of `tail -10 | tac`). It would be nice to instead just run `clog -f filter.log` (open filter.log in follow mode, just like `tail -f`), leave it open across dashboard refresh iterations, and maintain an in-memory buffer of the 10 most recent entries for display. It would presumably be more efficient. However this requires the ability to do a non-blocking read of the command output, which requires `PROCINFO["read_timeout"]`, which is gawk-specific and not available on FreeBSD's default awk. I don't mind it too much, because repeatedly doing `clog -f | tail -10r` doesn't use that much CPU to be concerned about anyway. Though I may investigate Perl to see if it's possible there.
- acqq 7y ago
  I guess you mean something like (in Perl): while (1) { try_reading_one_more_line( $file_in ); print_last_10_lines_reverse(); sleep( 1 ); # wait, in this case one second } and that doing on the opened file $file_in. You'd have to write both functions in the loop of course. But Perl is a real language which allows you to try reading from the point in the file you've reached the last time, without reading anything you've already read again. POSIX semantics allows you to observe that the file grew even if in the previous pass you've hit the end of file. And you even don't have to keep in memory the lines which aren't to be printed anymore. I prefer Perl to any shell scripts, as soon as the script is more than circa 10 lines.
  - Arnavion 7y ago
    >I guess you mean something like Yes. >But Perl is a real language which allows you to try reading from the point in the file you've reached the last time, without reading anything you've already read again. The input in this case is the stdout of a command (clog), so it's not a seekable file.
    - acqq 7y ago
      > it's not a seekable file. The flow I've described works on non-seekable files (and pipes). After you reach the end of file in one pass, you are allowed to try to read again in the next, and the file pointer stayed on the same position if you haven't closed the file (and of course you should not close it). If the other process wrote something, the next time read will succeed. If you don't want to display the partial line before the line feed is written, you have to collect the read bytes before you read the line feed. It's all doable in Perl. The only case I've seen something of what I've described not working was some case under some non-recent version of Wine, and I don't know if they fixed that. Otherwise Perl and POSIX support everything described.
      - acqq 7y ago
        A small demo: gen_lines.pl: #!/usr/bin/env perl $| = 1; while (1) { print "line $n\n"; $n++; sleep( 3 ); } show_10.pl: #!/usr/bin/env perl $u = 1; @lines = (); while (1) { $line = <>; push @lines, $line; shift @lines if ( scalar( @lines ) > 10 ); print "10 lines update $u:\n"; $u++; for $l ( reverse @lines ) { print $l; } sleep( 1 ); } ./gen_lines.pl | ./show_10.pl
        Arnavion 7y ago
        Okay, I thought you were trying to say that perl could resume reading from the last position of the file after closing and reopening it, which would require a seekable file to work. If you were referring to leaving the file open across multiple iterations, then yes that is already what I said in the first post that you responded to. It's obvious that perl can do it; any program can.
        acqq 7y ago
        > It's obvious that perl can do it; any program can. I'm not aware how shell scripts or awk could do that. Maybe you can explain. I knew about Perl and that's what I showed.
cstross 7y ago
I'd have gone for "can't use perl, that's obsolete (uses awk instead)" which is kind of ironic if you remember the days of a2p. As it is, a combination of sh/awk will do the job, but is likely slower and less flexible than doing it in perl because this is exactly the sort of job perl was designed for, back in the day.
- juped 7y ago
  Perl is not quite standard Unix, though (it should be imo)

More accurate title: Local Linux user discovers awk is a great and portable tool for handling structured text (it is)!

by juped 7y ago

More accurate title: Local Linux user discovers awk is a great and portable tool for handling structured text (it is)!

I did discover one minor bit of non-portability. To show the 10 most recent firewall log entries, the dashboard runs `clog filter.log | tail -10r` (`tail -10r` is the equivalent of `tail -10 | tac`). It would be nice to instead just run `clog -f filter.log` (open filter.log in follow mode, just like `tail -f`), leave it open across dashboard refresh iterations, and maintain an in-memory buffer of the 10 most recent entries for display. It would presumably be more efficient. However this requires the ability to do a non-blocking read of the command output, which requires `PROCINFO["read_timeout"]`, which is gawk-specific and not available on FreeBSD's default awk. I don't mind it too much, because repeatedly doing `clog -f | tail -10r` doesn't use that much CPU to be concerned about anyway. Though I may investigate Perl to see if it's possible there.

acqq 7y ago
I guess you mean something like (in Perl): while (1) { try_reading_one_more_line( $file_in ); print_last_10_lines_reverse(); sleep( 1 ); # wait, in this case one second } and that doing on the opened file $file_in. You'd have to write both functions in the loop of course. But Perl is a real language which allows you to try reading from the point in the file you've reached the last time, without reading anything you've already read again. POSIX semantics allows you to observe that the file grew even if in the previous pass you've hit the end of file. And you even don't have to keep in memory the lines which aren't to be printed anymore. I prefer Perl to any shell scripts, as soon as the script is more than circa 10 lines.
- Arnavion 7y ago
  >I guess you mean something like Yes. >But Perl is a real language which allows you to try reading from the point in the file you've reached the last time, without reading anything you've already read again. The input in this case is the stdout of a command (clog), so it's not a seekable file.
  - acqq 7y ago
    > it's not a seekable file. The flow I've described works on non-seekable files (and pipes). After you reach the end of file in one pass, you are allowed to try to read again in the next, and the file pointer stayed on the same position if you haven't closed the file (and of course you should not close it). If the other process wrote something, the next time read will succeed. If you don't want to display the partial line before the line feed is written, you have to collect the read bytes before you read the line feed. It's all doable in Perl. The only case I've seen something of what I've described not working was some case under some non-recent version of Wine, and I don't know if they fixed that. Otherwise Perl and POSIX support everything described.
    - acqq 7y ago
      A small demo: gen_lines.pl: #!/usr/bin/env perl $| = 1; while (1) { print "line $n\n"; $n++; sleep( 3 ); } show_10.pl: #!/usr/bin/env perl $u = 1; @lines = (); while (1) { $line = <>; push @lines, $line; shift @lines if ( scalar( @lines ) > 10 ); print "10 lines update $u:\n"; $u++; for $l ( reverse @lines ) { print $l; } sleep( 1 ); } ./gen_lines.pl | ./show_10.pl
      - Arnavion 7y ago
        Okay, I thought you were trying to say that perl could resume reading from the last position of the file after closing and reopening it, which would require a seekable file to work. If you were referring to leaving the file open across multiple iterations, then yes that is already what I said in the first post that you responded to. It's obvious that perl can do it; any program can.
        acqq 7y ago
        > It's obvious that perl can do it; any program can. I'm not aware how shell scripts or awk could do that. Maybe you can explain. I knew about Perl and that's what I showed.

cstross 7y ago

I'd have gone for "can't use perl, that's obsolete (uses awk instead)" which is kind of ironic if you remember the days of a2p. As it is, a combination of sh/awk will do the job, but is likely slower and less flexible than doing it in perl because this is exactly the sort of job perl was designed for, back in the day.

juped 7y ago
Perl is not quite standard Unix, though (it should be imo)