6 ms·

Moby Dick is very short compared to a normal server logfile.

• astine 16y ago
  Or rather, compared to a server log file that hasn't been properly rotated. I wrote a program a while back to incrementally parse a daily log file which was never rotated. It got say that it would take twenty minutes for for the program to just skip the lines that had been parsed previously. When those who had the power to do so started rotating the file, things speed up tremendously.
  • jonknee 16y ago
    Why didn't you split it?
    • fr0sty 16y ago
      Sparing that, why not use fseek() and do a binary search to get to where you wanted to go? Even better, save the prior offset somewhere and jump there immediately on T+1?
      • sigil 16y ago
        I used this trick the other day -- compute byte range partitions on a large file and use fseek when processing each. Is there a standard unix program that can output an arbitrary byte range from a seekable input? At first blush dd(1) looked like the ticket, but block-oriented operation means extra invocations to deal with a byte range that's not necessarily block-aligned.
        • jacquesm 16y ago
          tail -c offset [input] | head -c length will do the job.
        • pmjordan 16y ago
          You can set dd's "bs" parameter to 1, then seek, skip & count are all in terms of bytes.
      • astine 16y ago
        That, I should have done. I thought to save the number of lines previously read so that I could skip them; I don't know why it didn't occur to me to just save the whole damn offset.
    • astine 16y ago
      I couldn't, I didn't have control over the file.