3 ms·

The hardest part of this is not posting the version I made in 5 minutes ;) I have no interest in working for Reddit, but as a SysAdmin, I can't resist a challen

by e1ven 16y ago

The hardest part of this is not posting the version I made in 5 minutes ;) I have no interest in working for Reddit, but as a SysAdmin, I can't resist a challenge.

jedberg 16y ago
If you send it to me (jedberg@reddit.com) I'll tell you how awesome you are. :)
- dpritchett 16y ago
  Hey, this was actually easier and more fun than expected! I tried it out at the Python REPL and now I have a mostly-working implementation using os.path.getsize(), file.tell(), file.seek(), and file.readline(). The fun part is that I hadn't used the first three of those functions in Python before today. Layering a binary search on top of this looks as easy as *REDACTED* with some edge case checking and start-of-next-line seeking. I'd have thought there was more to this, but it seems pretty straightforward. Lessons learned for me: Python standard library docs are great, repl development is great, and text files are fun.
- e1ven 16y ago
  I thought it would be fun to do it entirely using unix tools. Sent it off using a recursive bash function, with du, dd, bc, and expr.
  - dpritchett 16y ago
    Oh nice! I had thought about using a script to isolate a rough-but-inclusive range of bytes from inside the file and then piping that into grep, but your way is certainly purer and more fun. edit And now I have a good reason to learn more about dd, du, and bc!
jmelloy 16y ago
Yeah, I had this working pretty well in about 20 minutes with python. Binary search is really pretty quick. I didn't have a 75 gigabyte log handy, so I had to test it on a measly 10 GB log. :( I had a secondary problem where our logs occasionally put a new line in, so I had to burn a couple fseeks checking for that. I'm not sure how I'd go about truly profiling it, but it returns pretty much instantly. Though it's not really useful for our logs to look for a per-minute line, because our logs are crap. What useful information do you find looking for that type of specific data?
- dpritchett 16y ago
  Note they also had a time range option. I imagine they start out finding the specific event line ("you broke reddit" at 4:07 am!") and then slowly widen the time range around it to see if anything interesting happened in the minutes leading up to it.
  - jmelloy 16y ago
    I suppose there's also tgrep 08:30-08:36 | grep REDDITBROKE

by e1ven 16y ago

The hardest part of this is not posting the version I made in 5 minutes ;) I have no interest in working for Reddit, but as a SysAdmin, I can't resist a challenge.

jedberg 16y ago

If you send it to me (jedberg@reddit.com) I'll tell you how awesome you are. :)

dpritchett 16y ago
Hey, this was actually easier and more fun than expected! I tried it out at the Python REPL and now I have a mostly-working implementation using os.path.getsize(), file.tell(), file.seek(), and file.readline(). The fun part is that I hadn't used the first three of those functions in Python before today. Layering a binary search on top of this looks as easy as *REDACTED* with some edge case checking and start-of-next-line seeking. I'd have thought there was more to this, but it seems pretty straightforward. Lessons learned for me: Python standard library docs are great, repl development is great, and text files are fun.
e1ven 16y ago
I thought it would be fun to do it entirely using unix tools. Sent it off using a recursive bash function, with du, dd, bc, and expr.
- dpritchett 16y ago
  Oh nice! I had thought about using a script to isolate a rough-but-inclusive range of bytes from inside the file and then piping that into grep, but your way is certainly purer and more fun. edit And now I have a good reason to learn more about dd, du, and bc!

jmelloy 16y ago

Yeah, I had this working pretty well in about 20 minutes with python. Binary search is really pretty quick. I didn't have a 75 gigabyte log handy, so I had to test it on a measly 10 GB log. :( I had a secondary problem where our logs occasionally put a new line in, so I had to burn a couple fseeks checking for that. I'm not sure how I'd go about truly profiling it, but it returns pretty much instantly. Though it's not really useful for our logs to look for a per-minute line, because our logs are crap. What useful information do you find looking for that type of specific data?

dpritchett 16y ago
Note they also had a time range option. I imagine they start out finding the specific event line ("you broke reddit" at 4:07 am!") and then slowly widen the time range around it to see if anything interesting happened in the minutes leading up to it.
- jmelloy 16y ago
  I suppose there's also tgrep 08:30-08:36 | grep REDDITBROKE