4 ms·

Show HN: My weekend project, realtime log viewer in Flask

58 points by llambda 15y ago

jiggity 15y ago
It's poetic. It's like walking into a room filled with mirrors.
llambda 15y ago
So if anyone's wondering how this works, it's actually quite simple thanks to juggernaut. I used Flask to build the web app but because it's basically just serving up a page that really wasn't necessary. You could actually write this all with something like gevent, however Flask lends itself to extensibility and you could certainly use this as a component of a larger app. The primary functionality goes like this: Essentially I wrote a small script to read off a log file, that runs concurrently as a greenlet via gevent. Anything it reads is pushed to juggernaut which uses socketio to publish that along a channel. The frontend reads off that channel, pushing in realtime. I actually spent more time styling it than I did coding it. The VT terminal style is very intentional and I'm pretty happy with the overall result of the look (hey, CSS3 is nice!). I also included a couple of alternate color schemes: blue and orange. Hope you like it! :)
- jemeshsu 15y ago
  I understand Flask can not do socketio because it is WSGI app. Any idea if socketio will be supported by WSGI at all?
  - llambda 15y ago
    There was/is an attempt to do this with gevent via gevent-socketio. However that project seems to have stalled and my last attempt to use it, a couple of weeks ago, did not have good results. Armin Ronacher, the author of Flask, actually used juggernaut in a sample application which led me to experiment with it.
  - ak217 15y ago
    In addition to this, I recently discovered that Flask doesn't support Transfer-Encoding: chunked (which is part of the HTTP/1.1 spec), and doesn't fail gracefully either (when receiving a chunked request). Then I discussed it with Armin and realized that the WSGI spec prevents Flask from doing either. (See also http://www.b-list.org/weblog/2009/aug/10/wsgi/ http://www.b-list.org/weblog/2009/aug/10/wsgi/) Time to fork WSGI? ETA: Upon further investigation, there's a big discussion out there on what to replace WSGI with, here's one entry point: http://dirtsimple.org/2011/08/is-wsgi-lite-library-or-protocol-and.html http://dirtsimple.org/2011/08/is-wsgi-lite-library-or-protoc...
septnuits 15y ago
It's fun to watch. You can communicate with your browser agent.
amix 15y ago
Why is it that Flask is highlighted when juggernaut, node.js and redis do the realtime communication? :-/
- llambda 15y ago
  Because it's not built in a JS framework such as Node (it just uses those things as a secondary service). Which up until recently was kind of a pain. Juggernaut makes life easier and as a result there has been an increase in realtime Flask applications which is pretty cool if you're a Python web dev.
jakubw 15y ago
Nice demo, thanks for enlightening me of juggernaut's existence! If I were to be picky, it's hard to read anything because the scroll position moves to the end even if you explicitly control it, which it shouldn't. Plus, there's this reliability problem that between your Flask code that serves the last 100 lines and the realtime communication, you can miss some entries. And it's a bit inefficient such as loading the whole log file just to grab the last 100 lines. Perhaps this comment should be on GitHub...
- llambda 15y ago
  Yeah it's definitely a little rough around the edges. If you're interested, I'd be more than happy to merge your suggested changes via a pull request on GitHub.
thehodge 15y ago
Seems to have some XSS vulnerabilities
- bluemoon 15y ago
  Yeah we had the data just return instead of escaping the value.
marcuswestin 15y ago
XSS By bflesch@bf-itservice.de :) nice work
- godDLL 15y ago
  https://github.com/maxcountryman/logmon/issues/1 https://github.com/maxcountryman/logmon/issues/1
- bflesch 15y ago
  I was playing a bit with the app and it seems some attack vectors went through. I really didn't want to mess up your experience of this POC and see that you've already filed a report on github, that's very impressive. Thanks. :-)
bflesch 15y ago
What is really bugging me as a programmer: We live in the 21st century and people still include user-supplied data in their pages as if Cross-Site Scripting (XSS) is totally unknown. I get a bad feeling when these simple attacks are possible; because this shows that there are fundamental flaws in some people's coding styles. I know this sounds harsh, and I recognize it is llamdba's weekend project, but these flaws show neglection of fundamentals in handling data. Apart from this, this live log is very entertaining!
- llambda 15y ago
  You're absolutely right, I overlooked the fact that the log data wasn't being escaped (something that I had realized in a previous chat application I had made last week). A quick fix is to use flask's built-in escape function, something like: line = escape(line). The server is now escaping lines off the log.
a_a_r_o_n 15y ago
Cool. And not to take away from it all, but ... You can do the same or better with multitail locally tailing a remote tail over ssh, and you get multitail's color schemes for various log types like mail or httpd. multitail -s 2 -sn 2,2 -C \ -l "ssh -t xxx.yyy.zzz.123 tail -n 100 -f /var/log/nginx/a*" \ -l "ssh -t xxx.yyy.zzz.123 tail -n 100 -f /var/log/nginx/e*" \ -l "ssh -t xxx.yyy.zzz.123 tail -n 100 -f /var/log/uwsgi/*" \ -l "ssh -t xxx.yyy.zzz.123 tail -n 100 -f /var/log/ufw.log"
- bluemoon 15y ago
  That's pretty cool, hopefully we will have better support for multiple logs soon
joshbaptiste 15y ago
I can immediately envision some use cases for this RTM at the workplace, yet our log file throughput is extremely high, so I would have to incorporate some type of throttling mechanism, Great Project...
pingswept 15y ago
I was planning on writing something like this in the next few weeks for my Flask app that runs on the Rascal (see profile for details), but now I won't have to. Thanks, Max.
- llambda 15y ago
  You're most welcome. Please feel free to contact me if you have questions about setting it up. Also if there's changes you'd like to make that would benefit the application in general, I'll be happy to accept pull requests on GitHub.
  - pingswept 15y ago
    Roger that. I'll let you know if I come up with anything good.
pingswept 15y ago
Thank you for the entertainment, unknown people of HN. X.X.X.X - - [27/Dec/2011:00:26:07 +0000] "GET /hey-i-can-send-secret-messages HTTP/1.1" 404 211 "-" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2" Y.Y.Y.Y - - [27/Dec/2011:00:26:19 +0000] "GET /I%20can%20recieve%20secret%20messages%20too HTTP/1.1" 404 211 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0" X.X.X.X - - [27/Dec/2011:00:26:25 +0000] "GET /hey-i-can-send-secret-messages/maybe-al-qaeda-could-use-this HTTP/1.1" 404 211 "-" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2" X.X.X.X - - [27/Dec/2011:00:26:50 +0000] "GET /ooh-you-can-recieve-them? HTTP/1.1" 404 211 "-" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2" Z.Z.Z.Z - - [27/Dec/2011:00:27:05 +0000] "GET /they+are+already+using+it HTTP/1.1" 404 211 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0" X.X.X.X - - [27/Dec/2011:00:27:34 +0000] "GET /hello-decatur! HTTP/1.1" 404 211 "-" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2" Y.Y.Y.Y - - [27/Dec/2011:00:27:56 +0000] "GET /this%20is%20alqaeda%20checking%20in...%20Hussain%20do%20you%20copy??? HTTP/1.1" 404 211 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0" X.X.X.X - - [27/Dec/2011:00:28:15 +0000] "GET /200-hussain-found HTTP/1.1" 404 211 "-" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2"
- FrankAbagnale 15y ago
  I coudn't respond to hussain :(
FrankAbagnale 15y ago
Response 500 :(. I was having fun chatting with GET requests..

58 points by llambda 15y ago

It's poetic. It's like walking into a room filled with mirrors.

So if anyone's wondering how this works, it's actually quite simple thanks to juggernaut. I used Flask to build the web app but because it's basically just serving up a page that really wasn't necessary. You could actually write this all with something like gevent, however Flask lends itself to extensibility and you could certainly use this as a component of a larger app. The primary functionality goes like this: Essentially I wrote a small script to read off a log file, that runs concurrently as a greenlet via gevent. Anything it reads is pushed to juggernaut which uses socketio to publish that along a channel. The frontend reads off that channel, pushing in realtime. I actually spent more time styling it than I did coding it. The VT terminal style is very intentional and I'm pretty happy with the overall result of the look (hey, CSS3 is nice!). I also included a couple of alternate color schemes: blue and orange. Hope you like it! :)

jemeshsu 15y ago
I understand Flask can not do socketio because it is WSGI app. Any idea if socketio will be supported by WSGI at all?
- llambda 15y ago
  There was/is an attempt to do this with gevent via gevent-socketio. However that project seems to have stalled and my last attempt to use it, a couple of weeks ago, did not have good results. Armin Ronacher, the author of Flask, actually used juggernaut in a sample application which led me to experiment with it.
- ak217 15y ago
  In addition to this, I recently discovered that Flask doesn't support Transfer-Encoding: chunked (which is part of the HTTP/1.1 spec), and doesn't fail gracefully either (when receiving a chunked request). Then I discussed it with Armin and realized that the WSGI spec prevents Flask from doing either. (See also http://www.b-list.org/weblog/2009/aug/10/wsgi/ http://www.b-list.org/weblog/2009/aug/10/wsgi/) Time to fork WSGI? ETA: Upon further investigation, there's a big discussion out there on what to replace WSGI with, here's one entry point: http://dirtsimple.org/2011/08/is-wsgi-lite-library-or-protocol-and.html http://dirtsimple.org/2011/08/is-wsgi-lite-library-or-protoc...

septnuits 15y ago

It's fun to watch. You can communicate with your browser agent.

amix 15y ago

Why is it that Flask is highlighted when juggernaut, node.js and redis do the realtime communication? :-/

llambda 15y ago
Because it's not built in a JS framework such as Node (it just uses those things as a secondary service). Which up until recently was kind of a pain. Juggernaut makes life easier and as a result there has been an increase in realtime Flask applications which is pretty cool if you're a Python web dev.

jakubw 15y ago

Nice demo, thanks for enlightening me of juggernaut's existence! If I were to be picky, it's hard to read anything because the scroll position moves to the end even if you explicitly control it, which it shouldn't. Plus, there's this reliability problem that between your Flask code that serves the last 100 lines and the realtime communication, you can miss some entries. And it's a bit inefficient such as loading the whole log file just to grab the last 100 lines. Perhaps this comment should be on GitHub...

llambda 15y ago
Yeah it's definitely a little rough around the edges. If you're interested, I'd be more than happy to merge your suggested changes via a pull request on GitHub.

thehodge 15y ago

Seems to have some XSS vulnerabilities

bluemoon 15y ago
Yeah we had the data just return instead of escaping the value.

marcuswestin 15y ago

XSS By bflesch@bf-itservice.de :) nice work

godDLL 15y ago
https://github.com/maxcountryman/logmon/issues/1 https://github.com/maxcountryman/logmon/issues/1
bflesch 15y ago
I was playing a bit with the app and it seems some attack vectors went through. I really didn't want to mess up your experience of this POC and see that you've already filed a report on github, that's very impressive. Thanks. :-)

bflesch 15y ago

What is really bugging me as a programmer: We live in the 21st century and people still include user-supplied data in their pages as if Cross-Site Scripting (XSS) is totally unknown. I get a bad feeling when these simple attacks are possible; because this shows that there are fundamental flaws in some people's coding styles. I know this sounds harsh, and I recognize it is llamdba's weekend project, but these flaws show neglection of fundamentals in handling data. Apart from this, this live log is very entertaining!

llambda 15y ago
You're absolutely right, I overlooked the fact that the log data wasn't being escaped (something that I had realized in a previous chat application I had made last week). A quick fix is to use flask's built-in escape function, something like: line = escape(line). The server is now escaping lines off the log.

a_a_r_o_n 15y ago

Cool. And not to take away from it all, but ... You can do the same or better with multitail locally tailing a remote tail over ssh, and you get multitail's color schemes for various log types like mail or httpd. multitail -s 2 -sn 2,2 -C \ -l "ssh -t xxx.yyy.zzz.123 tail -n 100 -f /var/log/nginx/a*" \ -l "ssh -t xxx.yyy.zzz.123 tail -n 100 -f /var/log/nginx/e*" \ -l "ssh -t xxx.yyy.zzz.123 tail -n 100 -f /var/log/uwsgi/*" \ -l "ssh -t xxx.yyy.zzz.123 tail -n 100 -f /var/log/ufw.log"

bluemoon 15y ago
That's pretty cool, hopefully we will have better support for multiple logs soon

joshbaptiste 15y ago

I can immediately envision some use cases for this RTM at the workplace, yet our log file throughput is extremely high, so I would have to incorporate some type of throttling mechanism, Great Project...

pingswept 15y ago

I was planning on writing something like this in the next few weeks for my Flask app that runs on the Rascal (see profile for details), but now I won't have to. Thanks, Max.

llambda 15y ago
You're most welcome. Please feel free to contact me if you have questions about setting it up. Also if there's changes you'd like to make that would benefit the application in general, I'll be happy to accept pull requests on GitHub.
- pingswept 15y ago
  Roger that. I'll let you know if I come up with anything good.

pingswept 15y ago

Thank you for the entertainment, unknown people of HN. X.X.X.X - - [27/Dec/2011:00:26:07 +0000] "GET /hey-i-can-send-secret-messages HTTP/1.1" 404 211 "-" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2" Y.Y.Y.Y - - [27/Dec/2011:00:26:19 +0000] "GET /I%20can%20recieve%20secret%20messages%20too HTTP/1.1" 404 211 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0" X.X.X.X - - [27/Dec/2011:00:26:25 +0000] "GET /hey-i-can-send-secret-messages/maybe-al-qaeda-could-use-this HTTP/1.1" 404 211 "-" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2" X.X.X.X - - [27/Dec/2011:00:26:50 +0000] "GET /ooh-you-can-recieve-them? HTTP/1.1" 404 211 "-" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2" Z.Z.Z.Z - - [27/Dec/2011:00:27:05 +0000] "GET /they+are+already+using+it HTTP/1.1" 404 211 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0" X.X.X.X - - [27/Dec/2011:00:27:34 +0000] "GET /hello-decatur! HTTP/1.1" 404 211 "-" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2" Y.Y.Y.Y - - [27/Dec/2011:00:27:56 +0000] "GET /this%20is%20alqaeda%20checking%20in...%20Hussain%20do%20you%20copy??? HTTP/1.1" 404 211 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0" X.X.X.X - - [27/Dec/2011:00:28:15 +0000] "GET /200-hussain-found HTTP/1.1" 404 211 "-" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2"

FrankAbagnale 15y ago
I coudn't respond to hussain :(

FrankAbagnale 15y ago

Response 500 :(. I was having fun chatting with GET requests..