10 ms·
Show HN: Piping logs, visualizing in a web app – just suffix "| npx logscreen"
- iJohnDoe 3y agoWanted to give kudos. This is one of those solutions that “just works” and was really nice to see. Well done!
- lionkor 3y agoNeat, thats a pretty good use of a web app. How is it handled when I want to view logs that are too large to fit in memory?
- number6 3y agoInstall more memory
- codetrotter 3y agoDownload more RAM
- soorajsanker 3y agoDepends on the size, for me it worked well with decent size JSON log lines, but if it is too much that doesn't fit on the browser memory, can think of index storage of sqlite in the future. Search and filtering can be then loaded to the server + sqlite setup.
- wiseowise 3y ago> This tool has been generated entirely with ChatGPT and hence it is also a demonstration of how useful products can be developed with collaborative reiteration with GPTs. All you need to know.
- CaptainFever 3y agoIf it works, I don't see anything wrong with it. It looks good. Hating on projects that disclose their GPT usage will only incentivize them to not disclose in the future.
- jasonjmcghee 3y agoI was going to ask about the strange combination of package.json and inlined cdn js dependencies (especially strange because it's a local server) Suppose that answers it.
- soorajsanker 3y agoHaha yeah, I wanted to frame this as a single file tool, and hence all the quick wins. But later moved the react app into a separate file because I was not getting the syntax highlights and all. Definitely this can be made better with proper packaging and tooling! As I mentioned, more than the idea, implementation was an experiment on the ability of collaborative iterations with GPTs- it was fun!
- jasonjmcghee 3y agoOh- how much manual intervention was there? It was not entirely generated?
- bilekas 3y agoYeah, there are better ways to do this, I wonder if it's purely an excersize in cgpt useages.
- soorajsanker 3y agoTrue, This was also a personal experiment in understanding the level of collaboration that someone can do with GPTs during the product iterations.
- fragmede 3y ago> All you need to know. For those of us not in the know, what does that mean?
- KronisLV 3y ago> For those of us not in the know, what does that mean? I reckon that some people are either critical of LLMs due to the data that they're trained on (concerns about the legality of using the outputs, or the ethics of taking open source projects with a variety of licenses and how transformative or not the output might be), or question the quality of the code that they might output. Oh, also there's hype cycles and right now people at large might overestimate the capabilities of what LLMs can or cannot do well, which can be tiring, as seeing constant posts about crypto was (where we had a solution in search of a problem). In my eyes, they can be used in conjunction with something like IntelliSense and IDEs to solve select problems (ones with solutions that already exist out there) more quickly, as well as save time on some boilerplate, as long as you still validate the outputs and actually check if everything works. Nowadays I'm using GitHub Copilot and the user experience is mostly okay, I'd say it actually lets me write code faster, at the expense of occasionally getting things pretty wrong, but is still a net positive, other considerations aside.
- fragmede 3y agoIn that case, I'd quote the guidelines to GGP poster: > Please don't fulminate. Please don't sneer, including at the rest of the community. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- h1fra 3y agoAs much as I love to hate on chatgpt engineer, the code is definitely okay; not much difference from a project you would write quickly in a single weekend.
- oneeyedpigeon 3y agoThe first thing that struck me when I looked at some of the code was: <div className="flex items-start border-b border-gray-300 py-2 hover:bg-blue-100 cursor-pointer transition-all" I'm not saying that humans never come up with this kind of crap, but it feels like it's going to be a LOT more common the more we resort to AI. I wonder if "... and make the code easy-to-read so that it's maintainable in future" is ever included in prompts for LLM-generated code.
- Kiro 3y agoThat's just Tailwind and exactly how my HTML looks as well.
- janpmz 3y agoI'm a clean code guy and tailwind made my code look terrible. But I gave it a chance because I heared so many positive things about it. Now I'm happy with it. I don't have to jump around from HTML to CSS and back to HTML.
- agos 3y agothat's the horror of Tailwind for you, which is different from the horror of AI
- addandsubtract 3y agoSoftware development is horror all the way down. The sooner you embrace the horror, the easier life gets.
- chrissoundz 3y agoThat is just tailwind CSS, one the most common CSS libs out there I would think. Your comment is actually a lot more telling of something as opposed to this AI written project though.
- karmakaze 3y ago> Features Web-Based Log Viewer: Get a cleaner and more organized view of command outputs. Real-Time Updates: Logs are displayed in real-time as the command executes. Interactive Interface: Search, filter, and navigate through logs easily. // TODO ChatGPT has its limits // TODO
- asplake 3y agoVery tempted! Installation instructions?
- soorajsanker 3y agoSimply suffix with any unix commands that emits stdout, ` | npx logscreen` (need npx though)
- tchetwin 3y ago`npx` is a tool that ships alongside `npm` (alongside `node`) that either finds a local installation or performs a just-in-time download, and then runs the default "bin" entrypoint for the package. Newer versions of `npx` (last few years) will prompt the user before installing a package. Older versions did not. Note that installing the package can run arbitrary scripts as a side effect with at least the level of permission of the current user, so there is implicit trust required of the author, and the authors of all transitive dependencies.
- asplake 3y agoThanks. Unfamiliar with the Node ecosystem, that escaped me
- jealousgelatin 3y agoLooks neat. Don’t listen to the haters
- fragmede 3y agoThis is pretty neat! I did brew upgrade | npx logscreen as the first thing I could think of that would generate logs to play with this with.
- mg 3y agoIs there a benefit over command | vim - ? I love doing that, because then I get all the power of Vim to look at the output. Search, regex search, sorting, bookmarks, counting, macros ... I started to write output that is indented, so I can use Vim's folding to comfortably look at complex data. For example if you have entries like this: 2023-04-12 17:22 Sighted and UFO Location: 27.1742424 12.137234 Temperature: 24°C Observer: Joe Actually it turned out to be a dove. And you press zM in Vim, they all turn into: 2023-04-12 17:22 Sighted and UFO + 4 lines So you have a nice overview and can open individual entries by putting the curser on them and hit zo. And while folded, you still have all the power of Vim to search within them. Regex search. Count by any measure you like etc etc. Actually, you can do anything, as Vim allows you to execute arbitrary vim commands, vim macros or shell commands on your buffer. Wanna know how many doves typically get mistaken for UFOs around New York grouped by month? Write a script for it and it will be forever at your fingertips.
- CGamesPlay 3y agoI think this is more designed for streamed output rather than viewing files. Vim doesn't seem to support that, in fact it just says "Vim: Reading from stdin..." forever.
- mg 3y agoHmm.. I guess you could write a little script (2 lines?), let's call it "vimlive" which reads from stdin, streams to a temp file and opens that temp file in vim. Then with: command | vimlive You get vim with the current state of the data and you can update it with :e
- darrenf 3y agoYou can have vim constantly refresh a buffer if it's modified externally, like this turns your buffer into a kinda-sorta-version of `tail -f`: :set autoread | au CursorHold * checktime | call feedkeys("G") So if it's already a file, just open it with vim; if it's a STDOUT/STDERR stream, redirect/`tee` to a file and open that.
- deleted 3y ago[deleted]
- gatolgaj 3y agoThis is very neat. It would be super cool if we could extent it to multiple processes. From the Looks of it , it should be very much possible.
- piterrro 3y agoCongrats on launching! Its interesting to see that ChatGPT is able to build something I've been working on (https://logdy.dev https://logdy.dev) for last two months. I guess new era is around the corner then.
- rnts08 3y agoTbf your tool looks 110% more useful and well built than a simple single screen showing the logs with an in-browser search. Keep at it!
- piterrro 3y agoThank you! I have a full roadmap of features to deliver :-)
- mittermayr 3y agoYour landing page looks fantastic, just wanted to throw that out (before ChatGPT starts making that, too)
- piterrro 3y agoThanks!
- marmaduke 3y agoI was looking for something like this for a while, and it could be just the landing page, but I much prefer a tool that's been built slowly with thought and consideration (which is evident from your landing page). How does your tool handle semi-permanent usage or very long logs? For instance, I would run this to view logs from an ssh server that's been up for 6 months, when systemd has rotated the log file etc.
- emadda 3y agoWhy would you watch logs for more than 6 months? Do you filter them? What does the system you are watching do?
- edudobay 3y agoNice, will give it a try! There is a similar tool that I've been using for years: https://github.com/mthenw/frontail/ https://github.com/mthenw/frontail/
- kevincox 3y agoI'm not super familiar with express but this seems to be listening on external interfaces as well: https://github.com/soorajshankar/logScreen/blob/f8b29aaef4284b92d1010ac372304c3765810404/index.js#L17 https://github.com/soorajshankar/logScreen/blob/f8b29aaef428... So this will be sharing your logs with the world if you run this outside of a trusted network. (And personally I avoid trusting any networks)
- soorajsanker 3y agoAre you talking about the difference between 0.0.0.0 and 127.0.0.1 /localhost (loopback address)? If so It should be a tiny thing to change to implement I can add that on the next update- I just went with the express JS defaults to keep things simple:) Or I can even make a flag to configure this :thinking
- ggpsv 3y ago`app.listen` uses the same signature as Node's `server.listen`. You can specify the `host` [0] or else it defaults to `0.0.0.0`. This means that when you run this service, it will be accessible to anyone that can reach the server. Typically, services allow you set the host address by configuration. This way you can set it to `127.0.0.1` so that it is only available locally. [0]: https://nodejs.org/api/net.html#serverlistenport-host-backlog-callback https://nodejs.org/api/net.html#serverlistenport-host-backlo...
- gothink 3y agoIf you do make it configurable, it's still a good idea to make the default `127.0.0.1` for the reasons @ggpsv mentioned. Since the link automatically opens at `http://localhost:PORT http://localhost:PORT` anyway, it's an easy security win.
- Calzifer 3y agoIt does, but express doesn't tell it directly in the docs. Instead http://expressjs.com/en/4x/api.html#app.listen http://expressjs.com/en/4x/api.html#app.listen sends you to https://nodejs.org/api/http.html#http_server_listen https://nodejs.org/api/http.html#http_server_listen which also won't give the answer but sends you to https://nodejs.org/api/net.html#serverlisten https://nodejs.org/api/net.html#serverlisten which at some points finally documents that default is 0.0.0.0. Welcome to the JavaScript ecosystem :/
- neomantra 3y agoThis is handy stuff, thanks for sharing. We are working on transporting tasks back and forth from TUI<>Web<>App<>VR ; this is a great and straightforward example of that. I use VSCode CoPilot constantly as simple augmentation and typing-saving — not for wholesale creation of code. I’ve seen my pre-novice daughter use ChatGPT and the results were mixed (good in some ways, horrible in others). Can you elaborate more on your ChatGPT/LLM working style for this?
- soorajsanker 3y agoSure thing! The collaboration with ChatGPT was like having a friendly coding buddy. I threw in ideas, got feedback, and we built this tool together. We chatted about names, features, and even how to post it. It's like having an AI pal who's great at brainstorming and coding. Fun times! ^ asked Chat GPT to generate this from my earlier prompts :) Some other notes, I am a developer who love to make usable products, and these years I learned the usability is most important especially when you are developing in the early stages of a product. So I fed some ideas, asked suggestions, gave some tech stacks (for example, I asked chatGPT to not go with webpack, instead simply load React from CDN for quick iteration. and suggested memoisation on React to render things faster). So process wise similar to any other product development, but it's like having someone listen to your feedback and instructions and do the work- I would like to call collaboration!
- neomantra 3y agoThanks for those details!
- corytheboyd 3y ago> Interactive Interface: Search, filter, and navigate through logs easily. // TODO This is exactly why you would show logs in the browser though— this is not a nice to have, it is THE core feature. The browser is perfect for building the complex UI necessary for filtering and making sense of log streams.
- reidjs 3y agoThat’s probably why it’s on their TODO list
- soorajsanker 3y agoAhh actually you can do search through the logs in real-time, I forgot to split that and remove TODO from the readme. Complex filtering based on the timestamp is yet to implement.
- simonw 3y agoFound the implementation: https://github.com/soorajshankar/logScreen/blob/f8b29aaef4284b92d1010ac372304c3765810404/public/app.jsx#L39-L44 https://github.com/soorajshankar/logScreen/blob/f8b29aaef428...
- simonw 3y ago"This tool has been generated entirely with ChatGPT and hence it is also a demonstration of how useful products can be developed with collaborative reiteration with GPTs." Any chance you could share a link to the ChatGPT conversation, or a copy of the prompt you used? I build little tools like this with ChatGPT all the time, it's one of my favourite use-cases for it. I often have an idea for a tool that wouldn't justify an hour of development work but is absolutely worth spending 5 minutes iterating on with ChatGPT.
- samsquire 3y agoGood idea. Thank you for sharing. One thing I've always wanted was simple filters on log lines in a GUI that let me collapse tags in the log line. I created a proof of concept that showed logs as a tree. Or I can say I want logs from the point where key=value and until method x. A bit like a debugger but for logs. Show logs that are between these facts.
- wfn 3y ago> showed logs as a tree. Yes, that is exactly what I want as well - a tree-like structure where you can trace events across multiple layers of abstraction as well as across systems (so both horizontal and vertical relations of events, so to speak). Some infra systems output syslog loglines and there's no changing that, and I still want to capture and relate these log messages to other events (incl. high-level business transactions / processes). Any way you could share your PoC? :) very interested.
- samsquire 3y agoThanks for your lovely reply. I don't have the code anymore might have been called "logtree", it was a HTML file and very small, it would parse dot "." separated lines such as (user.connected) and arrange them in a tree with indentation, if you clicked the dot syntax it would highlight in yellow other instances of that bucket of entries later in the log file. I think you want elegant navigation between linear sequences (next, previous) of log lines that are apart in time, but related. You might want to see all events of the internals of two different components in the tree structure. Need to process log lines into a graph and then provide quick navigation for them. It is indeed horizontal and vertical. This would be very useful to anti-abuse measures if you could trace to the cause of something and cancel all thing caused by it.
- wfn 3y agoThank you for helping me structure what I want, yes, exactly. (And noted re: your PoC, ok!) > You might want to see all events of the internals of two different components in the tree structure. > Need to process log lines into a graph and then provide quick navigation for them. Yes, looks like it, indeed. Yeah. If interested, this is how I'd hastily summarise my problem statement - it is I think a rather generic recurring problem and analysis pattern, not unique to any company (context: goal is to drive through this initiative as part of overall software architecture improvement needs, and bringing business maturity (as per CMM model) up in our particular case): - need to introduce and enforce structured event hierarchy (think: each high level business transaction (e.g. user signs document) is a parent which retains all related child events across all layers of abstraction (ideally down to DB connection and file handles, internal API endpoints hit, block storage i/o errors, etc.) - problem: while for own software Backend-generated events we can enforce and manage this, some infra pieces emit raw text syslog (sometimes even syslog message portion (log format) cannot be fully controlled) and do not contain cross-component technical identifiers for determining relations to events - there are solutions including ML solutions (of course) - there exist interesting papers to lose productivity and/or conduct deeper research - what do, how to balance, how to incrementally define, introduce into culture, incrementally realistically establish this (etc.) Some interesting papers for approaching the complicated part (assuming defining event ontology and hierarchy and overall semantics is the easy part) - the "relating unstructured logs to events and embedding them into graphs, and aligning and training everyone to analyse and work with "event trees" across multiple teams (but yes, ops, integration support, etc. - I mean is that not the dream, right...) do exist[1][2][3][4][5], and some interesting thoughts and many lofty ambitious questions[6]. Trying to gain insight into best practices here. Meanwhile, maybe links will look interesting :) (e.g. that 1849-2023 IEEE standard - just found it today...) [1] LogTree: A Framework for Generating System Events from Raw Textual Logs https://www.researchgate.net/publication/220766390_LogTree_A_Framework_for_Generating_System_Events_from_Raw_Textual_Logs https://www.researchgate.net/publication/220766390_LogTree_A... (https://ltangt.github.io/papers/icdm2010-logtree.pdf https://ltangt.github.io/papers/icdm2010-logtree.pdf) [2] Structural Event Detection from Log Messages https://faculty.ist.psu.edu/jessieli/Publications/2017-KDD-log.pdf https://faculty.ist.psu.edu/jessieli/Publications/2017-KDD-l... (this one may look bewildering at first but the math eqs are actually ~legible / straightforward from quick glance, even; also, has nice coverage of related work) (overall nice, might look smarter than appears, but not sure; see MCMC approach etc. in section 5.) [3] golang package https://pkg.go.dev/source.monogon.dev/metropolis/pkg/logtree https://pkg.go.dev/source.monogon.dev/metropolis/pkg/logtree - why i mention - see KLogParser func (https://pkg.go.dev/source.monogon.dev/metropolis/pkg/logtree#KLogParser https://pkg.go.dev/source.monogon.dev/metropolis/pkg/logtree...) (but also see limitations) (did not review in detail, maybe nothing useful) [4] 1849-2023 - IEEE Standard for eXtensible Event Stream (XES) for Achieving Interoperability in Event Logs and Event Streams https://www.xes-standard.org/ https://www.xes-standard.org/ [5] related to above (supports standard) https://pm4py.fit.fraunhofer.de/documentation https://pm4py.fit.fraunhofer.de/documentation -> see example of ontology: Object-Centric Event Logs [6] a structural approach to the problem of structure... use existing frameworks and languages like BPML to create or use existing event type ontologies from BPMN models; use a proper structured thought-through ontology for this, and then sort out that pesky "relating low level infra raw syslogs with none/few TIDs to some node on a graph" problem Anyway, thanks for making through my rant! Writing it in part because on the off chance someone bites and has insight - I'd be very interested to hear more; and partially just to share those links above because those may tickle your (Sam's fancy) :) thanks for reading!
- psim1 3y agoThis is a solution looking for a problem. The browser is a horrible place for viewing text logs. Companies like Splunk and Datadog put countless hours into trying to make it work and it's never not-painful to view logs in a browser.
- soorajsanker 3y agoWhat are the main specific issues that you felt difficult in viewing logs on a browser? imagine there is better visualisation, parsing, highlighting, filtering, collapse etc.? I feel like the terminal is where I spend more time looking for the relevant parts from the logs.
- dloss 3y agoSome CLI viewers for logs are listed on https://github.com/dloss/klp https://github.com/dloss/klp, as alternatives to my own little tool "klp".
- wetmore 3y agoWhat's better? I ask out of ignorance, but personally I think Datadog is amazing for viewing logs.
- liveoneggs 3y agoDid you know AWStats (first released 2003) could accept streaming logs and produce nice dashboards? https://awstats.sourceforge.io/ https://awstats.sourceforge.io/