6 ms·
Really cool idea but this gives me anxiety just thinking about how it has to be maintained. Taking into account versions, command flags changing output, etc. al
by kbknapp 3y ago
Really cool idea but this gives me anxiety just thinking about how it has to be maintained. Taking into account versions, command flags changing output, etc. all seems like a nightmare to maintain to the point where I'm assuming actual usage of this will work great for a few cases but quickly lose it's novelty beyond basic cases. Not to mention using `--<CMD>` for the tool seems like a poor choice as your help/manpage will end up being thousands of lines long because each new parser will require a new flag.
- verdverm 3y agoThis is one of the better use cases for LLMs, which have shown good capability at turning unstructured text into structured objects
- ninkendo 3y agoIf LLM’s were local and cheap, sure. They’re just too heavyweight of a tool to use for simple CLI output manipulation today. I don’t want to send everything to the cloud (and pay a fee), and even if it was a local LLM, I don’t want it to eat all my RAM and battery to do simple text manipulation. In 20 years, assuming some semblance of moore’s law still holds for storage/RAM/gpu, I’m right there with you.
- d3nj4l 3y agoOn my M1 Pro/16GB RAM mac I get decently fast, fully local LLMs which are good enough to do this sort of thing. I use them in scripts all the time. Granted, I haven’t checked the impact on the battery life I get, but I definitely haven’t noticed any differences in my regular use.
- mosselman 3y agoWhich models do you run and how?
- _joel 3y agonot op, but this is handy https://lmstudio.ai/ https://lmstudio.ai/
- mosselman 3y agoThanks!
- verdverm 3y agohttps://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp is a popular local first approach. LLaMa is a good place to start, though I typically use a model from Vertex AI via API
- mosselman 3y agoThanks. I have llama.cpp locally. How do you use it in scripts? As in how do you specifically, not how would one.
- d3nj4l 3y agoI have ollama's server running, and I interact with it via the REST API. My preferred model right now is Intel's neural chat, but I'm going to experiment with a few more over the holidays.
- verdverm 3y agoI tried ollama today and it is super easy, finding good models is definitely going to be challenging. I tried a few on some sample (JSON) tasks and it is... frustrating... how they ellide or are unable to follow instructions. Is there a good fine-tuning workflow with ollama?
- d3nj4l 3y agoI haven’t tried any fine tuning so I can’t help there, sorry. Though I will say that neural chat has been pretty good to me, even though I have definitely observed it ignoring instructions at times, like a “here’s your json:” preamble in response to a query that specifically requests only json.
- chongli 3y agoYeah, it would be much better if you could send a sample of the input and desired output and have the LLM write a highly optimized shell script for you, which you could then run locally on your multi-gigabyte log files or whatever.
- himinlomax 3y agoProblem: some crusty old tty command has dodgy output. Solution: throw a high end GPU with 24GB RAM and a million dollar of training at it. Yeah, great solution.
- verdverm 3y agoWith fine-tuning, you can get really good results on specific tasks that can run on regular cpu/mem. I'd suggest looking into the distillation research, where large model expertise can be transferred to much smaller models. Also, an LLM trained to be good at this task has many more applications than just turning command output into structured data. It's actually one of the most compelling business use cases for LLMs
- anonymous_sorry 3y agoThe complaint is less whether it would work, and more a question of taste. Obviously taste can be a personal thing. My opinions are my own and not those of the BBC, etc. You have a small C program that processes this data in memory, and dumps it to stdout in tabular text format. Rather than simplify by stripping out the problematic bit (the text output), you suggest adding a large, cutting-edge, hard to inspect and verify piece of technology that transforms that text through uncountable floating point operations back into differently-formatted UTF8. It might even work consistently (without you ever having 100% confidence it won't hallucinate at precisely the wrong moment). You can certainly see it being justified for one-off tasks that aren't worth automating. But to shove such byzantine inefficiency and complexity into an engineered system (rather than just modify the original program to give the format you want) offends my engineering sensibilities. Maybe I'm just getting old!
- verdverm 3y agoIf you can modify the original program, then that is by far the best way to go. More often than not, you cannot change the program, and in relation to the broader applicability, most unstructured content is not produced by programs.
- hnlmorg 3y agoAs someone who maintains a solution that solves similar problems to jc, I can assure you that you don’t need a LLM to parse most human readable output.
- verdverm 3y agoit's more about the maintenance cost, you don't have to write N parsers for M versions Maybe the best middle ground is to have an LLM write the parser. Lowers the development cost and runtime performance, in theory
- hnlmorg 3y agoYou don’t have to write dozens of parsers. I didn’t.
- verdverm 3y agoPart of the appeal is that people who don't know how to program or write parsers can use an LLM to solve their unstructured -> structured problem
- hnlmorg 3y agoSure. But we weren’t talking about non-programmers maintaining software.
- verdverm 3y ago> people who don't know how to program OR write parsers there are plenty of programmers who do not know how to write lexers, parsers, and grammars
- hnlmorg 3y agoWe are chatting about maintaining a software project written in a software programming language. Not some theoretical strawman argument youve just dreamt up because others have rightly pointed out that you don’t need a LLM to parse the output of a 20KB command line program. As I said before, I maintain a project like this. I also happen to work for a company that specialises in the use of generative AI. So I’m well aware of the power of LLMs as well as the problems of this very specific domain. The ideas you’ve expressed here are, at best, optimistic. by the time you’ve solved all the little quirks of ML you’ll have likely invested far more time on your LLM then you would have if you’d just written a simple parser and, ironically, needed someone far more specialised to write the LLM than your average developer. This simply isn’t a problem that needs a LLM chucked at it. You don’t even need to write lexers and grammars to parse 99% of application output. Again, I know this because I’ve written such software.
- keithalewis 3y agoGive a kid a hammer and he'll find something to fix.
- verdverm 3y agoWhat value does this comment add?
- otteromkram 3y agoI got a kick out of it. ¯\_(ツ)_/¯
- keithalewis 3y agoApproximately the same amount as the comment I replied to.
- verdverm 3y agoOne attempts to nudge a user towards the comment guidelines of HN (https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html) > Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes. > Comments should get more thoughtful and substantive, not less, as a topic gets more divisive. > Eschew flamebait. Avoid generic tangents. Omit internet tropes.
- cproctor 3y agoWould it be fair to think about this as a shim whose scope of responsibility will (hopefully) shrink over time, as command line utilities increasingly support JSON output? Once a utility commits to handling JSON export on its own, this tool can delegate to that functionality going forward.
- dan_quixote 3y agoI'd also assume that a CLI resisting JSON support is likely to have a very stable interface. Maybe wishful thinking...
- pydry 3y agoIt would but I can still see somebody launching this with great enthusiasm and then losing the passion to fix Yet Another Parsing Bug introduced on a new version of dig
- kbrazil 3y ago`jc` author here. I've been maintaining `jc` for nearly four years now. Most of the maintenance is choosing which new parsers to include. Old parsers don't seem to have too many problems (see the Github issues) and bugs are typically just corner cases that can be quickly addressed along with added tests. In fact there is a plugin architecture that allows users to get a quick fix so they don't need to wait for the next release for the fix. In practice it has worked out pretty well. Most of the commands are pretty old and do not change anymore. Many parsers are not even commands but standard filetypes (YAML, CSV, XML, INI, X509 certs, JWT, etc.) and string types (IP addresses, URLs, email addresses, datetimes, etc.) which don't change or use standard libraries to parse. Additionally, I get a lot of support from the community. Many new parsers are written and maintained by others, which spreads the load and accelerates development.
- majkinetor 3y agoThis requires collaboration. People submitting parsing info for the tool they need, and people that use it to easily keep it up to date. That is the only way.
- eichin 3y agoI'm sort of torn - yeah, one well-maintained "basket" beats having a bunch of ad-hoc output parsers all over the place, but I want direct json output because I'm doing something complicated and don't want parsing to add to the problem. (I suppose the right way to get comfortable with using this is to just make sure to submit PRs with additional test cases for everything I want to use it with, since I'd have to write those tests anyway...)
- zackmorris 3y agoKeep in mind that the maintenance responsibility you're anxious about is currently a cost imposed on all developers. <rant> Since I started programming in the 80s, I've noticed a trend where most software has adopted the Unix philosophy of "write programs that do one thing and do it well". Which is cool and everything, but has created an open source ecosystem of rugged individualism where the proceeds to the winners so vastly exceeds the crumbs left over for workers that there is no ecosystem to speak of, just exploitation. Reflected now in the wider economy. But curated open source solutions like jc approach problems at the systems level so that the contributions of an individual become available to society in a way that might be called "getting real work done". Because they prevent that unnecessary effort being repeated 1000 or 1 million times by others. Which feels alien in our current task-focussed reality where most developers never really escape maintenance minutia. So I'm all in favor of this inversion from "me" to "we". I also feel that open source is the tech analog of socialism. We just have it exactly backwards right now, that everyone has the freedom to contribute, but only a select few reap the rewards of those contributions. We can imagine what a better system might look like, as it would start with UBI. And we can start to think about delivering software resources by rail instead of the labor of countless individual developer "truck drivers". Some low-hanging fruit might be: maybe we need distributions that provide everything and the kitchen sink, then we run our software and a compiler strips out the unused code, rather than relying on luck to decide what we need before we've started like with Arch or Nix. We could explore demand-side economics, where humans no longer install software, but dependencies are met internally on the fly, so no more include paths or headers to babysit (how early programming languages worked before C++ imposed headers onto us). We could use declarative programming more instead of brittle imperative (hard coded) techniques. We could filter data through stateless self-contained code modules communicating via FIFO streams like Unix executables. We could use more #nocode approaches borrowed from FileMaker, MS Access and Airtable (or something like it). We could write software from least-privileges, asking the user for permission to access files or the networks outside the module's memory space, and then curate known-good permissions policies instead of reinventing the wheel for every single program. We could (will) write test-driven software where we design the spec as a series of tests and then AI writes the business logic until all of the tests pass. There's a lot here to unpack here and a wealth of experience available from older developers. But I sympathize with the cognitive dissonance, as that's how I feel every single day witnessing the frantic yak shaving of "modern" programming while having to suppress my desire to use these other proven techniques. Because there's simply no time to do so under the current status quo where FAANG has quite literally all of the trillions of dollars as the winners, so decides best practices while the open source community subsists on scraps in their parent's basement hoping to make rent someday.
- zimmund 3y ago> Not to mention using `--<CMD>` If you read further down in the documentation, you can just prefix your command with `jc` (e.g. `jc ls`). The `--cmd` param is actually a good idea, since it allows you to mangle the data before converting it (e.g. you want to grep a list before converting it). Regarding maintenance, most of the basic unix commands' output shouldn't change too much (they'd be breaking not only this tool but a lot of scripts). I wouldn't expect it to break as often as you imagine, at least not because of other binaries being updated.