6 ms·
This is one of the better use cases for LLMs, which have shown good capability at turning unstructured text into structured objects
by verdverm 3y ago
This is one of the better use cases for LLMs, which have shown good capability at turning unstructured text into structured objects
- ninkendo 3y agoIf LLM’s were local and cheap, sure. They’re just too heavyweight of a tool to use for simple CLI output manipulation today. I don’t want to send everything to the cloud (and pay a fee), and even if it was a local LLM, I don’t want it to eat all my RAM and battery to do simple text manipulation. In 20 years, assuming some semblance of moore’s law still holds for storage/RAM/gpu, I’m right there with you.
- d3nj4l 3y agoOn my M1 Pro/16GB RAM mac I get decently fast, fully local LLMs which are good enough to do this sort of thing. I use them in scripts all the time. Granted, I haven’t checked the impact on the battery life I get, but I definitely haven’t noticed any differences in my regular use.
- mosselman 3y agoWhich models do you run and how?
- _joel 3y agonot op, but this is handy https://lmstudio.ai/ https://lmstudio.ai/
- mosselman 3y agoThanks!
- verdverm 3y agohttps://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp is a popular local first approach. LLaMa is a good place to start, though I typically use a model from Vertex AI via API
- mosselman 3y agoThanks. I have llama.cpp locally. How do you use it in scripts? As in how do you specifically, not how would one.
- d3nj4l 3y agoI have ollama's server running, and I interact with it via the REST API. My preferred model right now is Intel's neural chat, but I'm going to experiment with a few more over the holidays.
- verdverm 3y agoI tried ollama today and it is super easy, finding good models is definitely going to be challenging. I tried a few on some sample (JSON) tasks and it is... frustrating... how they ellide or are unable to follow instructions. Is there a good fine-tuning workflow with ollama?
- d3nj4l 3y agoI haven’t tried any fine tuning so I can’t help there, sorry. Though I will say that neural chat has been pretty good to me, even though I have definitely observed it ignoring instructions at times, like a “here’s your json:” preamble in response to a query that specifically requests only json.
- d3nj4l 3y agoI use ollama (https://ollama.ai/ https://ollama.ai/) which supports most of the big new local models you might've heard of: llama2, mistral vicuna etc. Since I have 16GB of RAM, I stick to the 7b models.
- chongli 3y agoYeah, it would be much better if you could send a sample of the input and desired output and have the LLM write a highly optimized shell script for you, which you could then run locally on your multi-gigabyte log files or whatever.
- himinlomax 3y agoProblem: some crusty old tty command has dodgy output. Solution: throw a high end GPU with 24GB RAM and a million dollar of training at it. Yeah, great solution.
- verdverm 3y agoWith fine-tuning, you can get really good results on specific tasks that can run on regular cpu/mem. I'd suggest looking into the distillation research, where large model expertise can be transferred to much smaller models. Also, an LLM trained to be good at this task has many more applications than just turning command output into structured data. It's actually one of the most compelling business use cases for LLMs
- anonymous_sorry 3y agoThe complaint is less whether it would work, and more a question of taste. Obviously taste can be a personal thing. My opinions are my own and not those of the BBC, etc. You have a small C program that processes this data in memory, and dumps it to stdout in tabular text format. Rather than simplify by stripping out the problematic bit (the text output), you suggest adding a large, cutting-edge, hard to inspect and verify piece of technology that transforms that text through uncountable floating point operations back into differently-formatted UTF8. It might even work consistently (without you ever having 100% confidence it won't hallucinate at precisely the wrong moment). You can certainly see it being justified for one-off tasks that aren't worth automating. But to shove such byzantine inefficiency and complexity into an engineered system (rather than just modify the original program to give the format you want) offends my engineering sensibilities. Maybe I'm just getting old!
- verdverm 3y agoIf you can modify the original program, then that is by far the best way to go. More often than not, you cannot change the program, and in relation to the broader applicability, most unstructured content is not produced by programs.
- hnlmorg 3y agoAs someone who maintains a solution that solves similar problems to jc, I can assure you that you don’t need a LLM to parse most human readable output.
- verdverm 3y agoit's more about the maintenance cost, you don't have to write N parsers for M versions Maybe the best middle ground is to have an LLM write the parser. Lowers the development cost and runtime performance, in theory
- hnlmorg 3y agoYou don’t have to write dozens of parsers. I didn’t.
- verdverm 3y agoPart of the appeal is that people who don't know how to program or write parsers can use an LLM to solve their unstructured -> structured problem
- hnlmorg 3y agoSure. But we weren’t talking about non-programmers maintaining software.
- verdverm 3y ago> people who don't know how to program OR write parsers there are plenty of programmers who do not know how to write lexers, parsers, and grammars
- hnlmorg 3y agoWe are chatting about maintaining a software project written in a software programming language. Not some theoretical strawman argument youve just dreamt up because others have rightly pointed out that you don’t need a LLM to parse the output of a 20KB command line program. As I said before, I maintain a project like this. I also happen to work for a company that specialises in the use of generative AI. So I’m well aware of the power of LLMs as well as the problems of this very specific domain. The ideas you’ve expressed here are, at best, optimistic. by the time you’ve solved all the little quirks of ML you’ll have likely invested far more time on your LLM then you would have if you’d just written a simple parser and, ironically, needed someone far more specialised to write the LLM than your average developer. This simply isn’t a problem that needs a LLM chucked at it. You don’t even need to write lexers and grammars to parse 99% of application output. Again, I know this because I’ve written such software.
- keithalewis 3y agoGive a kid a hammer and he'll find something to fix.
- verdverm 3y agoWhat value does this comment add?
- otteromkram 3y agoI got a kick out of it. ¯\_(ツ)_/¯
- keithalewis 3y agoApproximately the same amount as the comment I replied to.
- verdverm 3y agoOne attempts to nudge a user towards the comment guidelines of HN (https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html) > Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes. > Comments should get more thoughtful and substantive, not less, as a topic gets more divisive. > Eschew flamebait. Avoid generic tangents. Omit internet tropes.