5 ms·

But isn't using LLM for that really expensive? Seems wasteful.

genewitch 2y ago
My desktop GPU can run small models at 185 tokens a second. Larger models with speculative decoding: 50t/s. With a small, finetuned model as the draft model, no, this won't take much power at all to run inference. Training, sure, but that's buy once cry once. Whether this means it's a good idea, I don't think so, but the energy usage for parsing isn't why.
- smokel 2y ago
  A simple text parser would probably be 10,000,000 times as fast. So the statement that this won't take much power at all, is a bit of an overstatement.
- Hercuros 2y ago
  It’s not just about the energy usage, but also purchase cost of the GPUs and opportunity cost of not using those GPUs for something more valuable (after you have bought them). Especially if you’re doing this at large scale and not just on a single desktop machine. Of course you were already saying it’s not a good idea, but I think the above definitely plays a role at scale as well.
  - SlightlyLeftPad 2y ago
    You’re right, I could be trying to get Crysis to run at 120 fps.
    - tmtvl 2y ago
      If you have spare GPU time you could donate it to projects like Folding@Home.
- boxed 2y ago
  50 tokens per second. Compared to a quick and dirty parser written in python or even a regex? That's going to be many many orders of magnitude slower+costlier.
  - anthk 2y ago
    awk would run millions times faster, not to mention mawk and awka.
- berkes 2y ago
  In order to make the point that > energy usage for parsing isn't why You'll need to provide actual figures and benchmark these against an actual parser. I've written parsers for larger-scale server stuff. And while I too don't have these benchmarks available, I'll dare to wager quite a lot that a dedicated parser for almost anything will outperform an LLM magnitudes. I won't be suprised if a parser written in rust uses upwards of 10k times less energy than the most efficient LLM setup today. Hell, even a sed/awk/bash monstrosity probably outperforms such an LLM hundreds of times, energy wise.
  - chgs 2y ago
    How many times would you need to parse to get an energy saving on using an lm to parse vs using an llm to write a parser, then using the parser to parse.
    - GTP 2y ago
      > using an llm to write a parser You're assuming OP needs an LLM to write a parser, since they mentions writing many during their career they probably don't need it ;)
      - berkes 2y ago
        I didn't use an LLM back then. But would totally do that today (copilot). Especially since the parser(s) I wrote were rather straightforward finite state machines with stream handling in front, parallel/async tooling around it, and at the core business logic (domain). Streaming, job/thread/mutex management, FSM are all solved and clear. And I'm convinced an LLM like copilot is very good at writing code for things that have been solved. The LLM, however, would get very much in the way in the domain/business layer. Because it hasn't got the statistical body of examples to handle our case. (Parsers I wrote were a.o.: IBAN, gps-trails, user-defined-calculations (simple math formulas), and a DSL to describe hierarchies. I wrote them in Ruby, PHP, rust and perl.)
      - chgs 2y ago
        I was thinking more of when a sufficiently advanced device would be able to “decide” the task would be worth using its own capabilities to write some code to tackle the problem rather than brute force. For small problems it’s not worthwhile, for large problems it is. It’s similar to choosing to manually do something vs automate it.
- anthk 2y ago
  My Atom n270 netbook with mawk and a few lines parsing the files with a simple regex will crush down your GPU+LLM's on both time and power usage.
genmon 2y ago
My assumption is that models are getting cheaper, fast. So you can build now with OpenAI/Anthropic/etc and swap it out for a local or hosted model in a year. This doesn't work for all use cases but data extraction is pretty safe. Treat it like a database query -- a slow but high availability and relatively cheap call.
- szundi 2y ago
  [dead]
- Cthulhu_ 2y ago
  While it will become cheaper, it will never be as fast / efficient as 'just' parsing the data the old-fashioned way. It feels like using AI to do computing things instead of writing code is just like when we moved to relatively inefficient web technology for front-ends, where we needed beefier systems to get the same performance as we used to have, or when cloud computing became a thing and efficiency / speed became a factor of credit card limit instead of code efficiency. Call me a luddite but I think as software developers we should do better, reduce waste, embrace mechanical sympathy, etc. Using AI to generate some code is fine - it's just the next step in code generators that I've been using throughout all my career IMO. But using AI to do tasks that can also be done 1000x more efficiently, like parsing / processing data, is going in the wrong direction.
  - relistan 2y ago
    I know this particular problem space well. AI is a reasonable solution. WHOIS records are intentionally made to be human readable and not be machine parseable without huge effort because so many people were scraping them. So the same registrar may return records in a huge range of text formats. You can write code to handle them all if you really want to, but if you are not doing it en masse, AI is going to probably be a cheaper solution. Example: https://github.com/weppos/whois https://github.com/weppos/whois is a very solid library for whois parsing but cannot handle all servers, as they say themselves. That has fifteen + years of work on it.
    - 2y ago
adrianmonk 2y ago
I wouldn't use LLMs, but if I did, I would try to get the LLM to write parser code instead. If it can convert from one format to another, then it can generate test cases for the parser. Then hopefully it can use those to iterate on parser code until it passes the tests. In a sense, asking it to automate the work isn't as straightforward as asking it to do the work. But if the approach does pan out, it might be easier overall since it's probably easier to deploy generated code to production (than deploying LLMs).
permo-w 2y ago
deepseek API costs are quite literally pennies per million tokens

My desktop GPU can run small models at 185 tokens a second. Larger models with speculative decoding: 50t/s. With a small, finetuned model as the draft model, no, this won't take much power at all to run inference. Training, sure, but that's buy once cry once. Whether this means it's a good idea, I don't think so, but the energy usage for parsing isn't why.

smokel 2y ago
A simple text parser would probably be 10,000,000 times as fast. So the statement that this won't take much power at all, is a bit of an overstatement.
Hercuros 2y ago
It’s not just about the energy usage, but also purchase cost of the GPUs and opportunity cost of not using those GPUs for something more valuable (after you have bought them). Especially if you’re doing this at large scale and not just on a single desktop machine. Of course you were already saying it’s not a good idea, but I think the above definitely plays a role at scale as well.
- SlightlyLeftPad 2y ago
  You’re right, I could be trying to get Crysis to run at 120 fps.
  - tmtvl 2y ago
    If you have spare GPU time you could donate it to projects like Folding@Home.
boxed 2y ago
50 tokens per second. Compared to a quick and dirty parser written in python or even a regex? That's going to be many many orders of magnitude slower+costlier.
- anthk 2y ago
  awk would run millions times faster, not to mention mawk and awka.
berkes 2y ago
In order to make the point that > energy usage for parsing isn't why You'll need to provide actual figures and benchmark these against an actual parser. I've written parsers for larger-scale server stuff. And while I too don't have these benchmarks available, I'll dare to wager quite a lot that a dedicated parser for almost anything will outperform an LLM magnitudes. I won't be suprised if a parser written in rust uses upwards of 10k times less energy than the most efficient LLM setup today. Hell, even a sed/awk/bash monstrosity probably outperforms such an LLM hundreds of times, energy wise.
- chgs 2y ago
  How many times would you need to parse to get an energy saving on using an lm to parse vs using an llm to write a parser, then using the parser to parse.
  - GTP 2y ago
    > using an llm to write a parser You're assuming OP needs an LLM to write a parser, since they mentions writing many during their career they probably don't need it ;)
    - berkes 2y ago
      I didn't use an LLM back then. But would totally do that today (copilot). Especially since the parser(s) I wrote were rather straightforward finite state machines with stream handling in front, parallel/async tooling around it, and at the core business logic (domain). Streaming, job/thread/mutex management, FSM are all solved and clear. And I'm convinced an LLM like copilot is very good at writing code for things that have been solved. The LLM, however, would get very much in the way in the domain/business layer. Because it hasn't got the statistical body of examples to handle our case. (Parsers I wrote were a.o.: IBAN, gps-trails, user-defined-calculations (simple math formulas), and a DSL to describe hierarchies. I wrote them in Ruby, PHP, rust and perl.)
    - chgs 2y ago
      I was thinking more of when a sufficiently advanced device would be able to “decide” the task would be worth using its own capabilities to write some code to tackle the problem rather than brute force. For small problems it’s not worthwhile, for large problems it is. It’s similar to choosing to manually do something vs automate it.
anthk 2y ago
My Atom n270 netbook with mawk and a few lines parsing the files with a simple regex will crush down your GPU+LLM's on both time and power usage.

genmon 2y ago

My assumption is that models are getting cheaper, fast. So you can build now with OpenAI/Anthropic/etc and swap it out for a local or hosted model in a year. This doesn't work for all use cases but data extraction is pretty safe. Treat it like a database query -- a slow but high availability and relatively cheap call.

szundi 2y ago
[dead]
Cthulhu_ 2y ago
While it will become cheaper, it will never be as fast / efficient as 'just' parsing the data the old-fashioned way. It feels like using AI to do computing things instead of writing code is just like when we moved to relatively inefficient web technology for front-ends, where we needed beefier systems to get the same performance as we used to have, or when cloud computing became a thing and efficiency / speed became a factor of credit card limit instead of code efficiency. Call me a luddite but I think as software developers we should do better, reduce waste, embrace mechanical sympathy, etc. Using AI to generate some code is fine - it's just the next step in code generators that I've been using throughout all my career IMO. But using AI to do tasks that can also be done 1000x more efficiently, like parsing / processing data, is going in the wrong direction.
- relistan 2y ago
  I know this particular problem space well. AI is a reasonable solution. WHOIS records are intentionally made to be human readable and not be machine parseable without huge effort because so many people were scraping them. So the same registrar may return records in a huge range of text formats. You can write code to handle them all if you really want to, but if you are not doing it en masse, AI is going to probably be a cheaper solution. Example: https://github.com/weppos/whois https://github.com/weppos/whois is a very solid library for whois parsing but cannot handle all servers, as they say themselves. That has fifteen + years of work on it.
  - 2y ago

adrianmonk 2y ago

I wouldn't use LLMs, but if I did, I would try to get the LLM to write parser code instead. If it can convert from one format to another, then it can generate test cases for the parser. Then hopefully it can use those to iterate on parser code until it passes the tests. In a sense, asking it to automate the work isn't as straightforward as asking it to do the work. But if the approach does pan out, it might be easier overall since it's probably easier to deploy generated code to production (than deploying LLMs).

permo-w 2y ago

deepseek API costs are quite literally pennies per million tokens