3 ms·
I belive that LLMs will become better and better in the near future and pipelines will replace classic approaches with LLM-enriched pipelines will drastically s
by galgia 2y ago
I belive that LLMs will become better and better in the near future and pipelines will replace classic approaches with LLM-enriched pipelines will drastically simplify the ETL flows.
- Yoric 2y agoBut at what cost? We're in an energy/environmental crisis, and we're replacing simple pipelines with (unreliable) gas factories?
- galgia 2y agoLLMs are not the most efficient way to solve the problem, but they can solve it.
- Yoric 2y agoThey can do it, they're just slower, less reliable and orders of magnitude more energy-expensive. But yes, they're potentially easier to setup.
- danielbln 2y agoCost per token has cratered a thousand percent over the last two years, and that's not just lighting VC on fire, efficiency gains are made left and right.
- Yoric 2y agoHow much do we need to progress before it becomes comparable in terms of energy to the (often already rather energy-inefficient) data pipelines we've been using so far? Recall that while the cost per token may decrease, CoT multiplies the number of tokens by several orders of magnitude.
- benrutter 2y agoYou may be right! I guess we'll find out soon. One thing I'd be wary of is what "LLM-enriched pipelines" look like. If it's "write a sentence and get a pipeline" then I think that does massively simplify the ammount of work, but there's another reality where people use LLMs to get more features out of existing data, rather than doing the same transformations we do now. Under that one, ETL pipelines would end up taking more time, and being more complex.
- isaacremuant 2y agoNot that I don't love LLMs and play with them and their potential but if we don't get proper mechanism that ensure quality and consistency then it's not really a substitute for what we have. It's very easy to produce something that seemingly works but you can't attest to its quality. The problem is producing something resilient, that is easy to adapt and describes the domain of what you want to do. If all these things are so great, them why do I still need to do so many things to integrate a bigtech cloud agent with popular tool? Why is it so costly or limited? UX matters, validation matters, reliability matters, cost matters. You can't simply wish for a problem not to happen. Someone owns the troubleshooting and the modification and they need to understand the system they're trying to modify. Replacing scrapers with LLM is an easy and obvious thing, specially when you don't care about quality to a high degree. Other systems such as financial ones don't have that luxury.