4 ms·

LLMs are having a moment in 2023 like self driving cars were having in 2015. Some really cool demos following a lot of hard work, too much hyperbolic speculatio

LLMs are having a moment in 2023 like self driving cars were having in 2015. Some really cool demos following a lot of hard work, too much hyperbolic speculation that mass real-world job-destroying deployments are right around the corner, not enough appreciation of how few commercial applications are ok with 99% (or even 99.9%) accurate solutions. Real value being created, but still requiring lots of human ingenuity and grunt work to unlock it. As a decent first-order metric - follow the ratio of companies getting money for using LLMs to do something, to companies getting money for providing LLMs and associated tooling to others. The bigger that ratio gets, the more real world impact LLMs are having.

famouswaffles 4y ago
almost nothing involving NLP requires solutions anywhere near that accuracy rate. I've seen the self driving comparisons a lot but they straight up make little sense. there's a reason microsoft's various copilot suites have already popped up (365, X, Bing). massive value to be gained already in the here and now.
- skepticATX 4y ago
  To play devil's advocate, we still have no idea how economically impactful the Copilot suites will be. I don't think this is the most likely outcome, but I can absolutely see a scenario in which these end up being minor features that are rarely used by the typical worker.
  - heyitsguay 4y ago
    I've messed with ChatGPT before but this weekend I tried to use it seriously to hack together a python demo in the computer vision space. I appreciate it and it's a way better rubber duck for me to talk my ideas through with and generate sample snippets, but hallucination is a problem, and the more intricate and customized the code being developed, the more prone it's been to misinterpretation. I'm getting some good code starting points, and talking the idea through step by step with a chatbot is really helping me clarify what i need to do, but I've still got API docs open for the libraries it uses because it likes to make up functions, including the core one on which the project logic hinges. (But that's ok, because now I know I just need to write a function that works that way and i can do that). Pretty cool and helpful! And I can imagine it getting better with GPT-4 or code-specific tooling. But it's generating value on the order of like.. many other SaaS offerings that have come onto the scene that try to ease pain points in coding workflows. Versus value of the sort that upends society and my entire way of life. A great new tool that I should learn about to make rote bits of my activities faster and easier, a story that's a bit more familiar in tech than some of the more breathless stories about AI make it sound.
- heyitsguay 4y ago
  For sure NLP writ large requires lower accuracy levels than AV, like how computer vision writ large has many applications that require lower accuracy levels than AV. And indeed over the past few years what we've seen play out in CV deployments in the real world is incremental gains in controlled environments built on gobs and gobs of application-specific engineering, vs breakout success steamrolling over industries with standardized turnkey solutions I say that as someone who works in the space and loves it tbc. And I fully expect to see some wild stuff make it IRL this decade in both NLP and CV, I just think the rubber hits the road a bit more slowly than pop social media discourse would have one think.
antibasilisk 4y ago
>As a decent first-order metric - follow the ratio of companies getting money for using LLMs to do something, to companies getting money for providing LLMs and associated tooling to others. The bigger that ratio gets, the more real world impact LLMs are having. This is a bad measure because the cost to specialize an LLM for a particular domain is so low, and the initial investment required to have the infrastructure so high, that it's going to lead to a radical centralization of technology. You will have maybe 5 companies crunching the entire world's data globally.

famouswaffles 4y ago

almost nothing involving NLP requires solutions anywhere near that accuracy rate. I've seen the self driving comparisons a lot but they straight up make little sense. there's a reason microsoft's various copilot suites have already popped up (365, X, Bing). massive value to be gained already in the here and now.

skepticATX 4y ago
To play devil's advocate, we still have no idea how economically impactful the Copilot suites will be. I don't think this is the most likely outcome, but I can absolutely see a scenario in which these end up being minor features that are rarely used by the typical worker.
- heyitsguay 4y ago
  I've messed with ChatGPT before but this weekend I tried to use it seriously to hack together a python demo in the computer vision space. I appreciate it and it's a way better rubber duck for me to talk my ideas through with and generate sample snippets, but hallucination is a problem, and the more intricate and customized the code being developed, the more prone it's been to misinterpretation. I'm getting some good code starting points, and talking the idea through step by step with a chatbot is really helping me clarify what i need to do, but I've still got API docs open for the libraries it uses because it likes to make up functions, including the core one on which the project logic hinges. (But that's ok, because now I know I just need to write a function that works that way and i can do that). Pretty cool and helpful! And I can imagine it getting better with GPT-4 or code-specific tooling. But it's generating value on the order of like.. many other SaaS offerings that have come onto the scene that try to ease pain points in coding workflows. Versus value of the sort that upends society and my entire way of life. A great new tool that I should learn about to make rote bits of my activities faster and easier, a story that's a bit more familiar in tech than some of the more breathless stories about AI make it sound.
heyitsguay 4y ago
For sure NLP writ large requires lower accuracy levels than AV, like how computer vision writ large has many applications that require lower accuracy levels than AV. And indeed over the past few years what we've seen play out in CV deployments in the real world is incremental gains in controlled environments built on gobs and gobs of application-specific engineering, vs breakout success steamrolling over industries with standardized turnkey solutions I say that as someone who works in the space and loves it tbc. And I fully expect to see some wild stuff make it IRL this decade in both NLP and CV, I just think the rubber hits the road a bit more slowly than pop social media discourse would have one think.

antibasilisk 4y ago

>As a decent first-order metric - follow the ratio of companies getting money for using LLMs to do something, to companies getting money for providing LLMs and associated tooling to others. The bigger that ratio gets, the more real world impact LLMs are having. This is a bad measure because the cost to specialize an LLM for a particular domain is so low, and the initial investment required to have the infrastructure so high, that it's going to lead to a radical centralization of technology. You will have maybe 5 companies crunching the entire world's data globally.