3 ms·

Unless you can define and measure expected business value and risk, walk away and let someone else spend money and take the risk on a mostly hype- and FOMO-driv

Unless you can define and measure expected business value and risk, walk away and let someone else spend money and take the risk on a mostly hype- and FOMO-driven fad. If LLMs ever actually deliver real business value we'll all have time to integrate them.

bruce511 2y ago
the problem with your position is the absolutness of it to such a wide set of input parameters. It's kinda like the folks saying Google is useless when there are clearly cases where it is useful. In the same way LLMs have enormous value and utility, but equally there are places it is not as useful. Simply dismissing it as "mostly hype" is to miss the potential value. For example, I've found it to be good at writing SQL queries. I can see how a human interface to a sql backend would make reading data much easier for my users. Equally I have a well documented data schema, and lots of canonical docs for the product. I'd like to integrate this specialised knowledge with the existing SQL knowledge. In that context I'm interested in actual answers to the original question. Is it even possible?
- gregjor 2y ago
  If you mean Google search, that had an immediate and obvious value from the day it went live. I didn’t call Google or anything else useless. I suggested defining and measuring expected benefits, costs, and risks. I would suggest that for any unproven technology, not just LLMs. We can revisit the graveyard of over-hyped technologies if you want, and catalog the failures and wasted money. Ride the wave if you like. Show the success stories of LLMs in terms of business value or innovation. Don’t just repeat the press releases and anecdotes. You won’t find many successes, though maybe they will come in time. > I’ve found [an LLM] good at writing SQL queries. I don’t mean snark, but how do you know it writes good queries? I will grant that it can write queries that execute, but do those give correct results? Will they look good on a large production database where performance starts to matter? When I played with LLMs writing SQL they worked well enough on toy schemas that resemble those found in tutorials. But faced with a real schema that requires joining tables and applying complex conditions they failed to produce usable SQL. However they did (with prodding) produce SQL that executed. It just gave the wrong results. That’s not a gain along any axis and could cost a lot of money for an organization blindly trusting such tools. LLMs can write “good” code for some constrained definition of “good.” If you also want correct, testable, maintainable, secure, etc. they don’t perform well enough. Not just my opinion, we’ve had time to study the output of coding LLMs and it’s not looking good.
  - uptownfunk 2y ago
    For me they provide a good jumping off point. I do find I still need to read it and tweak it. But it’s much easier than having to write the damn thing myself. I am not trying to be an engineer, I am trying to build a business and get an mvp off the ground. And if we get anywhere, then maybe we can have budget for real engineers.
    - gregjor 2y ago
      Good luck. Several of my clients have tried the same thing, using an LLM to build an MVP or something simple. In every case they’ve had to hire programmers and either start over or spend more money getting the AI-generated code to work than it would have cost to hire it out. Look out for security problems, because the bots certainly will. Please report your experience.
uptownfunk 2y ago
LLM (chatGPT) has mostly replaced 90% of Google search for me at this point. (Except browser, email, image, YouTube, google docs)
- gregjor 2y ago
  Until ChatGPT and related get inevitably enshittified with ads and other crap, gamed and polluted, as Google search has.
bloomingkales 2y ago
If LLMs ever actually deliver real business value Your typical LLM now days makes a lot of applications humans programmed thus far look silly. There will definitely be a denial phase, but entire companies and products just look literally silly compared to an llm.
- gregjor 2y ago
  Can you point to or describe any of those? The silly apps and the LLM-based non-silly versions? Almost unanimously CEOs/CTOs report no big gains in business value or measured productivity due to LLM adoption. The majority opinion seems to land at "We don't want to miss out, but we're not sure how to get any real benefit."
  - deleted 2y ago
    [deleted]

bruce511 2y ago

the problem with your position is the absolutness of it to such a wide set of input parameters. It's kinda like the folks saying Google is useless when there are clearly cases where it is useful. In the same way LLMs have enormous value and utility, but equally there are places it is not as useful. Simply dismissing it as "mostly hype" is to miss the potential value. For example, I've found it to be good at writing SQL queries. I can see how a human interface to a sql backend would make reading data much easier for my users. Equally I have a well documented data schema, and lots of canonical docs for the product. I'd like to integrate this specialised knowledge with the existing SQL knowledge. In that context I'm interested in actual answers to the original question. Is it even possible?

gregjor 2y ago
If you mean Google search, that had an immediate and obvious value from the day it went live. I didn’t call Google or anything else useless. I suggested defining and measuring expected benefits, costs, and risks. I would suggest that for any unproven technology, not just LLMs. We can revisit the graveyard of over-hyped technologies if you want, and catalog the failures and wasted money. Ride the wave if you like. Show the success stories of LLMs in terms of business value or innovation. Don’t just repeat the press releases and anecdotes. You won’t find many successes, though maybe they will come in time. > I’ve found [an LLM] good at writing SQL queries. I don’t mean snark, but how do you know it writes good queries? I will grant that it can write queries that execute, but do those give correct results? Will they look good on a large production database where performance starts to matter? When I played with LLMs writing SQL they worked well enough on toy schemas that resemble those found in tutorials. But faced with a real schema that requires joining tables and applying complex conditions they failed to produce usable SQL. However they did (with prodding) produce SQL that executed. It just gave the wrong results. That’s not a gain along any axis and could cost a lot of money for an organization blindly trusting such tools. LLMs can write “good” code for some constrained definition of “good.” If you also want correct, testable, maintainable, secure, etc. they don’t perform well enough. Not just my opinion, we’ve had time to study the output of coding LLMs and it’s not looking good.
- uptownfunk 2y ago
  For me they provide a good jumping off point. I do find I still need to read it and tweak it. But it’s much easier than having to write the damn thing myself. I am not trying to be an engineer, I am trying to build a business and get an mvp off the ground. And if we get anywhere, then maybe we can have budget for real engineers.
  - gregjor 2y ago
    Good luck. Several of my clients have tried the same thing, using an LLM to build an MVP or something simple. In every case they’ve had to hire programmers and either start over or spend more money getting the AI-generated code to work than it would have cost to hire it out. Look out for security problems, because the bots certainly will. Please report your experience.

uptownfunk 2y ago

LLM (chatGPT) has mostly replaced 90% of Google search for me at this point. (Except browser, email, image, YouTube, google docs)

gregjor 2y ago
Until ChatGPT and related get inevitably enshittified with ads and other crap, gamed and polluted, as Google search has.

bloomingkales 2y ago

If LLMs ever actually deliver real business value Your typical LLM now days makes a lot of applications humans programmed thus far look silly. There will definitely be a denial phase, but entire companies and products just look literally silly compared to an llm.

gregjor 2y ago
Can you point to or describe any of those? The silly apps and the LLM-based non-silly versions? Almost unanimously CEOs/CTOs report no big gains in business value or measured productivity due to LLM adoption. The majority opinion seems to land at "We don't want to miss out, but we're not sure how to get any real benefit."
- deleted 2y ago
  [deleted]