7 ms·

It’s fascinating how little people really understand how LLMs are impacting businesses. Almost any task that couldn’t be done because it was too expensive to hi

• jncfhnb 3y ago
  Like what
  • Gigachad 3y ago
    Just imagining some random ideas here and the use case come instantly. Imagine you have a user feedback system that has thousands of submisisons and you want to know how many are talking about a particular feature. How many are speaking positively about the product, how many seem to be asking for support rather than asking for feedback. Being able to sort through thousands of items for basically free which would have been so laborious previously that you would not have done it. And its a use where a small amount of error does not matter.
  • ikekkdcjkfke 3y ago
    Construction business
  • hn_throwaway_99 3y ago
    Lots of government contractors have junior analysts-type jobs whose work entails responding to government RFPs. I know of one specifically who found ChatGPT was great at these responses - it still took further human review and editing, but so did the initial human drafts. Just one example.
    • berkes 3y ago
      How big is the market of "junior analyst-type jobs" globally? Will disrupting or obliterating that industry make an impact on our lives? Will it add anything to the bottom line of OpenAI?
      • hn_throwaway_99 3y ago
        > How big is the market of "junior analyst-type jobs" globally? Hence the "Just one example" sentence I ended my comment with. There are tons and tons of others. The Washington Post did a whole long article months ago about people who were laid off _already_ due to ChatGPT. Folks like marketers, copywriters, data analysts, medical transcriptionists, etc. There are a large number of jobs out there that used to need, say, 3 or 4 people, where AI could bring that down to 1.
• hn_throwaway_99 3y ago
  Agreed. I thought the analysis in this article was exceptionally poor. For example, there is a small section handwaving away the potential impact on the search engine market, saying that ChatGPT has been out a year and hasn't made a sizable dent in Google. But the thing is, the way ChatGPT interface currently works and the way it is marketed, most people don't even think of it as a "Google replacement" in the first place - except perhaps some techies and folks on HN. But it's not hard to o imagine advancing tech (so LLMs can be more frequently updated) and a rethought UI (and I'm not just talking about plugging in some "ChatGPT-in-Bing"-type interface, either) that could radically change where people go when they want information.
  • sumedh 3y ago
    > there is a small section handwaving away the potential impact on the search engine market I would assume lot of IT folks might have moved to ChatGPT to ask IT (programming) questions instead of searching Google and lot of these folks also use Adblockers which blocked statcounter's trackers so the stats might be misleading by a bit.
  • Shawnj2 3y ago
    That’s also not considering people who “ask ChatGPT things” instead of googling and don’t consider it the same as search
• corethree 3y ago
  A lot of people are just in denial. It's some psychological mechanism. They have to call it a stochastic parrot, they have to think of the LLM as something that's only a toy at worst and at best equivalent to google search. They're wrong. This thing can code too. It won't replace a human coder yet, but it's part way there. It's only a matter of time before it's fully there.
  • otabdeveloper4 3y ago
    It won't be fully there until it gives a confidence interval and has weights to manipulate precision and recall. These are baseline non-negotiable features. Unfortunately the way LLM's are architected right now means they will never be "fully there".
    • ben_w 3y ago
      Why does it need things that are absent in humans? Sure, those things could make some things easier, but why are those things that humans don't have suddenly "non negotiable" when a machine also can't do them? Legal issues?
      • otabdeveloper4 3y ago
        Those things aren't absent in humans. We spend trillions of dollars to make sure humans are qualified and aren't just bullshitting plausible-sounding words. (In fact, that's really the whole point of "education", which some people spend 20 years being subjected to.)
        • ben_w 3y ago
          > confidence interval and has weights to manipulate precision and recall Neither of which are supplied by education, and I am unaware of any humans basing able to deliberately (let alone precisely) alter their own perception on the scale between precision and recall — look at something, it's "obviously" X or not X almost immediately after you know what the category X is, even when you're wrong. Apart from the very first few encounters it doesn't even matter how much of a noob or expert you are in the field of X-recognition, your confidence is the same. Worse: > qualified Given how well ChatGPT does on standardised tests, doing better than many actual humans even despite its many flaws and limitations, it should be clear that the qualifications are not good enough to do what you're expecting them to do. > and aren't just bullshitting plausible-sounding words. That's demonstrably how humans work (at least when it can be tested, perhaps people who need split brains are weird): all the indications are we do a thing first and then come up with a justification after. (And then we have people like Boris Johnson, 2:1 BA from Oxford, with a disconnect between reality and the words leaving his mouth that would be comical except he actually became Prime Minister in real life and not just a TV comedy blending 'Allo 'Allo with The Thick of It).
• otabdeveloper4 3y ago
  Somewhat of a hot take. Normal classification algorithms output a confidence weight, LLM's don't. This means you can only use LLM classification if you don't care at all about managing your precision and recall metrics. This means LLMs are useless for 90 percent of real business requirements. (Business actually wants some sort of binomial regressor, not a plausible-looking hallucination. Plausible-looking hallucinations are the domain of spam and copywriting.)
  • simbolit 3y ago
    Auto Label can do it even with closed-source LLMs: https://www.refuel.ai/blog-posts/labeling-with-confidence https://www.refuel.ai/blog-posts/labeling-with-confidence Also, your 90% number is a 'plausible-looking hallucination', which is mildly funny.
    • unraveller 3y ago
      >Explicitly prompting the LLM to output a confidence score is the least accurate They don't show the whole prompt they use for this so I am sceptical of the comparison that recommends their service which does the token level accuracy. In my short testing you can drill down some acceptable accuracy for specific domains if you lead it over some general "ask yourself" questions then "finally, output a % confidence"
      • simbolit 3y ago
        I couldn't care less about the specific implementation. My parent comment said "LLMs are useless for classification, because they can't give confidence scores". I think i have shown that to be demonstrably false.
        • fnordpiglet 3y ago
          My understanding is these are more “confidence in the tokens selected being appropriate” rather than the more fundamental “confidence the classification chosen is accurate given training data,” where training data is samples of the distribution being classified rather than language corpus. They seem similar at first blush but the first is more about appropriateness of phrasing and how esoteric the semantics are, and the latter more descriptive of some statistically observable process - which is something LLMs are literally incapable of reasoning about directly.
• jl6 3y ago
  From a personal perspective… I have a family photo collection going back decades that I used to put effort into tagging and describing, before realizing that automatic classification models would come eventually. After years of waiting, it feels like we’re close, but I’ve tried LLaVA-1.5 and it was underwhelming. Are there other more specialist models I should try? Must be open source and easy to invoke from a shell or Python script.
  • Philpax 3y ago
    Try Fuyu-8B or CogVLM. Failing that, there’s always the GPT-4V API. There are also other non-LLM solutions for image captioning / tagging (e.g. what the smartphone vendors do), but I’m less familiar with those.
    • stevenhuang 3y ago
      +1 on CogVLM. https://github.com/THUDM/CogVLM https://github.com/THUDM/CogVLM Recent discussions: https://github.com/ggerganov/llama.cpp/discussions/4350 https://github.com/ggerganov/llama.cpp/discussions/4350 It is the best open source vision language model out there that I'm aware of that's most comparable to gpt4v. Beats the pants off Llava1.5 and variants like bakllava. There's a demo here http://36.103.203.44:7861/ http://36.103.203.44:7861/
      • em-bee 3y ago
        i have been watching this subthread in hope that something interesting might come out of it and you just delivered. this demo is amazing. i am now dying to apply this to my photo collection. on that note, some time ago i came across an image management tool that uses machine learning in some form. i don't have time to look for it now, but i just wanted to mention that someone is working on something like that.
• RandomLensman 3y ago
  As you have seen it in action: how much more additional profit is in being able to do those task now? How much RoE increase is there? (For the megacorps).
  • fnordpiglet 3y ago
    In some cases as much as hundreds of millions of profit, in some cases its existential- one situation they needed to spin up a complex business process that would have required an entire organization of likely thousands of humans to comply with a regulatory requirement they were found in violation of and the only way to comply was to have humans review millions of documents and web sites manually to classify them. With a fair amount of fine tuning they were able to build an LLM infrastructure that did this role with the backing of a 20 person review and escalation team. They couldn’t have spun up the equivalent traditional org on time and would have had to had a potentially ruinous public regulatory issue. The cost savings on one hand is hard to gauge, particularly because it would have greatly impacted their culture and organization to add so many humans so quickly. Anyway it’s amazing how many ways these things can be used.
    • RandomLensman 3y ago
      Thank you! So it is rather an incremental change in profit, not transformational (when it isn't life or death of the corporation/a business model requirement)?
      • fnordpiglet 3y ago
        I think I would say this. It’s a technology that’s on radars for about 12 months, is immature in all aspects, and has already transformed large swaths of the business operations to be much cheaper, scalable, and precise. I’ve never seen that in 30+ years. In some cases it actually is transformational as in the case I mentioned that it talked directly to a core business problem that would have killed the org. In others it’s incremental, but at the scale of megacorp there everything is incremental and something that moves margins by 5% or something is earth shattering for them. It unlocks businesses they would have been too hard to comply with requirements, it stream lines operations, etc, which for many businesses are monumental changes. What happens in 5 years when models are much more powerful, interfaces more clear, and ability to customize easier?
• bambax 3y ago
  Can you elaborate? I think you're right that LLMs are very good at classification and structuring, but that's not how I see them used. The experiments I have witnessed resemble "hail Mary" attempts to replace human agents with a magic chatbot that would know it all and speak plainly. Those experiments almost always fail, because the bot hallucinates too much and/or because the training data is poor and not detailed enough to provide efficient fine-tuning. For example: a helpdesk handles incoming support requests for an internal, proprietary application. Support agents write the request down, then spend time investigating the problem and possible solutions, based on the symptoms. There is a weak link between the symptoms reported by the users and the actual underlying problem, so this takes time, esp. for newly hired support agents. One approach would be to build a semantic search engine to speed up the investigation phase, which would positively affect the productivity of the agents. But that's not what's been tried: the company hoped to get rid of support agents completely, and replace them with a bot that would interact with the users and solve their problems on the fly. When that doesn't work, everyone's disappointed and tends to simply give up.
  • d4rkp4ttern 3y ago
    I agree with the approach of augment-not-replace human agents. For example there’s a contact center management company using Langroid[1] (the Multi-Agent LLM framework from ex-CMU/UW-Madison researchers) in exactly this way: they use a Langroid RAG agent to come up with top 3 possible answers, and the human agent then picks a good one. Interestingly, they index past successful conversations of experienced human agents, and the Langroid agent’s suggestions are informed from these as well as company docs. [1] https://GitHub.com/Langroid/Langroid https://GitHub.com/Langroid/Langroid
  • fnordpiglet 3y ago
    So, imagine you need to evaluate applications for some line of credit by reading the documents associated and classifying the businesses in a taxonomy of business types. Depending on volume this could be millions of documents. You can prime an LLM context with the taxonomy and examples of each, then evaluate each document in a clean primed context asking for a classification if it’s clear and a request for review if it’s not clear. This works exceptionally well in practice, almost double the precision and recall of humans. Businesses are replete with similar relatively trivial classification problems that often require armies of low paid low skill workers but are fairly crucial to the ongoing operations of the business. Chat bots are the default thing people gravitate to with LLMs because they’re the example in chatgpt. But that’s a really uncreative use of the tech, and the domain they’re exposed to is pretty open ended leaving them subject to hallucination, prompt injection, etc. A more constrained domain with more business impact is where I see meaningful applications happening.
• yetanother12345 3y ago
  >> I’ve seen this now at five major megacorps and I have to believe it’s going on at most. I sense a potential misaglignment with the article headline that: The industries AI is disrupting are not lucrative While "major megacorps" may indeed be casually using AI for eg "classification and structuring of complex stuff..." I have no problem understanding why your averge mom 'n pop store might not even have problems suitable for this particular type of number-crunching
  • yetanother12345 3y ago
    I think this article makes some interesting points, but it's not exactly on the money (pun intended) Quote: What are AIs of the GPT-4 generation best at? It’s things like: - writing essays or short fictions - digital art - chatting - programming assistance The author lists these points yet fails to realize that these four activities succintly sum up the major part of Internet-as-we-know-it. Add media management for streaming and we're basically there. Also: The issue is that taking the job of a human illustrator just. . . doesn’t make you much money. Because human illustrators don’t make much money! This is direcly seeing things from the wrong perspective. On average illustrators may make little money individually due to a huge over supply in their market, but still those that depend on their services would probably not agree to calling these services "cheap" or that this part of the budget was neglible. This is not about wage competition at the producer stage - it is about cost savings at the consumer stage. As for the question: How precisely will AI capture a portion of the $300 billion movie and game market? ...the author tries to brush it off referring to some strike at one particular place called Hollywood. Well, that market is larger than any particular place, and... it is evolving too. And, at the core of it is exactly the services that LLMs excel at, be it storytelling or hallucinating. Clarifying my point it seems as if the author makes the claim that extremely lucrative fields are in fact not. Last, I should add one particular multi-billion dollar industry in which AI seems to have quite some potential: Crime. Specifically, but not limited to, fraud.