5 ms·

Why would you not expect diminishing returns? Surely that’s the default position of all AI non-experts?

TwoPhonesOneKid 2y ago
I thought we were supposed to see AI managing costs reductions by now. Instead I just see chatbots and frantic PMs looking to prove their utility.
iLoveOncall 2y ago
I don't know why people would downvote you. ChatGPT was released in 2022! It doesn't feel like that, but it's been out for a long time and we've only seen marginal improvements since, and the wider public has simply not seen ANY improvement. It's obvious that the technology has hit a brick wall and the farce which is to spend double the tokens to first come up with a plan and call that "reasoning" has not moved the needle either. I build systems with GenAI at work daily in a FAANG, I use LLMs in the real world, not in benchmarks. There hasn't been any improvement since ChatGPT first release and equivalent models. We haven't even bothered upgrading to newer models because our evals show they don't perform better at all.
- nyolfen 2y ago
  [flagged]
- KaoruAoiShiho 2y ago
  That's obviously untrue... Yes it hasn't hit the usable threshold for human replacement for many tasks, the majority of tasks, but that doesn't mean it hasn't improved massively for many others. Your evals seem to just be not fine enough, probably you're just waiting for agi. I do evals on a ton of tasks and my API/chatgpt usage has shot up drastically. It's now irreplaceable tool for me.
- eru 2y ago
  > It's obvious that the technology has hit a brick wall and the farce which is to spend double the tokens to first come up with a plan and call that "reasoning" has not moved the needle either. If nothing else, that technique has cut down drastically on hallucinations.
  - TwoPhonesOneKid 2y ago
    "Hallucinations" (ie a chatbot blatantly lying) have always struck me as a skill issue with bad prompting. Has this changed recently? to a skilled user of a model, the model won't just make shit up. Chatbots will of course answer unanswerable questions because they're still software. But why are you paying attention to software when you have the whole internet available to you? Are you dumb? You must be if you aren't on wikipedia right now. It's empowering to admit this. Say it with me: "i am so dumb wikipedia has no draw to me". If you can say this with a straight face, you're now equipped with everything you need to be a venture capitalist. You are now an employee of Y Combinator. Congratulations. Sometimes you have to admit the questions you're asking are unlikely to be answered by the core training documents and you'll get garbled responses. confabulations. Adjust your queries accordingly. This is the answer to 99% of issues product engineers have with llms. If you're regularly hitting random bullshit you're prompting it wrong. Models will only yield results if they get prompts they're already familiar with. Find a better model or ask better questions. Of course, none of this is news to people who actually, regularly talk to other humans. This is just normal behavior. Hey maybe if you hit the software more it'll respond kindly! Too bad you can't abuse a model.
    - eru 2y ago
      A sufficiently skilled person can use a warped and bent slide rule to design a jet engine. But that doesn't mean the warped slide rule and a super computer capable of finite element analysis are equally useful or powerful.
      - TwoPhonesOneKid 2y ago
        You can easily crush a person with a super computer. But what are you going to do with a slide rule—lightly tap someone?
        eru 2y ago
        You are lacking in imagination. (Alas, Dall-E also lacks, so I couldn't generate a picture of deadly slide rule kung fu. At least none that wasn't unintentionally hilarious.)
- dibujaron 2y ago
  I also work with genAI daily. to say there's no improvement between gpt3 and gpt4 and gpt4o is massively incorrect. the improvement in the latter two was measurable and obvious. Just the speed of answer alone is improved by an order of magnitude.
- anon373839 2y ago
  The boundaries of LLMs’ capabilities are really weird and unpredictable. I was doing a very basic info extraction task a few days ago and none of Gemini 2.0 Flash, Llama 3.3 70B, or GPT-4o could do it reliably without going off the rails. With the same prompt, I switched to the open-weights Gemma 2 27B - released last spring - and it nailed it.
  - al_hag 2y ago
    Is this something that structured propmpts a la DSPy are designed to address?
- JohnKemeny 2y ago
  GPT 2 has been out for more than 6 years.

I thought we were supposed to see AI managing costs reductions by now. Instead I just see chatbots and frantic PMs looking to prove their utility.

iLoveOncall 2y ago

I don't know why people would downvote you. ChatGPT was released in 2022! It doesn't feel like that, but it's been out for a long time and we've only seen marginal improvements since, and the wider public has simply not seen ANY improvement. It's obvious that the technology has hit a brick wall and the farce which is to spend double the tokens to first come up with a plan and call that "reasoning" has not moved the needle either. I build systems with GenAI at work daily in a FAANG, I use LLMs in the real world, not in benchmarks. There hasn't been any improvement since ChatGPT first release and equivalent models. We haven't even bothered upgrading to newer models because our evals show they don't perform better at all.

nyolfen 2y ago
[flagged]
KaoruAoiShiho 2y ago
That's obviously untrue... Yes it hasn't hit the usable threshold for human replacement for many tasks, the majority of tasks, but that doesn't mean it hasn't improved massively for many others. Your evals seem to just be not fine enough, probably you're just waiting for agi. I do evals on a ton of tasks and my API/chatgpt usage has shot up drastically. It's now irreplaceable tool for me.
eru 2y ago
> It's obvious that the technology has hit a brick wall and the farce which is to spend double the tokens to first come up with a plan and call that "reasoning" has not moved the needle either. If nothing else, that technique has cut down drastically on hallucinations.
- TwoPhonesOneKid 2y ago
  "Hallucinations" (ie a chatbot blatantly lying) have always struck me as a skill issue with bad prompting. Has this changed recently? to a skilled user of a model, the model won't just make shit up. Chatbots will of course answer unanswerable questions because they're still software. But why are you paying attention to software when you have the whole internet available to you? Are you dumb? You must be if you aren't on wikipedia right now. It's empowering to admit this. Say it with me: "i am so dumb wikipedia has no draw to me". If you can say this with a straight face, you're now equipped with everything you need to be a venture capitalist. You are now an employee of Y Combinator. Congratulations. Sometimes you have to admit the questions you're asking are unlikely to be answered by the core training documents and you'll get garbled responses. confabulations. Adjust your queries accordingly. This is the answer to 99% of issues product engineers have with llms. If you're regularly hitting random bullshit you're prompting it wrong. Models will only yield results if they get prompts they're already familiar with. Find a better model or ask better questions. Of course, none of this is news to people who actually, regularly talk to other humans. This is just normal behavior. Hey maybe if you hit the software more it'll respond kindly! Too bad you can't abuse a model.
  - eru 2y ago
    A sufficiently skilled person can use a warped and bent slide rule to design a jet engine. But that doesn't mean the warped slide rule and a super computer capable of finite element analysis are equally useful or powerful.
    - TwoPhonesOneKid 2y ago
      You can easily crush a person with a super computer. But what are you going to do with a slide rule—lightly tap someone?
      - eru 2y ago
        You are lacking in imagination. (Alas, Dall-E also lacks, so I couldn't generate a picture of deadly slide rule kung fu. At least none that wasn't unintentionally hilarious.)
dibujaron 2y ago
I also work with genAI daily. to say there's no improvement between gpt3 and gpt4 and gpt4o is massively incorrect. the improvement in the latter two was measurable and obvious. Just the speed of answer alone is improved by an order of magnitude.
anon373839 2y ago
The boundaries of LLMs’ capabilities are really weird and unpredictable. I was doing a very basic info extraction task a few days ago and none of Gemini 2.0 Flash, Llama 3.3 70B, or GPT-4o could do it reliably without going off the rails. With the same prompt, I switched to the open-weights Gemma 2 27B - released last spring - and it nailed it.
- al_hag 2y ago
  Is this something that structured propmpts a la DSPy are designed to address?
JohnKemeny 2y ago
GPT 2 has been out for more than 6 years.