3 ms·

I'm using the latest top paid models, gpt4o and claude 3.5. They work when there's a lot of examples on github or google, but once you get into something that

I'm using the latest top paid models, gpt4o and claude 3.5. They work when there's a lot of examples on github or google, but once you get into something that doesn't have a lot of examples like closed source code or rarely used libraries, it will start hallucinating and even mixing up different API versions to create a mess that doesn't work at all. I don't believe LLMs will get any better than this without a new major breakthrough, but this is already better than using Google search.

anon291 2y ago
> don't believe LLMs will get any better than this without a new major breakthrough, but this is already better than using Google search I mean .. every single org is invested in that research right now
- yoyohello13 2y ago
  Money helps, but doesn’t guarantee breakthroughs.
  - anon291 2y ago
    The sort of transformation needed here seems achievable. There are various roadmaps to success whereas there were not any roadmaps to general language agents just a few years ago.
    - codexon 2y ago
      I don't see how it seems achievable. Backprop was popularized in 1986. Transformers came out in 2017. It could be >10-20 years before the next breakthrough comes. Given all the progress so far, these things still don't seem to be learning any logic.
- majewsky 2y ago
  True, if by "every single org" you mean "maybe a dozen orgs". All that the vast majority of companies is doing is figuring out how to run an existing LLM or API thereof and shipping an RAG integration button for their application.
siodine 2y ago
I had the same problem with using the latest neo4j API and Sonnet 3.5 -- except it wasn't really a problem. I just created a project where I explain that its knowledge on neo4j is outdated and to instead use an API reference and changelog that I add to the project. It's not magic, you need to think what you would need in a similar situation and then provide that to the LLM. It definitely does suffer from severe overconfidence, though -- if you were to think of it as a person. Also, you need to break up your project into manageable portions and provide context to the other portions (without providing the entirety of them) for it to effectively work on the portion you want to work on.

anon291 2y ago

> don't believe LLMs will get any better than this without a new major breakthrough, but this is already better than using Google search I mean .. every single org is invested in that research right now

yoyohello13 2y ago
Money helps, but doesn’t guarantee breakthroughs.
- anon291 2y ago
  The sort of transformation needed here seems achievable. There are various roadmaps to success whereas there were not any roadmaps to general language agents just a few years ago.
  - codexon 2y ago
    I don't see how it seems achievable. Backprop was popularized in 1986. Transformers came out in 2017. It could be >10-20 years before the next breakthrough comes. Given all the progress so far, these things still don't seem to be learning any logic.
majewsky 2y ago
True, if by "every single org" you mean "maybe a dozen orgs". All that the vast majority of companies is doing is figuring out how to run an existing LLM or API thereof and shipping an RAG integration button for their application.

siodine 2y ago

I had the same problem with using the latest neo4j API and Sonnet 3.5 -- except it wasn't really a problem. I just created a project where I explain that its knowledge on neo4j is outdated and to instead use an API reference and changelog that I add to the project. It's not magic, you need to think what you would need in a similar situation and then provide that to the LLM. It definitely does suffer from severe overconfidence, though -- if you were to think of it as a person. Also, you need to break up your project into manageable portions and provide context to the other portions (without providing the entirety of them) for it to effectively work on the portion you want to work on.