8 ms·

Actually chat gpt is quite good at understanding some kinds of wiggliness. I built a restful api and documented it in a readme.md file in the wiggliest of ways.

Actually chat gpt is quite good at understanding some kinds of wiggliness. I built a restful api and documented it in a readme.md file in the wiggliest of ways. I then asked chatgpt to turn the readme into a swagger spec and then give me a page that read the spec and gave me a nice doc page with api exercise tool. Both tasks it performed really well and saved me a whole bunch of time.

pcthrowaway 4y ago
I have some meticulous API docs I've written, which I tried to get ChatGPT to convert into swagger It failed spectacularly I wonder if it's because the API is quite large, and I had to paste in ~10 messages worth of API docs before I was finished. It kept repeating segments of the same routes/paths and wasn't able to provide anything cohesive or useful to me. Was your API pretty small? Or were your docs pretty concise?
- kybernetikos 4y ago
  Chatgpt has a token limit. If you exceeded it then it would have no way of delivering a good result because it would simply have dirtied what you said at first. My api was not huge, about 8 endpoints.
- ilaksh 4y ago
  It can accept about 4k tokens, maybe 3000 words or 3500. GPT-4 can now accept 8k or 32k. The 32k version is 8 times larger than the one you tried. And these advances have come in a matter of a few months. Over the next several years we should expect at least one, quite easily two or more orders of magnitude improvements. I don't believe that this stuff can necessarily get a million times smarter. But 10 times? 100? In a few months the memory increased by a factor of 8. Pretty quickly we are going to get to the point where we have to question the wisdom of every advanced primate having a platoon of supergeniuses at their disposal. Probably as soon as the hardware scales out, or we get large scale memristor systems or whatever the next thing is which will be 1000 times more performant and efficient. Without exaggeration. Within about 10 years.
  - ChatGTP 4y ago
    So people want to build a nuclear reactor on the moon, I think these things should probably live on the moon or better yet Mars. That should be the place for experiments like this. Lowery latency links back to Earth and first see how it goes. Also you don’t think there will be resource constraints at some stage ? It’s funny we yelled at people for Bitcoin but when it’s ChstGPT, it’s fine to run probably tens of thousands of GPUs? In the middle of a climate crisis ? Not good.
    - pcthrowaway 4y ago
      Personally I don't think AI tools energy usage are comparable to BTC yet. Also, with BTC it's literally burning it in an unproductive way for "improved security". It's like lighting a forest on fire to keep warm. All the AI tools combined, last I heard, aren't consuming 0.5% of the world's energy usage. And even if they were, it would be absolutely bonkers to argue we should keep doing that when there were alternatives that accomplished similar goals without the energy usage (proof of stake)
      - ChatGTP 4y ago
        So, there is money to be made from LLMs, there will be advertising injected into the models responses etc. It's really the early days, but there's no way energy consumption won't grow exponentially now there is potential for earning money.
execveat 4y ago
Yeah, but now ask it to write a program that uses this API and then let it debug problems which arise from the swagger spec (or the backend) having bugs. I don't think LLMs have any way of recognizing and dealing with bad input data. That is I don't think they can recognize when something that is supposed to work in a particular way doesn't and fixing it is completely out of your reach, but you still need to get things working (by introducing workarounds).
- sebzim4500 4y ago
  Have you tried it? If you copy the errors back into the chat I could imagine it working quite well. Certainly you can give it contradictory instructions and it makes a decent effort at following them.
  - execveat 4y ago
    Yes, I'm subscribed to poe.com and am playing with all public models. They all suck at debugging issues with no known answers (I'm talking about typical problems every software developer, DevOps or infosec person solves every day). You need a real ability to reason and preserve context beyond inherent context window somehow (we humans do it by keeping notes, writing emails, and filing JIRA tickets). So while this doesn't require full AGI and some form of AI might be able to do it this century, it won't be LLMs.
    - ilaksh 4y ago
      If you think that the average public LLM is equivalent to ChatGPT or GPT-4 then you are completely mistaken. By a factor of say 500-10000%.
      - execveat 4y ago
        poe.com is a web interface (by Quora) to multiple LLMs. Right now it's ChatGPT, GPT-4, Claude, Claude+ as well as Sage and Dragonfly.

pcthrowaway 4y ago

I have some meticulous API docs I've written, which I tried to get ChatGPT to convert into swagger It failed spectacularly I wonder if it's because the API is quite large, and I had to paste in ~10 messages worth of API docs before I was finished. It kept repeating segments of the same routes/paths and wasn't able to provide anything cohesive or useful to me. Was your API pretty small? Or were your docs pretty concise?

kybernetikos 4y ago
Chatgpt has a token limit. If you exceeded it then it would have no way of delivering a good result because it would simply have dirtied what you said at first. My api was not huge, about 8 endpoints.
ilaksh 4y ago
It can accept about 4k tokens, maybe 3000 words or 3500. GPT-4 can now accept 8k or 32k. The 32k version is 8 times larger than the one you tried. And these advances have come in a matter of a few months. Over the next several years we should expect at least one, quite easily two or more orders of magnitude improvements. I don't believe that this stuff can necessarily get a million times smarter. But 10 times? 100? In a few months the memory increased by a factor of 8. Pretty quickly we are going to get to the point where we have to question the wisdom of every advanced primate having a platoon of supergeniuses at their disposal. Probably as soon as the hardware scales out, or we get large scale memristor systems or whatever the next thing is which will be 1000 times more performant and efficient. Without exaggeration. Within about 10 years.
- ChatGTP 4y ago
  So people want to build a nuclear reactor on the moon, I think these things should probably live on the moon or better yet Mars. That should be the place for experiments like this. Lowery latency links back to Earth and first see how it goes. Also you don’t think there will be resource constraints at some stage ? It’s funny we yelled at people for Bitcoin but when it’s ChstGPT, it’s fine to run probably tens of thousands of GPUs? In the middle of a climate crisis ? Not good.
  - pcthrowaway 4y ago
    Personally I don't think AI tools energy usage are comparable to BTC yet. Also, with BTC it's literally burning it in an unproductive way for "improved security". It's like lighting a forest on fire to keep warm. All the AI tools combined, last I heard, aren't consuming 0.5% of the world's energy usage. And even if they were, it would be absolutely bonkers to argue we should keep doing that when there were alternatives that accomplished similar goals without the energy usage (proof of stake)
    - ChatGTP 4y ago
      So, there is money to be made from LLMs, there will be advertising injected into the models responses etc. It's really the early days, but there's no way energy consumption won't grow exponentially now there is potential for earning money.

execveat 4y ago

Yeah, but now ask it to write a program that uses this API and then let it debug problems which arise from the swagger spec (or the backend) having bugs. I don't think LLMs have any way of recognizing and dealing with bad input data. That is I don't think they can recognize when something that is supposed to work in a particular way doesn't and fixing it is completely out of your reach, but you still need to get things working (by introducing workarounds).

sebzim4500 4y ago
Have you tried it? If you copy the errors back into the chat I could imagine it working quite well. Certainly you can give it contradictory instructions and it makes a decent effort at following them.
- execveat 4y ago
  Yes, I'm subscribed to poe.com and am playing with all public models. They all suck at debugging issues with no known answers (I'm talking about typical problems every software developer, DevOps or infosec person solves every day). You need a real ability to reason and preserve context beyond inherent context window somehow (we humans do it by keeping notes, writing emails, and filing JIRA tickets). So while this doesn't require full AGI and some form of AI might be able to do it this century, it won't be LLMs.
  - ilaksh 4y ago
    If you think that the average public LLM is equivalent to ChatGPT or GPT-4 then you are completely mistaken. By a factor of say 500-10000%.
    - execveat 4y ago
      poe.com is a web interface (by Quora) to multiple LLMs. Right now it's ChatGPT, GPT-4, Claude, Claude+ as well as Sage and Dragonfly.