3 ms·

Ah, sorry, totally didn't get that it was using them as users of the API. I would recommend API fuzzers here, which cycle through all available possibilities,

Ah, sorry, totally didn't get that it was using them as users of the API. I would recommend API fuzzers here, which cycle through all available possibilities, and generate 'garbage' inputs as well as structured inputs regarding the specifications the APIs implement (http, and the underlying structures provided to the api endpoints). an LLM would likely not perform an exhaustive test. There are several projects freely available to run such fuzz tests. Eventhough that does eat up considerable resources, and 'resetting' state during such tests can be problematic at best, the same would apply for LLM based testing. (database gets screwed in some way, and the next request is not on a 'clean' state.)

stavros 2y ago
> Ah, sorry, totally didn't get that it was using them as users of the API. Originally, it wasn't. From the GP post: > If there is a common hallucination API call, just add that API. This will make the API more "friendly" and resemble common similar APIs. If it's just users, you wouldn't be adding the APIs.
- IanCal 2y ago
  Users were calling their API.
  - sim7c00 2y ago
    programs call apis.
    - IanCal 2y ago
      This is the silliest argument I write API calls. I am a user of apis. If you want to be more pedantic than this I simply do not care and you should evaluate what benefit you are bringing to this conversation.
      - sim7c00 2y ago
        Silliness can also be beneficial :D. You are a user who 'writes code', and that code 'calls'. or are you using your vocal chords to magically retreive data from an API -> it could happen if you use voice-to-text and have an LLM make your code :'D. - sorry if the comment was too silly. i'll try to keep it for myself.
IanCal 2y ago
I think you're picturing this differently again. It's not testing the API for bugs, it's asking whether the API you have follows typical norms and isn't missing things. Is your API confusing or incomplete? Imagine you have a library dealing with chat rooms, and LLMs keep trying to call "room.getUser(id)" and "room.getUsers()" when your API is actually "room.user(filter:{id:id})[0]" and "room.user(filter:None)". Maybe that's a sign that your API, while totally functional is going to be confusing to other programmers. Maybe you don't have anything that lists all current users, or there's a flag or option somewhere that would make sense or be more typical. A fuzzer won't tell you that. For the original post, you could imagine that a user might want the price of a token in a different currency, but maybe it currently returns everything in dollars. An LLM might try and add ?currency=GBP . Maybe an LLM keeps expecting to see in the response data the last updated timestamp to check data freshness, but it isn't something you add right now.
- sim7c00 2y ago
  fair point again. i dont think this is the way to go in either case. of your api is easy to clearly document its good. if you struggle with that, a programmer trying to call it will also. regardless of what an LLM makes of it. it seems like trying to facilitate bad things. make good things instead.
  - IanCal 2y ago
    > of your api is easy to clearly document its good If it breaks a lot of conventions or norms, no it's probably not good. But also the point here is *how much do you need to really explain?". If you have apis that require a lot of re-explanation to llms to force them to do the right thing "no getUsers doesn't exist use users(filter) where... YOU MUST USE users(filter)!" That's a sign your API might be going counter to a broad estimation of existing code. > it seems like trying to facilitate bad things. make good things instead. I have seen so many people make something that is obvious to them and so few others that "simply write good things from the start" is pointless advice.
    - sim7c00 2y ago

stavros 2y ago

> Ah, sorry, totally didn't get that it was using them as users of the API. Originally, it wasn't. From the GP post: > If there is a common hallucination API call, just add that API. This will make the API more "friendly" and resemble common similar APIs. If it's just users, you wouldn't be adding the APIs.

IanCal 2y ago
Users were calling their API.
- sim7c00 2y ago
  programs call apis.
  - IanCal 2y ago
    This is the silliest argument I write API calls. I am a user of apis. If you want to be more pedantic than this I simply do not care and you should evaluate what benefit you are bringing to this conversation.
    - sim7c00 2y ago
      Silliness can also be beneficial :D. You are a user who 'writes code', and that code 'calls'. or are you using your vocal chords to magically retreive data from an API -> it could happen if you use voice-to-text and have an LLM make your code :'D. - sorry if the comment was too silly. i'll try to keep it for myself.

IanCal 2y ago

I think you're picturing this differently again. It's not testing the API for bugs, it's asking whether the API you have follows typical norms and isn't missing things. Is your API confusing or incomplete? Imagine you have a library dealing with chat rooms, and LLMs keep trying to call "room.getUser(id)" and "room.getUsers()" when your API is actually "room.user(filter:{id:id})[0]" and "room.user(filter:None)". Maybe that's a sign that your API, while totally functional is going to be confusing to other programmers. Maybe you don't have anything that lists all current users, or there's a flag or option somewhere that would make sense or be more typical. A fuzzer won't tell you that. For the original post, you could imagine that a user might want the price of a token in a different currency, but maybe it currently returns everything in dollars. An LLM might try and add ?currency=GBP . Maybe an LLM keeps expecting to see in the response data the last updated timestamp to check data freshness, but it isn't something you add right now.

sim7c00 2y ago
fair point again. i dont think this is the way to go in either case. of your api is easy to clearly document its good. if you struggle with that, a programmer trying to call it will also. regardless of what an LLM makes of it. it seems like trying to facilitate bad things. make good things instead.
- IanCal 2y ago
  > of your api is easy to clearly document its good If it breaks a lot of conventions or norms, no it's probably not good. But also the point here is *how much do you need to really explain?". If you have apis that require a lot of re-explanation to llms to force them to do the right thing "no getUsers doesn't exist use users(filter) where... YOU MUST USE users(filter)!" That's a sign your API might be going counter to a broad estimation of existing code. > it seems like trying to facilitate bad things. make good things instead. I have seen so many people make something that is obvious to them and so few others that "simply write good things from the start" is pointless advice.
  - sim7c00 2y ago