3 ms·

One challenge is that specific APIs are already being made that aren’t compatible between models. Right now, I’m referring to function calling and tools. I susp

One challenge is that specific APIs are already being made that aren’t compatible between models. Right now, I’m referring to function calling and tools. I suspect that these are likely to create lock in as vendors move away from general completion APIs.

dragonwriter 3y ago
Sure, you need a local implementation of functional calling, etc., to implement those APIs, but once you do, there is no reason that you can't have a model-agnostic API implementation. Obviously, if there are things that depend on specific model capabilities (e.g., image interpretation or other multimodal use), you'll need a model that supports that, but there are multimodel open models. You'll need a way to recognize and error on model/capability combos that are invalid though.
- danenania 3y ago
  Are any OSS models producing JSON with specific keys as reliably as OpenAI models do with function calling? That would be needed to implement function calling.
  - dragonwriter 3y ago
    > Are any OSS models producing JSON with specific keys as reliably as OpenAI models do with function calling? Yes, OSS models -- from my understanding, this is true in theory of any model over which you have fine-grained control of inference, which you do if the inference code isn't a black box, but this has been generally implemented in the frameworks for running OSS models -- can be constrained to generate according to a grammar of arbitrary specificity.
    - danenania 3y ago
      Cool, that makes sense. Are there any now doing this off-the-shelf where a schema can be passed in and the response will be valid JSON that matches the schema?
      - dragonwriter 3y ago
        Tooling for this was announced on HN 3 months ago, it now does a lot more: https://news.ycombinator.com/item?id=37125118 https://news.ycombinator.com/item?id=37125118 Here is an plugin for one of the popular tools for running multiple local models (it is a tool that presents an OpenAI-style API for downstream consumption, as well as presenting its own Web UI for direct interaction with the models.): https://github.com/hallucinate-games/oobabooga-jsonformer-plugin https://github.com/hallucinate-games/oobabooga-jsonformer-pl...
- deepsquirrelnet 3y ago
  It’s not that you can’t add it, but that it seems the reason openai added “tools” after function calling is to make it generic for more unique/proprietary features. Unless everybody is going to follow OpenAIs lead for every feature they implement, general compatibility is going to be challenging. Basically, the only reason this is possible now is due to the fact that there isn’t yet a lot of diversity in the way models are being used.
- deleted 3y ago
  [deleted]

dragonwriter 3y ago

Sure, you need a local implementation of functional calling, etc., to implement those APIs, but once you do, there is no reason that you can't have a model-agnostic API implementation. Obviously, if there are things that depend on specific model capabilities (e.g., image interpretation or other multimodal use), you'll need a model that supports that, but there are multimodel open models. You'll need a way to recognize and error on model/capability combos that are invalid though.

danenania 3y ago
Are any OSS models producing JSON with specific keys as reliably as OpenAI models do with function calling? That would be needed to implement function calling.
- dragonwriter 3y ago
  > Are any OSS models producing JSON with specific keys as reliably as OpenAI models do with function calling? Yes, OSS models -- from my understanding, this is true in theory of any model over which you have fine-grained control of inference, which you do if the inference code isn't a black box, but this has been generally implemented in the frameworks for running OSS models -- can be constrained to generate according to a grammar of arbitrary specificity.
  - danenania 3y ago
    Cool, that makes sense. Are there any now doing this off-the-shelf where a schema can be passed in and the response will be valid JSON that matches the schema?
    - dragonwriter 3y ago
      Tooling for this was announced on HN 3 months ago, it now does a lot more: https://news.ycombinator.com/item?id=37125118 https://news.ycombinator.com/item?id=37125118 Here is an plugin for one of the popular tools for running multiple local models (it is a tool that presents an OpenAI-style API for downstream consumption, as well as presenting its own Web UI for direct interaction with the models.): https://github.com/hallucinate-games/oobabooga-jsonformer-plugin https://github.com/hallucinate-games/oobabooga-jsonformer-pl...
deepsquirrelnet 3y ago
It’s not that you can’t add it, but that it seems the reason openai added “tools” after function calling is to make it generic for more unique/proprietary features. Unless everybody is going to follow OpenAIs lead for every feature they implement, general compatibility is going to be challenging. Basically, the only reason this is possible now is due to the fact that there isn’t yet a lot of diversity in the way models are being used.
deleted 3y ago
[deleted]