7 ms·

Ugh. Of course the Enterprise Architecture rears its ugly head here. Just here to say that you can quickly build a robust feature with only OpenAI's APIs, redi

Ugh. Of course the Enterprise Architecture rears its ugly head here. Just here to say that you can quickly build a robust feature with only OpenAI's APIs, redis, a text file for the prompt you parameterize (versioned), and a little bit of glue code (no LangChain). You can add instrumentation for observability around that like you would any other code. I would wager that most enterprise use cases don't need most of the tools listed in this article, and using them is complete overkill.

emmender 3y ago
how can they make a sale to enterprise IT without a giant architecture diagram ? The next thing you know, java may rear its ugly head as well just when we thought python eliminated it completely for these applications.
adamgordonbell 3y ago
Exactly! We are talking about calling an API here people. Maybe what is behind the API is magical and powerful seeming but it's just an API that takes some context, the tokens to generate and a temperature setting.
- tbrownaw 3y ago
  Part of that diagram is about fine-tuning / retraining. Part is about managing canned prompts, since those matter enough to have their own development cycle. Part is about caching, since the fancier models are very expensive. Part is about filtering the output to not upset people, which is built in to the hosted versions (currently called "AI safety"). Etc. Doing everything in that diagram is probably overkill for most uses. But using it as a starting point and trimming what you don't need will help a bit with triping over "oops I forgot to include that".
  - adamgordonbell 3y ago
    Makes sense. But caching, for instance, doesn't need to be it's own lib does it? I don't want 'semantic caching' I want to cache the exact same query and I can do that without being LLM specific: from joblib import Memory @memory.cache def call_chat_completion_api_cached(max_tokens, messages,temperature): ... I mean, I guess then I might want to store that somewhere central like redis and maybe slowly I need a specific cache tool. So I get your point. It's helpful to see the possibilities of approaching these problems. But it also does feel like an land grab of supporting libs and infrastructure.
jchanimal 3y ago
100% this. If you’re building apps that call LLMs, most of the magic is already in the model, and what you are concerned about is tracking inputs and outputs. A browser database and React is all I have needed for my LLM apps.
robbomacrae 3y ago
Sure you could and I bet there are a number of VC backed startups that do only this. But there's usually a lot of reasons why these architectures could be a useful reference. In my project, I host my own trained LLM and one of the cost efficiencies comes from being able to cache at every step along the way. Then there is a large private media hosting consideration. There is room for all sorts of setups and I kind of liked how the article mapped out some of the common paths.
- j45 3y ago
  Building flexibility in helps pivot easier.
satvikpendem 3y ago
Most enterprises likely won't be able to use OpenAI since they might have proprietary information, so setting up a good on-prem open source LLM can be necessary. In fact, I am doing so right now with my company.
- nonfamous 3y ago
  You can create your own instances of the OpenAI models in Azure to keep proprietary data with the bounds of your tenant there.
  - abrichr 3y ago
    Interesting, can you please provide a link to the relevant documentation? The closest I found is https://azure.microsoft.com/en-us/blog/chatgpt-is-now-available-in-azure-openai-service/ https://azure.microsoft.com/en-us/blog/chatgpt-is-now-availa... and https://learn.microsoft.com/en-us/azure/cognitive-services/openai/overview https://learn.microsoft.com/en-us/azure/cognitive-services/o... but wasn't able to find anything about the data provisioning terms you mention.
    - rpeden 3y ago
      This might be what you're looking for: https://learn.microsoft.com/en-us/legal/cognitive-services/openai/data-privacy https://learn.microsoft.com/en-us/legal/cognitive-services/o...
      - abrichr 3y ago
        Thank you! I believe you may be referring to this: > Fine-tuned OpenAI models. The Fine-tunes API allows customers to create their own fine-tuned version of the OpenAI models based on the training data that they have uploaded to the service via the Files APIs. The trained fine-tuned models are stored in Azure Storage in the same region, encrypted at rest and logically isolated with their Azure subscription and API credentials. Fine-tuned models can be deleted by the user by calling the DELETE API operation. My follow-up question is: how is this different from OpenAI's finetuning?
swyx 3y ago
> You can add instrumentation for observability around that like you would any other code. i built my own last weekend. https://github.com/smol-ai/logger https://github.com/smol-ai/logger dumps things to json files, or to a log store. all you need for prompt engineering and monitoring really! no VC needed, no DataDog of AI yet
hackernoteng 3y ago
"This work is based on conversations with AI startup founders and engineers." - when you have millions in venture funding there is pressure from the VC, CEO and board to show some non-trivial "architecture"... telling them you have a prompt text file, redid and a bit of glue code might not go over that well (unfortunately)
j45 3y ago
Simple or complex, API and workflow orchestration will always be a thing.
tbrownaw 3y ago
Don't read it as "you should use these tools", reat it as "if you want this feature, here are some example tools that provide it". . > quickly build a robust feature with only Much like how building Twitter is a weekend-sized project.
- phillipcarter 3y ago
  > Much like how building Twitter is a weekend-sized project. More like a month, but yes. That's what we did - a month to launch, then another month to harden and add several features. Rolled out to all customers. Building product features with LLMs is difficult, but not because of the architectural needs. It's an API you pass data to.

emmender 3y ago

how can they make a sale to enterprise IT without a giant architecture diagram ? The next thing you know, java may rear its ugly head as well just when we thought python eliminated it completely for these applications.

adamgordonbell 3y ago

Exactly! We are talking about calling an API here people. Maybe what is behind the API is magical and powerful seeming but it's just an API that takes some context, the tokens to generate and a temperature setting.

tbrownaw 3y ago
Part of that diagram is about fine-tuning / retraining. Part is about managing canned prompts, since those matter enough to have their own development cycle. Part is about caching, since the fancier models are very expensive. Part is about filtering the output to not upset people, which is built in to the hosted versions (currently called "AI safety"). Etc. Doing everything in that diagram is probably overkill for most uses. But using it as a starting point and trimming what you don't need will help a bit with triping over "oops I forgot to include that".
- adamgordonbell 3y ago
  Makes sense. But caching, for instance, doesn't need to be it's own lib does it? I don't want 'semantic caching' I want to cache the exact same query and I can do that without being LLM specific: from joblib import Memory @memory.cache def call_chat_completion_api_cached(max_tokens, messages,temperature): ... I mean, I guess then I might want to store that somewhere central like redis and maybe slowly I need a specific cache tool. So I get your point. It's helpful to see the possibilities of approaching these problems. But it also does feel like an land grab of supporting libs and infrastructure.

jchanimal 3y ago

100% this. If you’re building apps that call LLMs, most of the magic is already in the model, and what you are concerned about is tracking inputs and outputs. A browser database and React is all I have needed for my LLM apps.

robbomacrae 3y ago

Sure you could and I bet there are a number of VC backed startups that do only this. But there's usually a lot of reasons why these architectures could be a useful reference. In my project, I host my own trained LLM and one of the cost efficiencies comes from being able to cache at every step along the way. Then there is a large private media hosting consideration. There is room for all sorts of setups and I kind of liked how the article mapped out some of the common paths.

j45 3y ago
Building flexibility in helps pivot easier.

satvikpendem 3y ago

Most enterprises likely won't be able to use OpenAI since they might have proprietary information, so setting up a good on-prem open source LLM can be necessary. In fact, I am doing so right now with my company.

nonfamous 3y ago
You can create your own instances of the OpenAI models in Azure to keep proprietary data with the bounds of your tenant there.
- abrichr 3y ago
  Interesting, can you please provide a link to the relevant documentation? The closest I found is https://azure.microsoft.com/en-us/blog/chatgpt-is-now-available-in-azure-openai-service/ https://azure.microsoft.com/en-us/blog/chatgpt-is-now-availa... and https://learn.microsoft.com/en-us/azure/cognitive-services/openai/overview https://learn.microsoft.com/en-us/azure/cognitive-services/o... but wasn't able to find anything about the data provisioning terms you mention.
  - rpeden 3y ago
    This might be what you're looking for: https://learn.microsoft.com/en-us/legal/cognitive-services/openai/data-privacy https://learn.microsoft.com/en-us/legal/cognitive-services/o...
    - abrichr 3y ago
      Thank you! I believe you may be referring to this: > Fine-tuned OpenAI models. The Fine-tunes API allows customers to create their own fine-tuned version of the OpenAI models based on the training data that they have uploaded to the service via the Files APIs. The trained fine-tuned models are stored in Azure Storage in the same region, encrypted at rest and logically isolated with their Azure subscription and API credentials. Fine-tuned models can be deleted by the user by calling the DELETE API operation. My follow-up question is: how is this different from OpenAI's finetuning?

swyx 3y ago

> You can add instrumentation for observability around that like you would any other code. i built my own last weekend. https://github.com/smol-ai/logger https://github.com/smol-ai/logger dumps things to json files, or to a log store. all you need for prompt engineering and monitoring really! no VC needed, no DataDog of AI yet

hackernoteng 3y ago

"This work is based on conversations with AI startup founders and engineers." - when you have millions in venture funding there is pressure from the VC, CEO and board to show some non-trivial "architecture"... telling them you have a prompt text file, redid and a bit of glue code might not go over that well (unfortunately)

j45 3y ago

Simple or complex, API and workflow orchestration will always be a thing.

tbrownaw 3y ago

Don't read it as "you should use these tools", reat it as "if you want this feature, here are some example tools that provide it". . > quickly build a robust feature with only Much like how building Twitter is a weekend-sized project.

phillipcarter 3y ago
> Much like how building Twitter is a weekend-sized project. More like a month, but yes. That's what we did - a month to launch, then another month to harden and add several features. Rolled out to all customers. Building product features with LLMs is difficult, but not because of the architectural needs. It's an API you pass data to.