3 ms·

I don't understand how it's economically viable to build an AI business. 32,768 is nothing and using OpenAI as a price reference (which I presume is heavily sub

by Kiro 3y ago

I don't understand how it's economically viable to build an AI business. 32,768 is nothing and using OpenAI as a price reference (which I presume is heavily subsidised) just one chat hitting that will be ~$4 a message if you truncate the chat history down to 32k. That's $4 for every single message in your chat, even for mundane fillers such as "thank you".

kromem 3y ago
Maybe the use case isn't chat, and people need to stop looking at evolving tech solely through the lens of status quo applications of earlier incarnations? $4 to run two research papers through GPT-4 asking it to identify differences in methods in order to accelerate directing focus during a meta-analysis seems like a pretty amazing price point even if that same price point is expensive for a chatbot to answer a customer query about on a website. Which itself may not be an expensive price point if the good in question is an automobile or a travel package even if it is for laundry detergent. Markets tend to find an equilibrium pretty quickly, and I've yet to see any AI products priced such that I can't think of any applications where they'd be welcome as long as performance was adequate.
didgeoridoo 3y ago
Loading the whole history into the context window is, indeed, dumb. Most implementations I’ve seen recently involve using embeddings/RAG against the chat history (plus any other public, proprietary, or account-specific knowledge base you want to include) to only pull out the most relevant aspects of the history to include in the prompt envelope. Embeddings are cheap (don’t even need a transformer for them) and RAG lookup with a vector DB (e.g. pg-vector) is fast and efficient. You also don’t need to always his the 32k endpoint — you can save that for higher tier accounts, and even then only use it for queries where the prompt length requires it. Yes you’ll truncate some responses where prompt plus response > 32k, plus if your use case typically results in those kinds of situations then your pricing should probably be relatively high/“enterprisey”.

by Kiro 3y ago

kromem 3y ago

Maybe the use case isn't chat, and people need to stop looking at evolving tech solely through the lens of status quo applications of earlier incarnations? $4 to run two research papers through GPT-4 asking it to identify differences in methods in order to accelerate directing focus during a meta-analysis seems like a pretty amazing price point even if that same price point is expensive for a chatbot to answer a customer query about on a website. Which itself may not be an expensive price point if the good in question is an automobile or a travel package even if it is for laundry detergent. Markets tend to find an equilibrium pretty quickly, and I've yet to see any AI products priced such that I can't think of any applications where they'd be welcome as long as performance was adequate.

didgeoridoo 3y ago

Loading the whole history into the context window is, indeed, dumb. Most implementations I’ve seen recently involve using embeddings/RAG against the chat history (plus any other public, proprietary, or account-specific knowledge base you want to include) to only pull out the most relevant aspects of the history to include in the prompt envelope. Embeddings are cheap (don’t even need a transformer for them) and RAG lookup with a vector DB (e.g. pg-vector) is fast and efficient. You also don’t need to always his the 32k endpoint — you can save that for higher tier accounts, and even then only use it for queries where the prompt length requires it. Yes you’ll truncate some responses where prompt plus response > 32k, plus if your use case typically results in those kinds of situations then your pricing should probably be relatively high/“enterprisey”.