4 ms·

We've tried to sell variants of the open source models to our existing enterprise customers. I think the adage about "a solution needs to be 10x other solution

by xrd 3y ago

We've tried to sell variants of the open source models to our existing enterprise customers. I think the adage about "a solution needs to be 10x other solutions to make someone switch" applies here. Saying something performs slightly better than the industry standard offerings (OpenAI) means that OpenAI is going to laugh all the way to the bank. Everyone will just use their APIs over anything else. I'm excited about the LLM space and I can barely keep up with the model names, much less all the techniques for fine tuning. A customer is going to have an even worse time. No one will ever get fired for buying OpenAI (now that IBM is dead, and probably sad Watson never made a dent). I do use Mistral for all my personal projects but I'm not sure that is going to have the same effect on the industry as open source software did in the past.

turnsout 3y ago
There's a lot of truth to this, but I have seen clients get really interested in local models—mostly due to cost and/or confidentiality. For example, some healthcare clients will never upload medical records to OpenAI, regardless of the enterprise agreement.
esafak 3y ago
OpenAI is nothing like IBM in its heyday. I bet a very healthy proportion of companies will not share their data with OpenAI. I saw some numbers on this a while back I don't have the link handy. Trust has to be earned.
bugglebeetle 3y ago
The problem here is that the platform offering here is overly complicated to get started with and quite limited. 2000 dataset entries for $50 a month when I can do 10x as many as that on Colab for free with axolotl or unsloth? Yeah, no thanks.
wavemode 3y ago
> I think the adage about "a solution needs to be 10x other solutions to make someone switch" applies here. Cheaper and faster is also better. The cheapest version of GPT-4 costs $0.01/$0.03 per 1K input/output tokens [1]. Mistral AI is charging 0.14€/0.42€ per ONE MILLION input/output tokens for their 7B model [2]. It's night and day. If people can start fine-tuning a 7B model to do the same work they were doing with GPT-4, they will 100% switch. [1]: https://help.openai.com/en/articles/7127956-how-much-does-gpt-4-cost https://help.openai.com/en/articles/7127956-how-much-does-gp... [2]: https://docs.mistral.ai/platform/pricing/ https://docs.mistral.ai/platform/pricing/
oceanplexian 3y ago
> I think the adage about "a solution needs to be 10x other solutions to make someone switch" applies here. It's already superior to OpenAI because it doesn't require an API. You can run the model on your own hardware, in your own datacenter, and your data is guaranteed to remain confidential. Creating a one-off fine-tune is a different story than permanently joining your company at the hip to OpenAI. I know in our bubble, in the era of Cloud, it's easy to send confidential company data to some random API on the Internet and not worry about it, but that's absolutely not the case for anyone in Healthcare, Government, or even normal companies that are security conscious. For them, OpenAI was never a valid consideration in the first place.
- moneywoes 3y ago
  what is the most prominent use case for private LLMs, doctor notes?
  - miohtama 3y ago
    Anything related to the business or medium and large enterprises, government
  - noitpmeder 3y ago
    Definitely healthcare, or for certain industries (HFT/Finance/...) where for various reasons _everything_ must be run on prem.
    - Foobar8568 3y ago
      As long as you fit your regulatory requirements, it's incorrect.
  - sergiotapia 3y ago
    You could use it to query against any kind of B2B customer information and provide insight, citations and context without any of the data leaving your private server. When building something similar powered by OpenAI I had a real pain in the ass anonymizing the data, then de-anonymizing the answers before showing it to the customer. Also in my example, I'm sure using a string like "Pineapple Cave Inc." instead of the real business name hurt the AI's ability to contextualize the information and data and that hurt the LLM somewhat -- right?
moneywoes 3y ago
how are you using it for your project?
kcorbitt 3y ago
Hey, I'm the post author. This is a totally fair point! I do think though that depending on your specific requirements open-source models can be a 10x+ improvement. For example, we serve Mistral 7B for less than 1/10th the cost of GPT-4-Turbo, which is the model most of our users are comparing us to.
- xrd 3y ago
  This is the 10x I was looking for. Great post by the way!
- MacsHeadroom 3y ago
  I serve ~300tk/s of Mistral 7B for $0.60/hr by renting a cloud 3090. That's a lot cheaper than GPT-4-Turbo, though the quality is closer to GPT-3.5. Mixtral 8x7b is closer to GPT-4 quality though and only 2x the compute requirement of Mistral 7B.
jdwyah 3y ago
The real thing is the switching costs. Sure we start with openAI. But at some hackathon in 9 months somebody will try mistral and if that saves real money and still works it feels like any easy swap.
Joeri 3y ago
Actually, I think microsoft is going to laugh all the way to the bank, because probably most enterprises will use the Azure OpenAI service instead of directly buying OpenAI’s offerings.
ren_engineer 3y ago
all they need is an API compatible client library so there is no actual switching cost between models other than configuration. There's a reason OpenAI is adding all sorts of add-on features like assistants and file upload, because they know models themselves are going to be a commodity and they need something to lock developers on their platform
- visarga 3y ago
  Code execution and RAG are not going to lock people in. They are 1000x easier to replicate than the model, which as you say, is already becoming a commodity. My pet theory is that OpenAI are cooking high quality user data by empowering GPT with all these toys + human-in-the-loop. The purpose is to use this data as a sort of continual evaluation sifting for weak points and enhancing their fine-tuning datasets. Every human response can carry positive or negative connotation. The model can use that as a reward signal. They claimed to have 100M users, times let's say 10K tokens per month makes 1T synthetic tokens. In a whole year they generate about as much text as the original dataset, 13T. And we know that LLMs can benefit a lot from synthetic data when it is filtered/engineered for quality. So I think OpenAI's moat is the data they generate.
mmcwilliams 3y ago
I think at this point the "10x other solutions" should be measured for the cost. If I can process, in perpetuity, 100s of millions of tokens for the cost that OpenAI can do for 10s of millions of tokens one time, that is already past the threshold.
xrd 3y ago
I've read all the comments here, many of which contradict my points. I used to agree with those ideas but then tried to sell LLMs to customers. My takeaway is that customers will pretend they care about privacy and needs for on prem installations, but they will, at least right now, go with a vendor that tells them their data is protected and not investigate the truth of that. Zoom got away with it and still does and no one got fired for using zoom. I'm happy to have a debate with someone that has successfully sold those ideas to a customer, but I'm skeptical until then.
zmmmmm 3y ago
I have the opposite problem. We are beseiged by vendors promising the earth from their amazing AI tools and we peel back 1 surface layer and they are just shoving things wholesale into GPT-4. When I ask "can we please deploy this on a local model" they run off scared. I can't get any vendor to give us anything except OpenAI.

by xrd 3y ago

turnsout 3y ago

There's a lot of truth to this, but I have seen clients get really interested in local models—mostly due to cost and/or confidentiality. For example, some healthcare clients will never upload medical records to OpenAI, regardless of the enterprise agreement.

esafak 3y ago

OpenAI is nothing like IBM in its heyday. I bet a very healthy proportion of companies will not share their data with OpenAI. I saw some numbers on this a while back I don't have the link handy. Trust has to be earned.

bugglebeetle 3y ago

The problem here is that the platform offering here is overly complicated to get started with and quite limited. 2000 dataset entries for $50 a month when I can do 10x as many as that on Colab for free with axolotl or unsloth? Yeah, no thanks.

wavemode 3y ago

> I think the adage about "a solution needs to be 10x other solutions to make someone switch" applies here. Cheaper and faster is also better. The cheapest version of GPT-4 costs $0.01/$0.03 per 1K input/output tokens [1]. Mistral AI is charging 0.14€/0.42€ per ONE MILLION input/output tokens for their 7B model [2]. It's night and day. If people can start fine-tuning a 7B model to do the same work they were doing with GPT-4, they will 100% switch. [1]: https://help.openai.com/en/articles/7127956-how-much-does-gpt-4-cost https://help.openai.com/en/articles/7127956-how-much-does-gp... [2]: https://docs.mistral.ai/platform/pricing/ https://docs.mistral.ai/platform/pricing/

oceanplexian 3y ago

> I think the adage about "a solution needs to be 10x other solutions to make someone switch" applies here. It's already superior to OpenAI because it doesn't require an API. You can run the model on your own hardware, in your own datacenter, and your data is guaranteed to remain confidential. Creating a one-off fine-tune is a different story than permanently joining your company at the hip to OpenAI. I know in our bubble, in the era of Cloud, it's easy to send confidential company data to some random API on the Internet and not worry about it, but that's absolutely not the case for anyone in Healthcare, Government, or even normal companies that are security conscious. For them, OpenAI was never a valid consideration in the first place.

moneywoes 3y ago
what is the most prominent use case for private LLMs, doctor notes?
- miohtama 3y ago
  Anything related to the business or medium and large enterprises, government
- noitpmeder 3y ago
  Definitely healthcare, or for certain industries (HFT/Finance/...) where for various reasons _everything_ must be run on prem.
  - Foobar8568 3y ago
    As long as you fit your regulatory requirements, it's incorrect.
- sergiotapia 3y ago
  You could use it to query against any kind of B2B customer information and provide insight, citations and context without any of the data leaving your private server. When building something similar powered by OpenAI I had a real pain in the ass anonymizing the data, then de-anonymizing the answers before showing it to the customer. Also in my example, I'm sure using a string like "Pineapple Cave Inc." instead of the real business name hurt the AI's ability to contextualize the information and data and that hurt the LLM somewhat -- right?

moneywoes 3y ago

how are you using it for your project?

kcorbitt 3y ago

Hey, I'm the post author. This is a totally fair point! I do think though that depending on your specific requirements open-source models can be a 10x+ improvement. For example, we serve Mistral 7B for less than 1/10th the cost of GPT-4-Turbo, which is the model most of our users are comparing us to.

xrd 3y ago
This is the 10x I was looking for. Great post by the way!
MacsHeadroom 3y ago
I serve ~300tk/s of Mistral 7B for $0.60/hr by renting a cloud 3090. That's a lot cheaper than GPT-4-Turbo, though the quality is closer to GPT-3.5. Mixtral 8x7b is closer to GPT-4 quality though and only 2x the compute requirement of Mistral 7B.

jdwyah 3y ago

The real thing is the switching costs. Sure we start with openAI. But at some hackathon in 9 months somebody will try mistral and if that saves real money and still works it feels like any easy swap.

Joeri 3y ago

Actually, I think microsoft is going to laugh all the way to the bank, because probably most enterprises will use the Azure OpenAI service instead of directly buying OpenAI’s offerings.

ren_engineer 3y ago

all they need is an API compatible client library so there is no actual switching cost between models other than configuration. There's a reason OpenAI is adding all sorts of add-on features like assistants and file upload, because they know models themselves are going to be a commodity and they need something to lock developers on their platform

visarga 3y ago
Code execution and RAG are not going to lock people in. They are 1000x easier to replicate than the model, which as you say, is already becoming a commodity. My pet theory is that OpenAI are cooking high quality user data by empowering GPT with all these toys + human-in-the-loop. The purpose is to use this data as a sort of continual evaluation sifting for weak points and enhancing their fine-tuning datasets. Every human response can carry positive or negative connotation. The model can use that as a reward signal. They claimed to have 100M users, times let's say 10K tokens per month makes 1T synthetic tokens. In a whole year they generate about as much text as the original dataset, 13T. And we know that LLMs can benefit a lot from synthetic data when it is filtered/engineered for quality. So I think OpenAI's moat is the data they generate.

mmcwilliams 3y ago

I think at this point the "10x other solutions" should be measured for the cost. If I can process, in perpetuity, 100s of millions of tokens for the cost that OpenAI can do for 10s of millions of tokens one time, that is already past the threshold.

xrd 3y ago

I've read all the comments here, many of which contradict my points. I used to agree with those ideas but then tried to sell LLMs to customers. My takeaway is that customers will pretend they care about privacy and needs for on prem installations, but they will, at least right now, go with a vendor that tells them their data is protected and not investigate the truth of that. Zoom got away with it and still does and no one got fired for using zoom. I'm happy to have a debate with someone that has successfully sold those ideas to a customer, but I'm skeptical until then.

zmmmmm 3y ago

I have the opposite problem. We are beseiged by vendors promising the earth from their amazing AI tools and we peel back 1 surface layer and they are just shoving things wholesale into GPT-4. When I ask "can we please deploy this on a local model" they run off scared. I can't get any vendor to give us anything except OpenAI.