5 ms·

I wonder if the real market is actually bringing this stuff inhouse. Given the propensity for these big tech companies to hoover up/steal any information they

I wonder if the real market is actually bringing this stuff inhouse. Given the propensity for these big tech companies to hoover up/steal any information they can gather, running these models locally, with local fine tuning looks quite attractive.

rrix2 2y ago
> Given the propensity for these big tech companies to hoover up/steal any information they can gather at the end of the day you still have to sell this product to the sorts of companies that are far and away all microsoft 365/google workspace clients and we're gonna have to figure that out one day or another
antupis 2y ago
Pretty much especially in Europe there is lots of big companies and public sector institutions that would pay serious € if they could run these.
- morphle 2y ago
  Spot on! I concur most European business and public sector institutions would be eager to rent this because they are not allowed by law to use US datacenters like AWS or Azure.
  - kiviuq 2y ago
    That's not the only issue. They want a guarantee that the model wasn't trained on copyrighted material.
    - TeMPOraL 2y ago
      Now that is a real feature for now. A lot of hesitation in embracing generative AI in large enterprises stems from uncertainty about copyright issue. Anyone who trained an o1-level model from scratch on public/properly licensed data only would be able to provide a very valuable service to those enterprise customers. However, if both training and operating costs of a DeepSeek-like model are as small as they are, the companies best able to offer this service are... Microsoft, Amazon and Google. And second best are... teams inside the would-be customer enterprises themselves. $6M to train and $6K to run is effectively free for such companies; there is no moat here. The services that enterprise customers would happily buy instead of building are... operations, and assuming legal liability if the model turns out not to be safe from copyright infringement lawsuits. But those are exactly the services those companies are already buying from Microsoft, Amazon and Google.
    - fulafel 2y ago
      This would result in some refreshing models, I guess they would be trained mostly on out-of-copyright stuff from 75+ years ago and wouldn't have knowledge of the modern world. Maybe they could skin the robotic bureucrats in vintage scifi appearance as well to have the whole consistent experience when you go to the building permits bot, there could be small talk about the latest Beatles record etc.
      - TeMPOraL 2y ago
        Enforcing copyright on training data to this extent would actually create a temporary moat for the biggest players - they can afford to hire a lot of cheap labor to supplement the training dataset with human-authored original works that skirt IP protections by interpreting, parodying, commenting on or otherwise describing the protected works without actually infringing on them. As long as they keep those datasets private, everyone else is shit out of luck. (I'm reiterating my prediction wrt. AI and moats - the only mid-term moat there can be is in human labor. Hardware vendors benefit from selling better hardware to more people for less; software and research are cheap to scale, datasets eventually leak or get reproduced. Human labor is the one thing that doesn't scale, and except for an economic crisis, only ever gets more expensive with time. Whatever edge one can get by applying human labor that cannot be substituted by AI - like RLHF and its evolutions - is the one that will last all the way to AGI; past that, moats won't matter anymore.) One of the many reasons I'm firmly on the side of making the training of large neural models exempt of copyright considerations for everyone.
        fulafel 2y ago
        Isn't the training already exempt from copyright? Copyright is in the core about enabling licenses related to who's allowed to distribute copies of content (not ideas, but the exact same text, etc). edit: apparently in the EU the situation is complicated by new AI specific legislation in the works: https://www.morganlewis.com/pubs/2024/02/eu-ai-act-how-far-will-eu-copyright-principles-extend https://www.morganlewis.com/pubs/2024/02/eu-ai-act-how-far-w...
  - jhoechtl 2y ago
    Not true for two reasons! Azure data center in europe is fine if thr data stays there and generally there is the EU-US Data Privacy Framework https://en.m.wikipedia.org/wiki/EU%E2%80%93US_Data_Privacy_Framework https://en.m.wikipedia.org/wiki/EU%E2%80%93US_Data_Privacy_F...
themanmaran 2y ago
I'd wager this is the real market. Ship some company a server rack with Deepseek R1 for a $1M annual rental fee + upgrades to the latest models. think inside the box
- morphle 2y ago
  Cerebras already does this.
  - spacemanspiff01 2y ago
    I thought cerebras had moved to a cloud model so that they could more easily manage/patch their systems?
    - morphle 2y ago
      both. The cloud model is also for renting models across several Cerebras Wafer Scale Integrations.
      - spacemanspiff01 2y ago
        Oh, they will deploy the racks at customers sites?
        morphle 2y ago
        I don't know what their policy is now, but they talk about it in the Hot Chips 2024 presentation. I myself just have proof of a single customer having their own private Cerebras rack. There are rumors about several more customers with on-prem Cerebras.
willseth 2y ago
AWS already has platforms for running and fine tuning OSS models that can run privately inside a VPC. If Azure and GCP don’t have equivalent capabilities already, it is surely imminent. Seems pretty hard or impossible to beat cloud providers at their own game.
- HeatrayEnjoyer 2y ago
  If the hardware isn't in your physical possession you can't know that your data isn't being hoovered up. You can't end to end encrypt compute tasks (homomorphic processing is fiendishly uneconomical).
  - TeMPOraL 2y ago
    True, but at this point we're leaving the realm of cryptography and theoretical infosec, and enter the realm of real-world security. In this realm, permissions are established by armies of lawyers across organizations and governments defining who can or cannot do things, and what happens when transgressions occur; here, "defense in depth" carries all the way to the threat of men with guns escorting you to jail. So it's true that you can't encrypt compute tasks of this type end-to-end, so you can't know if unauthorized parties mine your data. However, Microsoft is very unlikely to mine your data (for "you" being e.g. any of the many multinational corporations that already run all their office work through Azure-hosted Outlook, Office, SharePoint, etc.), or to let others mine it, because if it ever came out, your customers' lawyers would be after you, your lawyers would be after Microsoft, and the whole thing would explode into a multiple-billion-dollars shitshow and might even get a government or two involved. That's the working assumption that makes Microsoft well-positioned to eat any fledgling self-hosted DeepSeek market in the business space. They already have things set up at a level that is trusted by governments as well as corporations in critical industrial sectors, with huge financial and legal exposure. (Presumably Google and Amazon are in a similar position here, though I've only seen this personally with Microsoft/Azure, so that's what I can comment on.)
    - zie 2y ago
      > "defense in depth" carries all the way to the threat of men with guns escorting you to jail. For contract breach civil crime like this, there is zero chance it ends with jail time.
valiant55 2y ago
This is what has me most excited. AI has it's limited uses for now but with the current requirement of handing over all your data to big brother it was not even worth considering. Now that on prem is reasonable and doesn't require you to beg Nvidia for H100s it might actually be usable.
- NemoNobody 2y ago
  It's virtually impossible to not hand over data now. I'd say at this point it's more likely that any attempt to draw less attention actually has the opposite effect in reality. The reality is that many datasets include stuff that has been absolutely stolen or haphazardly stored online, stuff like emails, texts, conversations, our credit histories, really any information about people that exists in a large quantity has already made its way into these LLMs one way or another. An AI trained solely/additionally on specific datasets, with the intent of predicting human behavior exactly, should yield successful and highly accurate predictions from rather general demographic data combined with an individuals location history/current their location, plus really any third substantial thing - could be browsing history, or purchases, activity on social media, that should be all that's necessary to rather accurately put everyone into a box. The most frustrating thing is that there is a box for the people that refuse to be put into a box.

rrix2 2y ago

> Given the propensity for these big tech companies to hoover up/steal any information they can gather at the end of the day you still have to sell this product to the sorts of companies that are far and away all microsoft 365/google workspace clients and we're gonna have to figure that out one day or another

antupis 2y ago

Pretty much especially in Europe there is lots of big companies and public sector institutions that would pay serious € if they could run these.

morphle 2y ago
Spot on! I concur most European business and public sector institutions would be eager to rent this because they are not allowed by law to use US datacenters like AWS or Azure.
- kiviuq 2y ago
  That's not the only issue. They want a guarantee that the model wasn't trained on copyrighted material.
  - TeMPOraL 2y ago
    Now that is a real feature for now. A lot of hesitation in embracing generative AI in large enterprises stems from uncertainty about copyright issue. Anyone who trained an o1-level model from scratch on public/properly licensed data only would be able to provide a very valuable service to those enterprise customers. However, if both training and operating costs of a DeepSeek-like model are as small as they are, the companies best able to offer this service are... Microsoft, Amazon and Google. And second best are... teams inside the would-be customer enterprises themselves. $6M to train and $6K to run is effectively free for such companies; there is no moat here. The services that enterprise customers would happily buy instead of building are... operations, and assuming legal liability if the model turns out not to be safe from copyright infringement lawsuits. But those are exactly the services those companies are already buying from Microsoft, Amazon and Google.
  - fulafel 2y ago
    This would result in some refreshing models, I guess they would be trained mostly on out-of-copyright stuff from 75+ years ago and wouldn't have knowledge of the modern world. Maybe they could skin the robotic bureucrats in vintage scifi appearance as well to have the whole consistent experience when you go to the building permits bot, there could be small talk about the latest Beatles record etc.
    - TeMPOraL 2y ago
      Enforcing copyright on training data to this extent would actually create a temporary moat for the biggest players - they can afford to hire a lot of cheap labor to supplement the training dataset with human-authored original works that skirt IP protections by interpreting, parodying, commenting on or otherwise describing the protected works without actually infringing on them. As long as they keep those datasets private, everyone else is shit out of luck. (I'm reiterating my prediction wrt. AI and moats - the only mid-term moat there can be is in human labor. Hardware vendors benefit from selling better hardware to more people for less; software and research are cheap to scale, datasets eventually leak or get reproduced. Human labor is the one thing that doesn't scale, and except for an economic crisis, only ever gets more expensive with time. Whatever edge one can get by applying human labor that cannot be substituted by AI - like RLHF and its evolutions - is the one that will last all the way to AGI; past that, moats won't matter anymore.) One of the many reasons I'm firmly on the side of making the training of large neural models exempt of copyright considerations for everyone.
      - fulafel 2y ago
        Isn't the training already exempt from copyright? Copyright is in the core about enabling licenses related to who's allowed to distribute copies of content (not ideas, but the exact same text, etc). edit: apparently in the EU the situation is complicated by new AI specific legislation in the works: https://www.morganlewis.com/pubs/2024/02/eu-ai-act-how-far-will-eu-copyright-principles-extend https://www.morganlewis.com/pubs/2024/02/eu-ai-act-how-far-w...
- jhoechtl 2y ago
  Not true for two reasons! Azure data center in europe is fine if thr data stays there and generally there is the EU-US Data Privacy Framework https://en.m.wikipedia.org/wiki/EU%E2%80%93US_Data_Privacy_Framework https://en.m.wikipedia.org/wiki/EU%E2%80%93US_Data_Privacy_F...

themanmaran 2y ago

I'd wager this is the real market. Ship some company a server rack with Deepseek R1 for a $1M annual rental fee + upgrades to the latest models. think inside the box

morphle 2y ago
Cerebras already does this.
- spacemanspiff01 2y ago
  I thought cerebras had moved to a cloud model so that they could more easily manage/patch their systems?
  - morphle 2y ago
    both. The cloud model is also for renting models across several Cerebras Wafer Scale Integrations.
    - spacemanspiff01 2y ago
      Oh, they will deploy the racks at customers sites?
      - morphle 2y ago
        I don't know what their policy is now, but they talk about it in the Hot Chips 2024 presentation. I myself just have proof of a single customer having their own private Cerebras rack. There are rumors about several more customers with on-prem Cerebras.

willseth 2y ago

AWS already has platforms for running and fine tuning OSS models that can run privately inside a VPC. If Azure and GCP don’t have equivalent capabilities already, it is surely imminent. Seems pretty hard or impossible to beat cloud providers at their own game.

HeatrayEnjoyer 2y ago
If the hardware isn't in your physical possession you can't know that your data isn't being hoovered up. You can't end to end encrypt compute tasks (homomorphic processing is fiendishly uneconomical).
- TeMPOraL 2y ago
  True, but at this point we're leaving the realm of cryptography and theoretical infosec, and enter the realm of real-world security. In this realm, permissions are established by armies of lawyers across organizations and governments defining who can or cannot do things, and what happens when transgressions occur; here, "defense in depth" carries all the way to the threat of men with guns escorting you to jail. So it's true that you can't encrypt compute tasks of this type end-to-end, so you can't know if unauthorized parties mine your data. However, Microsoft is very unlikely to mine your data (for "you" being e.g. any of the many multinational corporations that already run all their office work through Azure-hosted Outlook, Office, SharePoint, etc.), or to let others mine it, because if it ever came out, your customers' lawyers would be after you, your lawyers would be after Microsoft, and the whole thing would explode into a multiple-billion-dollars shitshow and might even get a government or two involved. That's the working assumption that makes Microsoft well-positioned to eat any fledgling self-hosted DeepSeek market in the business space. They already have things set up at a level that is trusted by governments as well as corporations in critical industrial sectors, with huge financial and legal exposure. (Presumably Google and Amazon are in a similar position here, though I've only seen this personally with Microsoft/Azure, so that's what I can comment on.)
  - zie 2y ago
    > "defense in depth" carries all the way to the threat of men with guns escorting you to jail. For contract breach civil crime like this, there is zero chance it ends with jail time.

valiant55 2y ago

This is what has me most excited. AI has it's limited uses for now but with the current requirement of handing over all your data to big brother it was not even worth considering. Now that on prem is reasonable and doesn't require you to beg Nvidia for H100s it might actually be usable.

NemoNobody 2y ago
It's virtually impossible to not hand over data now. I'd say at this point it's more likely that any attempt to draw less attention actually has the opposite effect in reality. The reality is that many datasets include stuff that has been absolutely stolen or haphazardly stored online, stuff like emails, texts, conversations, our credit histories, really any information about people that exists in a large quantity has already made its way into these LLMs one way or another. An AI trained solely/additionally on specific datasets, with the intent of predicting human behavior exactly, should yield successful and highly accurate predictions from rather general demographic data combined with an individuals location history/current their location, plus really any third substantial thing - could be browsing history, or purchases, activity on social media, that should be all that's necessary to rather accurately put everyone into a box. The most frustrating thing is that there is a box for the people that refuse to be put into a box.