6 ms·

> and seem to be perfectly willing to absorb costs to control the market That only works when the competing product incurs a cost that can be undercut, and can

> and seem to be perfectly willing to absorb costs to control the market That only works when the competing product incurs a cost that can be undercut, and can be pushed off market in the process. Self-hosten open source LLMs don't incur any cost beyond the utilities. They also cannot be pushed off the market. Trying this tactic would be like trying to replace Linux as the dominant server OS by lowering the licensing costs for Windows Server. > and maintain some very useful moats? https://www.semianalysis.com/p/google-we-have-no-moat-and-neither https://www.semianalysis.com/p/google-we-have-no-moat-and-ne... But even acknowledging the fact that larger models still have advantages in performance, how shall that moat be maintained over time? Even in their current state, smaller models are useful for specialised tasks. And I know I'm repeating myself, but they are also cheaper, work offline, and can be run on a laptop. And it isn't a question if there will be better open source base models, and better training data for RLHF it's only a question of when that happens. To wit, we are still waiting for the 15B and 30B checkpoints of stableLM. And other than with the giant models developed behind closed doors, development turnover for smaller LLMs happens in weeks, not months. Which isn't surprising, because the talent pool open source development can draw from, is basically limitless.

yyyk 3y ago
>That only works when the competing product incurs a cost that can be undercut You're right here, they can't destroy open source (unless they lobby/scare legislators, but that's an entirely different matter). But this subsidization means that the smaller OSS models don't appear cheaper to the client-side users. The companies can absorb the costs - it's a typical data for free service deal, and we already know users can be receptive to these deals. subsidization is also useful to scare off commercial competition. >... The Google 'leak' was a dumb spin and/or an example for why Google Research failed at converting its lead because it doesn't understand business. The important moats are not in raw performance following the initial training runs. That metric is secondary. The moats are in data and access, and both require productization. All of the specialized and local models need access to user data for their task, and BigCorp already has access and data from its products. Lots of telemetry! Everyone else are likely to get a scary user prompt for 'security' which they try to access user data. In the LLM world, data => performance, having better data could mean BigCorp keeps improving beyond OSS. Everything needs to be deployed, and BigCorp can just push it as an OS update. OSS needs word of mouth. BigCorp can aggregate data from multiple users on its remote end for retraining. Local models are likely to be intermittently updated (who's going to pay for that? And based on what data?) and have access only to local user data and what it saw in the original training. OSS catching up to GPT-4 performance will eventually happen, but by that time, BigCorp could achieve a strong product moat and improve its own performance beyond GPT-4. Right now, OSS is behind where it matters, and there's no guarantee this would change. One could hope...
- usrbinbash 3y ago
  > The moats are in data and access We can only speculate what data closed LLMs were trained on, but I'd be highly surprised if Google/openai had exclusive access to a bigger repository of written data than, well, the internet, as it presents itself to the world at large. Which is the reason why things like https://pile.eleuther.ai/ https://pile.eleuther.ai/ exist. So no, there is no data-moat. > , and both require productization. Products can be developed by basically every group with the passion to do so, even in an OSS setting. A great example is InvokeAI, a stable diffusion implementation that, while it doesn't (yet) offer the customization and extensability of AUTOMATIC1111 has a pretty superb UX. So no, there is no productization-moat either. > All of the specialized and local models need access to user data for their task What exactly would they require "user data" for? The LLM plugin I use for coding tasks requires access to my current vim-buffer, which the plugin provides. My script for generating documentation from API code requires only the API code. When I use an LLM to write an email, the data it requires is the prompt and some datetime information, which my script provides. Even the existing cloud based solutions don't need access to user-data either to perform their functions. > Everything needs to be deployed, and BigCorp can just push it as an OS update. And app providers can just update an app. LLMs don't have some special requirements that would make updateing integrated versions any more difficult than upgrading other software. > BigCorp could achieve a strong product moat and improve its own performance beyond GPT-4. By doing what, deploying ever larger models? Attention based transformers have O(n^2) scaling, so that's unlikely to happen unless there is some architectural breakthrough. Which is far more likely to happen in OSS first, due to the aforementioned next to limitless talent pool. > Right now, OSS is behind where it matters, and there's no guarantee this would change OSS powers basically everything in the world of computing minus office software, desktops and gaming PCs, and that isn't for a lack of capability. So I'd say that purely based on experience and history, I think it's very unlikely that this won't change, and quickly.
  - yyyk 3y ago
    >We can only speculate what data closed LLMs were trained on, but I'd be highly surprised if Google/openai had exclusive access to a bigger repository of written data than, well, the internet, as it presents itself to the world at large. >What exactly would they require "user data" for? There are several classes here: A) Total internet data. Google/OpenAI may have more data from Google Books/GSuite/etc. but maybe not. No way to know. Maybe even if they do, it's not significant compared to total data volume. Since we can't meaningfully compare, let's just ignore it. B) Global usage data. This is useful to further tune the model - we saw what the open models could do with a partial log of ChatGPT. OpenAI of course has the entire log. For example, it's possible that users in country X ask for stuff in a different manner, or that terms have a local meaning the model may not be aware of. Language evolves after all. A local model can at most update on current user data, or by much slower updates from the origin, and OSS has less resources here. C) Local usage data. For example, a company may wish an LLM to access all its documents to create a knowledge base. There's a good chance all the documents stored in Office 365/GSuite. You can guess who has easy access and who gets the scary permission prompts. Another example: The LLM writing an email may wish to be aware of the previous communication in the thread and your general tone. Or replace Spotlight/Windows search with an LLM, but the LLM needs access to all your data to properly search it. Some of this can be emulated with really long prompts, but it's more efficient to just let the LLM have access. >Even the existing cloud based solutions don't need access to user-data either to perform their functions. Currently no, but the personal assistant they want to build will require it. >Products can be developed by basically every group with the passion to do so >By doing what, deploying ever larger models? Alas, OSS devs tend to get bored on 'non-sexy' subjects. Meanwhile, Microsoft and Google will embed LLM in all their apps. The apps have their own moats (data migration, UX) and in turn act act as a moat for the LLM. Moat in action: Imagine Thunderbird worked with OSS LLM and Outlook works with OpenAI GPT. A user has meetings in Outlook and uses GPT to do various related planning. Say the user was willing to migrate to OSS LLM. But OSS LLM doesn't have easy interface with Outlook (Microsoft 'competitive' behaviour), and manually importing all the time is too messy. The user may even consider switching to Thunderbird, but Thunderbird doesn't do ActiveSync, and IT refuses to even consider allowing IMAP in its Exchange, so user is stuck with Outlook and in turn with OpenAI GPT. Doing ActiveSync is boring for OSS devs, so Microsoft gets an indirect moat: Exchange <=> Outlook <=> OpenAI. SD is way more in tune, subject is way more popular with devs I guess. These people have a chance. They don't however need to brag about inevitable victory of SD or how Adobe is going down. >> Everything needs to be deployed, and BigCorp can just push it as an OS update. >And app providers can just update an app. Deploying an app requires more friction. How do you get users to install it in the first place? Not impossible (see Google Chrome over Internet Explorer) but a struggle where the OS maker has a built-in advantage. >By doing what, deploying ever larger models? A bit of that, but I expect more effective tuning because they have way more usage data. >Attention based transformers have O(n^2) scaling There are numerous papers trying to improve that. We'll see.

yyyk 3y ago

>That only works when the competing product incurs a cost that can be undercut You're right here, they can't destroy open source (unless they lobby/scare legislators, but that's an entirely different matter). But this subsidization means that the smaller OSS models don't appear cheaper to the client-side users. The companies can absorb the costs - it's a typical data for free service deal, and we already know users can be receptive to these deals. subsidization is also useful to scare off commercial competition. >... The Google 'leak' was a dumb spin and/or an example for why Google Research failed at converting its lead because it doesn't understand business. The important moats are not in raw performance following the initial training runs. That metric is secondary. The moats are in data and access, and both require productization. All of the specialized and local models need access to user data for their task, and BigCorp already has access and data from its products. Lots of telemetry! Everyone else are likely to get a scary user prompt for 'security' which they try to access user data. In the LLM world, data => performance, having better data could mean BigCorp keeps improving beyond OSS. Everything needs to be deployed, and BigCorp can just push it as an OS update. OSS needs word of mouth. BigCorp can aggregate data from multiple users on its remote end for retraining. Local models are likely to be intermittently updated (who's going to pay for that? And based on what data?) and have access only to local user data and what it saw in the original training. OSS catching up to GPT-4 performance will eventually happen, but by that time, BigCorp could achieve a strong product moat and improve its own performance beyond GPT-4. Right now, OSS is behind where it matters, and there's no guarantee this would change. One could hope...

usrbinbash 3y ago
> The moats are in data and access We can only speculate what data closed LLMs were trained on, but I'd be highly surprised if Google/openai had exclusive access to a bigger repository of written data than, well, the internet, as it presents itself to the world at large. Which is the reason why things like https://pile.eleuther.ai/ https://pile.eleuther.ai/ exist. So no, there is no data-moat. > , and both require productization. Products can be developed by basically every group with the passion to do so, even in an OSS setting. A great example is InvokeAI, a stable diffusion implementation that, while it doesn't (yet) offer the customization and extensability of AUTOMATIC1111 has a pretty superb UX. So no, there is no productization-moat either. > All of the specialized and local models need access to user data for their task What exactly would they require "user data" for? The LLM plugin I use for coding tasks requires access to my current vim-buffer, which the plugin provides. My script for generating documentation from API code requires only the API code. When I use an LLM to write an email, the data it requires is the prompt and some datetime information, which my script provides. Even the existing cloud based solutions don't need access to user-data either to perform their functions. > Everything needs to be deployed, and BigCorp can just push it as an OS update. And app providers can just update an app. LLMs don't have some special requirements that would make updateing integrated versions any more difficult than upgrading other software. > BigCorp could achieve a strong product moat and improve its own performance beyond GPT-4. By doing what, deploying ever larger models? Attention based transformers have O(n^2) scaling, so that's unlikely to happen unless there is some architectural breakthrough. Which is far more likely to happen in OSS first, due to the aforementioned next to limitless talent pool. > Right now, OSS is behind where it matters, and there's no guarantee this would change OSS powers basically everything in the world of computing minus office software, desktops and gaming PCs, and that isn't for a lack of capability. So I'd say that purely based on experience and history, I think it's very unlikely that this won't change, and quickly.
- yyyk 3y ago
  >We can only speculate what data closed LLMs were trained on, but I'd be highly surprised if Google/openai had exclusive access to a bigger repository of written data than, well, the internet, as it presents itself to the world at large. >What exactly would they require "user data" for? There are several classes here: A) Total internet data. Google/OpenAI may have more data from Google Books/GSuite/etc. but maybe not. No way to know. Maybe even if they do, it's not significant compared to total data volume. Since we can't meaningfully compare, let's just ignore it. B) Global usage data. This is useful to further tune the model - we saw what the open models could do with a partial log of ChatGPT. OpenAI of course has the entire log. For example, it's possible that users in country X ask for stuff in a different manner, or that terms have a local meaning the model may not be aware of. Language evolves after all. A local model can at most update on current user data, or by much slower updates from the origin, and OSS has less resources here. C) Local usage data. For example, a company may wish an LLM to access all its documents to create a knowledge base. There's a good chance all the documents stored in Office 365/GSuite. You can guess who has easy access and who gets the scary permission prompts. Another example: The LLM writing an email may wish to be aware of the previous communication in the thread and your general tone. Or replace Spotlight/Windows search with an LLM, but the LLM needs access to all your data to properly search it. Some of this can be emulated with really long prompts, but it's more efficient to just let the LLM have access. >Even the existing cloud based solutions don't need access to user-data either to perform their functions. Currently no, but the personal assistant they want to build will require it. >Products can be developed by basically every group with the passion to do so >By doing what, deploying ever larger models? Alas, OSS devs tend to get bored on 'non-sexy' subjects. Meanwhile, Microsoft and Google will embed LLM in all their apps. The apps have their own moats (data migration, UX) and in turn act act as a moat for the LLM. Moat in action: Imagine Thunderbird worked with OSS LLM and Outlook works with OpenAI GPT. A user has meetings in Outlook and uses GPT to do various related planning. Say the user was willing to migrate to OSS LLM. But OSS LLM doesn't have easy interface with Outlook (Microsoft 'competitive' behaviour), and manually importing all the time is too messy. The user may even consider switching to Thunderbird, but Thunderbird doesn't do ActiveSync, and IT refuses to even consider allowing IMAP in its Exchange, so user is stuck with Outlook and in turn with OpenAI GPT. Doing ActiveSync is boring for OSS devs, so Microsoft gets an indirect moat: Exchange <=> Outlook <=> OpenAI. SD is way more in tune, subject is way more popular with devs I guess. These people have a chance. They don't however need to brag about inevitable victory of SD or how Adobe is going down. >> Everything needs to be deployed, and BigCorp can just push it as an OS update. >And app providers can just update an app. Deploying an app requires more friction. How do you get users to install it in the first place? Not impossible (see Google Chrome over Internet Explorer) but a struggle where the OS maker has a built-in advantage. >By doing what, deploying ever larger models? A bit of that, but I expect more effective tuning because they have way more usage data. >Attention based transformers have O(n^2) scaling There are numerous papers trying to improve that. We'll see.