3 ms·

>I think we don't need to argue the fact that an on premise solution is cheaper than a cloud solution for a lot of tasks In the absolute sense where we look at

by yyyk 3y ago

>I think we don't need to argue the fact that an on premise solution is cheaper than a cloud solution for a lot of tasks In the absolute sense where we look at the total cost of running the model (and not care how it's distributed or include profits), you may be right even with scale efficiencies - making a determination requires data about cloud server costs we do not have. But I can make an informed guess about the dollar cost the user sees. The cost the user sees is influenced by the factor called 'Microsoft and Google (etc.) have a lot of money, and seem to be perfectly willing to absorb costs to control the market and get user data', and that's enough to get user costs very low when calling to Cloud LLMs. >>>I can easily see scenarious where big tech fail to pivot into AI properly and go down. >>I don't think you have much to base scenarios where big tech goes down upon, but I'll be glad to hear an argument. Hopefully being based on more than predictions of local LLM performance. >What else should they be based on, pray, given the fact that smaller models suitable for on-premise and even on-machine use are improving rapidly? My consistent point is that technical performance is not enough. While the small open models have a tendency to overclaim[0], they'll get to GPT-4 level in time. Success, however, is not determined by technical performance alone. There are some very big hurdles ahead. Why should big tech fall when they pivoted ahead in time, and maintain some very useful moats? [0] https://arxiv.org/abs/2305.15717 https://arxiv.org/abs/2305.15717

usrbinbash 3y ago
> and seem to be perfectly willing to absorb costs to control the market That only works when the competing product incurs a cost that can be undercut, and can be pushed off market in the process. Self-hosten open source LLMs don't incur any cost beyond the utilities. They also cannot be pushed off the market. Trying this tactic would be like trying to replace Linux as the dominant server OS by lowering the licensing costs for Windows Server. > and maintain some very useful moats? https://www.semianalysis.com/p/google-we-have-no-moat-and-neither https://www.semianalysis.com/p/google-we-have-no-moat-and-ne... But even acknowledging the fact that larger models still have advantages in performance, how shall that moat be maintained over time? Even in their current state, smaller models are useful for specialised tasks. And I know I'm repeating myself, but they are also cheaper, work offline, and can be run on a laptop. And it isn't a question if there will be better open source base models, and better training data for RLHF it's only a question of when that happens. To wit, we are still waiting for the 15B and 30B checkpoints of stableLM. And other than with the giant models developed behind closed doors, development turnover for smaller LLMs happens in weeks, not months. Which isn't surprising, because the talent pool open source development can draw from, is basically limitless.
- yyyk 3y ago
  >That only works when the competing product incurs a cost that can be undercut You're right here, they can't destroy open source (unless they lobby/scare legislators, but that's an entirely different matter). But this subsidization means that the smaller OSS models don't appear cheaper to the client-side users. The companies can absorb the costs - it's a typical data for free service deal, and we already know users can be receptive to these deals. subsidization is also useful to scare off commercial competition. >... The Google 'leak' was a dumb spin and/or an example for why Google Research failed at converting its lead because it doesn't understand business. The important moats are not in raw performance following the initial training runs. That metric is secondary. The moats are in data and access, and both require productization. All of the specialized and local models need access to user data for their task, and BigCorp already has access and data from its products. Lots of telemetry! Everyone else are likely to get a scary user prompt for 'security' which they try to access user data. In the LLM world, data => performance, having better data could mean BigCorp keeps improving beyond OSS. Everything needs to be deployed, and BigCorp can just push it as an OS update. OSS needs word of mouth. BigCorp can aggregate data from multiple users on its remote end for retraining. Local models are likely to be intermittently updated (who's going to pay for that? And based on what data?) and have access only to local user data and what it saw in the original training. OSS catching up to GPT-4 performance will eventually happen, but by that time, BigCorp could achieve a strong product moat and improve its own performance beyond GPT-4. Right now, OSS is behind where it matters, and there's no guarantee this would change. One could hope...
  - usrbinbash 3y ago
    > The moats are in data and access We can only speculate what data closed LLMs were trained on, but I'd be highly surprised if Google/openai had exclusive access to a bigger repository of written data than, well, the internet, as it presents itself to the world at large. Which is the reason why things like https://pile.eleuther.ai/ https://pile.eleuther.ai/ exist. So no, there is no data-moat. > , and both require productization. Products can be developed by basically every group with the passion to do so, even in an OSS setting. A great example is InvokeAI, a stable diffusion implementation that, while it doesn't (yet) offer the customization and extensability of AUTOMATIC1111 has a pretty superb UX. So no, there is no productization-moat either. > All of the specialized and local models need access to user data for their task What exactly would they require "user data" for? The LLM plugin I use for coding tasks requires access to my current vim-buffer, which the plugin provides. My script for generating documentation from API code requires only the API code. When I use an LLM to write an email, the data it requires is the prompt and some datetime information, which my script provides. Even the existing cloud based solutions don't need access to user-data either to perform their functions. > Everything needs to be deployed, and BigCorp can just push it as an OS update. And app providers can just update an app. LLMs don't have some special requirements that would make updateing integrated versions any more difficult than upgrading other software. > BigCorp could achieve a strong product moat and improve its own performance beyond GPT-4. By doing what, deploying ever larger models? Attention based transformers have O(n^2) scaling, so that's unlikely to happen unless there is some architectural breakthrough. Which is far more likely to happen in OSS first, due to the aforementioned next to limitless talent pool. > Right now, OSS is behind where it matters, and there's no guarantee this would change OSS powers basically everything in the world of computing minus office software, desktops and gaming PCs, and that isn't for a lack of capability. So I'd say that purely based on experience and history, I think it's very unlikely that this won't change, and quickly.

by yyyk 3y ago

usrbinbash 3y ago

> and seem to be perfectly willing to absorb costs to control the market That only works when the competing product incurs a cost that can be undercut, and can be pushed off market in the process. Self-hosten open source LLMs don't incur any cost beyond the utilities. They also cannot be pushed off the market. Trying this tactic would be like trying to replace Linux as the dominant server OS by lowering the licensing costs for Windows Server. > and maintain some very useful moats? https://www.semianalysis.com/p/google-we-have-no-moat-and-neither https://www.semianalysis.com/p/google-we-have-no-moat-and-ne... But even acknowledging the fact that larger models still have advantages in performance, how shall that moat be maintained over time? Even in their current state, smaller models are useful for specialised tasks. And I know I'm repeating myself, but they are also cheaper, work offline, and can be run on a laptop. And it isn't a question if there will be better open source base models, and better training data for RLHF it's only a question of when that happens. To wit, we are still waiting for the 15B and 30B checkpoints of stableLM. And other than with the giant models developed behind closed doors, development turnover for smaller LLMs happens in weeks, not months. Which isn't surprising, because the talent pool open source development can draw from, is basically limitless.

yyyk 3y ago
>That only works when the competing product incurs a cost that can be undercut You're right here, they can't destroy open source (unless they lobby/scare legislators, but that's an entirely different matter). But this subsidization means that the smaller OSS models don't appear cheaper to the client-side users. The companies can absorb the costs - it's a typical data for free service deal, and we already know users can be receptive to these deals. subsidization is also useful to scare off commercial competition. >... The Google 'leak' was a dumb spin and/or an example for why Google Research failed at converting its lead because it doesn't understand business. The important moats are not in raw performance following the initial training runs. That metric is secondary. The moats are in data and access, and both require productization. All of the specialized and local models need access to user data for their task, and BigCorp already has access and data from its products. Lots of telemetry! Everyone else are likely to get a scary user prompt for 'security' which they try to access user data. In the LLM world, data => performance, having better data could mean BigCorp keeps improving beyond OSS. Everything needs to be deployed, and BigCorp can just push it as an OS update. OSS needs word of mouth. BigCorp can aggregate data from multiple users on its remote end for retraining. Local models are likely to be intermittently updated (who's going to pay for that? And based on what data?) and have access only to local user data and what it saw in the original training. OSS catching up to GPT-4 performance will eventually happen, but by that time, BigCorp could achieve a strong product moat and improve its own performance beyond GPT-4. Right now, OSS is behind where it matters, and there's no guarantee this would change. One could hope...
- usrbinbash 3y ago
  > The moats are in data and access We can only speculate what data closed LLMs were trained on, but I'd be highly surprised if Google/openai had exclusive access to a bigger repository of written data than, well, the internet, as it presents itself to the world at large. Which is the reason why things like https://pile.eleuther.ai/ https://pile.eleuther.ai/ exist. So no, there is no data-moat. > , and both require productization. Products can be developed by basically every group with the passion to do so, even in an OSS setting. A great example is InvokeAI, a stable diffusion implementation that, while it doesn't (yet) offer the customization and extensability of AUTOMATIC1111 has a pretty superb UX. So no, there is no productization-moat either. > All of the specialized and local models need access to user data for their task What exactly would they require "user data" for? The LLM plugin I use for coding tasks requires access to my current vim-buffer, which the plugin provides. My script for generating documentation from API code requires only the API code. When I use an LLM to write an email, the data it requires is the prompt and some datetime information, which my script provides. Even the existing cloud based solutions don't need access to user-data either to perform their functions. > Everything needs to be deployed, and BigCorp can just push it as an OS update. And app providers can just update an app. LLMs don't have some special requirements that would make updateing integrated versions any more difficult than upgrading other software. > BigCorp could achieve a strong product moat and improve its own performance beyond GPT-4. By doing what, deploying ever larger models? Attention based transformers have O(n^2) scaling, so that's unlikely to happen unless there is some architectural breakthrough. Which is far more likely to happen in OSS first, due to the aforementioned next to limitless talent pool. > Right now, OSS is behind where it matters, and there's no guarantee this would change OSS powers basically everything in the world of computing minus office software, desktops and gaming PCs, and that isn't for a lack of capability. So I'd say that purely based on experience and history, I think it's very unlikely that this won't change, and quickly.