8 ms·

Whether scaling laws hold or not, is up for debate. What isn't up for debate, is that: 1) Giant serverfarms are expensive 2) People want on premises/o

Whether scaling laws hold or not, is up for debate. What isn't up for debate, is that: 1) Giant serverfarms are expensive 2) People want on premises/on machine solutions 3) LoRA Tuning of small models continues to excel 4) Thus specialized models continue to evolve at a fast pace 5) Open source foundation models on which tuning can be done are accelerating by the week 6) Performance doesn't matter once a model is "good enough" for a given task The long and short of it is this: LLMs are a fantastic technology, and people want to have it. And they want to have it local, private, and as cheap as possible. Just one example, how many devs are out there who would love having a local LLM model integrated into their IDE? The answer: Yes. And again I draw your attention to point 6): For such use-cases, It doesn't matter if a gargantuan model performs a bit better ... if the small model is good enough for the task, then it wins, purely by being private, offline and is free to use. We are already at a point, where small models are competitive for specialized tasks, and aren't performing bad at all as general models either. What will it take for bigger models to out-perform them so enourmously that customers simply have no other choice than to use them? A 5x increase in size? 10x? 100x? Where do we run models with trillions of params? What will that cost, and how available will that be?

YetAnotherNick 3y ago
> And they want to have it local, private, and as cheap as possible. Most people don't care about local and private. They care about cheap and free ChatGPT is the cheapest they could get, which is better than all open access models. > Open source foundation models on which tuning can be done are accelerating by the week Based on the (unverified)information I have been hearing in the research circle is that OpenAI invested heavily in data collection. One thing I heard is that they have hired experts to label lot of codes with information like complexity, correctness etc. Also the questions and RLHF data they get from real users is something open access models can't match. Palm 2 chat is significantly worse than ChatGPT(free), even though it is larger and trained by experts in the field.
- mr_mitm 3y ago
  > Most people don't care about local and private. Right, but corporations do. And AI is a revolution because it makes workers substantially more efficient, not because it makes for a slightly better Google. If corporations want to leverage AI to make their workforce more efficient, it must be local and private in many cases. OpenAI is already experiencing friction in the EU.
  - yyyk 3y ago
    Same corporations happily use Office 365 and GSuite for all their documents rather than a local and private solution. I'm sure they'll come to a similar arrangement with regard to AI.
    - usrbinbash 3y ago
      That doesn't change the fact that these corporations have an interest in data privacy and cost reduction. To wit, if MS wouldn't guarantee data security in european datacenters, they would be way less competitive in the EU. Many corporations run their own datacenters, have on premises mailservers and run their own backup solutions. Not everything is in the cloud.
yyyk 3y ago
1) Giant serverfarms are expensive, but Microsoft has money and savings on scale. This can compare favourably to running locally. 2) People use what LLMs software makers give them and don't care at all about on premise. Software makers want something reliable they can deploy at scale, and not end up debugging problems on a client 4GB machine with an iGPU and multitude of OSs. The actually deciding people want to delegate to an API, and the only offerings available right now are from OpenAI, Google, Anthropic (aka Google again) etc.. 3) Open source LLM products and APIs are at the zero point and remain so by the week. "Download from GitHub and compile" is not going to fly. 4) Right now the question is whether the Closed Source and Propriety product people will actually use will be provided by a web API (openAI or maybe Meta) or by the OS (Apple, Microsoft, Google, etc.). Either way it will be controlled by giant corporations. Unless 3) ever changes, but doing product is icky for OSS people for some reason so that will never happen. [EDIT: Note that the Stable Diffusion area is in way better shape. There's a one-click installer for Auto111! There are some products and sites using SD! There are some usecases important commercial interests don't want! These OSS people also don't engage in triumphalism much less unearned triumphalism! They are probably still going to lose to Adobe and Midjourney but at least it will be a fight.]
- senttoschool 3y ago
  A long time ago, in order to get computers to do anything useful, the hardware required huge rooms. Today, a little cell phone is significantly more powerful than a room sized computer back in the days. LLMs shouldn't be any different. Today, they require giant server farms to run. Tomorrow, they will run on little robots/cell phones. A few things will allow this: 1. Chip makers like Apple, Qualcomm, AMD, Intel, Nvidia, will heavily emphasize AI performance in their future chips. Expect accelerators like the Neural Engine to get significantly bigger. In the future, I expect that most of the transistors in a SoC will be dedicated to AI acceleration - and not CPUs/GPUs. 2. Moore's law is not dead. We will continue to get more transistors in a given area. For example, TSMC has plans for 1nm which will likely have 4-5x more transistors per area than the 5nm Apple M2 or around 100 billion transistors. For comparison, the 4090 has 76 billion transistors. A 1nm Apple Silicon Max chip could theoretically have around 300 billion transistors, which makes it 4x more than a 4090. If GPT-4 requires 4x 4090 to run, then a single 1nm M Max might be able to do it in the future. 3. Larger models will optimize to require less resources. 4. Smaller models will become more capable. These forces will converge and we will have local LLMs that are top of the line.
moffkalast 3y ago
> How many devs are out there who would love having a local LLM model integrated into their IDE? The answer: Yes. Tbf, local models are still completely crap for that because the base model isn't as good. Software development is expensive so if a 1% better model cuts 10 hours of work it's already worth it. The problem is more in that you can't send private company data to OpenAI and back because they'll use it for training and leak your IP.
evc123 3y ago
> Whether scaling laws hold or not, is up for debate. The scaling laws have broken: https://arxiv.org/abs/2210.14891 https://arxiv.org/abs/2210.14891

YetAnotherNick 3y ago

> And they want to have it local, private, and as cheap as possible. Most people don't care about local and private. They care about cheap and free ChatGPT is the cheapest they could get, which is better than all open access models. > Open source foundation models on which tuning can be done are accelerating by the week Based on the (unverified)information I have been hearing in the research circle is that OpenAI invested heavily in data collection. One thing I heard is that they have hired experts to label lot of codes with information like complexity, correctness etc. Also the questions and RLHF data they get from real users is something open access models can't match. Palm 2 chat is significantly worse than ChatGPT(free), even though it is larger and trained by experts in the field.

mr_mitm 3y ago
> Most people don't care about local and private. Right, but corporations do. And AI is a revolution because it makes workers substantially more efficient, not because it makes for a slightly better Google. If corporations want to leverage AI to make their workforce more efficient, it must be local and private in many cases. OpenAI is already experiencing friction in the EU.
- yyyk 3y ago
  Same corporations happily use Office 365 and GSuite for all their documents rather than a local and private solution. I'm sure they'll come to a similar arrangement with regard to AI.
  - usrbinbash 3y ago
    That doesn't change the fact that these corporations have an interest in data privacy and cost reduction. To wit, if MS wouldn't guarantee data security in european datacenters, they would be way less competitive in the EU. Many corporations run their own datacenters, have on premises mailservers and run their own backup solutions. Not everything is in the cloud.

yyyk 3y ago

1) Giant serverfarms are expensive, but Microsoft has money and savings on scale. This can compare favourably to running locally. 2) People use what LLMs software makers give them and don't care at all about on premise. Software makers want something reliable they can deploy at scale, and not end up debugging problems on a client 4GB machine with an iGPU and multitude of OSs. The actually deciding people want to delegate to an API, and the only offerings available right now are from OpenAI, Google, Anthropic (aka Google again) etc.. 3) Open source LLM products and APIs are at the zero point and remain so by the week. "Download from GitHub and compile" is not going to fly. 4) Right now the question is whether the Closed Source and Propriety product people will actually use will be provided by a web API (openAI or maybe Meta) or by the OS (Apple, Microsoft, Google, etc.). Either way it will be controlled by giant corporations. Unless 3) ever changes, but doing product is icky for OSS people for some reason so that will never happen. [EDIT: Note that the Stable Diffusion area is in way better shape. There's a one-click installer for Auto111! There are some products and sites using SD! There are some usecases important commercial interests don't want! These OSS people also don't engage in triumphalism much less unearned triumphalism! They are probably still going to lose to Adobe and Midjourney but at least it will be a fight.]

senttoschool 3y ago
A long time ago, in order to get computers to do anything useful, the hardware required huge rooms. Today, a little cell phone is significantly more powerful than a room sized computer back in the days. LLMs shouldn't be any different. Today, they require giant server farms to run. Tomorrow, they will run on little robots/cell phones. A few things will allow this: 1. Chip makers like Apple, Qualcomm, AMD, Intel, Nvidia, will heavily emphasize AI performance in their future chips. Expect accelerators like the Neural Engine to get significantly bigger. In the future, I expect that most of the transistors in a SoC will be dedicated to AI acceleration - and not CPUs/GPUs. 2. Moore's law is not dead. We will continue to get more transistors in a given area. For example, TSMC has plans for 1nm which will likely have 4-5x more transistors per area than the 5nm Apple M2 or around 100 billion transistors. For comparison, the 4090 has 76 billion transistors. A 1nm Apple Silicon Max chip could theoretically have around 300 billion transistors, which makes it 4x more than a 4090. If GPT-4 requires 4x 4090 to run, then a single 1nm M Max might be able to do it in the future. 3. Larger models will optimize to require less resources. 4. Smaller models will become more capable. These forces will converge and we will have local LLMs that are top of the line.

moffkalast 3y ago

> How many devs are out there who would love having a local LLM model integrated into their IDE? The answer: Yes. Tbf, local models are still completely crap for that because the base model isn't as good. Software development is expensive so if a 1% better model cuts 10 hours of work it's already worth it. The problem is more in that you can't send private company data to OpenAI and back because they'll use it for training and leak your IP.

evc123 3y ago

> Whether scaling laws hold or not, is up for debate. The scaling laws have broken: https://arxiv.org/abs/2210.14891 https://arxiv.org/abs/2210.14891