4 ms·

You could probably run a model but not train a new one, especially one bigger than GPT-4. You'd need ten of thousand of A100 as a rough estimate.

by luxcem 4y ago

deleted 4y ago
[deleted]
bob1029 4y ago
I suspect some of us are going to be really disappointed when the actual parameter count for GPT-4 is discovered. Has anyone considered the opposite possibility around the closed nature of OpenAI's work? I.e. Instead of worrying that someone will reproduce their 10k+ GPU GPT-4 training super farm, maybe the opposite happened - They discovered a way to increase performance while simultaneously reducing the required parameter counts and/or hardware. Would that realization be more or less dangerous with this A100 constraint in mind?
- esperent 4y ago
  > They discovered a way to increase performance while simultaneously reducing the required parameter counts and/or hardware. In that case, why would they be so severely limiting access? If GPT4 was cheaper to run and better, surely they would have immediately transitioned away from 3.5?
- politician 4y ago
  (1) Some people speculate that GPT-4 is actually an ensemble of models selected by a supervisory model based on response timings. (2) GPT4All is using improved training data with LoRA and GPT-J to build something that competes with GPT-4 at a fraction of the size and parameter count. (3) The emergence of new capabilities strongly corresponds to parameter count. Transformers with 6.7B parameters are apparently where the first such emergence occurs associated with a global structural change in the hidden layers (the so-called “phase change”). There will be a strong incentive to train larger models to identify additional capabilities. The first and second observations imply that federal regulation of A100-class GPUs would be ineffective. However, the third observation suggests that it would be highly effective at preventing untrusted parties from discovering novel and potentially more disruptive capabilities.

You could probably run a model but not train a new one, especially one bigger than GPT-4. You'd need ten of thousand of A100 as a rough estimate.

by luxcem 4y ago

You could probably run a model but not train a new one, especially one bigger than GPT-4. You'd need ten of thousand of A100 as a rough estimate.

deleted 4y ago

[deleted]

bob1029 4y ago

I suspect some of us are going to be really disappointed when the actual parameter count for GPT-4 is discovered. Has anyone considered the opposite possibility around the closed nature of OpenAI's work? I.e. Instead of worrying that someone will reproduce their 10k+ GPU GPT-4 training super farm, maybe the opposite happened - They discovered a way to increase performance while simultaneously reducing the required parameter counts and/or hardware. Would that realization be more or less dangerous with this A100 constraint in mind?

esperent 4y ago
> They discovered a way to increase performance while simultaneously reducing the required parameter counts and/or hardware. In that case, why would they be so severely limiting access? If GPT4 was cheaper to run and better, surely they would have immediately transitioned away from 3.5?
politician 4y ago
(1) Some people speculate that GPT-4 is actually an ensemble of models selected by a supervisory model based on response timings. (2) GPT4All is using improved training data with LoRA and GPT-J to build something that competes with GPT-4 at a fraction of the size and parameter count. (3) The emergence of new capabilities strongly corresponds to parameter count. Transformers with 6.7B parameters are apparently where the first such emergence occurs associated with a global structural change in the hidden layers (the so-called “phase change”). There will be a strong incentive to train larger models to identify additional capabilities. The first and second observations imply that federal regulation of A100-class GPUs would be ineffective. However, the third observation suggests that it would be highly effective at preventing untrusted parties from discovering novel and potentially more disruptive capabilities.