5 ms·

Surprisingly, their scaling law analysis still focuses on training FLOPs instead of training + inference FLOPs. That said, they do mention this: > The largest

Surprisingly, their scaling law analysis still focuses on training FLOPs instead of training + inference FLOPs. That said, they do mention this: > The largest model in the PaLM 2 family, PaLM 2-L, is significantly smaller than the largest PaLM model but uses more training compute. [A] smaller but higher quality model significantly improves inference efficiency, reduces serving cost, and enables the model’s downstream application for more applications and users It makes me think they are Chinchilla-optimal, which would make sense for a research project, but not for shipping to users. I am surprised they didn’t train to the validation loss plateau.

jumpCastle 3y ago
Optimazing for training could help distillation also.
- sanxiyn 3y ago
  I agree distillation is the wild card. The question is whether distillation works for LLM. I am not aware of any public report of successful distillation of LLM (I searched quite hard for this; if you know of any and can tell me I would be very grateful), and I interpreted it to mean that it doesn't work yet and negative results are not published due to publication bias.
  - mrbungie 3y ago
    This was published here in HN last week: https://news.ycombinator.com/item?id=35810663 https://news.ycombinator.com/item?id=35810663 Don't know if there any public technical reports by any of the big AI companies about this, as its pretty new.
    - sanxiyn 3y ago
      No, distilling step-by-step https://arxiv.org/abs/2305.02301 https://arxiv.org/abs/2305.02301 distills LLM to task specific model. That works, and I know of multiple successes. But it doesn't relate to choice of optimizing training FLOP vs training and inference FLOP, since the resulting distilled model is not LLM.
  - fpgaminer 3y ago
    Off the top of my head there's DistilBERT from awhile back. I also recall distilled GPT-2 models from before the GPT-3 times.
    - sanxiyn 3y ago
      Yes, DistilBERT https://arxiv.org/abs/1910.01108 https://arxiv.org/abs/1910.01108 is in fact the closest case I know of. But it is too small (distilling from 110M to 66M) and both BERT and DistilBERT is intended to be used (and benchmarked) with separate fine tuning for specific tasks, so they are not general.
  - jumpCastle 3y ago
    The name 3.5-turbo sounds to me like it implies distillation. The release notes at the time also hinted at it IIRC.
    - sanxiyn 3y ago
      Well, that's why I said public. Personally, I don't think release notes https://help.openai.com/en/articles/6825453-chatgpt-release-notes https://help.openai.com/en/articles/6825453-chatgpt-release-... hinted at any such thing, and I think quantization is more likely than distillation.
      - kristianp 3y ago
        Does the turbo API being 10 times cheaper than davinci imply anything? It implies more than just quantisation to me.
        cubefox 3y ago
        "davinci" is the original GPT-3 (175B) which had too many parameters per Chinchilla scaling law. And parameter count is strongly correlated with inference cost. GPT-3.5 is likely Chinchilla optimal and much smaller than davinci. Though this theory has the defect that GPT-4 is, I think, more expensive than GPT-3, but as I recall it was considered unlikely that GPT-4 is larger than 175 billion parameters. Not sure.
      - ImprobableTruth 3y ago
        Turbo uses a different vocabulary (same one as gpt-4). Indicates that it's not the same model as the original 3.5, so I would be very surprised if it wasn't distilled.
deleted 3y ago
[deleted]
haldujai 3y ago
Depends on your goal, if it's to overtake OpenAI as having the best model overall it makes sense to optimize for training loss alone (assuming a fixed upfront compute budget). Optimizing for inference to achieve the same loss would require more compute overall so you're either paying upfront with higher training costs or kicking the can down the road to inference. News articles estimates of GPT4 cost seem to peg it at ~8 months of inference to achieve 1:1 cost with training. Life span of these models is TBD but it's a pretty safe bet we'll have new ones by then. Of course GPT3.5 is still getting used but probably won't cross 2:1ish in its lifetime. Might as well roll the dice and kick the can down the road if you're Google, I imagine they would happily pay an extra 500k/day in inference compute to be market leaders, whats 183mill for them? But if they don't get any real market share or the model sucks they saved substantially on training. > It makes me think they are Chinchilla-optimal, They elaborate in the appendix but they empirically determine PaLM-optimal, which concurs with Chinchilla-optimal (more or less).
cubefox 3y ago
They also mention this: > Moreover, there are several other considerations besides the optimal training loss, such as training throughput and serving latency, which affect the decision regarding the optimal model size. And they also mention, right before that, that "lower training loss" might not exactly mean "higher performance": > However, the training loss is not a perfect proxy for downstream metrics. For example, the 8.95B model, which shows the lowest loss (Table 1) and is closest to the optimal model, slightly underperforms the 14.7B model on downstream tasks. This suggests that while scaling laws can be used to achieve optimal training loss for a given quantity of FLOPs, this does not necessarily transfer to achieving optimal performance for a given task. That might be a random outlier, but ... The Chinchilla scaling law describes how to balance parameters and training tokens to achieve minimal training loss for a given amount of compute. Low training loss is a good proxy for model performance (intelligence) but perhaps it is somewhat off? For example, Chinchilla says that for optimal loss, we have to scale training tokens and parameters equally (50%/50%). But perhaps for optimal model "intelligence" we need something slightly different, e.g. 60% parameters and 40% training tokens. Of course this seems somewhat unlikely, since it would mean such models are systematically smarter but systematically worse at predicting text compared to Chinchilla optimal models trained with the same amount of compute.
Herring 3y ago
"Surprisingly, their scaling law analysis still focuses on training FLOPs instead of training + inference FLOPs." It's kind of weird. In the conclusion, they say >With PaLM 2, we have independently verified the scaling laws from Hoffmann et al. (2022) at large scales; we have shown that training tokens should grow at roughly the same rate as the number of model parameters. then a few lines later >In effect, we find that it is generally more efficient to train a smaller model with more tokens, for a fixed inference and training budget. Without more architecture details, it's hard to tell what they're going on about.

jumpCastle 3y ago

Optimazing for training could help distillation also.

sanxiyn 3y ago
I agree distillation is the wild card. The question is whether distillation works for LLM. I am not aware of any public report of successful distillation of LLM (I searched quite hard for this; if you know of any and can tell me I would be very grateful), and I interpreted it to mean that it doesn't work yet and negative results are not published due to publication bias.
- mrbungie 3y ago
  This was published here in HN last week: https://news.ycombinator.com/item?id=35810663 https://news.ycombinator.com/item?id=35810663 Don't know if there any public technical reports by any of the big AI companies about this, as its pretty new.
  - sanxiyn 3y ago
    No, distilling step-by-step https://arxiv.org/abs/2305.02301 https://arxiv.org/abs/2305.02301 distills LLM to task specific model. That works, and I know of multiple successes. But it doesn't relate to choice of optimizing training FLOP vs training and inference FLOP, since the resulting distilled model is not LLM.
- fpgaminer 3y ago
  Off the top of my head there's DistilBERT from awhile back. I also recall distilled GPT-2 models from before the GPT-3 times.
  - sanxiyn 3y ago
    Yes, DistilBERT https://arxiv.org/abs/1910.01108 https://arxiv.org/abs/1910.01108 is in fact the closest case I know of. But it is too small (distilling from 110M to 66M) and both BERT and DistilBERT is intended to be used (and benchmarked) with separate fine tuning for specific tasks, so they are not general.
- jumpCastle 3y ago
  The name 3.5-turbo sounds to me like it implies distillation. The release notes at the time also hinted at it IIRC.
  - sanxiyn 3y ago
    Well, that's why I said public. Personally, I don't think release notes https://help.openai.com/en/articles/6825453-chatgpt-release-notes https://help.openai.com/en/articles/6825453-chatgpt-release-... hinted at any such thing, and I think quantization is more likely than distillation.
    - kristianp 3y ago
      Does the turbo API being 10 times cheaper than davinci imply anything? It implies more than just quantisation to me.
      - cubefox 3y ago
        "davinci" is the original GPT-3 (175B) which had too many parameters per Chinchilla scaling law. And parameter count is strongly correlated with inference cost. GPT-3.5 is likely Chinchilla optimal and much smaller than davinci. Though this theory has the defect that GPT-4 is, I think, more expensive than GPT-3, but as I recall it was considered unlikely that GPT-4 is larger than 175 billion parameters. Not sure.
    - ImprobableTruth 3y ago
      Turbo uses a different vocabulary (same one as gpt-4). Indicates that it's not the same model as the original 3.5, so I would be very surprised if it wasn't distilled.

deleted 3y ago

[deleted]

haldujai 3y ago

Depends on your goal, if it's to overtake OpenAI as having the best model overall it makes sense to optimize for training loss alone (assuming a fixed upfront compute budget). Optimizing for inference to achieve the same loss would require more compute overall so you're either paying upfront with higher training costs or kicking the can down the road to inference. News articles estimates of GPT4 cost seem to peg it at ~8 months of inference to achieve 1:1 cost with training. Life span of these models is TBD but it's a pretty safe bet we'll have new ones by then. Of course GPT3.5 is still getting used but probably won't cross 2:1ish in its lifetime. Might as well roll the dice and kick the can down the road if you're Google, I imagine they would happily pay an extra 500k/day in inference compute to be market leaders, whats 183mill for them? But if they don't get any real market share or the model sucks they saved substantially on training. > It makes me think they are Chinchilla-optimal, They elaborate in the appendix but they empirically determine PaLM-optimal, which concurs with Chinchilla-optimal (more or less).

cubefox 3y ago

They also mention this: > Moreover, there are several other considerations besides the optimal training loss, such as training throughput and serving latency, which affect the decision regarding the optimal model size. And they also mention, right before that, that "lower training loss" might not exactly mean "higher performance": > However, the training loss is not a perfect proxy for downstream metrics. For example, the 8.95B model, which shows the lowest loss (Table 1) and is closest to the optimal model, slightly underperforms the 14.7B model on downstream tasks. This suggests that while scaling laws can be used to achieve optimal training loss for a given quantity of FLOPs, this does not necessarily transfer to achieving optimal performance for a given task. That might be a random outlier, but ... The Chinchilla scaling law describes how to balance parameters and training tokens to achieve minimal training loss for a given amount of compute. Low training loss is a good proxy for model performance (intelligence) but perhaps it is somewhat off? For example, Chinchilla says that for optimal loss, we have to scale training tokens and parameters equally (50%/50%). But perhaps for optimal model "intelligence" we need something slightly different, e.g. 60% parameters and 40% training tokens. Of course this seems somewhat unlikely, since it would mean such models are systematically smarter but systematically worse at predicting text compared to Chinchilla optimal models trained with the same amount of compute.

Herring 3y ago

"Surprisingly, their scaling law analysis still focuses on training FLOPs instead of training + inference FLOPs." It's kind of weird. In the conclusion, they say >With PaLM 2, we have independently verified the scaling laws from Hoffmann et al. (2022) at large scales; we have shown that training tokens should grow at roughly the same rate as the number of model parameters. then a few lines later >In effect, we find that it is generally more efficient to train a smaller model with more tokens, for a fixed inference and training budget. Without more architecture details, it's hard to tell what they're going on about.