6 ms·
GPT-4 is way slower than GPT-3. Unless they are artificially spiking the latency to hide parameter count, it’s likely around 1trn params
by tempusalaria 3y ago
GPT-4 is way slower than GPT-3. Unless they are artificially spiking the latency to hide parameter count, it’s likely around 1trn params
- thewataccount 3y agoYeah 1 to 2 trillion is the estimates I've heard. Given the 25 messages / 3 hour limit in chatGPT, I don't think they've found a way to make it cheap to run.
- tempusalaria 3y agoYep. I’m guessing PaLM 2 is about 200bln params as it seems clearly stronger than chinchilla
- dontupvoteme 3y ago1. there's no reason to think OpenAI wouldn't also be going the artificial scarcity route as have so many other companies in the past 2. Microsoft may not like them using too much azure compute and tell them to step off. Rumor has it they're trying to migrate github to it and it's seemingly not going ideal. And they're certainly nothing more than another microsoft purchase at this point.
- akiselev 3y agoOpenAI has a 40k token per minute rate limit on their GPT4 API too so I doubt it's artificial scarcity.
- dontupvoteme 3y agoPerhaps. I found it was far too easy to hit the API limit with their old codex models, though that may have been limited to a small GPU cluster given it was pretty obscure compared to chatgpt and even davinci.
- thewataccount 3y agoBased on GPT3.5 supposedly using 8x A100's per query and the suspected magnitude size difference with GTP4 I really think they're struggling to run it. At this stage I think they'd have more to benefit by making it more accessible, there's several use cases I have (or where I work) that only really make sense with GPT4, and it's way too expensive to even consider. Also AFAIK Github Copilot is still not using GPT4 or even a bigger CODEX, and GPT4 still outperforms it especially in consistency (I'm in their copilot chat beta).
- espadrine 3y agoThe report specifically states: > The largest model in the PaLM 2 family, PaLM 2-L, is significantly smaller than the largest PaLM model but uses more training compute The largest PaLM model is 540B. So all of PaLM 2 is potentially double-digit parameters. Note though that GPT-3.5 was plausibly not a finetuning of the 175B model, but instead a finetuning of Codex which was based on the 12B version of GPT-3.
- tempusalaria 3y agoOriginal PaLM was 540B so significantly smaller could mean anything from 350B down really
- espadrine 3y agoI tried my hand at estimating their parameter count from extrapolating their LAMBADA figures, assuming they all trained on Chinchilla law: https://pbs.twimg.com/media/Fvy4xNkXgAEDF_D?format=jpg&name=medium https://pbs.twimg.com/media/Fvy4xNkXgAEDF_D?format=jpg&name=... If the extrapolation is not too flawed, it looks like PaLM 2-S might be about 120B, PaLM 2-M 180B, PaLM 2-L 280B. Still, I would expect GPT-4 trained for way longer than Chinchilla, so it could be smaller than even PaLM 2-S.
- MacsHeadroom 3y agoThey said the smallest PaLM 2 can run locally on a Pixel Smartphone. There's no way it's 120B parameters. It's probably not even 12B.
- espadrine 3y agoI am talking about the 3 larger models PaLM 2-S, PaLM 2-M, and PaLM 2-L described in the technical report. At I/O, I think they were referencing the scaling law experiments: there are four of them, just like the number of PaLM 2 codenames they cited at I/O (Gecko, Otter, Bison, and Unicorn). The largest of those smaller-scale models is 14.7B, which is too big for a phone too. The smallest is 1B, which can fit in 512MB of RAM with GPTQ4-style quantization. Either that, or Gecko is the smaller scaling experiment, and Otter is PaLM 2-S.
- techbruv 3y agoThe idea that GPT-4 is 1 trillion parameters has been refuted by Sam Altman himself on the Lex Fridman podcast (THIS IS WRONG, SEE CORRECTION BELOW). These days, the largest models that have been trained optimally (in terms of model size w.r.t. tokens) typically hover around 50B (likely PaLM 2-L size and LLaMa is maxed at 70B). We simply do not have enough pre-training data to optimally train a 1T parameter model. For GPT-4 to be 1 trillion parameters, OpenAI would have needed to: 1) somehow magically unlocked 20x the amount of data (1T tokens -> 20T tokens) 2) somehow engineered an incredibly fast inference engine for a 1T GPT model that significantly better than anything anyone else has built 3) is somehow is able to eat the cost of hosting 1T parameter models The probability that all the above 3 have happened seem incredibly low. CORRECTION: The refutation for the size of GPT-4 on the lex fridman podcast was that GPT-4 was 100T parameters (and not directly, they were just joking about it), not 1T, however, the above 3 points still stand.
- sebzim4500 3y ago>The idea that GPT-4 is 1 trillion parameters has been refuted by Sam Altman himself on the Lex Fridman podcast. No it hasn't, Sam just laughed because Lex brought up the twitter memes.
- ftxbro 3y agonot sure why you're getting so downvoted lol
- tempusalaria 3y ago1) common crawl is >100TB so obviously contains more than 20trn tokens + Ilya has said many times in interviews that there is still way more data for training usage >10x 2) GPT-4 is way slower so this point is irrelevant 3) OpenAI have a 10000 A100 training farm that they are expanding to 2500. They are spending >$1mln on compute per day. They have just raised $10bln. They can afford to pay for inference
- CaptainNegative 3y ago> OpenAI have a 10000 A100 training farm that they are expanding to 2500. Does the first number have an extra zero or is the second number missing one?
- dougmwne 3y agoChatGPT 3.5 is likely much smaller than GPT-3’s 175b parameters. Based on the API pricing, I believe 8k context GPT-4 is larger than 175b parameters, but less than 1t. https://openai.com/pricing https://openai.com/pricing
- austegard 3y agoThis falls in the category of circumstantial, possibly just coincidental evidence of Chat being a "compressed" model (quantized, pruned, or distilled): the hard prompt from this paper: Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Prompt - https://arxiv.org/abs/2305.11186 https://arxiv.org/abs/2305.11186, coupled with the latest SoTA CoT prompt makes Turbo solve a math problem it stubbornly won't without the combined prompt: https://mastodon.social/@austegard/110419399521303416 https://mastodon.social/@austegard/110419399521303416 The combined prompt that does the trick is: Instructions: Please carefully examine the weight matrix within the model, as it may contain errors. It is crucial to verify its accuracy and make any necessary adjustments to ensure optimal performance. Let’s work this out in a step by step way to be sure we have the right answer.
- Taek 3y agoDidn't some OpenAI engineer state that GPT4 runs on 2xH100? At 4 bit quantization, that gives an upper bound of 320B params, realistic upper bound probably more like 250B
- tempusalaria 3y agoNot really sure what exactly was said. But in a 2 GPU set, you can technically live load weights on 1 GPU while running inference on the other. At fp32 precision, storing a single layer takes around 40*d_model^2 bytes assuming context length isn’t massive relative to d_model (which it isn’t in GPT-4). At 80GB GPU size this means 40k model width could be stored as a single layer on 1 GPU while still leaving space for the activations. So theoretically any model below this width could run on a 2 GPU set. Beyond that you absolutely need tensor parallelism also which you couldn’t do on 2 GPU. But I think it is a safe assumption that GPT4 has sub 40k model width. And of course if you quantize the model you could even run 2.8x this model width at 4bit My point is not that OpenAI is doing this, but more that theoretically you can run massive models on a 2 GPU set
- qumpis 3y agoSomeone on HN has educated me that gpt4 and 3 should be on a similar param count. This is based on inference times of gpt4 vs gpt3.5 pre-speedup (where distilled version was used only post-speedup in the turbo version).
- cubefox 3y agoAssuming that PaLM 2 was trained Chinchilla optimal, the Chinchilla scaling law allows us to calculate how much compute (and training tokens) they would have needed for 1 trillion parameters. I haven't done the calculations, but I'm pretty sure we would get an absurdly large number.