5 ms·

During every GPT-4 request, eight of these[0] cards are utilized to 100%. EUV wafers are borderline black magic, only so many can be produced in a day. [0] htt

During every GPT-4 request, eight of these[0] cards are utilized to 100%. EUV wafers are borderline black magic, only so many can be produced in a day. [0] https://www.newegg.com/p/1FT-0004-00814 https://www.newegg.com/p/1FT-0004-00814

BizarroLand 3y ago
That is entirely a problem that can be solved by throwing money at it. Why are they complaining to the public about their own organizational torpor? Microsoft should have the maturity, money, and any other resources needed to solve their own problems without making a public spectacle of it. It's embarrassing and childish. I am being critical of them, but their net worth is nearly 1/10th of the debt of the entire United States of America. If they can't get it together with that level of resources, then they should be ashamed of themselves.
copperx 3y ago
I've heard this line before, but I have a hard time understanding it. Does that mean that a single user has an exclusive lock on 8 of these things for the duration of the request? I'm surely missing something.
- soulofmischief 3y ago
  Yes, these large language models are extremely constrained by RAM. Traditional PC architecture separate RAM and VRAM, so GPUs are required to have the appropriate amount of memory onboard, in this case achieved via 8 A100s synchronized to act as one compute module. This is why developers such as myself have opted for Apple's new unified architecture to run smaller models locally, as the GPU can access all 64GB. But these small models are still a fraction of the size of GPT-3.5-Turbo and GPT-4.
  - thefurdrake 3y ago
    I was very sad to discover that my 10GB 3080 can't run any compelling LLMs on initial overview, and that the A100s are well above 5grand in price. I wish something like mythic ai's analog chips would come out in the consumer arena.
- flangola7 3y ago
  > Does that mean that a single user has an exclusive lock on 8 of these things for the duration of the request? I'm surely missing something. You're not missing anything, that's exactly what it means. There is a reason OpenAI severely ratelimits the number of GPT-4 prompts users can make. Every token (approximately 4 letters) requires many trillions of floating point operations. Large language models are only possible because of decades of exponential transistor density growth.

BizarroLand 3y ago

That is entirely a problem that can be solved by throwing money at it. Why are they complaining to the public about their own organizational torpor? Microsoft should have the maturity, money, and any other resources needed to solve their own problems without making a public spectacle of it. It's embarrassing and childish. I am being critical of them, but their net worth is nearly 1/10th of the debt of the entire United States of America. If they can't get it together with that level of resources, then they should be ashamed of themselves.

copperx 3y ago

I've heard this line before, but I have a hard time understanding it. Does that mean that a single user has an exclusive lock on 8 of these things for the duration of the request? I'm surely missing something.

soulofmischief 3y ago
Yes, these large language models are extremely constrained by RAM. Traditional PC architecture separate RAM and VRAM, so GPUs are required to have the appropriate amount of memory onboard, in this case achieved via 8 A100s synchronized to act as one compute module. This is why developers such as myself have opted for Apple's new unified architecture to run smaller models locally, as the GPU can access all 64GB. But these small models are still a fraction of the size of GPT-3.5-Turbo and GPT-4.
- thefurdrake 3y ago
  I was very sad to discover that my 10GB 3080 can't run any compelling LLMs on initial overview, and that the A100s are well above 5grand in price. I wish something like mythic ai's analog chips would come out in the consumer arena.
flangola7 3y ago
> Does that mean that a single user has an exclusive lock on 8 of these things for the duration of the request? I'm surely missing something. You're not missing anything, that's exactly what it means. There is a reason OpenAI severely ratelimits the number of GPT-4 prompts users can make. Every token (approximately 4 letters) requires many trillions of floating point operations. Large language models are only possible because of decades of exponential transistor density growth.