3 ms·

I've heard this line before, but I have a hard time understanding it. Does that mean that a single user has an exclusive lock on 8 of these things for the dura

• soulofmischief 3y ago
  Yes, these large language models are extremely constrained by RAM. Traditional PC architecture separate RAM and VRAM, so GPUs are required to have the appropriate amount of memory onboard, in this case achieved via 8 A100s synchronized to act as one compute module. This is why developers such as myself have opted for Apple's new unified architecture to run smaller models locally, as the GPU can access all 64GB. But these small models are still a fraction of the size of GPT-3.5-Turbo and GPT-4.
  • thefurdrake 3y ago
    I was very sad to discover that my 10GB 3080 can't run any compelling LLMs on initial overview, and that the A100s are well above 5grand in price. I wish something like mythic ai's analog chips would come out in the consumer arena.
• flangola7 3y ago
  > Does that mean that a single user has an exclusive lock on 8 of these things for the duration of the request? I'm surely missing something. You're not missing anything, that's exactly what it means. There is a reason OpenAI severely ratelimits the number of GPT-4 prompts users can make. Every token (approximately 4 letters) requires many trillions of floating point operations. Large language models are only possible because of decades of exponential transistor density growth.