4 ms·

Compute being the majority of costs is the reality for every single LLM in existence. This isn't an opinion i'm trying to convince you of.

imtringued 3y ago
Compute isn't the majority of costs at all. It's primarily memory capacity and memory bandwidth. Tenstorrent's Grayskull chip has 384 TOPS and only a paltry 8GB memory capacity with roughly 120GB/s memory bandwidth. The compute vs memory ratio is so heavily skewed towards compute its ridiculous. 3200 OPS per byte/s of memory bandwidth. 48000 OPS per byte of memory capacity. How can compute possibly ever be a bottleneck? The moment you start thinking about connecting these together to form a 100 TB system, you need 12.5k nodes or $10 million. A billion dollar budget means you can afford multiple one petabyte systems. Admittedly, I am ignoring the cost of the data center itself, but given enough money, there is no bottleneck on the compute or memory side at all! How exactly are you going to find enough data to feed these monstrous petabyte scale models? You won't! Your models will stay small and become cheap over time!
- famouswaffles 3y ago
  Memory is part of compute. I have no idea why you've separated them. >How exactly are you going to find enough data to feed these monstrous petabyte scale models? You won't! Your models will stay small and become cheap over time! People have this idea that we're stretching the limit of tokens to feed to models. Not even close. This is a 30T token dataset https://www.together.ai/blog/redpajama-data-v2 https://www.together.ai/blog/redpajama-data-v2 It's just web scrapped which means all the writing largely inaccessible on the web (swaths of fiction, textbooks, papers etc) and in books are not a part of it. It doesn't contain content in some of the most written languages on earth (e.g Chinese. You can easily get trilions of tokens of Chinese alone)

imtringued 3y ago

Compute isn't the majority of costs at all. It's primarily memory capacity and memory bandwidth. Tenstorrent's Grayskull chip has 384 TOPS and only a paltry 8GB memory capacity with roughly 120GB/s memory bandwidth. The compute vs memory ratio is so heavily skewed towards compute its ridiculous. 3200 OPS per byte/s of memory bandwidth. 48000 OPS per byte of memory capacity. How can compute possibly ever be a bottleneck? The moment you start thinking about connecting these together to form a 100 TB system, you need 12.5k nodes or $10 million. A billion dollar budget means you can afford multiple one petabyte systems. Admittedly, I am ignoring the cost of the data center itself, but given enough money, there is no bottleneck on the compute or memory side at all! How exactly are you going to find enough data to feed these monstrous petabyte scale models? You won't! Your models will stay small and become cheap over time!

famouswaffles 3y ago
Memory is part of compute. I have no idea why you've separated them. >How exactly are you going to find enough data to feed these monstrous petabyte scale models? You won't! Your models will stay small and become cheap over time! People have this idea that we're stretching the limit of tokens to feed to models. Not even close. This is a 30T token dataset https://www.together.ai/blog/redpajama-data-v2 https://www.together.ai/blog/redpajama-data-v2 It's just web scrapped which means all the writing largely inaccessible on the web (swaths of fiction, textbooks, papers etc) and in books are not a part of it. It doesn't contain content in some of the most written languages on earth (e.g Chinese. You can easily get trilions of tokens of Chinese alone)