5 ms·
Wire typically means internet connection, and it’s hardly the bottleneck
by Kerbonut 2mo ago
Wire typically means internet connection, and it’s hardly the bottleneck
- deleted 2mo ago[deleted]
- 317070 2mo agoin the case of large language models, the wire is the communication of your parameters between your layers of memory that is often the bottleneck. To do a forward pass, you need to use all parameters once, and so the communication between the compute and the storage is the bottleneck, and that bottleneck is also a bunch of wires.
- mitxela 2mo agoThe other bottleneck is the amount of fast storage, which compression also improves.