4 ms·
which is important though since sending it across the wire over and over and over is actually the main bottleneck.
by mitxela 1mo ago
which is important though since sending it across the wire over and over and over is actually the main bottleneck.
- Kerbonut 1mo agoWire typically means internet connection, and it’s hardly the bottleneck
- deleted 1mo ago[deleted]
- 317070 1mo agoin the case of large language models, the wire is the communication of your parameters between your layers of memory that is often the bottleneck. To do a forward pass, you need to use all parameters once, and so the communication between the compute and the storage is the bottleneck, and that bottleneck is also a bunch of wires.
- mitxela 1mo agoThe other bottleneck is the amount of fast storage, which compression also improves.