3 ms·
I am missing a mention of ROM in the article. Keeping read-only weights in RAM is rather wasteful, as ROM can be implemented more cheaply. Approaches like High
by jononor 18d ago
I am missing a mention of ROM in the article. Keeping read-only weights in RAM is rather wasteful, as ROM can be implemented more cheaply. Approaches like High Bandwidth Flash (HBF) are relevant here, and should come to market in a few years.
Further optimization may be possible by tailoring for sequential access, since inference of a particular model is very predictable.
- Melatonic 18d agoExactly. If we ever get models that stabilize in terms of capabilities (and arent leapfrogging each other month after month) you have a "cartridge" like thing that just plugs in with the latest model in a read only ROM thats super fast. As far as the interconnect to the GPU/CPU - thats a different story. But with Nvidia acquiring Mellanox and Nvlink getting faster and faster I assume well get there
- abrookewood 18d agoModern ROM is not locked in stone. For example, Flash memory is considered a modern, advanced type of Read-Only Memory (ROM), specifically a specialized form of Electrically Erasable Programmable Read-Only Memory (EEPROM), and can be re-written.
- Melatonic 17d agoIsnt that slower though? Maybe Im not thinking of ROM specifically (the architecture/chip) but a different form of read only cache that can be located close to the compute and is super fast