2 ms·
Fair point, but you’ll still be bounded by disk read speed on an SSD. The access pattern itself matters less than the read cache being << the parameter set size
by toxik 4y ago
Fair point, but you’ll still be bounded by disk read speed on an SSD. The access pattern itself matters less than the read cache being << the parameter set size.
- lostmsu 4y agoTop SSDs do over 4GB/s so you can infer in 50 seconds if disk bound. You can also infer a few tokens at once, so it will be more than 1 char a minute. Probably more like sentence a minute.
- toxik 4y agoYou can read bits at that rate yes, but keep in mind that it’s 250 GiB /parameters/, and matrix-matrix multiplication is typically somewhere between quadratic and cubic in complexity. Then you get to wait for the page out of your intermediate result etc etc. It’s difficult to estimate how slow it would be, but I’m guessing unusably slow.
- lostmsu 4y agoThe intermediate result will all fit into a relatively small amount of memory. During inference you only need to keep layer outputs until the next layer's outputs are computed. If we talk about memory bandwidth, it is space requirements that are important, not so much time complexity.