An excellent article! >"During inference, layers are executed sequentially. The output of the previous layer is the input to the next. Only one layer executes

An excellent article! >"During inference, layers are executed sequentially. The output of the previous layer is the input to the next. Only one layer executes at a time. Therefore, it is completely unnecessary to keep all layers in GPU memory. We can load whichever layer is needed from disk..." ...Or from the RAM of the main computer the Graphics Card is attached to, via the PCIe Bus and DMA (or whatever other methods exist now or in the future)... >"...when executing that layer, do all the calculations, and then completely free the memory after." ...Or keep it in the adjacent RAM of the main computer, for more speed... (Random idea here for hardware designers incidentally: 1) Graphics Card/GPU manufacturers -- put a high-speed CXL (Compute Express Link) interface on your Graphics card (https://en.wikipedia.org/wiki/Compute_Express_Link https://en.wikipedia.org/wiki/Compute_Express_Link). 2) Independent Hardware Designers -- design a stand-alone RAM board (or RAM board rack/tower) which has a high-speed CXL interface on it, which can be interfaced to #1, above. If #1 and #2 exist, now we have Graphics Cards/GPU's that can be interfaced to very large high-speed banks of external RAM -- effectively making the limited local memory of a GPU, a thing of the past... Yes, there might be a slight latency, slightly above that of local GPU memory when accessing this adjacent memory. But I'm guessing that if this latency exists, then over time this will be engineered away... In the short term, well designed software and/or caching strategies could compensate for any latency overhead, if there is one... In conclusion, Graphics Card / GPU + CXL + External Rack o' RAM -- could be a very winning combination...) >"This way, the GPU memory required per layer is only about the parameter size of one transformer layer, 1/80 of the full model, around 1.6GB." A brilliant observation and a brilliantly written article! Well done!