> That is a step up from current model limitations which require ram or vram to hold the model. Current? Apple recently published a neat paper on how they opti

> That is a step up from current model limitations which require ram or vram to hold the model. Current? Apple recently published a neat paper on how they optimise for both inference (cpu/gpu) and memory use: Our method involves constructing an inference cost model that takes into account the characteristics of flash memory, guiding us to optimize in two critical areas: reducing the volume of data transferred from flash and reading data in larger, more contiguous chunks. Within this hardware-informed framework, we introduce two principal techniques. First, “windowing” strategically reduces data transfer by reusing previously activated neurons, and second, “row-column bundling”, tailored to the sequential data access strengths of flash memory, increases the size of data chunks read from flash memory. These methods collectively enable running models up to twice the size of the available DRAM, with up to 4x and 20x increase in inference speed compared to naive loading approaches in CPU and GPU, respectively. https://news.ycombinator.com/item?id=38704982 https://news.ycombinator.com/item?id=38704982