i think this matters more for lower batch sizes (local llm and private enterprise deployment where there wont be big user at specific time for big batch size) g

i think this matters more for lower batch sizes (local llm and private enterprise deployment where there wont be big user at specific time for big batch size) going from mem Io bottleneck to compute.