Compute is the latency for LLMs :) And in general, your inference code will be compiled to a CPU/Architecture target - so you can know ahead of time what instr

Compute is the latency for LLMs :) And in general, your inference code will be compiled to a CPU/Architecture target - so you can know ahead of time what instructions you'll have access to when writing your code for that target. For example in the case of AWS Lambda, you can choose graviton2 (ARM with NEON), or x86_64 (AVX). The trick is that for some processors such as Xeon3+ there is AVX 512, and others you will top out at AVX 256. You might be able to figure out what exact instruction set your serverless target supports.