Batching just uses hardware capabilities efficiently. Either vector instructions in CPUs or batch capabilities in GPUs. For the same hardware load, you can pro

Batching just uses hardware capabilities efficiently. Either vector instructions in CPUs or batch capabilities in GPUs. For the same hardware load, you can process several samples instead of just one. Pre-computation means running your model on samples in advance, before the model result is needed, so it's ready to use instantly when needed. Caching works probably because there are model results that are reused again and again, so it makes sense to cache them. For example, there are deep models that process room pictures, room and customer characteristics. Only customer characteristics change between customers, so it makes sense to cache the features output by the deep CNN that processes the room pictures. Once you start doing prediction at scale, there are lots of these optimizations to pick up.