The library which runs your ML model will do multithreading. Depending on model size, it may perform the best when passed multiple samples at a time to process.

The library which runs your ML model will do multithreading. Depending on model size, it may perform the best when passed multiple samples at a time to process. For this reason, it can be useful to implement "micro batching", which combines multiple samples from independent requests into batches that are processed together. In such a model one would have n_processes << n_cpus. Which tends to help with RAM usage also.