You should be able to easily sustain below 5ms buffer size, and RT_PREEMPT Linux has no problems keeping a few hundred microseconds of buffer filled (a suitable

You should be able to easily sustain below 5ms buffer size, and RT_PREEMPT Linux has no problems keeping a few hundred microseconds of buffer filled (a suitable PCIe soundcard should easily make 100us buffer level reliable; that's 9.6 samples at 96kHz sample rate). Yes, sub-ms latency is hard. But the infrastructure exists.