From experience trying to minimise latency of Raspberry Camera stream, two biggest factors of latency are H264 encoding (maybe because it's temporal) and RTMP.

From experience trying to minimise latency of Raspberry Camera stream, two biggest factors of latency are H264 encoding (maybe because it's temporal) and RTMP. MJPEG and UDP/TCP streaming work better (for me at least).