That's a good point, though these folks do specifically call this a "Streaming Processor", so I'm not sure that using anything besides a batch size of 1 for inf

That's a good point, though these folks do specifically call this a "Streaming Processor", so I'm not sure that using anything besides a batch size of 1 for inference is entirely fair, but perhaps I'm misunderstanding what they mean by streaming. I was also thinking that the A100 comparison would be very interesting.