3 ms·No one who is using this for home use cares about anything except batch size 1 sequence size 1.by Der_Einzige 2y agoNo one who is using this for home use cares about anything except batch size 1 sequence size 1.ein0p 2y agoWhat if you're doing bulk inference? The efficiency and throughput of bs=1 s=1 is truly abysmal.saagarjha 2y agoPeople want to talk to their computer, not service requests for a thousand users.