3 ms·
What do you mean by commodity hardware? Single server single CPU socket x86/ARM boxes? Anything that does not have a GPU?
by c_o_n_v_e_x 3y ago
What do you mean by commodity hardware? Single server single CPU socket x86/ARM boxes? Anything that does not have a GPU?
- deleted 3y ago[deleted]
- KronisLV 3y ago> What do you mean by commodity hardware? In my case, my local workstation has a Ryzen 5 1600 desktop CPU from 2017 (first generation Zen, 14nm) and it still worked decently. Of course, response times would grow with longer inputs and outputs or larger models, but getting a response in less than a minute when running off of purely CPU is encouraging in of itself.