3 ms·
LLM latency is a huge no go for most apps except for chat apps. I’ve try to build apps based on OpenAI and that itself creates a bad experience no matter how m
by m3kw9 3y ago
LLM latency is a huge no go for most apps except for chat apps. I’ve try to build apps based on OpenAI and that itself creates a bad experience no matter how much elevator music/mirrors/spinners you place. Then you need proper error correction when dealing with structured responses/occasional hallucinations
- PUSH_AX 3y agoLatency is acceptable when the value of the result outweighs the time you believe to be excess. If AI can build me a useful marketing plan in 15 mins with multiple agents doing work this seems fine to me. It’s going to take much longer to get a human involved.
- m3kw9 3y agoYou are assuming it gets you the result first time, it rarely does and you always need many back and forths. It’s still ok in your case if the results can kick start something, but you’d be nuts to assume you can use that all the way
- PUSH_AX 3y agoUltimately the same concept applies in my opinion, ROI for both effort and time to be clear. Even today it's in a good place for a lot of things, and will only get better.