Either you’re queuing the prompts to smooth out traffic variability without growing the cluster size, or Little’s Law kills you here. If queries actually take a

Either you’re queuing the prompts to smooth out traffic variability without growing the cluster size, or Little’s Law kills you here. If queries actually take an hour on average, and my customers are starting one task per second, then I have 3600 tasks in progress at the same time. I think the main advantage to slow queries would be in discouraging users from monopolizing the system by shotgunning many requests in a short period. Which is probably one of those cost cutting measures that costs you customers. If I asked it to write me a block of code and it told me one hour, I’d just write it myself and tell my boss we aren’t really using the service.