3 ms·
Does it need to reinitialize for each request or is there a warm start / cold start model like lambda? I don't really understand how you can charge per request.
by spullara 5y ago
Does it need to reinitialize for each request or is there a warm start / cold start model like lambda? I don't really understand how you can charge per request.
- rubatuga 5y agoI guess GPU loading is quick? Like ~10 seconds?
- theo31 5y agoThere is no cold start! We keep your service hot all the time.
- spullara 5y agoWell, I guess I know where I am going to host GPT-J-6B then. I don't think it is sustainable.
- maldeh 5y agoThe pricing appears to be static per model with a ceiling on the monthly request count, not charged per request. Edit: Actually, I didn't spot the free tier of 1000 requests. I wonder how you avoid the problem of a lot of users leaving defunct/disused models running while still keeping them hot - presumably some kind of limit to the model count?