4 ms·
To make sure I understand…this would provide a private API endpoint for a developer to call an LLM model in a serverless way? They could call it and just pay f
by iambateman 3y ago
To make sure I understand…this would provide a private API endpoint for a developer to call an LLM model in a serverless way?
They could call it and just pay for the time spent, not a persistent server.
- dartos 3y agoSure, there are platforms that do this (replicate) The issue is cold start time for custom models. It takes time to pull in dependencies, the model, and load the model into memory. It’s difficult because someone has to pay for the cold start time.