4 ms·

There is actual, live compute costs every time you submit a prompt, and they're not unsubstantial. If you're not paying for them on your own hardware, you have

There is actual, live compute costs every time you submit a prompt, and they're not unsubstantial. If you're not paying for them on your own hardware, you have to rent someone else's hardware to do it on a token-by-token basis or all-you-can-eat-basis. There is no way to save or cache any of this because it all works by predicting what comes next after the entire corpus plus your newly-written prompt. Unless you have that entire model stored, you can't do the calculation.

hitsy00 2y ago
So I am talking about allowing users to bring their own keys. Business model like Typemind.
- dtagames 2y ago
  Allowing a user to bring their own key to a backing LLM over-complicates the product for the developer. Now, instead of managing your account with them, they also have to manage your account with the LLM provider, and they will be restricted by the limits of that account. They can't take advantage of any economy of scale by using their own keys and accounts. It's a huge mess and security nightmare to hold keys for other users and usually not worth it, unless that itself is your business.
- talldayo 2y ago
  You could try that, but I think 99% of SaaS owners would rather have 10 monthly subscriptions than 100 lifetime key owners. There isn't a huge incentive to split out your product into it's constituent parts when you can market the whole thing and charge full price. This doesn't just extend to AI, most API-based interfaces don't offer you a choice between your provider.
  - hitsy00 2y ago
    True. But my logic was that in an environment where SAAS is going hyperbole towards 0, this would be a great way to set ourselves apart. Oir product is definitely better than most of our competitors. But this could be a way to sink the costs entirely and set ourselves apart from our competitors.

4 ms·

There is actual, live compute costs every time you submit a prompt, and they're not unsubstantial. If you're not paying for them on your own hardware, you have

by dtagames 2y ago

hitsy00 2y ago
So I am talking about allowing users to bring their own keys. Business model like Typemind.
- dtagames 2y ago
  Allowing a user to bring their own key to a backing LLM over-complicates the product for the developer. Now, instead of managing your account with them, they also have to manage your account with the LLM provider, and they will be restricted by the limits of that account. They can't take advantage of any economy of scale by using their own keys and accounts. It's a huge mess and security nightmare to hold keys for other users and usually not worth it, unless that itself is your business.
- talldayo 2y ago
  You could try that, but I think 99% of SaaS owners would rather have 10 monthly subscriptions than 100 lifetime key owners. There isn't a huge incentive to split out your product into it's constituent parts when you can market the whole thing and charge full price. This doesn't just extend to AI, most API-based interfaces don't offer you a choice between your provider.
  - hitsy00 2y ago
    True. But my logic was that in an environment where SAAS is going hyperbole towards 0, this would be a great way to set ourselves apart. Oir product is definitely better than most of our competitors. But this could be a way to sink the costs entirely and set ourselves apart from our competitors.