3 ms·

How much would you pay per month, reasonable? $10 a month is kind of reasonable given you need a cluster of GPUs to service the requests. The cheapest single

by fswd 4y ago

How much would you pay per month, reasonable? $10 a month is kind of reasonable given you need a cluster of GPUs to service the requests. The cheapest single GPU capable of responding relatively fast at the same performance of OpenAI is about $275/month. So basically 30 users at $10 a month just to break even on that cost. However, that GPU could probably handle only half dozen users, and 6 < 30. I could use cheaper older 1080ti but the response time/latency is going to be slower than copilot. And the electricity costs are now 4x. I work with language models and build SaaS. If you can help me figure it out, I could build a solution for you. The only angle I can think is charging $100-500/month, but for an ultra fine-tuned version specific to a framework tailored to a "no-code" crowd.

thaeli 4y ago
The other angle I'd really like to see is license safety. If it was possible to create a Copilot type tool trained exclusively on permissively licensed code (or even proprietary code that was licensed such that the tool subscription could include a very broad license grant) and with a business model that was willing to indemnify customers against copyright and patent claims - now that would be worth a hefty subscription fee.
- WithinReason 4y ago
  I wonder what percentage of open source code doesn't require attribution when copied
- chrismorgan 4y ago
  You’re probably making the already-classic blunder here, thinking that the GPL is somehow special. If the license requires attribution (which almost all do), you’re stuck if the “fair use” exemption Copilot claims falls down. The quantity of public-domain or public-domain-equivalent-licensed code out there is almost certainly not enough to train such a tool (by several orders of magnitude, most likely—you can tweak an existing model with small amounts of code, but I believe generating a useful model from scratch will require a lot more than you are likely to find). This kind of thing cannot succeed legally without the fair use exemption from copyright.
danpalmer 4y ago
> However, that GPU could probably handle only half dozen users It may only be able to host a handful of users making requests within the same second, but there's a lot of downtime when users are not triggering inference requests – time reading code, navigating around, etc. Add to that the fact that most engineers are having meetings, writing docs, testing things, they do a lot that isn't writing code. Then you add the fact that users don't work 24/7. Between all of this, I'd be surprised if a GPU wasn't able to handle ~hundreds to ~thousands of monthly active users.

by fswd 4y ago

thaeli 4y ago

The other angle I'd really like to see is license safety. If it was possible to create a Copilot type tool trained exclusively on permissively licensed code (or even proprietary code that was licensed such that the tool subscription could include a very broad license grant) and with a business model that was willing to indemnify customers against copyright and patent claims - now that would be worth a hefty subscription fee.

WithinReason 4y ago
I wonder what percentage of open source code doesn't require attribution when copied
chrismorgan 4y ago
You’re probably making the already-classic blunder here, thinking that the GPL is somehow special. If the license requires attribution (which almost all do), you’re stuck if the “fair use” exemption Copilot claims falls down. The quantity of public-domain or public-domain-equivalent-licensed code out there is almost certainly not enough to train such a tool (by several orders of magnitude, most likely—you can tweak an existing model with small amounts of code, but I believe generating a useful model from scratch will require a lot more than you are likely to find). This kind of thing cannot succeed legally without the fair use exemption from copyright.

danpalmer 4y ago

> However, that GPU could probably handle only half dozen users It may only be able to host a handful of users making requests within the same second, but there's a lot of downtime when users are not triggering inference requests – time reading code, navigating around, etc. Add to that the fact that most engineers are having meetings, writing docs, testing things, they do a lot that isn't writing code. Then you add the fact that users don't work 24/7. Between all of this, I'd be surprised if a GPU wasn't able to handle ~hundreds to ~thousands of monthly active users.