2 ms·
A) is just a sign, that further engineering, both model and framework, is required. Also, 50-60 words per second does not sound too bad. It amounts to 4M words
by lostmsu 6y ago
A) is just a sign, that further engineering, both model and framework, is required. Also, 50-60 words per second does not sound too bad. It amounts to 4M words per day. I imagine most people only type a thousand words per day. So with a single GPU you can handle 4000 users (assuming batch-style processing).
B) This specific example actually shows ML toolchains are orders of magnitude better, than existing ones. Imagine you wanted to make a football game from scratch, and this technique would let you do that by just feeding a lot of football game recordings to a program, as opposed to hiring software devs and graphics designers, and iterating for a year.
- lumost 6y agoOn A: Using a p3.2xl for that task leaves you with a minimum ec2 cost of goods sold of 50 cents per user per month for the target task. To make a reasonable SaaS business out of this you would need to be charging a minimum of $5.00 per user month for the service that this GPU is providing, assuming all other tasks have a negligible impact on Cost of Goods Sold, and that you are able to spread the load optimally throughout the day - and thus don't have to purchase higher capacity during peak. This isn't too bad of a situation if this GPU provides 100% of your value proposition. but, if you needed your product to perform 5 such tasks - or competition/customer demand forces you to move to a bigger model with a 10 words per second inference rate then you may find it hard to balance customer value vs. price vs. margin.
- chx 6y ago> On A: Using a p3.2xl for that task leaves you with a minimum ec2 cost of goods sold of 50 cents per user per month for the target task https://twitter.com/br_/status/979442438254166016 https://twitter.com/br_/status/979442438254166016 > "selling AWS at a loss" is crisp shorthand for a lot of startups' business models!