7 ms·

Together AI raises a $102.5M Series A

70 points by bratao 3y ago

nittanymount 3y ago
the price is pretty low: https://www.together.ai/pricing https://www.together.ai/pricing 4B MODEL, PRICE 1K TOKENS: $0.0001 register with an email, test account has 25$ credit, python API as well, the model is smaller, but good to have some fun with the API integrated with other systems, haha...
- valine 3y ago
  Not really fair to compare a 4B model and a 1.7T model. Per flop the 4B model here is slightly more expensive than GPT4.
  - The_Contrarian 3y ago
    The theory that GPT-4 is 1.7T also posits that GPT-4 is composed of eight 220B experts, meaning once you've loaded the model, inference costs aren't akin to a 1.7T model, but instead to a 220B model. If prices scale linearly, we reach 0.03 / 1K tokens (lower end of GPT-4's price range) at about 0.0001 * 300, or about 1.2T parameters (and that's dense - no MoE here).
    - valine 3y ago
      I guess the question is how the mix of experts works. Do they predict one token from each model? If so you’re still doing 1.7T worth of computation.
- erikaww 3y ago
  clause 2.1 is $8/million tokens for prompt or .008/1k tokens. interesting
datadrivenangel 3y ago
I wonder how NVIDIA gets to report their investment here from an accounting perspective? Could they book the money TogetherAI spends on GPUs as revenue now, and then when TogetherAI flops and returns five cents on the investment dollar write it off in a different line of business?
- HPMOR 3y ago
  NVIDIA probably recognizes revenue upon delivery of product. The investment write down would probably impact their P&L and have some associated tax benefits there.
- JonChesterfield 3y ago
  Nvidia spending money on nvidia hardware is less annoying than the model of raising money from VCs to give to nvidia. It's an interesting question though - does it still count as a high risk investment worthy of tax advantages if lots of the investment is spent on the hardware the invester sells?
- cma 3y ago
  Likewise Microsoft's $10 billion OpenAI investment has a large (majority?) portion in Azure cloud credits. Microsoft has around a 70% margin at Azure.
  - sebzim4500 3y ago
    > Microsoft has around a 70% margin at Azure. I find this impossible to believe, given they are far from a monopoly here. Where are you getting the figure?
    - vlovich123 3y ago
      I don’t find it surprising that SaaS list price margins could be that high. The big purchasers do volume discounts to get massive discounts while the smaller players are forced to pay more.
      - 3y ago
esafak 3y ago
Anybody using them got feedback?
- tikkun 3y ago
  I've explored most of the inference as a service platforms, theirs is particularly nice. (Clear/simple UI, good pricing, good speed.)
nibab 3y ago
The growth of together.ai makes a lot of sense given that 99% of current AI infrastructure spend is in training (vs inference). As far as navigating the product maze, this is where you want to be in AI right now. The training revenue stream can sustain the company over the next couple of years as it develops other product lines. They get 2 big bonuses as part of this too: 1. training/finetuning llms for a bunch of companies exposes them to the big problems in this area that customers are willing to spend $ on 2. they're building a big distribution network with AI buyers This is defensible too. They have a horde of big customers/spenders that are likely incapable of undertaking these efforts on their own and have very high switching costs. Something that you can not say about the big llm providers. Together is sort of flying under the radar serving a great market segment/need while there is a lot of expensive battles being fought everywhere else in the AI space. This is definitely a company to watch. I wouldn't be surprised to see together.ai becoming a top 3 player in this space.
- htrp 3y ago
  > I wouldn't be surprised to see together.ai becoming a top 3 player in this space. I'd like some more detail on how they differ from the other 50 or so inference providers currently out there. I always figured inference was something GCP/AWS would eventually get into their platforms
  - nibab 3y ago
    Even though they offer inference, training is their primary focus. Inference hasn't really picked up revenue-wise (across the space) comparing to training and it's not a great market to be in. As you mentioned, it's crowded and the barriers to entry are minimal. Anyone with experience in spinning up containers and scaling them can offer this servicer. Paradoxically, it's also the market where the big cloud providers are very well positioned to dominate. Spiky and unpredictable workloads is where their bread and butter is. Their whole economic and infrastructural model is pretty much tailored to this traffic pattern. Training is a totally different ball game. It is a model that is disruptive to big cloud providers given that it follows very different traffic patterns. Training LLMs involves spinning up 100-1000s of machines for a relatively short period of time and with interconnect that doesn't typically exist in data centers. That is a very unique workload. Additionally you need significantly more specialized ML knowledge in tensor parallelism, optimizations, CUDA etc. That is not as common as scaling a container based workload.. Fun fact: Oracle is surprisingly well positioned in terms of their interconnect fabric. Even Microsoft is partnering with CoreWeave for GPU clusters because they dont have as much capacity interconnected in the right way.
lijok 3y ago
So this is where all the VC money in FinTech has gone to.

70 points by bratao 3y ago

the price is pretty low: https://www.together.ai/pricing https://www.together.ai/pricing 4B MODEL, PRICE 1K TOKENS: $0.0001 register with an email, test account has 25$ credit, python API as well, the model is smaller, but good to have some fun with the API integrated with other systems, haha...

valine 3y ago
Not really fair to compare a 4B model and a 1.7T model. Per flop the 4B model here is slightly more expensive than GPT4.
- The_Contrarian 3y ago
  The theory that GPT-4 is 1.7T also posits that GPT-4 is composed of eight 220B experts, meaning once you've loaded the model, inference costs aren't akin to a 1.7T model, but instead to a 220B model. If prices scale linearly, we reach 0.03 / 1K tokens (lower end of GPT-4's price range) at about 0.0001 * 300, or about 1.2T parameters (and that's dense - no MoE here).
  - valine 3y ago
    I guess the question is how the mix of experts works. Do they predict one token from each model? If so you’re still doing 1.7T worth of computation.
erikaww 3y ago
clause 2.1 is $8/million tokens for prompt or .008/1k tokens. interesting

datadrivenangel 3y ago

I wonder how NVIDIA gets to report their investment here from an accounting perspective? Could they book the money TogetherAI spends on GPUs as revenue now, and then when TogetherAI flops and returns five cents on the investment dollar write it off in a different line of business?

HPMOR 3y ago
NVIDIA probably recognizes revenue upon delivery of product. The investment write down would probably impact their P&L and have some associated tax benefits there.
JonChesterfield 3y ago
Nvidia spending money on nvidia hardware is less annoying than the model of raising money from VCs to give to nvidia. It's an interesting question though - does it still count as a high risk investment worthy of tax advantages if lots of the investment is spent on the hardware the invester sells?
cma 3y ago
Likewise Microsoft's $10 billion OpenAI investment has a large (majority?) portion in Azure cloud credits. Microsoft has around a 70% margin at Azure.
- sebzim4500 3y ago
  > Microsoft has around a 70% margin at Azure. I find this impossible to believe, given they are far from a monopoly here. Where are you getting the figure?
  - vlovich123 3y ago
    I don’t find it surprising that SaaS list price margins could be that high. The big purchasers do volume discounts to get massive discounts while the smaller players are forced to pay more.
    - 3y ago

esafak 3y ago

Anybody using them got feedback?

tikkun 3y ago
I've explored most of the inference as a service platforms, theirs is particularly nice. (Clear/simple UI, good pricing, good speed.)

nibab 3y ago

The growth of together.ai makes a lot of sense given that 99% of current AI infrastructure spend is in training (vs inference). As far as navigating the product maze, this is where you want to be in AI right now. The training revenue stream can sustain the company over the next couple of years as it develops other product lines. They get 2 big bonuses as part of this too: 1. training/finetuning llms for a bunch of companies exposes them to the big problems in this area that customers are willing to spend $ on 2. they're building a big distribution network with AI buyers This is defensible too. They have a horde of big customers/spenders that are likely incapable of undertaking these efforts on their own and have very high switching costs. Something that you can not say about the big llm providers. Together is sort of flying under the radar serving a great market segment/need while there is a lot of expensive battles being fought everywhere else in the AI space. This is definitely a company to watch. I wouldn't be surprised to see together.ai becoming a top 3 player in this space.

htrp 3y ago
> I wouldn't be surprised to see together.ai becoming a top 3 player in this space. I'd like some more detail on how they differ from the other 50 or so inference providers currently out there. I always figured inference was something GCP/AWS would eventually get into their platforms
- nibab 3y ago
  Even though they offer inference, training is their primary focus. Inference hasn't really picked up revenue-wise (across the space) comparing to training and it's not a great market to be in. As you mentioned, it's crowded and the barriers to entry are minimal. Anyone with experience in spinning up containers and scaling them can offer this servicer. Paradoxically, it's also the market where the big cloud providers are very well positioned to dominate. Spiky and unpredictable workloads is where their bread and butter is. Their whole economic and infrastructural model is pretty much tailored to this traffic pattern. Training is a totally different ball game. It is a model that is disruptive to big cloud providers given that it follows very different traffic patterns. Training LLMs involves spinning up 100-1000s of machines for a relatively short period of time and with interconnect that doesn't typically exist in data centers. That is a very unique workload. Additionally you need significantly more specialized ML knowledge in tensor parallelism, optimizations, CUDA etc. That is not as common as scaling a container based workload.. Fun fact: Oracle is surprisingly well positioned in terms of their interconnect fabric. Even Microsoft is partnering with CoreWeave for GPU clusters because they dont have as much capacity interconnected in the right way.

lijok 3y ago

So this is where all the VC money in FinTech has gone to.