4 ms·

Far fewer than 600,000 of those are pretrained. Most are finetuned which is much easier. You can finetune a 7B model on gamer cards. There is basically the big

by mlsu 2y ago

Far fewer than 600,000 of those are pretrained. Most are finetuned which is much easier. You can finetune a 7B model on gamer cards. There is basically the big guys that everyone's heard of (google, meta, microsoft/openAI, and anthropic) and then a handful of smaller players who are training foundation models mostly so that they can prove to VCs that they are capable of doing so -- to acquire more funding/access to compute so that they may eventually dethrone openAI and take a piece of the multi-billion dollar "enterprise AI" market for themselves. Below that, there is a frothing ocean of mostly 7B finetunes created mostly by individuals who want to jailbreak base models for... reasons, plus the occasional research group. The most oddball one I have seen is the databricks LLM which seems to have been an exercise of pure marketing. Those I suspect will disappear when the bubble deflates a bit.

cs702 2y ago
> an exercise of pure marketing Yes. Great choice of words. A lot of non-frontier models look like "an exercise of pure marketing" to me. Still, I fail to see the rationale for telling the world, "Look at us! We can do it too!"
- mlsu 2y ago
  Mid-level managers at a lot of companies still have no clue what LLMs are or how they work. These companies (like databricks) want to have their salespeople upsell such companies on "business AI." They have the base model in their back pocket just in case one of the customers in the room has heard the name Andrej Karpathy before and starts asking questions about how good their AI solution is... they can point to their model and its benchmarks to say "we know what we are doing with this AI stuff." It's just standard marketing stuff which works right now because of how difficult it is to actually objectively benchmark LLMs.
theturtletalks 2y ago
Yep, seems like every company is taking a longshot on a AI project. Even companies like Databricks (MosaicML) and Vercel (v0 and ai.sdk) are seeing if they can take a piece of this every growing pie. Snowflake and the like are training and releasing new models because they intend to integrate the AI into their existing product down the line. Why not use and fine-tune an existing model? Their in-grown model maybe better suited for their product. This can also fail like Bloomberg's financial model being inferior to GPT-4, but these companies have to try.
- vineyardmike 2y ago
  > Why not use and fine-tune an existing model? Not all of them have permissive licenses for whatever the companies may want (or their clients want). Kind of a funny situation where everyone would benefit, but no one wants to burn their money for the greater good.
- dudus 2y ago
  Their biggest competitor release a model. They must follow suit.
  - newshackr 2y ago
    It isn't like they could have started this after the release and been done by now.
grahamgooch 2y ago
600k?
ignoramous 2y ago
> oddball one I have seen is the databricks LLM Interesting you'd say that in a discussion on Snowflake's LLM, no less. As someone who has a good opinion of Databricks, genuinely curious what made you arrive at such a damning conclusion.
hnthrowaway9812 2y ago
You've nerdsniped me so hard that I had to make an account. There are DOZENS of orgs releasing foundational models, not "a handful." Salesforce, EleuthierAI, NVIDIA, Amazon, Stanford, RedPajama, Cohere, Mistral, MosaicML, Yandex, Huawei StabilityLM, ... https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYpb63e1ZR3aePczz3zlbJW-Y4/edit#gid=741531996 https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp... It's completely bonkers and a huge waste of resources. Most of them will see barely any use at all.
- andy99 2y ago
  Competition isn't a waste of resources, it's the best mechanism we have to ensure quality. Furthermore, I'm happy to be in a golden age with lots of orgs trying things and many options. It's going to suck once the market eventually consolidates us and we have to take whatever enshittified thing the ologopolists feed us.
  - RhodesianHunter 2y ago
    Gods if this isn't exactly how it'll turn out.
  - hnthrowaway9812 2y ago
    It's a wast if they are mostly all trying the SAME things. Which is mostly what is happening. I want someone to spend a million on a Chess LLM so we can get a sense of how sophisticated they can get at non-linguistic pattern matching. I want someone to spend a million on an LLM trained on Python program traces so we can try to teach it cause and effect and "debugging". Maybe it will emulate a Python interpreter and get highly reliable at predicting the outcome of Python code. etc.
- mlsu 2y ago
  Interesting! That is more than I thought. Honored to have caused a nerdsnipe. In the grand scheme of things, though, most of these are quite small -- 7b range. A 7b model is nothing to sneeze at but it's not megacorp resources either. It's in the range of "VC check" size. The "big boys" who are training 70b plus are FAANG or government-scale entities. Microsoft, Google, and Meta have multiple entries on that "big" LLM foundation list -- it's because the GPUs are already bought, have to train something to keep utilization up. Also bear in mind that training of these things is still something closer to an art than a science; you put terabytes of data into the cauldron, let it brew, and only after it's done can you taste what you've made. Makes sense that some of these models will be junk.

by mlsu 2y ago

cs702 2y ago

> an exercise of pure marketing Yes. Great choice of words. A lot of non-frontier models look like "an exercise of pure marketing" to me. Still, I fail to see the rationale for telling the world, "Look at us! We can do it too!"

mlsu 2y ago
Mid-level managers at a lot of companies still have no clue what LLMs are or how they work. These companies (like databricks) want to have their salespeople upsell such companies on "business AI." They have the base model in their back pocket just in case one of the customers in the room has heard the name Andrej Karpathy before and starts asking questions about how good their AI solution is... they can point to their model and its benchmarks to say "we know what we are doing with this AI stuff." It's just standard marketing stuff which works right now because of how difficult it is to actually objectively benchmark LLMs.

theturtletalks 2y ago

Yep, seems like every company is taking a longshot on a AI project. Even companies like Databricks (MosaicML) and Vercel (v0 and ai.sdk) are seeing if they can take a piece of this every growing pie. Snowflake and the like are training and releasing new models because they intend to integrate the AI into their existing product down the line. Why not use and fine-tune an existing model? Their in-grown model maybe better suited for their product. This can also fail like Bloomberg's financial model being inferior to GPT-4, but these companies have to try.

vineyardmike 2y ago
> Why not use and fine-tune an existing model? Not all of them have permissive licenses for whatever the companies may want (or their clients want). Kind of a funny situation where everyone would benefit, but no one wants to burn their money for the greater good.
dudus 2y ago
Their biggest competitor release a model. They must follow suit.
- newshackr 2y ago
  It isn't like they could have started this after the release and been done by now.

grahamgooch 2y ago

600k?

ignoramous 2y ago

> oddball one I have seen is the databricks LLM Interesting you'd say that in a discussion on Snowflake's LLM, no less. As someone who has a good opinion of Databricks, genuinely curious what made you arrive at such a damning conclusion.

hnthrowaway9812 2y ago

You've nerdsniped me so hard that I had to make an account. There are DOZENS of orgs releasing foundational models, not "a handful." Salesforce, EleuthierAI, NVIDIA, Amazon, Stanford, RedPajama, Cohere, Mistral, MosaicML, Yandex, Huawei StabilityLM, ... https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYpb63e1ZR3aePczz3zlbJW-Y4/edit#gid=741531996 https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp... It's completely bonkers and a huge waste of resources. Most of them will see barely any use at all.

andy99 2y ago
Competition isn't a waste of resources, it's the best mechanism we have to ensure quality. Furthermore, I'm happy to be in a golden age with lots of orgs trying things and many options. It's going to suck once the market eventually consolidates us and we have to take whatever enshittified thing the ologopolists feed us.
- RhodesianHunter 2y ago
  Gods if this isn't exactly how it'll turn out.
- hnthrowaway9812 2y ago
  It's a wast if they are mostly all trying the SAME things. Which is mostly what is happening. I want someone to spend a million on a Chess LLM so we can get a sense of how sophisticated they can get at non-linguistic pattern matching. I want someone to spend a million on an LLM trained on Python program traces so we can try to teach it cause and effect and "debugging". Maybe it will emulate a Python interpreter and get highly reliable at predicting the outcome of Python code. etc.
mlsu 2y ago
Interesting! That is more than I thought. Honored to have caused a nerdsnipe. In the grand scheme of things, though, most of these are quite small -- 7b range. A 7b model is nothing to sneeze at but it's not megacorp resources either. It's in the range of "VC check" size. The "big boys" who are training 70b plus are FAANG or government-scale entities. Microsoft, Google, and Meta have multiple entries on that "big" LLM foundation list -- it's because the GPUs are already bought, have to train something to keep utilization up. Also bear in mind that training of these things is still something closer to an art than a science; you put terabytes of data into the cauldron, let it brew, and only after it's done can you taste what you've made. Makes sense that some of these models will be junk.