3 ms·

If you read around, training a 7B model costs on the order of $85,000; the 1.4 stable diffusion release cost around $600,000 to train. You don't see a lot of 7

If you read around, training a 7B model costs on the order of $85,000; the 1.4 stable diffusion release cost around $600,000 to train. You don't see a lot of 70B or larger models being released for the same reason; it's expensive. We should just be grateful for what we're getting right now: basically, people are spending 100s of thousands of dollars on training and giving the results away for free. Hugging face is hosting them for free. ollama is hosting them for free. People are writing free inference engines (eg. llama.cpp) and giving them away. Don't complain. We've got it pretty damn good right now.

alchemist1e9 3y ago
> If you read around, training a 7B model costs on the order of $85,000; the 1.4 stable diffusion release cost around $600,000 to train. That seems remarkably cheap actually and likely getting cheaper fairly quickly with improvements in training efficiencies I’d imagine.
sjwhevvvvvsj 3y ago
On the other hand, the systems are trained on “free” data so it kinda should be public property by default. Claiming it’s fair use to suck up the entire web and pay wall the derived result is absurd argument. We all created the lifeblood of LLM and we’re entitled to the product.
- visarga 3y ago
  > We all created the lifeblood of LLM and we’re entitled to the product. sounds so nice, yet there are going to be objections, NYT for example doesn't think we all should be entitled to the product
  - sjwhevvvvvsj 3y ago
    Of course, that is partially my point: if OpenAI et al wants to make the argument anything online is fair game, then they should release the weights. If not, they have no leg to stand on.
- vlovich123 3y ago
  You’ve just described Google which derives most of its ad revenue from ads it places on the search engine that’s crawling the public web. It has always been thus that derivative products that provide a meaningful transformation of the input is a wholly separate piece of copyright.
  - sjwhevvvvvsj 3y ago
    No, this is very different. Google will link you to the NYT, you read there, and see ads. If GPT eats the web and pay walls it, they are 100% free riding. Now, I also think the Google model is proven at this point to be a bad model since the web is 90% ads and SEO dogshit. They strip mined the value, took them a while, but its nearly decimated.
    - nl 3y ago
      The value of ChatGPT isn't that it regurgitates the NYT. The value is that it will read the NYT and the Washington Post and Fox News and The Guardian and everything else for you and synthesise a new view from it all that represents the viewpoint you ask for. That's completely different to Google a d completely different to anything done before. It's as transformative as a human expert news analyst giving you a new perspective on a story.
- wokwokwok 3y ago
  Whether that’s true or not, the fact remains that a lot of people are spending real money in astonishing large amounts and not asking for anything in return. Seriously, complaining they haven’t spent enough money or didn’t spend 600k making exactly you the model you wanted is… Let’s just say, ungracious. Got some cake for my birthday, but it wasn’t the chocolate deluxe cream cake I wanted. …just remember, the cake is pretty good, and it’s free. :) Over time the cost of training models will come down and bigger open models will turn up, eventually.
dragonwriter 3y ago
> If you read around, training a 7B model costs on the order of $85,000; the 1.4 stable diffusion release cost around $600,000 to train. SD 1.x is a ~1B parameter model, so its interesting that it cost so much more than a 7B LLM.
- senseiV 3y ago
  yes the size is different, but training a diffusion model and a language model are really different, like how RL models can be small but take a long time to train aswell
swyx 3y ago
does ollama actually host the models or is it a set of aliases to huggingface? and is it llama.cpp under the hood? trying to figure out how thick this layer is

alchemist1e9 3y ago

> If you read around, training a 7B model costs on the order of $85,000; the 1.4 stable diffusion release cost around $600,000 to train. That seems remarkably cheap actually and likely getting cheaper fairly quickly with improvements in training efficiencies I’d imagine.

sjwhevvvvvsj 3y ago

On the other hand, the systems are trained on “free” data so it kinda should be public property by default. Claiming it’s fair use to suck up the entire web and pay wall the derived result is absurd argument. We all created the lifeblood of LLM and we’re entitled to the product.

visarga 3y ago
> We all created the lifeblood of LLM and we’re entitled to the product. sounds so nice, yet there are going to be objections, NYT for example doesn't think we all should be entitled to the product
- sjwhevvvvvsj 3y ago
  Of course, that is partially my point: if OpenAI et al wants to make the argument anything online is fair game, then they should release the weights. If not, they have no leg to stand on.
vlovich123 3y ago
You’ve just described Google which derives most of its ad revenue from ads it places on the search engine that’s crawling the public web. It has always been thus that derivative products that provide a meaningful transformation of the input is a wholly separate piece of copyright.
- sjwhevvvvvsj 3y ago
  No, this is very different. Google will link you to the NYT, you read there, and see ads. If GPT eats the web and pay walls it, they are 100% free riding. Now, I also think the Google model is proven at this point to be a bad model since the web is 90% ads and SEO dogshit. They strip mined the value, took them a while, but its nearly decimated.
  - nl 3y ago
    The value of ChatGPT isn't that it regurgitates the NYT. The value is that it will read the NYT and the Washington Post and Fox News and The Guardian and everything else for you and synthesise a new view from it all that represents the viewpoint you ask for. That's completely different to Google a d completely different to anything done before. It's as transformative as a human expert news analyst giving you a new perspective on a story.
wokwokwok 3y ago
Whether that’s true or not, the fact remains that a lot of people are spending real money in astonishing large amounts and not asking for anything in return. Seriously, complaining they haven’t spent enough money or didn’t spend 600k making exactly you the model you wanted is… Let’s just say, ungracious. Got some cake for my birthday, but it wasn’t the chocolate deluxe cream cake I wanted. …just remember, the cake is pretty good, and it’s free. :) Over time the cost of training models will come down and bigger open models will turn up, eventually.

dragonwriter 3y ago

> If you read around, training a 7B model costs on the order of $85,000; the 1.4 stable diffusion release cost around $600,000 to train. SD 1.x is a ~1B parameter model, so its interesting that it cost so much more than a 7B LLM.

senseiV 3y ago
yes the size is different, but training a diffusion model and a language model are really different, like how RL models can be small but take a long time to train aswell

swyx 3y ago

does ollama actually host the models or is it a set of aliases to huggingface? and is it llama.cpp under the hood? trying to figure out how thick this layer is