3 ms·

This is amazing work and bravo on to the people working on redpajama. This is fantastic for the world, this means LLMs will not be controlled by a couple of co

This is amazing work and bravo on to the people working on redpajama. This is fantastic for the world, this means LLMs will not be controlled by a couple of companies with the associated rents. Yes, private LLMs will likely be a couple of years ahead of 'free' alternatives, but that's OK, we want to incentivize for profit research so long as the services are low priced in time (and in this case in short order). AMAZING WORK.

swyx 3y ago
to be clear this work is not based on redpajama - though we did discuss that in the previous episode https://twitter.com/swyx/status/1648080532734087168?s=46&t=90xQ8sGy63D2OtiaoGJuww https://twitter.com/swyx/status/1648080532734087168?s=46&t=9...
- GreedClarifies 3y ago
  Oh my bad! I thought I read that, is it based upon: https://arxiv.org/abs/2211.15533 https://arxiv.org/abs/2211.15533 (The Stack) ?
  - swyx 3y ago
    partially. Reza discussed their data pipeline in the blogpost that we reference in the show notes
m3kw9 3y ago
Have you even tried it? It’s pretty bad
- GreedClarifies 3y ago
  But that's fine it can be a year or two behind the state of the art. That's not the point. The point is that there will be alternatives and that will reduce the price in time further increasing the impact of the technology. There was a possible future where only MSFT and maybe GOOG and maybe one or two other companies had this technology and extracted massive rents.
laweijfmvo 3y ago
My first reaction was, "why is replit building LLMs," but I guess it fits their needs to have one optimized for their use. But I wonder, is this the beginning of another wave of "every company is an AI company?" Are we going to see a spike in tech hiring around AI/LLM, money starting to flow again, etc? And how many years until it all blows up and the layoffs start?
- dpflan 3y ago
  Finetuning models and LLMs (and any model) is going to a be common practice . Each company is its own domain, which domain knowledge and data to specialize open sourced models or used other models to distill/teach their own proprietary model (home grown or modify someone else's).

swyx 3y ago

to be clear this work is not based on redpajama - though we did discuss that in the previous episode https://twitter.com/swyx/status/1648080532734087168?s=46&t=90xQ8sGy63D2OtiaoGJuww https://twitter.com/swyx/status/1648080532734087168?s=46&t=9...

GreedClarifies 3y ago
Oh my bad! I thought I read that, is it based upon: https://arxiv.org/abs/2211.15533 https://arxiv.org/abs/2211.15533 (The Stack) ?
- swyx 3y ago
  partially. Reza discussed their data pipeline in the blogpost that we reference in the show notes

m3kw9 3y ago

Have you even tried it? It’s pretty bad

GreedClarifies 3y ago
But that's fine it can be a year or two behind the state of the art. That's not the point. The point is that there will be alternatives and that will reduce the price in time further increasing the impact of the technology. There was a possible future where only MSFT and maybe GOOG and maybe one or two other companies had this technology and extracted massive rents.

laweijfmvo 3y ago

My first reaction was, "why is replit building LLMs," but I guess it fits their needs to have one optimized for their use. But I wonder, is this the beginning of another wave of "every company is an AI company?" Are we going to see a spike in tech hiring around AI/LLM, money starting to flow again, etc? And how many years until it all blows up and the layoffs start?

dpflan 3y ago
Finetuning models and LLMs (and any model) is going to a be common practice . Each company is its own domain, which domain knowledge and data to specialize open sourced models or used other models to distill/teach their own proprietary model (home grown or modify someone else's).