2 ms·

On one hand this really sucks, particularly for newcomers starting from scratch, as now only the larger companies that already scraped the web can carry on in t

On one hand this really sucks, particularly for newcomers starting from scratch, as now only the larger companies that already scraped the web can carry on in this vein, finding ways to improve the architecture to better utilize the data they already have. On the other hand, I see this being a forcing function to move away from generative AI sooner, which I've always considered to be a dead end for AGI, to future architectures that train on data similar to how humans learn, ie raw audio, video, etc streams from the environment.

ChrisMarshallNY 2y ago
> now only the larger companies that already scraped the web can carry on in this vein That will have a finite lifetime. We've already seen issues with just the couple of years' difference that the first LLMs have, and it will definitely get worse, as time goes on. They will, however, be able to improve what they do with the data, and some of them will probably get enough funding to buy the data. Don't be shocked to find that the reason so many organizations are restricting access to their data, has nothing, whatsoever, to do with things like privacy and copyright. They just see a lot of money sloshing around, and want to hold out for their share. The pipes will reopen, but the free lunch is probably done. One big issue will probably be, that the free data will be extremely low-quality (possibly poisonous) data, so that might be very bad for folks on the lower tiers.
- skeledrew 2y ago
  Without that "free lunch", the incumbents will still likely have the advantage in perpetuity. There will be fewer competitors as the cost will be a non-starter. Unless backed by some deep-pocketed entity that doesn't mind the possibility of losing a lot. The tech is already proven, and there's not enough value in the incremental improvements that'd make a new entrant sustainable. So even if the pipes were reopened, behind walls there won't be much demand.

ChrisMarshallNY 2y ago

> now only the larger companies that already scraped the web can carry on in this vein That will have a finite lifetime. We've already seen issues with just the couple of years' difference that the first LLMs have, and it will definitely get worse, as time goes on. They will, however, be able to improve what they do with the data, and some of them will probably get enough funding to buy the data. Don't be shocked to find that the reason so many organizations are restricting access to their data, has nothing, whatsoever, to do with things like privacy and copyright. They just see a lot of money sloshing around, and want to hold out for their share. The pipes will reopen, but the free lunch is probably done. One big issue will probably be, that the free data will be extremely low-quality (possibly poisonous) data, so that might be very bad for folks on the lower tiers.

skeledrew 2y ago
Without that "free lunch", the incumbents will still likely have the advantage in perpetuity. There will be fewer competitors as the cost will be a non-starter. Unless backed by some deep-pocketed entity that doesn't mind the possibility of losing a lot. The tech is already proven, and there's not enough value in the incremental improvements that'd make a new entrant sustainable. So even if the pipes were reopened, behind walls there won't be much demand.