5 ms·

The point is, OpenAI spent a lot of money on training on all these copyrighted materials ordinary individuals/companies don't have access to, so replicating the

The point is, OpenAI spent a lot of money on training on all these copyrighted materials ordinary individuals/companies don't have access to, so replicating their effort would mean that you either 1) spend a ridiculous amount of money, 2) use Library Genesis (and still pay millions for GPU usage). So we have very little choice now. Open Source LLMs might be getting close to ChatGPT3 (opinions vary), but OpenAI is still far ahead.

btbuildem 3y ago
Don't underestimate Big Corp's resistance to using OpenAI's hosted solutions (even on Azure) for anything that's not marketing fluff.
- phillipcarter 3y ago
  You can say that about anything, though. BigCorps aren't exactly known for adopting useful tech on a reasonable timeline, let alone at all. I don't think anyone is under the impression that orgs who refuse to migrate off of Java 5 will be looking at OpenAI for anything.
  - swagempire 3y ago
    No, this is silly reasoning. A middle manager somewhere has no clue what Java 5 is. But he does know -- or let's say IMAGINES what he knows about ChatGPT. And unlike Java 5-- he just needs to use his departmental budget and instantly mandate that his team now use ChatGPT. Whatever that means you can argue it. But ChatGPT is a front line technology and super accessible. Java 5 is super back end and very specialized. The adoption you say won't happen: it will come from the middle -> up.
    - hashtag-til 3y ago
      Honest question: do you really mean Java 5 when you say Java 5? It sounds a bit 2000s to me.
      - swagempire 3y ago
        Parent used "Java 5" as an example. Java 5 somehow in my mind is from like the 200x era. But no. I practically mean any complicated back end technology that takes corporations months or years to migrate off of because its quite complicated and requires an intense amount of technical savoir-faire. My point was that ChatGPT bypasses all this and any middle manager can start using it anywhere for a small hit to his departmental budget.
      - eddtries 3y ago
        If you care about the security of OpenAI, you care about the EOL of 14 year old Java 5
      - cornel_io 3y ago
        In 2016 I worked on a project with a client who still mandated that all code was written to the Java 1.1 language specification - no generics, no enums, no annotations, etc., not to even mention all the stuff that's come since 1.5 (or Java 5, or whatever you want to call it). They had Reasons(tm), which after filtering through the nonsense mostly boiled down to the CTO being curmudgeonly and unwilling to approve replacing a hand-written code transformer that he had personally written back in the stone ages and that he 1) considered core to their product, and 2) considered too risky to replace, because obviously there were no tests covering any of the core systems...sigh. At least they ran it all on a modern JVM. But no, it would not surprise me to find a decent handful of large companies still writing Java 5 code; it would surprise me a bit more to find many still using that JVM, since you can't even get paid support through Oracle anymore, but I'm sure someone out there is doing it. Never underestimate the "don't touch it, you might break it" sentiment at non-tech companies, even big ones with lots of revenue, they routinely understaff their tech departments and the people who built key systems may have retired 20 years ago at this point so it's really risky to do any sort of big system migration. That's why so many lines of COBOL are still running.
    - Turing_Machine 3y ago
      > But he does know -- or let's say IMAGINES what he knows about ChatGPT. And unlike Java 5-- Those of us who've been around for a long time know that's pretty much how Java worked as well. All of the non-technical "manager" magazines started running advertorials (no doubt heavily astroturfed by Sun) about how great Java was. Those managers didn't know what Java was either. All they knew (or thought they knew) was that all the "smart managers" were using Java (according to their "smart manager" magazines), and the rest was history.
- outside415 3y ago
  chatgpt will have an on prem solution eventually. in the mean time players like NVIDIA are working on that as well.
- tsunamifury 3y ago
  Marketing fluff is what 90% of tech is... it amazes me how many people think otherwise on hacker news. Unless you are building utility systems that run power plants, at the end of the day -- you're doing marketing fluff or the tools for it.
  - TeMPOraL 3y ago
    > Unless you are building utility systems that run power plants, at the end of the day -- you're doing marketing fluff or the tools for it. Even when you are building utility systems for critical infrastructure, you'll still be dealing with a disheartening amount of focus on marketing fluff and sales trickery.
colinsane 3y ago
the choice is to live 2 years behind (e.g. integrate the open source stuff and ride that wave of improvement). for businesses in a competitive space, that’s perhaps untenable. but for individuals and anywhere else where this stuff is just a “nice to have”, that’s really just the long-term sustainable approach. it reminds me of a choice like “do i host my website on a Windows Server, or a Linux box” at a time when both of these things are new.
- hfjjbf 3y ago
  [dead]
- BiteCode_dev 3y ago
  2 years behind in terms of timeline, but what factor in terms of productivity and quality of life? Not to mention openai's lead compounds, so 2 years now and 4 years in 2025 may be 10 times the original prod/qol gain.
  - verdverm 3y ago
    The gap seems to be shrinking, not growing. The OSS models have reach new capabilities faster than most thought
- TeMPOraL 3y ago
  For individuals, this is a very short window of time where we have cheap access to an actually useful, and relatively unshackled SOTA model[0]. This is the rare time individuals can empower themselves, become briefly better at whatever it is they're doing, expand their skills, cut through tedium, let their creativity bloom. It's only a matter of time before many a corporation and startup parcel it all between themselves, enshittify the living shit out of AI, disenfranchise individuals again and sell them as services what they just took away. No, it's exactly the individuals who can't afford to live "2 years behind". Benefits are too great, and worst that can happen is... going back to where one is now. -- [0] - I'm not talking the political bias and using the idea of alignment to give undue weigh to corporate reputation management issues. I'm talking about gutting the functionality to establish revenue channels. Like, imagine ChatGPT telling you it won't help you with your programming question, until you subscribe to Premium Dev Package for $language, or All Seasons Pass for all languages.
zarzavat 3y ago
OpenAI is obviously using libgen. Libgen is necessary but not sufficient for a top AI model. I believe that Google's corporate reluctance to use it is what's holding them back.
- nvm0n2 3y ago
  How do you know? Has someone created a set of test questions that can definitively prove it one way or another?
  - benterix 3y ago
    I don't believe it's even possible to create such a set of test questions. You would need to specifically test for differences between the versions found on Libgen and those that aren't available there. But even then, the evidence would be inconclusive.
weinzierl 3y ago
I won't say I disagree because only time can tell, but what you wrote sounds a lot like what people said before open source software took off. All these companies spend so much money on software development and they hire the best people available, how can a bunch of unorganized volunteers ever compete? We saw how they could and I hope we will see the same in AI.
giancarlostoro 3y ago
Kind of hilarious that the vast capabilities of an LLM are held back by copyright infringement.
- joquarky 3y ago
  The original purpose of copyright was to promote progress, and now it seems to hinder it.
- TeMPOraL 3y ago
  Intellectual property rights will yet turn out to be one of the Great Filters.
- zozbot234 3y ago
  I'd love to see a language model that was only trained on public domain and openly available content. It would probably be way too little data to give it ChatGPT-like generality, but even a GPT-2 scale model would be interesting.
  - TeMPOraL 3y ago
    If, hypothetically, libraries in the US - including in particular the Library of Congress - were to scan and OCR every book, newspaper and magazine they have with copyright protection already expired, would that be enough? Is there some estimate for the size of such dataset?
    - Turing_Machine 3y ago
      Much of that material is already available at https://archive.org https://archive.org. It might be good enough for some purposes, but limiting it to stuff before 1928 (in the United Sates) isn't going to be very helpful for (e.g.) coding. Maybe if you added github projects with permissive licenses?

btbuildem 3y ago

Don't underestimate Big Corp's resistance to using OpenAI's hosted solutions (even on Azure) for anything that's not marketing fluff.

phillipcarter 3y ago
You can say that about anything, though. BigCorps aren't exactly known for adopting useful tech on a reasonable timeline, let alone at all. I don't think anyone is under the impression that orgs who refuse to migrate off of Java 5 will be looking at OpenAI for anything.
- swagempire 3y ago
  No, this is silly reasoning. A middle manager somewhere has no clue what Java 5 is. But he does know -- or let's say IMAGINES what he knows about ChatGPT. And unlike Java 5-- he just needs to use his departmental budget and instantly mandate that his team now use ChatGPT. Whatever that means you can argue it. But ChatGPT is a front line technology and super accessible. Java 5 is super back end and very specialized. The adoption you say won't happen: it will come from the middle -> up.
  - hashtag-til 3y ago
    Honest question: do you really mean Java 5 when you say Java 5? It sounds a bit 2000s to me.
    - swagempire 3y ago
      Parent used "Java 5" as an example. Java 5 somehow in my mind is from like the 200x era. But no. I practically mean any complicated back end technology that takes corporations months or years to migrate off of because its quite complicated and requires an intense amount of technical savoir-faire. My point was that ChatGPT bypasses all this and any middle manager can start using it anywhere for a small hit to his departmental budget.
    - eddtries 3y ago
      If you care about the security of OpenAI, you care about the EOL of 14 year old Java 5
    - cornel_io 3y ago
      In 2016 I worked on a project with a client who still mandated that all code was written to the Java 1.1 language specification - no generics, no enums, no annotations, etc., not to even mention all the stuff that's come since 1.5 (or Java 5, or whatever you want to call it). They had Reasons(tm), which after filtering through the nonsense mostly boiled down to the CTO being curmudgeonly and unwilling to approve replacing a hand-written code transformer that he had personally written back in the stone ages and that he 1) considered core to their product, and 2) considered too risky to replace, because obviously there were no tests covering any of the core systems...sigh. At least they ran it all on a modern JVM. But no, it would not surprise me to find a decent handful of large companies still writing Java 5 code; it would surprise me a bit more to find many still using that JVM, since you can't even get paid support through Oracle anymore, but I'm sure someone out there is doing it. Never underestimate the "don't touch it, you might break it" sentiment at non-tech companies, even big ones with lots of revenue, they routinely understaff their tech departments and the people who built key systems may have retired 20 years ago at this point so it's really risky to do any sort of big system migration. That's why so many lines of COBOL are still running.
  - Turing_Machine 3y ago
    > But he does know -- or let's say IMAGINES what he knows about ChatGPT. And unlike Java 5-- Those of us who've been around for a long time know that's pretty much how Java worked as well. All of the non-technical "manager" magazines started running advertorials (no doubt heavily astroturfed by Sun) about how great Java was. Those managers didn't know what Java was either. All they knew (or thought they knew) was that all the "smart managers" were using Java (according to their "smart manager" magazines), and the rest was history.
outside415 3y ago
chatgpt will have an on prem solution eventually. in the mean time players like NVIDIA are working on that as well.
tsunamifury 3y ago
Marketing fluff is what 90% of tech is... it amazes me how many people think otherwise on hacker news. Unless you are building utility systems that run power plants, at the end of the day -- you're doing marketing fluff or the tools for it.
- TeMPOraL 3y ago
  > Unless you are building utility systems that run power plants, at the end of the day -- you're doing marketing fluff or the tools for it. Even when you are building utility systems for critical infrastructure, you'll still be dealing with a disheartening amount of focus on marketing fluff and sales trickery.

colinsane 3y ago

the choice is to live 2 years behind (e.g. integrate the open source stuff and ride that wave of improvement). for businesses in a competitive space, that’s perhaps untenable. but for individuals and anywhere else where this stuff is just a “nice to have”, that’s really just the long-term sustainable approach. it reminds me of a choice like “do i host my website on a Windows Server, or a Linux box” at a time when both of these things are new.

hfjjbf 3y ago
[dead]
BiteCode_dev 3y ago
2 years behind in terms of timeline, but what factor in terms of productivity and quality of life? Not to mention openai's lead compounds, so 2 years now and 4 years in 2025 may be 10 times the original prod/qol gain.
- verdverm 3y ago
  The gap seems to be shrinking, not growing. The OSS models have reach new capabilities faster than most thought
TeMPOraL 3y ago
For individuals, this is a very short window of time where we have cheap access to an actually useful, and relatively unshackled SOTA model[0]. This is the rare time individuals can empower themselves, become briefly better at whatever it is they're doing, expand their skills, cut through tedium, let their creativity bloom. It's only a matter of time before many a corporation and startup parcel it all between themselves, enshittify the living shit out of AI, disenfranchise individuals again and sell them as services what they just took away. No, it's exactly the individuals who can't afford to live "2 years behind". Benefits are too great, and worst that can happen is... going back to where one is now. -- [0] - I'm not talking the political bias and using the idea of alignment to give undue weigh to corporate reputation management issues. I'm talking about gutting the functionality to establish revenue channels. Like, imagine ChatGPT telling you it won't help you with your programming question, until you subscribe to Premium Dev Package for $language, or All Seasons Pass for all languages.

zarzavat 3y ago

OpenAI is obviously using libgen. Libgen is necessary but not sufficient for a top AI model. I believe that Google's corporate reluctance to use it is what's holding them back.

nvm0n2 3y ago
How do you know? Has someone created a set of test questions that can definitively prove it one way or another?
- benterix 3y ago
  I don't believe it's even possible to create such a set of test questions. You would need to specifically test for differences between the versions found on Libgen and those that aren't available there. But even then, the evidence would be inconclusive.

weinzierl 3y ago

I won't say I disagree because only time can tell, but what you wrote sounds a lot like what people said before open source software took off. All these companies spend so much money on software development and they hire the best people available, how can a bunch of unorganized volunteers ever compete? We saw how they could and I hope we will see the same in AI.

giancarlostoro 3y ago

Kind of hilarious that the vast capabilities of an LLM are held back by copyright infringement.

joquarky 3y ago
The original purpose of copyright was to promote progress, and now it seems to hinder it.
TeMPOraL 3y ago
Intellectual property rights will yet turn out to be one of the Great Filters.
zozbot234 3y ago
I'd love to see a language model that was only trained on public domain and openly available content. It would probably be way too little data to give it ChatGPT-like generality, but even a GPT-2 scale model would be interesting.
- TeMPOraL 3y ago
  If, hypothetically, libraries in the US - including in particular the Library of Congress - were to scan and OCR every book, newspaper and magazine they have with copyright protection already expired, would that be enough? Is there some estimate for the size of such dataset?
  - Turing_Machine 3y ago
    Much of that material is already available at https://archive.org https://archive.org. It might be good enough for some purposes, but limiting it to stuff before 1928 (in the United Sates) isn't going to be very helpful for (e.g.) coding. Maybe if you added github projects with permissive licenses?