4 ms·

> The main way in which IBM mainframes in the 1950s-1970s were "firewalled" was simply by being fiendishly expensive – most people's houses cost significantly l

> The main way in which IBM mainframes in the 1950s-1970s were "firewalled" was simply by being fiendishly expensive – most people's houses cost significantly less. This was the primary aspect I was referring to, in the same way that training a ChatGPT-like NN can be (or could become) prohibitively expensive. But your comments about openness are relevant on an entirely different axis.

skissane 4y ago
> This was the primary aspect I was referring to, in the same way that training a ChatGPT-like NN can be (or could become) prohibitively expensive. It is fundamentally different though - let's say it costs US$5 million to train a ChatGPT-like system. Someone only has to pay that once, and open source the results, and then everyone else gets it for free. US$5 million is a lot of money for the average person, but a drop in the bucket as far as corporations/governments/universities/research labs/etc go. By contrast, IBM's 1964 S/360 announcement priced the top-of-the-line model at US$5.5 million – which is over US$50 million in today's money – and that only bought you one mainframe, a second one would cost about as much as the first. A mainframe is hardware, but ChatGPT is software. ChatGPT's runtime (post-training) hardware requirements are hefty, but (on a per user basis) still cost less than a car does.
- AstixAndBelix 4y ago
  The problem is that AI research is moving incredibly fast. You might train a LLN today for $5M but a year from now the competition will have implemented an absolutely killer feature that needs $10M worth of training
  - skissane 4y ago
    AI research isn't particularly expensive. US$10 million to train a new model? Other fields have R&D budgets measured in the billions. I bet if you were a senior researcher at OpenAI, and you decided to quit and start a competing firm, there'd be a whole line of investors wanting to give you a lot more than US$10 million. And you don't need to be coming first in the technology race to make money. A lot of people would be willing to pay for something ChatGPT-level with less restrictions on use. And then next year OpenAI will come out with something even more advanced, and they'll ask themselves "do I want a 2023-level solution which I'm free to use as I like, or a 2024-level solution with all these strings attached?", and many of them will decide the former is superior to the latter. Maybe GPT-10 will cost US$10 billion to train? Anything could happen. Even if it does, the US government will ban China from using it, and then Beijing will spend US$10 billion to clone it. Even 10 billion isn't that much money if we are talking about nation-states pursuing their national interests, like not being left behind in the AI arms race. And then maybe China will outcompete OpenAI by offering an equivalent product but with far less limitations on how you use it.
    - gpderetta 4y ago
      Agree completely. The state of the art is probably going to always be closed and proprietary, but, especially with hardware becoming more and more powerful, training a custom model is not going to be beyond the budget and capabilities of even small organizations.

skissane 4y ago

> This was the primary aspect I was referring to, in the same way that training a ChatGPT-like NN can be (or could become) prohibitively expensive. It is fundamentally different though - let's say it costs US$5 million to train a ChatGPT-like system. Someone only has to pay that once, and open source the results, and then everyone else gets it for free. US$5 million is a lot of money for the average person, but a drop in the bucket as far as corporations/governments/universities/research labs/etc go. By contrast, IBM's 1964 S/360 announcement priced the top-of-the-line model at US$5.5 million – which is over US$50 million in today's money – and that only bought you one mainframe, a second one would cost about as much as the first. A mainframe is hardware, but ChatGPT is software. ChatGPT's runtime (post-training) hardware requirements are hefty, but (on a per user basis) still cost less than a car does.

AstixAndBelix 4y ago
The problem is that AI research is moving incredibly fast. You might train a LLN today for $5M but a year from now the competition will have implemented an absolutely killer feature that needs $10M worth of training
- skissane 4y ago
  AI research isn't particularly expensive. US$10 million to train a new model? Other fields have R&D budgets measured in the billions. I bet if you were a senior researcher at OpenAI, and you decided to quit and start a competing firm, there'd be a whole line of investors wanting to give you a lot more than US$10 million. And you don't need to be coming first in the technology race to make money. A lot of people would be willing to pay for something ChatGPT-level with less restrictions on use. And then next year OpenAI will come out with something even more advanced, and they'll ask themselves "do I want a 2023-level solution which I'm free to use as I like, or a 2024-level solution with all these strings attached?", and many of them will decide the former is superior to the latter. Maybe GPT-10 will cost US$10 billion to train? Anything could happen. Even if it does, the US government will ban China from using it, and then Beijing will spend US$10 billion to clone it. Even 10 billion isn't that much money if we are talking about nation-states pursuing their national interests, like not being left behind in the AI arms race. And then maybe China will outcompete OpenAI by offering an equivalent product but with far less limitations on how you use it.
  - gpderetta 4y ago
    Agree completely. The state of the art is probably going to always be closed and proprietary, but, especially with hardware becoming more and more powerful, training a custom model is not going to be beyond the budget and capabilities of even small organizations.