6 ms·

Asymptotic improvements are flattening the cost curves so fast that AI regulation might become practically meaningless by the end of the year. If you want unreg

Asymptotic improvements are flattening the cost curves so fast that AI regulation might become practically meaningless by the end of the year. If you want unregulated output you'll have tons of offshore models to choose from. The risk is that the good guys end up being the only ones hampered by it. Hopefully it won't be so large a burden that the bad guys and especially the so-so guys (those with a real chance, e.g. Alibaba) get a massive leg up.

FeepingCreature 2y ago
Unregulated output at small scales. The really big training runs will still cost millions.
- worstspotgain 2y ago
  Not when we're talking asymptotically. The linked paper for instance claims 14- to 118-fold cost reductions. 1-2 GPU generations from now you'll train this model for $0.12.
  - impossiblefork 2y ago
    Surely not $0.12. Maybe $100.
    - moffkalast 2y ago
      People are casually dropping thousands on cloud GPUs making random fine tunes over at r/localllama, the threshold will be met far sooner. Plus datacenters selling away their collection of A100s and eventually H100s when they become EoL for their standards.
      - impossiblefork 2y ago
        Yes, but that doesn't mean that things are suddenly incredibly cheap. It just means that people have hardware. $0.12 I feel is extremely superlative and I feel that $100 is more reasonable.
uyzstvqs 2y ago
> Asymptotic improvements are flattening the cost curves so fast that AI regulation might become practically meaningless by the end of the year. Awesome. This will mean actually good open-source models, not just API endpoints by big tech which are unusable because of dataset censorship and bias alignment (SD3, Gemini). In other words, big tech will actually need to make good stuff to be competitive, not trash protected by a granted monopoly.
- sigmoid10 2y ago
  Those improvements are definitely real, but we also have pretty solidly established and confirmed scaling laws by now. Until someone utterly breaks those, big players will always have an edge, simply because they can spend more compute on training and inference. The only way to change this is with a new architecture that benefits more from intelligent adjustments in a space than cannot be searched efficiently with raw compute. And even then we are not far from the point where these models could try out those adjustments themselves. So by the time you get to tune your own GAI in your home like you could do with a human, corporations might have millions of them improving themselves to something you could never achieve on your own.
  - worstspotgain 2y ago
    We're still in phase 1, where human-directed improvement has the highest potential. Papers are still getting published and the cells interlinked. (I'm not sure the scaling picture is at all clear, given that papers like this can turn up casually with 15x savings, but let's put that aside for now.) Phase 2 begins when patents break stealth, unsettling the picture. If some patent impairs research or operations in IP-solid countries, the lower-level stuff might move to local inference, and maybe some minor Pirate Bay-style outfits. Phase 3 begins when the costly research goes dark (well, darker.) Everyone is Apple now. The research papers are replaced by white papers, then by PR communiqués. Phase 4 begins when the AI AI researchers take over. The old AI researchers turn into their managers. Some of the path is compute-bound. Some of it is IP-, luck-, and genius-bound.
whywhywhywhy 2y ago
You can't really stop it at this point anyway without completely locking down any code that resembles AI at a processor level to only signed and allowed models and making owning hardware before the lock illegal and destroying any thats ceased.
- pk-protect-ai 2y ago
  Please, don't give them ideas!!!!
- eru 2y ago
  > [...] locking down any code that resembles AI at a processor level Just outlaw matrix multiplication, I guess?
Y_Y 2y ago
Who are the good guys and bad guys again?
- benopal64 2y ago
  I almost commented the same thing. Framing things as "good/bad/so-so" is kind of moving the target. If we are focusing on who might use the model, rather than considering that when focusing on a model that accurately represents reality and altruistically aids humans... we will lose sight of the really valuable things in life. The reality is that I do not think that people are good/so-so/bad as humans are equipped with extremely complex and diverse adaptive systems with near-limitless capabilities. Sure, I am just re-framing, however from my perspective, we are not reducing humans to "good/bad/so-so." What does HK think?

FeepingCreature 2y ago

Unregulated output at small scales. The really big training runs will still cost millions.

worstspotgain 2y ago
Not when we're talking asymptotically. The linked paper for instance claims 14- to 118-fold cost reductions. 1-2 GPU generations from now you'll train this model for $0.12.
- impossiblefork 2y ago
  Surely not $0.12. Maybe $100.
  - moffkalast 2y ago
    People are casually dropping thousands on cloud GPUs making random fine tunes over at r/localllama, the threshold will be met far sooner. Plus datacenters selling away their collection of A100s and eventually H100s when they become EoL for their standards.
    - impossiblefork 2y ago
      Yes, but that doesn't mean that things are suddenly incredibly cheap. It just means that people have hardware. $0.12 I feel is extremely superlative and I feel that $100 is more reasonable.

uyzstvqs 2y ago

> Asymptotic improvements are flattening the cost curves so fast that AI regulation might become practically meaningless by the end of the year. Awesome. This will mean actually good open-source models, not just API endpoints by big tech which are unusable because of dataset censorship and bias alignment (SD3, Gemini). In other words, big tech will actually need to make good stuff to be competitive, not trash protected by a granted monopoly.

sigmoid10 2y ago
Those improvements are definitely real, but we also have pretty solidly established and confirmed scaling laws by now. Until someone utterly breaks those, big players will always have an edge, simply because they can spend more compute on training and inference. The only way to change this is with a new architecture that benefits more from intelligent adjustments in a space than cannot be searched efficiently with raw compute. And even then we are not far from the point where these models could try out those adjustments themselves. So by the time you get to tune your own GAI in your home like you could do with a human, corporations might have millions of them improving themselves to something you could never achieve on your own.
- worstspotgain 2y ago
  We're still in phase 1, where human-directed improvement has the highest potential. Papers are still getting published and the cells interlinked. (I'm not sure the scaling picture is at all clear, given that papers like this can turn up casually with 15x savings, but let's put that aside for now.) Phase 2 begins when patents break stealth, unsettling the picture. If some patent impairs research or operations in IP-solid countries, the lower-level stuff might move to local inference, and maybe some minor Pirate Bay-style outfits. Phase 3 begins when the costly research goes dark (well, darker.) Everyone is Apple now. The research papers are replaced by white papers, then by PR communiqués. Phase 4 begins when the AI AI researchers take over. The old AI researchers turn into their managers. Some of the path is compute-bound. Some of it is IP-, luck-, and genius-bound.

whywhywhywhy 2y ago

You can't really stop it at this point anyway without completely locking down any code that resembles AI at a processor level to only signed and allowed models and making owning hardware before the lock illegal and destroying any thats ceased.

pk-protect-ai 2y ago
Please, don't give them ideas!!!!
eru 2y ago
> [...] locking down any code that resembles AI at a processor level Just outlaw matrix multiplication, I guess?

Y_Y 2y ago

Who are the good guys and bad guys again?

benopal64 2y ago
I almost commented the same thing. Framing things as "good/bad/so-so" is kind of moving the target. If we are focusing on who might use the model, rather than considering that when focusing on a model that accurately represents reality and altruistically aids humans... we will lose sight of the really valuable things in life. The reality is that I do not think that people are good/so-so/bad as humans are equipped with extremely complex and diverse adaptive systems with near-limitless capabilities. Sure, I am just re-framing, however from my perspective, we are not reducing humans to "good/bad/so-so." What does HK think?