8 ms·

The pace of collaborative OSS development on these projects is amazing, but the rate of optimisations being achieved is almost unbelievable. What has everyone b

The pace of collaborative OSS development on these projects is amazing, but the rate of optimisations being achieved is almost unbelievable. What has everyone been doing wrong all these years cough sorry, I mean to say weeks? Ok I answered my own question.

datadeft 4y ago
I have predicted that LLaMA will be available on mobile phones before the end of this year. We are very close.
- terafo 4y ago
  You mean in contained app? It can already run on a phone. GPU acceleration would be nice at this point, though.
  - datadeft 4y ago
    I was thinking about the limitation by the mobile HW. Yeah GPU support would be nice.
- rickrollin 4y ago
  People have actually ran it on phones.
  - datadeft 4y ago
    This is what my comment is about. It happened much sooner than I thought.
politician 4y ago
Roughly: OpenAIs don’t employ enough jarts. In other words, the groups of folks working on training models don’t necessarily have access to the sort of optimization engineers that are working in other areas. When all of this leaked into the open, it caused a lot of people knowledgeable in different areas to put their own expertise to the task. Some of those efforts (mmap) pay off spectacularly. Expect industry to copy the best of these improvements.
- bee_rider 4y ago
  The professional optimizes well enough to get management off their back, the hobbyist can be irrationally good.
  - fallous 4y ago
    The professional operates within prioritization tranches. Make it work, make it reliable, make it fast, make it pretty. If you're still iterating on proof-of-concept/prototyping you'll generally confine yourself to the first and/or second levels. Once you've settled on a finalized prototype you then follow the rest of the prioritization levels to achieve shippable product.
- hedgehog 4y ago
  They have very good people but those people have other priorities.
- dsjoerg 4y ago
  ~Whats a jart?~ Ah I see https://news.ycombinator.com/user?id=jart https://news.ycombinator.com/user?id=jart
  - sp332 4y ago
    In March 2014, Tunney petitioned the US government on We the People to hold a referendum asking for support to retire all government employees with full pensions, transfer administrative authority to the technology industry, and appoint the executive chairman of Google Eric Schmidt as CEO of America https://en.m.wikipedia.org/wiki/Justine_Tunney https://en.m.wikipedia.org/wiki/Justine_Tunney
    - hackernewds 4y ago
      what's your point? also interestingly JART is in the thread here, so they might have read your comment :)
      - 4y ago
kmeisthax 4y ago
>What has everyone been doing wrong all these years So it's important to note that all of these improvements are the kinds of things that are cheap to run on a pretrained model. And all of the developments involving large language models recently have been the product of hundreds of thousands of dollars in rented compute time. Once you start putting six digits on a pile of model weights, that becomes a capital cost that the business either needs to recuperate or turn into a competitive advantage. So everyone who scales up to this point doesn't release model weights. The model in question - LLaMA - isn't even a public model. It leaked and people copied[0] it. But because such a large model leaked, now people can actually work on iterative improvements again. Unfortunately we don't really have a way for the FOSS community to pool together that much money to buy compute from cloud providers. Contributions-in-kind through distributed computing (e.g. a "GPT@home" project) would require significant changes to training methodology[1]. Further compounding this, the state-of-the-art is actually kind of a trade secret now. Exact training code isn't always available, and OpenAI has even gone so far as to refuse to say anything about GPT-4's architecture or training set to prevent open replication. [0] I'm avoiding the use of the verb "stole" here, not just because I support filesharing, but because copyright law likely does not protect AI model weights alone. [1] AI training has very high minimum requirements to get in the door. If your GPU has 12GB of VRAM and your model and gradients require 13GB, you can't train the model. CPUs don't have this limitation but they are ridiculously inefficient for any training task. There are techniques like ZeRO to give pagefile-like state partitioning to GPU training, but that requires additional engineering.
- terafo 4y ago
  AI training has very high minimum requirements to get in the door. If your GPU has 12GB of VRAM and your model and gradients require 13GB, you can't train the model. CPUs don't have this limitation but they are ridiculously inefficient for any training task. There are techniques like ZeRO to give pagefile-like state partitioning to GPU training, but that requires additional engineering. You can't if you have one 12gb gpu. You can if you have couple of dozens. And then petals-style training is possible. It is all very very new and there are many unsolved hurdles, but I think it can be done.
  - 4y ago
xienze 4y ago
> but the rate of optimisations being achieved is almost unbelievable. What has everyone been doing wrong all these years cough sorry, I mean to say weeks? It’s several things: * Cutting-edge code, not overly concerned with optimization * Code written by scientists, who aren’t known for being the world’s greatest programmers * The obsession the research world has with using Python Not surprising that there’s a lot of low-hanging fruit that can be optimized.
- Miraste 4y ago
  Why does Python get so much flak for inefficiencies? It's really not that slow, and in ML the speed-sensitive parts are libraries in lower level languages anyway. Half of the optimization from this very post is in Python.
  - chatmasta 4y ago
    It might not be slow in general, but it's easy to write slow code in it.
    - Closi 4y ago
      In ML python is effectively the interface rather than the bit that is doing the heavy lifting. The interface is designed to be easy to use (python) and the bit that is actually doing the work is designed to be heavily performant (which is C & CUDA and may even be running on a TPU).
  - chaorace 4y ago
    Python has the misfortune of competing against JS in this arena, which just so happens to have the most obsessively optimized JIT ever.
    - Max-Limelihood 4y ago
      In ML? No, the best competition for Python in ML is... well, it's either C++ or Julia, depending on how you define "competition," given Python is effectively a glorified C++ interface.
lannisterstark 4y ago
And THIS is why I always advocate for democratization of AI models, getting these into hands of open source community and letting people use it as they wish etc. But a lot of people would rather only have govt or corp control of it...

datadeft 4y ago

I have predicted that LLaMA will be available on mobile phones before the end of this year. We are very close.

terafo 4y ago
You mean in contained app? It can already run on a phone. GPU acceleration would be nice at this point, though.
- datadeft 4y ago
  I was thinking about the limitation by the mobile HW. Yeah GPU support would be nice.
rickrollin 4y ago
People have actually ran it on phones.
- datadeft 4y ago
  This is what my comment is about. It happened much sooner than I thought.

politician 4y ago

Roughly: OpenAIs don’t employ enough jarts. In other words, the groups of folks working on training models don’t necessarily have access to the sort of optimization engineers that are working in other areas. When all of this leaked into the open, it caused a lot of people knowledgeable in different areas to put their own expertise to the task. Some of those efforts (mmap) pay off spectacularly. Expect industry to copy the best of these improvements.

bee_rider 4y ago
The professional optimizes well enough to get management off their back, the hobbyist can be irrationally good.
- fallous 4y ago
  The professional operates within prioritization tranches. Make it work, make it reliable, make it fast, make it pretty. If you're still iterating on proof-of-concept/prototyping you'll generally confine yourself to the first and/or second levels. Once you've settled on a finalized prototype you then follow the rest of the prioritization levels to achieve shippable product.
hedgehog 4y ago
They have very good people but those people have other priorities.
dsjoerg 4y ago
~Whats a jart?~ Ah I see https://news.ycombinator.com/user?id=jart https://news.ycombinator.com/user?id=jart
- sp332 4y ago
  In March 2014, Tunney petitioned the US government on We the People to hold a referendum asking for support to retire all government employees with full pensions, transfer administrative authority to the technology industry, and appoint the executive chairman of Google Eric Schmidt as CEO of America https://en.m.wikipedia.org/wiki/Justine_Tunney https://en.m.wikipedia.org/wiki/Justine_Tunney
  - hackernewds 4y ago
    what's your point? also interestingly JART is in the thread here, so they might have read your comment :)
    - 4y ago

kmeisthax 4y ago

>What has everyone been doing wrong all these years So it's important to note that all of these improvements are the kinds of things that are cheap to run on a pretrained model. And all of the developments involving large language models recently have been the product of hundreds of thousands of dollars in rented compute time. Once you start putting six digits on a pile of model weights, that becomes a capital cost that the business either needs to recuperate or turn into a competitive advantage. So everyone who scales up to this point doesn't release model weights. The model in question - LLaMA - isn't even a public model. It leaked and people copied[0] it. But because such a large model leaked, now people can actually work on iterative improvements again. Unfortunately we don't really have a way for the FOSS community to pool together that much money to buy compute from cloud providers. Contributions-in-kind through distributed computing (e.g. a "GPT@home" project) would require significant changes to training methodology[1]. Further compounding this, the state-of-the-art is actually kind of a trade secret now. Exact training code isn't always available, and OpenAI has even gone so far as to refuse to say anything about GPT-4's architecture or training set to prevent open replication. [0] I'm avoiding the use of the verb "stole" here, not just because I support filesharing, but because copyright law likely does not protect AI model weights alone. [1] AI training has very high minimum requirements to get in the door. If your GPU has 12GB of VRAM and your model and gradients require 13GB, you can't train the model. CPUs don't have this limitation but they are ridiculously inefficient for any training task. There are techniques like ZeRO to give pagefile-like state partitioning to GPU training, but that requires additional engineering.

terafo 4y ago
AI training has very high minimum requirements to get in the door. If your GPU has 12GB of VRAM and your model and gradients require 13GB, you can't train the model. CPUs don't have this limitation but they are ridiculously inefficient for any training task. There are techniques like ZeRO to give pagefile-like state partitioning to GPU training, but that requires additional engineering. You can't if you have one 12gb gpu. You can if you have couple of dozens. And then petals-style training is possible. It is all very very new and there are many unsolved hurdles, but I think it can be done.
- 4y ago

xienze 4y ago

> but the rate of optimisations being achieved is almost unbelievable. What has everyone been doing wrong all these years cough sorry, I mean to say weeks? It’s several things: * Cutting-edge code, not overly concerned with optimization * Code written by scientists, who aren’t known for being the world’s greatest programmers * The obsession the research world has with using Python Not surprising that there’s a lot of low-hanging fruit that can be optimized.

Miraste 4y ago
Why does Python get so much flak for inefficiencies? It's really not that slow, and in ML the speed-sensitive parts are libraries in lower level languages anyway. Half of the optimization from this very post is in Python.
- chatmasta 4y ago
  It might not be slow in general, but it's easy to write slow code in it.
  - Closi 4y ago
    In ML python is effectively the interface rather than the bit that is doing the heavy lifting. The interface is designed to be easy to use (python) and the bit that is actually doing the work is designed to be heavily performant (which is C & CUDA and may even be running on a TPU).
- chaorace 4y ago
  Python has the misfortune of competing against JS in this arena, which just so happens to have the most obsessively optimized JIT ever.
  - Max-Limelihood 4y ago
    In ML? No, the best competition for Python in ML is... well, it's either C++ or Julia, depending on how you define "competition," given Python is effectively a glorified C++ interface.

lannisterstark 4y ago

And THIS is why I always advocate for democratization of AI models, getting these into hands of open source community and letting people use it as they wish etc. But a lot of people would rather only have govt or corp control of it...