4 ms·

I have some experience with ML and I have no idea what you're talking about. It kinda sounds like neural architecture search and sparse models created using wei

by NavinF 4y ago

I have some experience with ML and I have no idea what you're talking about. It kinda sounds like neural architecture search and sparse models created using weight pruning. Lots of people are working on both of those things, but IMO the latter (if that's what you mean by "Can you add weights non uniformly?") is a dead end for most use cases where you have some sort of accelerator or deep learning instructions available like on every modern desktop, laptop, phone, and server. Models that use their weights "efficiently" in terms of FLOPs tend to perform like crap on real hardware.

sillysaurusx 4y ago
I’m unfortunately the sickest I’ve been in years, so this will have to wait. Maybe it’s part of why my comment sounded strange. There is an idea here, and it’s a mistake to dismiss it out of hand. Adding weights non uniformly during training (not after) is the key to smaller models that outperform present day GPT3. A sketch of the algorithm is to start with a 2x2 block of weights, sum the gradients across 10 training steps, then subdivide the quadrant with the highest delta. Doing this recursively is prohibitive, which is where megatexture comes in. Many advantages. At runtime you don’t need weight compression because you can simply switch to a lower miplevel if running on a phone. Different accelerators during training can focus on different areas of the network. Weight dropout is an automatic feature. Etc. If you’ll excuse me, it’s back to hugging the porcelain bowl.
- joxel 4y ago
  You mention this technique has been shown to outperform GPT3, do you have a citation for that? Would love to read more details about this interesting concept.
  - fl0id 4y ago
    they didn't say it was shown, he stated that those to be developed models would do that.
    - joxel 4y ago
      "Adding weights non uniformly during training (not after) is the key to smaller models that outperform present day GPT3." seems to greatly imply a certainty of result that has already been discovered. Though this commenter is familiar to me, and I know he has made silly claims in other threads throughout the years here.
      - sillysaurusx 4y ago
        I would hope so. It’s in my name! I ended up calling an ambulance so I’ll postpone this until later. Feeling a little better but a full explanation will have to wait. The answer is that of course it’s not proved yet, since no one has implemented it (or at least efficiently). It’s fine to be skeptical. Current techniques are blocked by the technical challenge of getting 10GB+ to fit on a pod. Very few people have those skills. If there’s even a chance that this will work, it’s worth exploring, so I will be.
- NavinF 4y ago
  Sounds kinda like progressive growing except you're not doubling the resolution uniformly. See ProGAN and its successors. You'd still need to add a large block of weights at a time for performance reasons. Edit: Ah I checked your profile and you already know all this. You probably should have mentioned that lol

by NavinF 4y ago

sillysaurusx 4y ago

I’m unfortunately the sickest I’ve been in years, so this will have to wait. Maybe it’s part of why my comment sounded strange. There is an idea here, and it’s a mistake to dismiss it out of hand. Adding weights non uniformly during training (not after) is the key to smaller models that outperform present day GPT3. A sketch of the algorithm is to start with a 2x2 block of weights, sum the gradients across 10 training steps, then subdivide the quadrant with the highest delta. Doing this recursively is prohibitive, which is where megatexture comes in. Many advantages. At runtime you don’t need weight compression because you can simply switch to a lower miplevel if running on a phone. Different accelerators during training can focus on different areas of the network. Weight dropout is an automatic feature. Etc. If you’ll excuse me, it’s back to hugging the porcelain bowl.

joxel 4y ago
You mention this technique has been shown to outperform GPT3, do you have a citation for that? Would love to read more details about this interesting concept.
- fl0id 4y ago
  they didn't say it was shown, he stated that those to be developed models would do that.
  - joxel 4y ago
    "Adding weights non uniformly during training (not after) is the key to smaller models that outperform present day GPT3." seems to greatly imply a certainty of result that has already been discovered. Though this commenter is familiar to me, and I know he has made silly claims in other threads throughout the years here.
    - sillysaurusx 4y ago
      I would hope so. It’s in my name! I ended up calling an ambulance so I’ll postpone this until later. Feeling a little better but a full explanation will have to wait. The answer is that of course it’s not proved yet, since no one has implemented it (or at least efficiently). It’s fine to be skeptical. Current techniques are blocked by the technical challenge of getting 10GB+ to fit on a pod. Very few people have those skills. If there’s even a chance that this will work, it’s worth exploring, so I will be.
NavinF 4y ago
Sounds kinda like progressive growing except you're not doubling the resolution uniformly. See ProGAN and its successors. You'd still need to add a large block of weights at a time for performance reasons. Edit: Ah I checked your profile and you already know all this. You probably should have mentioned that lol