6 ms·

Learning to Play the Chaos Game

149 points by hardmath123 6y ago

dstick 6y ago
Upvoted for the pure passion that just oozes through the sentences. Didn’t understand half of it but that did not detract from the reading fun! :)
- simonebrunozzi 6y ago
  "comfortably numbered" (as a reference to Pink Floyd's "Comfortably numb") is fun as well.
- hardmath123 6y ago
  Thank you! That means a lot to me. :)
MrXOR 6y ago
Nice!
chris_st 6y ago
That's both amazing and beautiful. Well done!
skulk 6y ago
Hey, I remember you from the Scratch forums back in the early '10s. It's great to see that you're still producing excellent content!
skinner_ 6y ago
This is so great! Frankly, I believe that this kind of low-parameter-count high complexity optimization task is the least suitable kind of task for SGD. Bad local optima everywhere. But I didn't let this opinion of mine spoil the fun: I changed Chamfer distance to unbiased Sinkhorn divergence (via GeomLoss), bumped arity to 4, moved randomness out of the training loop (with the goal of making training more stable), and added a LR scheduler. Here's my notebook: https://colab.research.google.com/drive/154ffvEWpD7tTW_AIqTDWq6kkeGgmhf7P https://colab.research.google.com/drive/154ffvEWpD7tTW_AIqTD... This tree parameter set is quite nice and interpretable: https://users.renyi.hu/~daniel/tmp/ifs-christmas-tree-arity-4.gif https://users.renyi.hu/~daniel/tmp/ifs-christmas-tree-arity-...
- hardmath123 6y ago
  How cool! I was on the fence about whether or not to put up the source code— I'm _glad_ I did! Do you have a recommendation for a good reference that teaches about the various metrics for point-cloud distance? (I only used Chamfer distance because I hazily recalled it from some undergrad class taken a while ago...)
  - skinner_ 6y ago
    > How cool! I was on the fence about whether or not to put up the source code— I'm _glad_ I did! I'm glad you did, thank you for that! Disclaimer: I'm not claiming that any of my modifications actually help, there's too much randomness introduced by local minima, and I only did a few training runs. Unbiased Sinkhorn is fancier than Chamfer, but who knows if it's better or not for this use case. Starting from a much higher learning rate did speed up convergence, though. Re point cloud distance, there's lots of good stuff referenced in the GeomLoss documentation: https://www.kernel-operations.io/geomloss/api/geomloss.html https://www.kernel-operations.io/geomloss/api/geomloss.html , for example the author's GTTI 2019 slides are an excellent overview. For a very deep dive into Optimal Transport there is Computational Optimal Transport by Peyré and Cuturi: https://arxiv.org/abs/1803.00567 https://arxiv.org/abs/1803.00567 . Note: these mention MMD and Hausdorff, but it's all very Optimal Transport centric.

149 points by hardmath123 6y ago

Upvoted for the pure passion that just oozes through the sentences. Didn’t understand half of it but that did not detract from the reading fun! :)

simonebrunozzi 6y ago
"comfortably numbered" (as a reference to Pink Floyd's "Comfortably numb") is fun as well.
hardmath123 6y ago
Thank you! That means a lot to me. :)

MrXOR 6y ago

Nice!

chris_st 6y ago

That's both amazing and beautiful. Well done!

skulk 6y ago

Hey, I remember you from the Scratch forums back in the early '10s. It's great to see that you're still producing excellent content!

skinner_ 6y ago

This is so great! Frankly, I believe that this kind of low-parameter-count high complexity optimization task is the least suitable kind of task for SGD. Bad local optima everywhere. But I didn't let this opinion of mine spoil the fun: I changed Chamfer distance to unbiased Sinkhorn divergence (via GeomLoss), bumped arity to 4, moved randomness out of the training loop (with the goal of making training more stable), and added a LR scheduler. Here's my notebook: https://colab.research.google.com/drive/154ffvEWpD7tTW_AIqTDWq6kkeGgmhf7P https://colab.research.google.com/drive/154ffvEWpD7tTW_AIqTD... This tree parameter set is quite nice and interpretable: https://users.renyi.hu/~daniel/tmp/ifs-christmas-tree-arity-4.gif https://users.renyi.hu/~daniel/tmp/ifs-christmas-tree-arity-...

hardmath123 6y ago
How cool! I was on the fence about whether or not to put up the source code— I'm _glad_ I did! Do you have a recommendation for a good reference that teaches about the various metrics for point-cloud distance? (I only used Chamfer distance because I hazily recalled it from some undergrad class taken a while ago...)
- skinner_ 6y ago
  > How cool! I was on the fence about whether or not to put up the source code— I'm _glad_ I did! I'm glad you did, thank you for that! Disclaimer: I'm not claiming that any of my modifications actually help, there's too much randomness introduced by local minima, and I only did a few training runs. Unbiased Sinkhorn is fancier than Chamfer, but who knows if it's better or not for this use case. Starting from a much higher learning rate did speed up convergence, though. Re point cloud distance, there's lots of good stuff referenced in the GeomLoss documentation: https://www.kernel-operations.io/geomloss/api/geomloss.html https://www.kernel-operations.io/geomloss/api/geomloss.html , for example the author's GTTI 2019 slides are an excellent overview. For a very deep dive into Optimal Transport there is Computational Optimal Transport by Peyré and Cuturi: https://arxiv.org/abs/1803.00567 https://arxiv.org/abs/1803.00567 . Note: these mention MMD and Hausdorff, but it's all very Optimal Transport centric.