5 ms·

Generative Flow Networks

168 points by selimonder 5y ago

soraki_soladead 5y ago
Here's the referenced tutorial: https://milayb.notion.site/GFlowNet-Tutorial-919dcf0a0f0c4e978916a2f509938b00 https://milayb.notion.site/GFlowNet-Tutorial-919dcf0a0f0c4e9... (Some of the links appear to require signing in to Notion but this one doesn't.)
deleted 5y ago
[deleted]
awelkie 5y ago
Having just skimmed the linked tutorial, it looks like GFlowNets could be useful for automated theorem proving. Anyone with more knowledge know if this is possible?
- andrewnc 5y ago
  I don't see why not. I think they can be used for anything a GNN / Transformer could be used for, but with better probabilistic qualities.
  - mcbuilder 5y ago
    What do you see as the application to GNN / Transformer tasks? Both of those networks are best suited for different tasks, and GFlowNetworks seem well suited to quickly approximating the partition function of say an Deep RL reward distribution, something that MCMC is ill suited for because of the incredibly high dimensional space. I see these all as different techniques, perhaps the ideas will combine at some point but GFlowNetworks are not an evolution of GNNs or Transformers.
    - manux 5y ago
      Yeah the "net" in GFlowNet refers to how the underlying state is interpreted, not to an architecture. It is a way to train generative models, on any kind of discrete data (like graphs). Source: am first author of original GFlowNet paper.
      - mcbuilder 5y ago
        Thanks for your comment! I originally finished my PhD in computational neuroscience and have been riding the deep learning wave for the past five years. I listened to the audio Machine Learning Street Talk episode on GFlowNets, and that was my general introduction. It wasn't until I looked at the image of the GFlowNet on your blog post a connection to actually biological plausible architectures became apparent. The individual firing of a neuron can be interpreted as a Poisson process, but with populations of neurons we often would approximate the firing rate of the entire population as a Gaussian distribution. That would be one patch of cortical column, but these populations are linked together to form larger scale functional brain networks. Anyway, thinking about that scale you could imagine the neural activity communicated between these patches of activity as a type of flow from one region to another. I think your paper raises interesting questions about how this inference engine in our brain might be doing some sort of MCMC like sampling and constructing different belief hypothesis.
        manux 5y ago
        Yeah! This kind of reasoning is why it's exciting to think that we could use the GFlowNet machinery to construct latent representations--ones that map more directly to the notion of "beliefs" that we're used to think about as humans, something "discrete" and relational. Concretely what this could mean is using these tools to generate causal hypotheses, like what's been done here: https://arxiv.org/abs/2202.13903 https://arxiv.org/abs/2202.13903
- manux 5y ago
  Yes, although they may not be ideal unless you're able to define a "distance" between a proof that the agent is proposing and whether that proof is correct (i.e. unless you're able to define a reward or energy function). It may be possible to infer/learn a score from existing proofs though. We have a paper that manages to both learn a flow and an energy function (the score) from data: https://arxiv.org/abs/2202.01361 https://arxiv.org/abs/2202.01361 I don't know much about theorem proving though. Can some value be attributed to partial proofs?
mark_l_watson 5y ago
I started reading through the linked tutorial, but I think I need to first back up and review reinforcement learning (I took 2 Coursera classes in RL, and helped with one project at a previous job, so I am at a beginner stage). I love deep learning, a practical way to solve many problems (and most of my work over the last 6 years) but DL seems incomplete as far as explainability, causation, etc. Anyway, I just retired last week (mostly to take care of my wife who is having a health crisis) so I have plenty of time to study GFN, probably starting with re-taking the 2 Coursera classes on RL and then finishing the RL specialization.
cs702 5y ago
For a quick, easy-to-digest introduction to the topic, there's a fantastic video with explanations and an interview with Bengio that uses Galton peg boards as a starting point for understanding generative flow networks. Highly recommended if you're a researcher or practitioner interested in learning about this: https://www.youtube.com/watch?v=M49TMqK5uCE https://www.youtube.com/watch?v=M49TMqK5uCE
ramesh31 5y ago
Correct me if I'm wrong, but it seems like SOTA for AI now is entirely in developing more and more complex sets of convolutions for deep learning?
- MathYouF 5y ago
  And what is the SoTA for biological intelligence? While not exactly this, it seems to be pretty similar (even between members of the same species). Snark aside, with a few nuances, you're mostly correct.
manux 5y ago
Hi, first author of the original NeurIPS paper here! Someone just shared this HN post with me. I'll go through the comments but happy to try to answer questions as well.
- igorkraw 5y ago
  Could you highlight the difference between this and training a permutation invariant or equivariant policy network using standard supervised or RL methods? Assuming I also have a way of having an invariant/equivariant loss function
  - manux 5y ago
    What the permutation invariance gets you is that the model doesn't arbitrarily prefer one (graph) configuration over another, but this seems tangential. The difference between this and RL is in what we do with the reward: - RL says, give me a reward and I'll give you its max. - GFlowNet says, give me a reward and I'll give you all its modes (via p(x) \propto R(x)). Yes you would ideally have a loss (well, a reward/energy) that is invariant and operates e.g. directly on the molecule rather than on some arbitrary ordering of the nodes.
- cs702 5y ago
  I'm still wrapping my head around this (it seems rather magical; can it really be just that?)... but I find the idea of "sampling from a distribution of trajectories that reach a terminal state whose probability is proportional to a positive reward function, by minimizing the difference between the flow coming into the trajectories and the flow coming out of them, which by construction must be equal" to be both beautiful and elegant -- like all great ideas. How did you and your coauthors come up with this? Trial and error? Or was there a moment of serendipitous creative insight? -- To the moderators: My comment is now at the top of the page, but manux's comment above is more deserving of the top spot. I just upvoted it. Please consider pinning it at the top.
  - manux 5y ago
    I think the inspiration came to me from looking at SumTrees and from having worked on Temporal Difference learning for a long time. The idea of flows came to Yoshua and I from the realization that we wanted some kind of energy conservation/preservation mechanism from having multiple paths lead to the same state. But, yes in the moment it felt like some very serendipitous insight!

168 points by selimonder 5y ago

Here's the referenced tutorial: https://milayb.notion.site/GFlowNet-Tutorial-919dcf0a0f0c4e978916a2f509938b00 https://milayb.notion.site/GFlowNet-Tutorial-919dcf0a0f0c4e9... (Some of the links appear to require signing in to Notion but this one doesn't.)

deleted 5y ago

[deleted]

awelkie 5y ago

Having just skimmed the linked tutorial, it looks like GFlowNets could be useful for automated theorem proving. Anyone with more knowledge know if this is possible?

andrewnc 5y ago
I don't see why not. I think they can be used for anything a GNN / Transformer could be used for, but with better probabilistic qualities.
- mcbuilder 5y ago
  What do you see as the application to GNN / Transformer tasks? Both of those networks are best suited for different tasks, and GFlowNetworks seem well suited to quickly approximating the partition function of say an Deep RL reward distribution, something that MCMC is ill suited for because of the incredibly high dimensional space. I see these all as different techniques, perhaps the ideas will combine at some point but GFlowNetworks are not an evolution of GNNs or Transformers.
  - manux 5y ago
    Yeah the "net" in GFlowNet refers to how the underlying state is interpreted, not to an architecture. It is a way to train generative models, on any kind of discrete data (like graphs). Source: am first author of original GFlowNet paper.
    - mcbuilder 5y ago
      Thanks for your comment! I originally finished my PhD in computational neuroscience and have been riding the deep learning wave for the past five years. I listened to the audio Machine Learning Street Talk episode on GFlowNets, and that was my general introduction. It wasn't until I looked at the image of the GFlowNet on your blog post a connection to actually biological plausible architectures became apparent. The individual firing of a neuron can be interpreted as a Poisson process, but with populations of neurons we often would approximate the firing rate of the entire population as a Gaussian distribution. That would be one patch of cortical column, but these populations are linked together to form larger scale functional brain networks. Anyway, thinking about that scale you could imagine the neural activity communicated between these patches of activity as a type of flow from one region to another. I think your paper raises interesting questions about how this inference engine in our brain might be doing some sort of MCMC like sampling and constructing different belief hypothesis.
      - manux 5y ago
        Yeah! This kind of reasoning is why it's exciting to think that we could use the GFlowNet machinery to construct latent representations--ones that map more directly to the notion of "beliefs" that we're used to think about as humans, something "discrete" and relational. Concretely what this could mean is using these tools to generate causal hypotheses, like what's been done here: https://arxiv.org/abs/2202.13903 https://arxiv.org/abs/2202.13903
manux 5y ago
Yes, although they may not be ideal unless you're able to define a "distance" between a proof that the agent is proposing and whether that proof is correct (i.e. unless you're able to define a reward or energy function). It may be possible to infer/learn a score from existing proofs though. We have a paper that manages to both learn a flow and an energy function (the score) from data: https://arxiv.org/abs/2202.01361 https://arxiv.org/abs/2202.01361 I don't know much about theorem proving though. Can some value be attributed to partial proofs?

mark_l_watson 5y ago

I started reading through the linked tutorial, but I think I need to first back up and review reinforcement learning (I took 2 Coursera classes in RL, and helped with one project at a previous job, so I am at a beginner stage). I love deep learning, a practical way to solve many problems (and most of my work over the last 6 years) but DL seems incomplete as far as explainability, causation, etc. Anyway, I just retired last week (mostly to take care of my wife who is having a health crisis) so I have plenty of time to study GFN, probably starting with re-taking the 2 Coursera classes on RL and then finishing the RL specialization.

cs702 5y ago

For a quick, easy-to-digest introduction to the topic, there's a fantastic video with explanations and an interview with Bengio that uses Galton peg boards as a starting point for understanding generative flow networks. Highly recommended if you're a researcher or practitioner interested in learning about this: https://www.youtube.com/watch?v=M49TMqK5uCE https://www.youtube.com/watch?v=M49TMqK5uCE

ramesh31 5y ago

Correct me if I'm wrong, but it seems like SOTA for AI now is entirely in developing more and more complex sets of convolutions for deep learning?

MathYouF 5y ago
And what is the SoTA for biological intelligence? While not exactly this, it seems to be pretty similar (even between members of the same species). Snark aside, with a few nuances, you're mostly correct.

manux 5y ago

Hi, first author of the original NeurIPS paper here! Someone just shared this HN post with me. I'll go through the comments but happy to try to answer questions as well.

igorkraw 5y ago
Could you highlight the difference between this and training a permutation invariant or equivariant policy network using standard supervised or RL methods? Assuming I also have a way of having an invariant/equivariant loss function
- manux 5y ago
  What the permutation invariance gets you is that the model doesn't arbitrarily prefer one (graph) configuration over another, but this seems tangential. The difference between this and RL is in what we do with the reward: - RL says, give me a reward and I'll give you its max. - GFlowNet says, give me a reward and I'll give you all its modes (via p(x) \propto R(x)). Yes you would ideally have a loss (well, a reward/energy) that is invariant and operates e.g. directly on the molecule rather than on some arbitrary ordering of the nodes.
cs702 5y ago
I'm still wrapping my head around this (it seems rather magical; can it really be just that?)... but I find the idea of "sampling from a distribution of trajectories that reach a terminal state whose probability is proportional to a positive reward function, by minimizing the difference between the flow coming into the trajectories and the flow coming out of them, which by construction must be equal" to be both beautiful and elegant -- like all great ideas. How did you and your coauthors come up with this? Trial and error? Or was there a moment of serendipitous creative insight? -- To the moderators: My comment is now at the top of the page, but manux's comment above is more deserving of the top spot. I just upvoted it. Please consider pinning it at the top.
- manux 5y ago
  I think the inspiration came to me from looking at SumTrees and from having worked on Temporal Difference learning for a long time. The idea of flows came to Yoshua and I from the realization that we wanted some kind of energy conservation/preservation mechanism from having multiple paths lead to the same state. But, yes in the moment it felt like some very serendipitous insight!