6 ms·

This is one of those links where just seeing the title sets you off, thinking about the implications. I'm going to have to spend more time digesting the articl

This is one of those links where just seeing the title sets you off, thinking about the implications. I'm going to have to spend more time digesting the article, but one thing that jumps out at me, and maybe it's answered in the article and I don't understand it, is the role of time. Generally in physics, you're talking about a quantity being conserved over time, and I'm not sure what plays the role of time when you're talking about conserved quantities in machine learning -- is it conserved over training iterations or over inference layers, or what? edit: now that i've read it again, I just saw that they described in the second paragraph. I'm now wondering if in something like Sora that can do a kind of physical modeling, if there's some conserved quantity in the neural network that is _directly analagous_ to conserved quantities in physics -- if there is, for example, something that represents momentum, that operates exactly as momentum as it progresses through the layers.

Raro 3y ago
Yeah, I've been thinking about similar concepts in a different context. Fascinating. Regarding the role of time, the idea of a purely conserved quantity is that it is conserved under the conditions of the system (that's why the article frequently references Newton's First Law), so they're generally held "for all time that these symmetries exist in the system". Specifically on time: the invariant for systems that exhibit continuous time symmetries (i.e. you move a little bit forward or backward in time and the system looks exactly the same) is energy.
- dustingetz 3y ago
  Here's my ELI5 attempt of the time/energy relation: imagine a spring at rest (not moving) strike the spring, it's now oscillating the system now contains energy like a battery what is energy? it's stored work potential the battery is storing the energy, which can then be taken out at some future time the spring is transporting the energy through time in fact how do we measure time? with clocks. What's a clock? It's an oscillator. The energized spring is the clock. When system energy is zero, what is time even? There's no baseline against which to measure change when nothing is changing
  - PaulHoule 3y ago
    Symmetry exists abstractly, apart from time. There are many machine learning problems which should have symmetries: a picture of a cow rotated 135 degrees is still a picture of a cow, the meaning of spoken words shouldn't change with the audio level, etc. If they were doing machine learning on tracks from the LHC the system ought to take account of relativistic momentum and energy. Can a model learn a symmetry? Or should a symmetry just be built into the model from the beginning?
    - sdenton4 3y ago
      Equivariant machine learning is a thing that people have tried... Tends to be expensive and slow, though, and imposes invariances that our model (a universal function approximator, recall) should just learn anyway: If you don't have enough pictures of upside down cows, just train a normal model with augmentations.
  - slingnow 3y ago
    [flagged]
Raro 3y ago
Ha, my previous comment was before your new edit mentioning Sora. There is a good reason why the accompanying research report to the Sora demo isn't titled "Awesome Generative Video," but references world models. The interesting feature is how many apparently (approximations to) physical properties emerge (object permanence, linear motion, partially elastic collisions, as well as many of the elements of grammar of film), and which do not (notably material properties of solid and fluids, creation of objects from nothing, etc.)
Communitivity 3y ago
"I'm now wondering if in something like Sora that can do a kind of physical modeling, if there's some conserved quantity in the neural network that is _directly analogous_ to conserved quantities in physics" My first thought on reading that was that if there was it would be interesting to see if there was some way it tied into the concept of us living in a simulation, i.e. we're all living in a complex ML network simulation.
nostrademons 3y ago
In physics, the conserved quantity isn't always time. Invariance over time translation is specifically conservation of energy. Invariance over spatial translation is conservation of momentum, invariance over spatial rotation is conservation of conservation of angular momentum, invariance of electromagnetic field is conservation of current, and invariance of wave function phase is conservation of charge. I think the analogue in machine learning is conservation over changes in the training data. After all, the point of machine learning is to find general models that describe the training data given, and minimize the loss function. Assuming that a useful model can be trained, the whole point is that it generalizes to new, unseen instances with minimal losses, i.e. the model remains invariant under shifts in the instances seen. The more interesting part to me is what this says about philosophy of physics. Noether's Theorem can be restated as "The laws of physics are invariant under X transformation", where X is the gauge symmetry associated with the conservation law. But maybe this is simply a consequence of how we do physics. After all, the point of science is to produce generalized laws from empirical observations. It's trivially easy to find a real-world situation where conservation of energy does not hold (any system with friction, which is basically all of them), but the math gets very messy if you try to actually model the real data, so we rely on approximations that are close enough most of the time. And if many people take empirical measurements at many different points in space, and time, and orientations, you get generalized laws that hold regardless of where/when/who takes the measurement. Machine learning could be viewed as doing science on empirically measurable social quantities. It won't always be accurate, as individual machine-learning fails show. But it's accurate enough that it can provide useful models for civilization-scale quantities.
- sdenton4 3y ago
  A nice way to formulate (most) data augmentations is: a family of functions A = {a} such that our optimized neural network f obeys f(x) ~= f(a(x)). So in this case, we're explicitly defining the set of desired invariances.
- Aardwolf 3y ago
  Is there any way to deduce which invariance gives which conservation? I mean for example: how can you tell that time invariance is the one paired with conservation of energy? Why is e.g. time invariance not paired with momentum, current, or anything else, but specifically energy? I know that I can remember momentum is paired with translation simply because there's both the angular momentum and the non-angular momentum one and in space you have translation and rotation, so for time energy is the only one that's left over, but I'm not looking for a trick to remember it, I'm looking for the fundamental reason, as well as how to tell what will be paired with some invariance when looking at some other new invariance
shiandow 3y ago
A convolutional neural network ought to have translational symmetry, which should lead to a generalized version of momentum. If I understood the article correctly the conserved quantity would be <gx, dx>, where dx is the finite difference gradient of x. This gives a vector with dimensions equal to however many directions you can translate a layer in and which is conserved over all (convolutional) layers.
- cgadski 3y ago
  Exactly right! In fact, because that symmetry does not include an action on the parameters of the layer, your conserved quantity <gx, dx> should hold whether or not the network is stationary for a loss. This means that it'll be stationary on every single data point. (In an image classification model, these values are just telling you whether or not the loss would be improved if the input image were translated.)
  - empath-nirvana 3y ago
    Everything in the paper is talking about global symmetries, is there also the possibility of gauge symmetries?
deleted 3y ago
[deleted]
nurple 3y ago
I think the most profound insight I've come across while studying this particular topic is the insight that information theory ended up being the answer to conserving the 2nd law with respect to Maxwell's demon thought experiment. Not to put too fine a point, but essentially the knowledge organized in the mind of the demon, about the particles in its system, was calculated to offset the creation of the energy gradient. I found the thinking of William Sidis to be particularly thought provoking perspective on Noether's benchmark work, in his paper The Animate and the Inanimate he posits--at a high level--that life is a "reversal of the second law of thermodynamics"; not that the 2nd law is a physical symmetry, but a mental one in an existence where energy reversibly flows between positive and negative states. Indeed, when considering machine learning, I think it's quite interesting to consider how the organizing of information/knowledge done during training in some real way mirrors the energy-creating information interred in the mind of Maxwell's demon. When taking into account the possible transitive benefits of knowledge organized via machine learning, and its attendant oracle through application, it's easy to see a world where this results in a net entropy loss, the creation of a previously non-existent energy gradient. In my mind this has interesting implications for Fermi's paradox as it seems to imply the inevitibility of the organization of information. Taken further into my own personal dogma, I think it's inevitable that we create--what we would consider--a sentient being as I believe this is the cycle of our own origin in the larger evolutionary timeline.
- Jerrrry 3y ago
  >at a high level--that life is a "reversal of the second law of thermodynamics"; Life temporarily displaces entropy, locally. Life wins battles, chaos wins the war. >Indeed, when considering machine learning, I think it's quite interesting to consider how the organizing of information/knowledge done during training in some real way mirrors the energy-creating information interred in the mind of Maxwell's demon. This is our human bias favoring the common myth of ever-expanding complexity is an "inevitable" result of the passage of time; refer to Stephen Jay Gould's "Full House: The Spread of Excellence from Plato to Darwin"[0] for the only palatable refute modern evolutionists can offer. >When taking into account the possible transitive benefits of knowledge organized via machine learning, and its attendant oracle through application, it's easy to see a world where this results in a net entropy loss, the creation of a previously non-existent energy gradient. Because it is. Randomness combined with a sieve, like a generator and a discriminator, like the primordial protein soup and our own existence as a selector, like chaos and order themselves, MAY - but DOES NOT have to - lead to temporary, localized areas of complexity, that we call 'life'. This "energy gradient" you speak of is literally gravity pulling baryonic matter foward thru space time. All work requires a temperature gradient - Hawking's musings on the second law of thermodynamics and your own intuition can reason why. >In my mind this has interesting implications for Fermi's paradox as it seems to imply the inevitibility of the organization of information. Taken further into my own personal dogma, I think it's inevitable that we create--what we would consider--a sentient being as I believe this is the cycle of our own origin in the larger evolutionary timeline. Over cosmological time spans, it is a near-mathematical certainty, that we are to either reach the universe's Omega point[1] on "our" own accord, perish to our own, by our own creation, or by our own son's, hands. [0]: https://www.amazon.com/Full-House-Spread-Excellence-Darwin/dp/0674061616 https://www.amazon.com/Full-House-Spread-Excellence-Darwin/d... [1]: https://www.youtube.com/watch?v=eOxHRFN4rs0 https://www.youtube.com/watch?v=eOxHRFN4rs0
jungturk 3y ago
> now wondering...if there's some conserved quantity in the neural network that is _directly analagous_ to conserved quantities in physics Isn't the model attempting to conserve information during training? And isn't information a physical quantity?
rnhmjoj 3y ago
Time is not special regarding symmetries and conserved quantities. In general you can consider any family of continuous transformations parametrised by some real variable s: be it translations by a distance x, rotations by an angle φ, etc. These are technically one-parameter subgroups of a Lie group. Then, if your dynamical system is symmetrical under these transformations you can construct a quantity whose derivative wrt s is zero.

Raro 3y ago

Yeah, I've been thinking about similar concepts in a different context. Fascinating. Regarding the role of time, the idea of a purely conserved quantity is that it is conserved under the conditions of the system (that's why the article frequently references Newton's First Law), so they're generally held "for all time that these symmetries exist in the system". Specifically on time: the invariant for systems that exhibit continuous time symmetries (i.e. you move a little bit forward or backward in time and the system looks exactly the same) is energy.

dustingetz 3y ago
Here's my ELI5 attempt of the time/energy relation: imagine a spring at rest (not moving) strike the spring, it's now oscillating the system now contains energy like a battery what is energy? it's stored work potential the battery is storing the energy, which can then be taken out at some future time the spring is transporting the energy through time in fact how do we measure time? with clocks. What's a clock? It's an oscillator. The energized spring is the clock. When system energy is zero, what is time even? There's no baseline against which to measure change when nothing is changing
- PaulHoule 3y ago
  Symmetry exists abstractly, apart from time. There are many machine learning problems which should have symmetries: a picture of a cow rotated 135 degrees is still a picture of a cow, the meaning of spoken words shouldn't change with the audio level, etc. If they were doing machine learning on tracks from the LHC the system ought to take account of relativistic momentum and energy. Can a model learn a symmetry? Or should a symmetry just be built into the model from the beginning?
  - sdenton4 3y ago
    Equivariant machine learning is a thing that people have tried... Tends to be expensive and slow, though, and imposes invariances that our model (a universal function approximator, recall) should just learn anyway: If you don't have enough pictures of upside down cows, just train a normal model with augmentations.
- slingnow 3y ago
  [flagged]

Raro 3y ago

Ha, my previous comment was before your new edit mentioning Sora. There is a good reason why the accompanying research report to the Sora demo isn't titled "Awesome Generative Video," but references world models. The interesting feature is how many apparently (approximations to) physical properties emerge (object permanence, linear motion, partially elastic collisions, as well as many of the elements of grammar of film), and which do not (notably material properties of solid and fluids, creation of objects from nothing, etc.)

Communitivity 3y ago

"I'm now wondering if in something like Sora that can do a kind of physical modeling, if there's some conserved quantity in the neural network that is _directly analogous_ to conserved quantities in physics" My first thought on reading that was that if there was it would be interesting to see if there was some way it tied into the concept of us living in a simulation, i.e. we're all living in a complex ML network simulation.

nostrademons 3y ago

In physics, the conserved quantity isn't always time. Invariance over time translation is specifically conservation of energy. Invariance over spatial translation is conservation of momentum, invariance over spatial rotation is conservation of conservation of angular momentum, invariance of electromagnetic field is conservation of current, and invariance of wave function phase is conservation of charge. I think the analogue in machine learning is conservation over changes in the training data. After all, the point of machine learning is to find general models that describe the training data given, and minimize the loss function. Assuming that a useful model can be trained, the whole point is that it generalizes to new, unseen instances with minimal losses, i.e. the model remains invariant under shifts in the instances seen. The more interesting part to me is what this says about philosophy of physics. Noether's Theorem can be restated as "The laws of physics are invariant under X transformation", where X is the gauge symmetry associated with the conservation law. But maybe this is simply a consequence of how we do physics. After all, the point of science is to produce generalized laws from empirical observations. It's trivially easy to find a real-world situation where conservation of energy does not hold (any system with friction, which is basically all of them), but the math gets very messy if you try to actually model the real data, so we rely on approximations that are close enough most of the time. And if many people take empirical measurements at many different points in space, and time, and orientations, you get generalized laws that hold regardless of where/when/who takes the measurement. Machine learning could be viewed as doing science on empirically measurable social quantities. It won't always be accurate, as individual machine-learning fails show. But it's accurate enough that it can provide useful models for civilization-scale quantities.

sdenton4 3y ago
A nice way to formulate (most) data augmentations is: a family of functions A = {a} such that our optimized neural network f obeys f(x) ~= f(a(x)). So in this case, we're explicitly defining the set of desired invariances.
Aardwolf 3y ago
Is there any way to deduce which invariance gives which conservation? I mean for example: how can you tell that time invariance is the one paired with conservation of energy? Why is e.g. time invariance not paired with momentum, current, or anything else, but specifically energy? I know that I can remember momentum is paired with translation simply because there's both the angular momentum and the non-angular momentum one and in space you have translation and rotation, so for time energy is the only one that's left over, but I'm not looking for a trick to remember it, I'm looking for the fundamental reason, as well as how to tell what will be paired with some invariance when looking at some other new invariance

shiandow 3y ago

A convolutional neural network ought to have translational symmetry, which should lead to a generalized version of momentum. If I understood the article correctly the conserved quantity would be <gx, dx>, where dx is the finite difference gradient of x. This gives a vector with dimensions equal to however many directions you can translate a layer in and which is conserved over all (convolutional) layers.

cgadski 3y ago
Exactly right! In fact, because that symmetry does not include an action on the parameters of the layer, your conserved quantity <gx, dx> should hold whether or not the network is stationary for a loss. This means that it'll be stationary on every single data point. (In an image classification model, these values are just telling you whether or not the loss would be improved if the input image were translated.)
- empath-nirvana 3y ago
  Everything in the paper is talking about global symmetries, is there also the possibility of gauge symmetries?

deleted 3y ago

[deleted]

nurple 3y ago

I think the most profound insight I've come across while studying this particular topic is the insight that information theory ended up being the answer to conserving the 2nd law with respect to Maxwell's demon thought experiment. Not to put too fine a point, but essentially the knowledge organized in the mind of the demon, about the particles in its system, was calculated to offset the creation of the energy gradient. I found the thinking of William Sidis to be particularly thought provoking perspective on Noether's benchmark work, in his paper The Animate and the Inanimate he posits--at a high level--that life is a "reversal of the second law of thermodynamics"; not that the 2nd law is a physical symmetry, but a mental one in an existence where energy reversibly flows between positive and negative states. Indeed, when considering machine learning, I think it's quite interesting to consider how the organizing of information/knowledge done during training in some real way mirrors the energy-creating information interred in the mind of Maxwell's demon. When taking into account the possible transitive benefits of knowledge organized via machine learning, and its attendant oracle through application, it's easy to see a world where this results in a net entropy loss, the creation of a previously non-existent energy gradient. In my mind this has interesting implications for Fermi's paradox as it seems to imply the inevitibility of the organization of information. Taken further into my own personal dogma, I think it's inevitable that we create--what we would consider--a sentient being as I believe this is the cycle of our own origin in the larger evolutionary timeline.

Jerrrry 3y ago
>at a high level--that life is a "reversal of the second law of thermodynamics"; Life temporarily displaces entropy, locally. Life wins battles, chaos wins the war. >Indeed, when considering machine learning, I think it's quite interesting to consider how the organizing of information/knowledge done during training in some real way mirrors the energy-creating information interred in the mind of Maxwell's demon. This is our human bias favoring the common myth of ever-expanding complexity is an "inevitable" result of the passage of time; refer to Stephen Jay Gould's "Full House: The Spread of Excellence from Plato to Darwin"[0] for the only palatable refute modern evolutionists can offer. >When taking into account the possible transitive benefits of knowledge organized via machine learning, and its attendant oracle through application, it's easy to see a world where this results in a net entropy loss, the creation of a previously non-existent energy gradient. Because it is. Randomness combined with a sieve, like a generator and a discriminator, like the primordial protein soup and our own existence as a selector, like chaos and order themselves, MAY - but DOES NOT have to - lead to temporary, localized areas of complexity, that we call 'life'. This "energy gradient" you speak of is literally gravity pulling baryonic matter foward thru space time. All work requires a temperature gradient - Hawking's musings on the second law of thermodynamics and your own intuition can reason why. >In my mind this has interesting implications for Fermi's paradox as it seems to imply the inevitibility of the organization of information. Taken further into my own personal dogma, I think it's inevitable that we create--what we would consider--a sentient being as I believe this is the cycle of our own origin in the larger evolutionary timeline. Over cosmological time spans, it is a near-mathematical certainty, that we are to either reach the universe's Omega point[1] on "our" own accord, perish to our own, by our own creation, or by our own son's, hands. [0]: https://www.amazon.com/Full-House-Spread-Excellence-Darwin/dp/0674061616 https://www.amazon.com/Full-House-Spread-Excellence-Darwin/d... [1]: https://www.youtube.com/watch?v=eOxHRFN4rs0 https://www.youtube.com/watch?v=eOxHRFN4rs0

jungturk 3y ago

> now wondering...if there's some conserved quantity in the neural network that is _directly analagous_ to conserved quantities in physics Isn't the model attempting to conserve information during training? And isn't information a physical quantity?

rnhmjoj 3y ago

Time is not special regarding symmetries and conserved quantities. In general you can consider any family of continuous transformations parametrised by some real variable s: be it translations by a distance x, rotations by an angle φ, etc. These are technically one-parameter subgroups of a Lie group. Then, if your dynamical system is symmetrical under these transformations you can construct a quantity whose derivative wrt s is zero.