3 ms·

It makes sense that allowing for things many consider to be incorrect or impossible, like through removing valence restrictions, improves prediction quality. As

It makes sense that allowing for things many consider to be incorrect or impossible, like through removing valence restrictions, improves prediction quality. As an example, pentavalent carbon is very much real, and so disallowing it means whatever chemical system is learned by the model is less aligned with reality. It doesn't matter if pentavalent carbon is extremely conditional to specific circumstances, many novel and interesting molecules are.

Bluestein 2y ago
Excuse my ignorance but, would those resulting molecules be real - in the sense of being realizable?
- Grimblewald 2y ago
  No excuses necessary I think that's a great question. Yes, like in the article - you can remove extremely unlikely to be real things, unlikely to be real could be because of things such as being extremely conditional (pentavalent carbon for example) or simply implausible. The point is that by preventing the model from learning a distribution that captures the extremely rare though real instances of things, you end up with a distribution that departs from whatever the reality of the situation is. You end up with a biased model, and that is highly unlikely to be a good thing. Bottom line, it is better to let a model learn the full distribution then clip predictions that come from extreme ends of the distribution, rather than preventing the model from considering these extremes in the first place. Either way you don't get predictions from the extremes, but the model allowed to consider extremes is more likely to do interesting things.
  - Bluestein 2y ago
    Makes sense, thanks for taking the time.- > The point is that by preventing the model from learning a distribution that captures the extremely rare though real instances of things, you end up with a distribution that departs from whatever the reality of the situation is. You end up with a biased model, and that is highly unlikely to be a good thing. I (think I) understand. On top, the model being able to generate for conditions able to bring forth a "materially" (in realizable fact) unlikely condition, might yield results that are interesting, -particularly- because the conditions that it would be contingent upon -are- so hard, or unlikely in actual fact. Makes sense.-
  - iboisvert 2y ago
    I think I may have heard of a similar problem in a different field, computational thermodynamics. Solutions to a set of equations may be real or complex. The nonlinear solvers worked with real numbers exclusively and from time to time would not converge to any solution. Apparently if one used a solver that worked with complex numbers, the solver would reach a solution in real space more often than exclusively real solvers. Of course solving was much more expensive. Your explanation seems to fit here too
    - Grimblewald 2y ago
      Interesting, thanks for sharing! Do you have any papers or other publicly available works to share that discuss this? I'd be very interested in reading about it.
      - iboisvert 2y ago
        I originally heard about this many years ago at an informal talk, but I was able to find this paper that sounds like the same problem and should have pretty good references https://www.sciencedirect.com/science/article/abs/pii/S009813540700292X https://www.sciencedirect.com/science/article/abs/pii/S00981...
    - 11101010001100 2y ago
      I'm guessing the complex numbers add the right kind of 'fuzziness'. But please share any references if you have.
nick__m 2y ago
in the article they talk about syntactically invalid smile formula (mismatch parenthesis and cie, check figure 1b), impossible molecules can be expressed with syntactically valid SMILE. But nonetheless your explanation appears valid to my non chemist eyes. from the paper: Structural biases limit generalization An ideal generative model would sample evenly from the chemical space surrounding the molecules in the training set. The observation of structural biases in the outputs of language models trained on SELFIES is at odds with this goal.

Bluestein 2y ago

Excuse my ignorance but, would those resulting molecules be real - in the sense of being realizable?

Grimblewald 2y ago
No excuses necessary I think that's a great question. Yes, like in the article - you can remove extremely unlikely to be real things, unlikely to be real could be because of things such as being extremely conditional (pentavalent carbon for example) or simply implausible. The point is that by preventing the model from learning a distribution that captures the extremely rare though real instances of things, you end up with a distribution that departs from whatever the reality of the situation is. You end up with a biased model, and that is highly unlikely to be a good thing. Bottom line, it is better to let a model learn the full distribution then clip predictions that come from extreme ends of the distribution, rather than preventing the model from considering these extremes in the first place. Either way you don't get predictions from the extremes, but the model allowed to consider extremes is more likely to do interesting things.
- Bluestein 2y ago
  Makes sense, thanks for taking the time.- > The point is that by preventing the model from learning a distribution that captures the extremely rare though real instances of things, you end up with a distribution that departs from whatever the reality of the situation is. You end up with a biased model, and that is highly unlikely to be a good thing. I (think I) understand. On top, the model being able to generate for conditions able to bring forth a "materially" (in realizable fact) unlikely condition, might yield results that are interesting, -particularly- because the conditions that it would be contingent upon -are- so hard, or unlikely in actual fact. Makes sense.-
- iboisvert 2y ago
  I think I may have heard of a similar problem in a different field, computational thermodynamics. Solutions to a set of equations may be real or complex. The nonlinear solvers worked with real numbers exclusively and from time to time would not converge to any solution. Apparently if one used a solver that worked with complex numbers, the solver would reach a solution in real space more often than exclusively real solvers. Of course solving was much more expensive. Your explanation seems to fit here too
  - Grimblewald 2y ago
    Interesting, thanks for sharing! Do you have any papers or other publicly available works to share that discuss this? I'd be very interested in reading about it.
    - iboisvert 2y ago
      I originally heard about this many years ago at an informal talk, but I was able to find this paper that sounds like the same problem and should have pretty good references https://www.sciencedirect.com/science/article/abs/pii/S009813540700292X https://www.sciencedirect.com/science/article/abs/pii/S00981...
  - 11101010001100 2y ago
    I'm guessing the complex numbers add the right kind of 'fuzziness'. But please share any references if you have.

nick__m 2y ago

in the article they talk about syntactically invalid smile formula (mismatch parenthesis and cie, check figure 1b), impossible molecules can be expressed with syntactically valid SMILE. But nonetheless your explanation appears valid to my non chemist eyes. from the paper: Structural biases limit generalization An ideal generative model would sample evenly from the chemical space surrounding the molecules in the training set. The observation of structural biases in the outputs of language models trained on SELFIES is at odds with this goal.