3 ms·

I don't think people here realize how amazing this is. You are providing very little data and the machine is able to fit the closed-form recurrent relation out

I don't think people here realize how amazing this is. You are providing very little data and the machine is able to fit the closed-form recurrent relation out of it. Very little data and the clean simple expressions is the key. It's just matter of adding more ops like sinusoidals, primes etc to make it recognize more and more complex expressions.

amelius 5y ago
Wouldn't it become slower, the more functions you add? And exponentially slower if you start nesting those functions?
- ShamelessC 5y ago
  There's not any execution of the functions occurring here. It's modeled as a sequence-to-sequence problem where the training sequences have been pre-computed programmatically. I think in this case the attention operation bounds the runtime by the longest possible sequence that can go in at once. Since you have a positional embedding, you can add arbitrarily many more functions without incurring a runtime penalty, although it may begin to interfere with the model's ability to effectively generalize as well, particularly when used with very long sequence lengths (so chains of operations, lists of ints). Since each operation is mapped to an integer, the maximum number of tokens is bound by the precision of the system.
- CasperDern 5y ago
  They used a fixed size transformer, where the vocab determines the functions and input/output range. So unless the model needs more 'memory' for your class of expression there wouldn't necessarily be a big change in performance. They have experiments in the paper with bigger/smaller vocabs.
verdverm 5y ago
People have been doing Symbolic Regression for a long time. I don't really see any novelty in this paper. They have left out a large part of the history and comparative results. How good is this compared to other methods?
- robrenaud 5y ago
  Have people been using transformers for it? Their method beats mathematica by a large margin on OEIS.
  - verdverm 5y ago
    There are much better options than mathematica for SR. The even cite the paper that made SR famous again (the nature one by Hod and Mike Schmidt). They could have at least compared to them. They barely mention the standard method and leave out an entire class of top tier solutions, the more recent deterministic stratgeies. Shows they did not do their background reading and know the history. More like someone with a tool looking for an interesting application than someone deeply interested in a problem and finding all of the tools which can be used. They go on to leave out all of the common test problems used in the field. Makes one wonder if the tests were too difficult for neural nets to produce publishable results with, or if they are just ignorant of the research and test problems. They almost state as much in the next paragraph. They don't even .mention how the other algos are capable where they are not. It's also worth thinking about the environmental cost of the, now three, methods.

amelius 5y ago

Wouldn't it become slower, the more functions you add? And exponentially slower if you start nesting those functions?

ShamelessC 5y ago
There's not any execution of the functions occurring here. It's modeled as a sequence-to-sequence problem where the training sequences have been pre-computed programmatically. I think in this case the attention operation bounds the runtime by the longest possible sequence that can go in at once. Since you have a positional embedding, you can add arbitrarily many more functions without incurring a runtime penalty, although it may begin to interfere with the model's ability to effectively generalize as well, particularly when used with very long sequence lengths (so chains of operations, lists of ints). Since each operation is mapped to an integer, the maximum number of tokens is bound by the precision of the system.
CasperDern 5y ago
They used a fixed size transformer, where the vocab determines the functions and input/output range. So unless the model needs more 'memory' for your class of expression there wouldn't necessarily be a big change in performance. They have experiments in the paper with bigger/smaller vocabs.

verdverm 5y ago

People have been doing Symbolic Regression for a long time. I don't really see any novelty in this paper. They have left out a large part of the history and comparative results. How good is this compared to other methods?

robrenaud 5y ago
Have people been using transformers for it? Their method beats mathematica by a large margin on OEIS.
- verdverm 5y ago
  There are much better options than mathematica for SR. The even cite the paper that made SR famous again (the nature one by Hod and Mike Schmidt). They could have at least compared to them. They barely mention the standard method and leave out an entire class of top tier solutions, the more recent deterministic stratgeies. Shows they did not do their background reading and know the history. More like someone with a tool looking for an interesting application than someone deeply interested in a problem and finding all of the tools which can be used. They go on to leave out all of the common test problems used in the field. Makes one wonder if the tests were too difficult for neural nets to produce publishable results with, or if they are just ignorant of the research and test problems. They almost state as much in the next paragraph. They don't even .mention how the other algos are capable where they are not. It's also worth thinking about the environmental cost of the, now three, methods.