4 ms·

First, congratulations to you and the other authors - this is a very informative and accessible paper with impressive results. > On both datasets, we find that

First, congratulations to you and the other authors - this is a very informative and accessible paper with impressive results. > On both datasets, we find that synthesis performance scales log-linearly with model size This "scaling law" observation is starting to become a major trend in NLP [1] and other other modalities such as speech recognition [2] and protein structure/function prediction [3]. Do you have any insight or commentary to offer regarding the next step in improving program synthesis? For example, will techniques to scale up model size continue to be a primary focus (eg as in [4]), or do you see improvements in Transformer and attention based architectures as essential for pushing the limits of what has been achieved by you and your colleagues? > We find that even our best models are generally unable to predict the output of a program given a specific input. What do you think about leveraging unsupervised training to improve program synthesis? Could synthesized programs be executed on generated input in a way that supports contrastive learning [5]? Thanks in advance for your time and comments here. [1] Scaling Laws for Neural Language Models https://arxiv.org/abs/2001.08361 https://arxiv.org/abs/2001.08361 [2] Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition https://arxiv.org/abs/2010.10504 https://arxiv.org/abs/2010.10504 [3] ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Learning https://arxiv.org/abs/2007.06225 https://arxiv.org/abs/2007.06225 [4] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity https://arxiv.org/abs/2101.03961 https://arxiv.org/abs/2101.03961 [5] A Simple Framework for Contrastive Learning of Visual Representations https://arxiv.org/pdf/2108.00587.pdf https://arxiv.org/pdf/2108.00587.pdf

tasdfqwer0897 5y ago
> do you see improvements in Transformer or attention based architectures as essential... I do personally, but there is some disagreement about this in the field. In fact, I would go further and say that (in addition to using large pre-trained models) we will need methods of training that are pretty substantially different in order to elicit robust reasoning behavior. Even supposing I'm wrong about this, if you go and look at the scaling plots in figure 3 and try to figure out how big your model would need to be in order to be solving most of these problems, you'd get a really big number. Even if you had such a big model, it would still require post-processing of the samples to actually get the right answers. From the perspective of applications, that's fine, but it's a little unsatisfying from the perspective of studying intelligence. Even with those caveats (!) these problems aren't that hard compared to general software engineering tasks... > What do you think about leveraging unsupervised training to improve program synthesis? Could synthesized programs be executed on generated input in a way that supports contrastive learning [5]? I think this is an interesting idea and someone should try it! I do think that, even restricting our attention to just getting neural networks to execute programs, that we will need to do something a little more drastic to robustly get the results we want.
- mjburgess 5y ago
  > it's a little unsatisfying from the perspective of studying intelligence This power-law behaviour is exactly what you get when modelling a non-computable function. Eg., consider approximating the mean of sin^2(x) over -pi to +pi. If you sample x 10^s times (s = 1, 2, 3, ...) the difference from the non-computable analytic answer (0.5) scales log-linear (ie., power-law). { pwr : np.log(0.5 - ( np.sin(np.linspace(-np.pi, +np.pi, 10**pwr) ) ** 2).mean()) for pwr in range(1, 8) } It is my view that intelligence isn't computable, and that any approximation method to some layer of intelligence will run-up against this problem.
  - nextaccountic 5y ago
    I think you're mixing up terminology here, because sin^2(x) is computable. I mean, if there is a process to progressively give better estimates of the output of a function, then this function is computable.
    - mjburgess 5y ago
      sine is a real-valued function There is a computable version for computable inputs, ie., we ask "what is sin(x)?" for some computable-number x_computable. But there is no computable version for a non-computable real number, x_real.
      - nextaccountic 5y ago
        Thank you for the clarification. > But there is no computable version for a non-computable real number, x_real. That's true[0]. But consider that you will never receive something non-computable as input to a program, ever. (if you're allowed to approximate the input until you have enough digits to compute what you need, then the input is computable) Really, I think the best way to view sin(x) is as a function that receives a stream of digits 0.2345345323.. and returns a stream of digits 0.0040933884... - this computable version of sine completely captures every thing we could possibly do with it in a program. Operating with floating point numbers, then, is just mostly truncating the input and output stream. [0] At least in classical logic; in intuitionistic logic, sin : Real -> Real is computable and is equivalent to this idea of receiving and returning streams of digits.

tasdfqwer0897 5y ago

> do you see improvements in Transformer or attention based architectures as essential... I do personally, but there is some disagreement about this in the field. In fact, I would go further and say that (in addition to using large pre-trained models) we will need methods of training that are pretty substantially different in order to elicit robust reasoning behavior. Even supposing I'm wrong about this, if you go and look at the scaling plots in figure 3 and try to figure out how big your model would need to be in order to be solving most of these problems, you'd get a really big number. Even if you had such a big model, it would still require post-processing of the samples to actually get the right answers. From the perspective of applications, that's fine, but it's a little unsatisfying from the perspective of studying intelligence. Even with those caveats (!) these problems aren't that hard compared to general software engineering tasks... > What do you think about leveraging unsupervised training to improve program synthesis? Could synthesized programs be executed on generated input in a way that supports contrastive learning [5]? I think this is an interesting idea and someone should try it! I do think that, even restricting our attention to just getting neural networks to execute programs, that we will need to do something a little more drastic to robustly get the results we want.

mjburgess 5y ago
> it's a little unsatisfying from the perspective of studying intelligence This power-law behaviour is exactly what you get when modelling a non-computable function. Eg., consider approximating the mean of sin^2(x) over -pi to +pi. If you sample x 10^s times (s = 1, 2, 3, ...) the difference from the non-computable analytic answer (0.5) scales log-linear (ie., power-law). { pwr : np.log(0.5 - ( np.sin(np.linspace(-np.pi, +np.pi, 10**pwr) ) ** 2).mean()) for pwr in range(1, 8) } It is my view that intelligence isn't computable, and that any approximation method to some layer of intelligence will run-up against this problem.
- nextaccountic 5y ago
  I think you're mixing up terminology here, because sin^2(x) is computable. I mean, if there is a process to progressively give better estimates of the output of a function, then this function is computable.
  - mjburgess 5y ago
    sine is a real-valued function There is a computable version for computable inputs, ie., we ask "what is sin(x)?" for some computable-number x_computable. But there is no computable version for a non-computable real number, x_real.
    - nextaccountic 5y ago
      Thank you for the clarification. > But there is no computable version for a non-computable real number, x_real. That's true[0]. But consider that you will never receive something non-computable as input to a program, ever. (if you're allowed to approximate the input until you have enough digits to compute what you need, then the input is computable) Really, I think the best way to view sin(x) is as a function that receives a stream of digits 0.2345345323.. and returns a stream of digits 0.0040933884... - this computable version of sine completely captures every thing we could possibly do with it in a program. Operating with floating point numbers, then, is just mostly truncating the input and output stream. [0] At least in classical logic; in intuitionistic logic, sin : Real -> Real is computable and is equivalent to this idea of receiving and returning streams of digits.