4 ms·

I wonder if this requires a 7B parameter model.Assembly has a small grammar and is very constrained compared to natural language.

• kaba0 3y ago
  I don’t see why would the output language’s complexity matter - that’s clearly not the hard part. You need plenty of “thinking” to do for outputting sensible assembly, let alone whole programs. With that said, it is not doing neural compilation as others have mentioned, it’s only about ordering/enabling different phases of the compiler based on ML, over the current, simpler heuristics.
  • quadrature 3y ago
    Exactly, since its not doing whole program synthesis im thinking it could be done with fewer parameters. However program synthesis is part of the loss function.
    • cec 3y ago
      Program synthesis is part of the loss function, which is what makes it a auxiliary learning task. We haven’t experimented with model size yet, we just used the same configuration as the smallest Code Llama. We did play with dataset size and found thah performance tracks the usual scaling laws. Details in the paper
      • quadrature 3y ago
        Thanks for the reply!. This is really interesting research, hope to see more from your team.