9 ms·

Meta LLM Compiler: neural optimizer and disassembler

248 points by foobazgt 2y ago

HanClinto 2y ago
Huh. This is a very... "interesting" application for an LLM. I'm not the brightest crayon in the box, but if anyone else would like to follow along with my non-expert opinion as I read through the paper, here's my take on it. It's pretty important for compilers / decompilers to be reliable and accurate -- compilers behaving in a deterministic and predictable way is an important fundamental of pipelines. LLMs are inherently unpredictable, and so using an LLM for compilation / decompilation -- even an LLM that has 99.99% accuracy -- feels a bit odd to include as a piece in my build pipeline. That said, let's look at the paper and see what they did. They essentially started with CodeLlama, and then went further to train the model on three tasks -- one primary, and two downstream. The first task is compilation: given input code and a set of compiler flags, can we predict the output assembly? Given the inability to verify correctness without using a traditional compiler, this feels like it's of limited use on its own. However, training a model on this as a primary task enables a couple of downstream tasks. Namely: The second task (and first downstream task) is compiler flag prediction / optimization to predict / optimize for smaller assembly sizes. It's a bit disappointing that they only seem to be able to optimize for assembly size (and not execution speed), but it's not without its uses. Because the output of this task (compiler flags) are then passed to a deterministic function (a traditional compiler), then the instability of the LLM is mitigated. The third task (second downstream task) is decompilation. This is not the first time that LLMs have been trained to do better decompilation -- however, because of the pretraining that they did on the primary task, they feel that this provides some advantages over previous approaches. Sadly, they only compare LLM Compiler to Code Llama and GPT-4 Turbo, and not against any other LLMs fine-tuned for the decompilation task, so it's difficult to see in context how much better their approach is. Regarding the verifiability of the disassembly approach, the authors note that there are issues regarding correctness. So the authors employ round-tripping -- recompiling the decompiled code (using the same compiler flags) to verify correctness / exact-match. This still puts accuracy in the 45% or so (if I understand their output numbers), so it's not entirely trustworthy yet, but it might be able to still be useful (especially if used alongside a traditional decompiler, and this model's outputs only used when they are verifiably correct). Overall I'm happy to see this model be released as it seems like an interesting use-case. I may need to read more, but at first blush I'm not immediately excited by the possibilities that this unlocks. Most of all, I would like to see it explored if these methods could be extended to optimize for performance -- not just size of assembly.
- riedel 2y ago
  It is normally not a necessary feature of a compiler to be determistic. A compiler should be correct against a specification. If the specification allows indeterminism a compiler should be able to exploit them. I remember the story of the sather-k compiler that did things differently based on the phase of the moon.
  - munificent 2y ago
    It's technically correct that a language specification is rarely precise enough to require compiler output to be deterministic. But it's pragmatically true that engineers will want to murder you if your compiler is non-deterministic. All sorts of build systems, benchmark harnesses, supply chain validation tools, and other bits of surrounding ecosystem will shit the bed if the compiler doesn't produce bitwise identical output on the same input and compiler flags.
    - foobazgt 2y ago
      Can vouch for this having fixed non-determinism bugs in a compiler. Nobody is happy if your builds aren't reproducible. You'll also suffer crazy performance problems as everything downstream rebuilds randomly and all your build caches randomly miss.
      - SushiHippie 2y ago
        NixOS with its nixpkgs [0] and cache [1] would also not work if compilers weren't reproducible. Though they won't use something like PGO or some specific optimization flags as these would very likely lead to unreproducible builds. For example most distros ship a PGO optimized build of Python while nixos does not. [0] https://github.com/nixos/nixpkgs https://github.com/nixos/nixpkgs [1] https://cache.nixos.org/ https://cache.nixos.org/
        boomanaiden154 2y ago
        PGO can be used in such situations, but the profile needs to be checked in. Same code + same profile -> same binary (assuming the compiler is deterministic, which is tested quite extensively). There are several big projects that use PGO (like Chrome), and you can get a deterministic build at whatever revision using PGO as the profiles are checked in to the repository.
mmphosis 2y ago
https://xcancel.com/AIatMeta/status/1806361623831171318 https://xcancel.com/AIatMeta/status/1806361623831171318
chad1n 2y ago
As usual, Twitter is impressed by this, but I'm very skeptical, the chance of it breaking your program is pretty high. The thing that makes optimizations so hard to make is that they have to match the behavior without optimizations (unless you have UBs), which is something that LLMs probably will struggle with since they can't exactly understand the code and execution tree.
- ramon156 2y ago
  Let's be real, at least 40% of those comments are bots
  - extheat 2y ago
    People simply have no idea what they're talking about. It's just jumping on to the latest hype train. My first impression here was per the name that it was actually some sort of compiler in it of itself--ie programming language in and pure machine code or some other IR out. It's got bits and pieces of that here and there but that's not what it really is at all. It's more of a predictive engine for an optimizer and not a very generalized one for that. What would be more interesting is training a large model on pure (code, assembly) pairs like a normal translation task. Presumably a very generalized model would be good at even doing the inverse: given some assembly, write code that will produce the given assembly. Unlike human language there is a finite set of possible correct answers here and you have the convenience of being able to generate synthetic data for cheap. I think optimizations would arise as a natural side effect this way: if there's multiple trees of possible generations (like choosing between logits in an LLM) you could try different branches to see what's smaller in terms of byte code or faster in terms of execution.
    - quonn 2y ago
      > Presumably a very generalized model would be good at even doing the inverse: given some assembly, write code that will produce the given assembly. ChatGPT does this, unreliably.
    - hughleat 2y ago
ldjkfkdsjnv 2y ago
I love this company. Advancing ai and keeping the rest of us in the loop.
- LoganDark 2y ago
  I hate the company (Facebook), but I still think them having been publicly releasing a bunch of the research they've been doing (and models they've been making) has been a net good for almost everybody, at least in terms of exploring the field of LLMs.
- ein0p 2y ago
  My love for Meta is strictly confined to FAIR and the PyTorch team. The rest of the company is basically cancer.
- Slyfox33 2y ago
  Is this a bot comment?
muglug 2y ago
Unlike many other AI-themed papers at Meta this one omits any mention of the model output getting used at Instagram, Facebook or Meta. Research is great! But doesn't seem all that actionable today.
- boomanaiden154 2y ago
  This would be difficult to deploy as-is in production. There are correctness issues mentioned in the paper regarding adjusting phase orderings away from the well-trodden O0/O1/O2/O3/Os/Oz path. Their methodology works for a research project quite well, but I personally wouldn't trust it in production. While some obvious issues can be caught by a small test suite and unit tests, there are others that won't be, and that's really risky in production scenarios. There are also some practical software engineering things like deployment in the compiler. There is actually tooling in upstream LLVM to do this (https://www.youtube.com/watch?v=mQu1CLZ3uWs https://www.youtube.com/watch?v=mQu1CLZ3uWs), but running models on a GPU would be difficult and I would expect CPU inference to massively blow up compile times.
soist 2y ago
How do they verify the output preserves semantics of the input?
- hughleat 2y ago
  For the disassembler we round trip. x86 ->(via model) IR ->(via clang) x86. If they are identical then the IR is correct. Could be correct even if not identical, but then you need to check. For the auto-tuning, we suggest the best passes to use in LLVM. We take some effort to weed out bad passes, but LLVM has bugs. This is in common with any auto-tuner. We train it to emulate the compiler. The compiler does that better already. We do it because it helps the LLM understand the compiler better and it auto-tunes better as a result. We hope people will use this model to fine-tune for other heuristics. E.g. an inliner which accepts the IR or the caller and callee to decide profitability. We think things like that will be vastly cheaper for people if they can start from LLM Compiler. Training LLMs from scratch is expensive :-) IMO, right now, AI should be used to decide profitability not correctness.
  - tezheng 2y ago
    > Could be correct even if not identical Can you be 100% sure that the model-gen-IR is correct if output x86 is identical against the input?
  - soloist11 2y ago
    Have you guys applied this work internally to optimize Meta's codebase?
zitterbewegung 2y ago
Some previous work in the space is at https://github.com/albertan017/LLM4Decompile https://github.com/albertan017/LLM4Decompile
LoganDark 2y ago
Reading the title, I thought this was a tool for optimizing and disassembling LLMs, not an LLM designed to optimize and disassemble. Seeing it's just a model is a little disappointing in comparison.
jameshart 2y ago
Pretty sure I remember trading 300 creds for a Meta Technologies Neural Optimizer and Disassembler in one of the early Deus Ex games.
nothrowaways 2y ago
It is so funny that meta has to post it on X.
- rising-sky 2y ago
  https://www.threads.net/@aiatmeta/post/C8ubaKupPwC https://www.threads.net/@aiatmeta/post/C8ubaKupPwC
- deleted 2y ago
  [deleted]
0x1ceb00da 2y ago
Wouldn't "Compiler LLM" be a more accurate name than "LLM Compiler"?
- hughleat 2y ago
  Never let a computer scientist name anything :-)
Havoc 2y ago
I don’t understand the purpose of this. Feels like a task for function calling and sending it to an actual compiler. Is there an obvious use case I’m missing?
- singularity2001 2y ago
  GPT 6 can write software directly (as assembly) instead of writing c first. Lots of training data for binary, and it can train itself by seeing if the program does what it expects it to do.
  - dunefox 2y ago
    Is this GPT-6 in the room with us now?
- killerstorm 2y ago
  This is not a product, it's a research project. They don't expect you to use this. Applications might require further research. And the main takeaway might be not "here's a tool to generate code", but "LLMs are able to understand binary code, and thus we can train them to do ...".
zellyn 2y ago
I continue to be fascinated about what the next qualitative iteration of models will be, marrying the language processing and broad knowledge of LLMs with an ability to reason rigorously. If I understand correctly, this work (or the most obvious productionized version of it) is similar to the work Deep Mind released a while back: the LLM is essentially used for “intuition”—-to pick the approach—-and then you hand off to something mechanical/rigorous. I think we’re going to see a huge growth in that type of system. I still think it’s kind of weird and cool that our meat brains with spreading activation can (with some amount of effort/concentration) switch over into math mode and manipulate symbols and inferences rigorously.
hgl71964 2y ago
I am curious about CUDA assembly, does it work on CUDA -> ptx level? or ptx -> sass? I have done some work on SASS optimization and it would be a lot easier if LLM could be applied at SASS level
rldjbpin 2y ago
my knowledge of compilers don't extend beyond a 101 course done ages ago, but i wonder how the researchers enriched the dataset for improving these features. did they just happen to find a way to format the heuristics of major compilers in half-code, half-language mix? confusingly enough, another use case where a (potential) tool that let us veer into the solution with some work is being replaced by an llm.

248 points by foobazgt 2y ago

Huh. This is a very... "interesting" application for an LLM. I'm not the brightest crayon in the box, but if anyone else would like to follow along with my non-expert opinion as I read through the paper, here's my take on it. It's pretty important for compilers / decompilers to be reliable and accurate -- compilers behaving in a deterministic and predictable way is an important fundamental of pipelines. LLMs are inherently unpredictable, and so using an LLM for compilation / decompilation -- even an LLM that has 99.99% accuracy -- feels a bit odd to include as a piece in my build pipeline. That said, let's look at the paper and see what they did. They essentially started with CodeLlama, and then went further to train the model on three tasks -- one primary, and two downstream. The first task is compilation: given input code and a set of compiler flags, can we predict the output assembly? Given the inability to verify correctness without using a traditional compiler, this feels like it's of limited use on its own. However, training a model on this as a primary task enables a couple of downstream tasks. Namely: The second task (and first downstream task) is compiler flag prediction / optimization to predict / optimize for smaller assembly sizes. It's a bit disappointing that they only seem to be able to optimize for assembly size (and not execution speed), but it's not without its uses. Because the output of this task (compiler flags) are then passed to a deterministic function (a traditional compiler), then the instability of the LLM is mitigated. The third task (second downstream task) is decompilation. This is not the first time that LLMs have been trained to do better decompilation -- however, because of the pretraining that they did on the primary task, they feel that this provides some advantages over previous approaches. Sadly, they only compare LLM Compiler to Code Llama and GPT-4 Turbo, and not against any other LLMs fine-tuned for the decompilation task, so it's difficult to see in context how much better their approach is. Regarding the verifiability of the disassembly approach, the authors note that there are issues regarding correctness. So the authors employ round-tripping -- recompiling the decompiled code (using the same compiler flags) to verify correctness / exact-match. This still puts accuracy in the 45% or so (if I understand their output numbers), so it's not entirely trustworthy yet, but it might be able to still be useful (especially if used alongside a traditional decompiler, and this model's outputs only used when they are verifiably correct). Overall I'm happy to see this model be released as it seems like an interesting use-case. I may need to read more, but at first blush I'm not immediately excited by the possibilities that this unlocks. Most of all, I would like to see it explored if these methods could be extended to optimize for performance -- not just size of assembly.

riedel 2y ago
It is normally not a necessary feature of a compiler to be determistic. A compiler should be correct against a specification. If the specification allows indeterminism a compiler should be able to exploit them. I remember the story of the sather-k compiler that did things differently based on the phase of the moon.
- munificent 2y ago
  It's technically correct that a language specification is rarely precise enough to require compiler output to be deterministic. But it's pragmatically true that engineers will want to murder you if your compiler is non-deterministic. All sorts of build systems, benchmark harnesses, supply chain validation tools, and other bits of surrounding ecosystem will shit the bed if the compiler doesn't produce bitwise identical output on the same input and compiler flags.
  - foobazgt 2y ago
    Can vouch for this having fixed non-determinism bugs in a compiler. Nobody is happy if your builds aren't reproducible. You'll also suffer crazy performance problems as everything downstream rebuilds randomly and all your build caches randomly miss.
    - SushiHippie 2y ago
      NixOS with its nixpkgs [0] and cache [1] would also not work if compilers weren't reproducible. Though they won't use something like PGO or some specific optimization flags as these would very likely lead to unreproducible builds. For example most distros ship a PGO optimized build of Python while nixos does not. [0] https://github.com/nixos/nixpkgs https://github.com/nixos/nixpkgs [1] https://cache.nixos.org/ https://cache.nixos.org/
      - boomanaiden154 2y ago
        PGO can be used in such situations, but the profile needs to be checked in. Same code + same profile -> same binary (assuming the compiler is deterministic, which is tested quite extensively). There are several big projects that use PGO (like Chrome), and you can get a deterministic build at whatever revision using PGO as the profiles are checked in to the repository.

mmphosis 2y ago

https://xcancel.com/AIatMeta/status/1806361623831171318 https://xcancel.com/AIatMeta/status/1806361623831171318

chad1n 2y ago

As usual, Twitter is impressed by this, but I'm very skeptical, the chance of it breaking your program is pretty high. The thing that makes optimizations so hard to make is that they have to match the behavior without optimizations (unless you have UBs), which is something that LLMs probably will struggle with since they can't exactly understand the code and execution tree.

ramon156 2y ago
Let's be real, at least 40% of those comments are bots
- extheat 2y ago
  People simply have no idea what they're talking about. It's just jumping on to the latest hype train. My first impression here was per the name that it was actually some sort of compiler in it of itself--ie programming language in and pure machine code or some other IR out. It's got bits and pieces of that here and there but that's not what it really is at all. It's more of a predictive engine for an optimizer and not a very generalized one for that. What would be more interesting is training a large model on pure (code, assembly) pairs like a normal translation task. Presumably a very generalized model would be good at even doing the inverse: given some assembly, write code that will produce the given assembly. Unlike human language there is a finite set of possible correct answers here and you have the convenience of being able to generate synthetic data for cheap. I think optimizations would arise as a natural side effect this way: if there's multiple trees of possible generations (like choosing between logits in an LLM) you could try different branches to see what's smaller in terms of byte code or faster in terms of execution.
  - quonn 2y ago
    > Presumably a very generalized model would be good at even doing the inverse: given some assembly, write code that will produce the given assembly. ChatGPT does this, unreliably.
  - hughleat 2y ago

ldjkfkdsjnv 2y ago

I love this company. Advancing ai and keeping the rest of us in the loop.

LoganDark 2y ago
I hate the company (Facebook), but I still think them having been publicly releasing a bunch of the research they've been doing (and models they've been making) has been a net good for almost everybody, at least in terms of exploring the field of LLMs.
ein0p 2y ago
My love for Meta is strictly confined to FAIR and the PyTorch team. The rest of the company is basically cancer.
Slyfox33 2y ago
Is this a bot comment?

muglug 2y ago

Unlike many other AI-themed papers at Meta this one omits any mention of the model output getting used at Instagram, Facebook or Meta. Research is great! But doesn't seem all that actionable today.

boomanaiden154 2y ago
This would be difficult to deploy as-is in production. There are correctness issues mentioned in the paper regarding adjusting phase orderings away from the well-trodden O0/O1/O2/O3/Os/Oz path. Their methodology works for a research project quite well, but I personally wouldn't trust it in production. While some obvious issues can be caught by a small test suite and unit tests, there are others that won't be, and that's really risky in production scenarios. There are also some practical software engineering things like deployment in the compiler. There is actually tooling in upstream LLVM to do this (https://www.youtube.com/watch?v=mQu1CLZ3uWs https://www.youtube.com/watch?v=mQu1CLZ3uWs), but running models on a GPU would be difficult and I would expect CPU inference to massively blow up compile times.

soist 2y ago

How do they verify the output preserves semantics of the input?

hughleat 2y ago
For the disassembler we round trip. x86 ->(via model) IR ->(via clang) x86. If they are identical then the IR is correct. Could be correct even if not identical, but then you need to check. For the auto-tuning, we suggest the best passes to use in LLVM. We take some effort to weed out bad passes, but LLVM has bugs. This is in common with any auto-tuner. We train it to emulate the compiler. The compiler does that better already. We do it because it helps the LLM understand the compiler better and it auto-tunes better as a result. We hope people will use this model to fine-tune for other heuristics. E.g. an inliner which accepts the IR or the caller and callee to decide profitability. We think things like that will be vastly cheaper for people if they can start from LLM Compiler. Training LLMs from scratch is expensive :-) IMO, right now, AI should be used to decide profitability not correctness.
- tezheng 2y ago
  > Could be correct even if not identical Can you be 100% sure that the model-gen-IR is correct if output x86 is identical against the input?
- soloist11 2y ago
  Have you guys applied this work internally to optimize Meta's codebase?

zitterbewegung 2y ago

Some previous work in the space is at https://github.com/albertan017/LLM4Decompile https://github.com/albertan017/LLM4Decompile

LoganDark 2y ago

Reading the title, I thought this was a tool for optimizing and disassembling LLMs, not an LLM designed to optimize and disassemble. Seeing it's just a model is a little disappointing in comparison.

jameshart 2y ago

Pretty sure I remember trading 300 creds for a Meta Technologies Neural Optimizer and Disassembler in one of the early Deus Ex games.

nothrowaways 2y ago

It is so funny that meta has to post it on X.

rising-sky 2y ago
https://www.threads.net/@aiatmeta/post/C8ubaKupPwC https://www.threads.net/@aiatmeta/post/C8ubaKupPwC
deleted 2y ago
[deleted]

0x1ceb00da 2y ago

Wouldn't "Compiler LLM" be a more accurate name than "LLM Compiler"?

hughleat 2y ago
Never let a computer scientist name anything :-)

Havoc 2y ago

I don’t understand the purpose of this. Feels like a task for function calling and sending it to an actual compiler. Is there an obvious use case I’m missing?

singularity2001 2y ago
GPT 6 can write software directly (as assembly) instead of writing c first. Lots of training data for binary, and it can train itself by seeing if the program does what it expects it to do.
- dunefox 2y ago
  Is this GPT-6 in the room with us now?
killerstorm 2y ago
This is not a product, it's a research project. They don't expect you to use this. Applications might require further research. And the main takeaway might be not "here's a tool to generate code", but "LLMs are able to understand binary code, and thus we can train them to do ...".

zellyn 2y ago

I continue to be fascinated about what the next qualitative iteration of models will be, marrying the language processing and broad knowledge of LLMs with an ability to reason rigorously. If I understand correctly, this work (or the most obvious productionized version of it) is similar to the work Deep Mind released a while back: the LLM is essentially used for “intuition”—-to pick the approach—-and then you hand off to something mechanical/rigorous. I think we’re going to see a huge growth in that type of system. I still think it’s kind of weird and cool that our meat brains with spreading activation can (with some amount of effort/concentration) switch over into math mode and manipulate symbols and inferences rigorously.

hgl71964 2y ago

I am curious about CUDA assembly, does it work on CUDA -> ptx level? or ptx -> sass? I have done some work on SASS optimization and it would be a lot easier if LLM could be applied at SASS level

rldjbpin 2y ago

my knowledge of compilers don't extend beyond a 101 course done ages ago, but i wonder how the researchers enriched the dataset for improving these features. did they just happen to find a way to format the heuristics of major compilers in half-code, half-language mix? confusingly enough, another use case where a (potential) tool that let us veer into the solution with some work is being replaced by an llm.