6 ms·

Weights are kind of like a compiled binary, because they are an incomprehensible blob of bits. But they are also unlike a compiled binary, because they can be f

Weights are kind of like a compiled binary, because they are an incomprehensible blob of bits. But they are also unlike a compiled binary, because they can be fine-tuned.

dartos 2y ago
You can decompile binaries. You can also edit binaries by hand.
- behrlich 2y ago
  Comparing fine tuning to editing binaries by hand is not a fair comparison. If I could show the decompiler some output I liked and it edited the binary for me to make the output match, then the comparison would be closer.
  - dartos 2y ago
    With regard to the argument about open source, it’s pretty much the same. Especially with dynamically linked binaries like many games.
  - carom 2y ago
    It is a fair comparison. Normal programming takes inputs and a function and produces outputs. Deep learning takes inputs and outputs and derives a functions. Of course the decompilers for traditional programs do not work on inputs and outputs, it is a different paradigm!
  - _ea1k 2y ago
    I feel like a lot of people in this thread have never done continued training on an LLM and it shows. Seriously, a set of weights that already works really well is basically the ideal basis for a _lot_ of ML tasks.
    - zelphirkalt 2y ago
      The question is not, whether it is ideal to do some ML tasks with it, the question is, whether you can do the things you could typically do with open sourced software, including looking at the source and build it, or modify the source and build it. If you don't have the original training data, or mechanism of getting the training data, the compiled result is not reproducible, like normal code would be, and you cannot make a version saying for example: "I want just the same, but without it ever learning from CCP prop."
  - TeMPOraL 2y ago
    > If I could show the decompiler some output I liked and it edited the binary for me to make the output match, then the comparison would be closer. That's fundamentally the same thing though - you run an optimization algorithm on a binary blob. I don't see why this couldn't work. Sure, a neural net is designed to be differentiable, while ELF and PE executables aren't, but then backprop isn't the be-all, end-all of optimization algorithms. Off the top of my head, you could reframe the task as a special kind of genetic programming problem, one that starts with a large program instead of starting from scratch, and that works on an assembly instead of an abstract syntax tree. Hell, you could first decompile the executable and then have the genetic programming solver run on decompiled code. I'd be really surprised if no one tried that before. Or, if such functionality isn't already available in some RE tools (or as a plugin for one). My own hands-on experience with reverse engineering is limited to a few attempts at adding extra UI and functionality to StarCraft by writing some assembly, turning it into object code, and injecting it straight into the running game process[0] - but that was me doing exactly what you described, just by hand. I imagine doing such things is common practice in RE that someone already automated finding the specific parts of the binary that produce the outputs you want to modify. -- [0] - I sometimes miss the times before Data Execution Prevention became a thing.
    - sitkack 2y ago
      Grab an AI model and keep going.
kragen 2y ago
I've fine-tuned compiled binaries on occasion. It used to be a common pastime among teenagers; that's where the demoscene came from.
sksrbWgbfK 2y ago
Ghidra (https://ghidra-sre.org/ https://ghidra-sre.org/) can fine-tune executables way more easily than your models.
- mistercheph 2y ago
  Actually it can't, you can fine tune models with training data, parameters, time and compute, ghidra won't "fine-tune" anything for you.
  - TeMPOraL 2y ago
    How hard can it be to wrap it in a loop and apply some off-the-shelf good old fashioned AI^H^H optimization technique? "Given specific inputs X and outputs Y, have a computer automatically find modifications to F so that F(X) gives Y" is a problem that's been studied for nearly a century now (longer, if relax the meaning of "computer"), with plenty of well-known solutions, most of which don't require F to be differentiable. Isn't "operational research" a standard part of undergrad CS curriculum? It was at my alma mater.
    - mistercheph 2y ago
      There's billions of dollars at the end of the rainbow you're gesturing towards
edflsafoiewq 2y ago
GPL defines "source code" as "the preferred form of the work for making modifications to it", which certainly describes the weights.
- deleted 2y ago
  [deleted]
- fsflover 2y ago
  https://news.ycombinator.com/item?id=42869403 https://news.ycombinator.com/item?id=42869403
- dietr1ch 2y ago
  Training being a one-way function that drops knowledge should tell you that the weights are not the form you want to start with. This is like saying, hey, a regular binary executable is fine because I can edit it with hexl-mode.
  - furyofantares 2y ago
    If hexl-mode on the binary works on my home PC but compiling the source code costs me millions of dollars in compute then I want the binary. Someone with millions of dollars to spend on compute may have a differing opinion.
    - TeMPOraL 2y ago
      This argument only barely holds water for those big SOTA models like llama derivatives, and that's only because of practical costs involved. Or should I say, it held water until few days ago. Personally though, I never bought it. Saying that weights are the "preferred form of the work for making modifications to it" because a) approximately no one can afford to start with the training data, and b) fine-tuning and training LoRAs are cheap enough, is basically like saying binary blobs are "open source" as long as they provide an API (or ABI) for other programs to use. By this line of reasoning, NVIDIA GPU stack and Broadcom chipset firmware would qualify as open source, too.

Weights are kind of like a compiled binary, because they are an incomprehensible blob of bits. But they are also unlike a compiled binary, because they can be fine-tuned.

dartos 2y ago

You can decompile binaries. You can also edit binaries by hand.

behrlich 2y ago
Comparing fine tuning to editing binaries by hand is not a fair comparison. If I could show the decompiler some output I liked and it edited the binary for me to make the output match, then the comparison would be closer.
- dartos 2y ago
  With regard to the argument about open source, it’s pretty much the same. Especially with dynamically linked binaries like many games.
- carom 2y ago
  It is a fair comparison. Normal programming takes inputs and a function and produces outputs. Deep learning takes inputs and outputs and derives a functions. Of course the decompilers for traditional programs do not work on inputs and outputs, it is a different paradigm!
- _ea1k 2y ago
  I feel like a lot of people in this thread have never done continued training on an LLM and it shows. Seriously, a set of weights that already works really well is basically the ideal basis for a _lot_ of ML tasks.
  - zelphirkalt 2y ago
    The question is not, whether it is ideal to do some ML tasks with it, the question is, whether you can do the things you could typically do with open sourced software, including looking at the source and build it, or modify the source and build it. If you don't have the original training data, or mechanism of getting the training data, the compiled result is not reproducible, like normal code would be, and you cannot make a version saying for example: "I want just the same, but without it ever learning from CCP prop."
- TeMPOraL 2y ago
  > If I could show the decompiler some output I liked and it edited the binary for me to make the output match, then the comparison would be closer. That's fundamentally the same thing though - you run an optimization algorithm on a binary blob. I don't see why this couldn't work. Sure, a neural net is designed to be differentiable, while ELF and PE executables aren't, but then backprop isn't the be-all, end-all of optimization algorithms. Off the top of my head, you could reframe the task as a special kind of genetic programming problem, one that starts with a large program instead of starting from scratch, and that works on an assembly instead of an abstract syntax tree. Hell, you could first decompile the executable and then have the genetic programming solver run on decompiled code. I'd be really surprised if no one tried that before. Or, if such functionality isn't already available in some RE tools (or as a plugin for one). My own hands-on experience with reverse engineering is limited to a few attempts at adding extra UI and functionality to StarCraft by writing some assembly, turning it into object code, and injecting it straight into the running game process[0] - but that was me doing exactly what you described, just by hand. I imagine doing such things is common practice in RE that someone already automated finding the specific parts of the binary that produce the outputs you want to modify. -- [0] - I sometimes miss the times before Data Execution Prevention became a thing.
  - sitkack 2y ago
    Grab an AI model and keep going.

kragen 2y ago

I've fine-tuned compiled binaries on occasion. It used to be a common pastime among teenagers; that's where the demoscene came from.

sksrbWgbfK 2y ago

Ghidra (https://ghidra-sre.org/ https://ghidra-sre.org/) can fine-tune executables way more easily than your models.

mistercheph 2y ago
Actually it can't, you can fine tune models with training data, parameters, time and compute, ghidra won't "fine-tune" anything for you.
- TeMPOraL 2y ago
  How hard can it be to wrap it in a loop and apply some off-the-shelf good old fashioned AI^H^H optimization technique? "Given specific inputs X and outputs Y, have a computer automatically find modifications to F so that F(X) gives Y" is a problem that's been studied for nearly a century now (longer, if relax the meaning of "computer"), with plenty of well-known solutions, most of which don't require F to be differentiable. Isn't "operational research" a standard part of undergrad CS curriculum? It was at my alma mater.
  - mistercheph 2y ago
    There's billions of dollars at the end of the rainbow you're gesturing towards

edflsafoiewq 2y ago

GPL defines "source code" as "the preferred form of the work for making modifications to it", which certainly describes the weights.

deleted 2y ago
[deleted]
fsflover 2y ago
https://news.ycombinator.com/item?id=42869403 https://news.ycombinator.com/item?id=42869403
dietr1ch 2y ago
Training being a one-way function that drops knowledge should tell you that the weights are not the form you want to start with. This is like saying, hey, a regular binary executable is fine because I can edit it with hexl-mode.
- furyofantares 2y ago
  If hexl-mode on the binary works on my home PC but compiling the source code costs me millions of dollars in compute then I want the binary. Someone with millions of dollars to spend on compute may have a differing opinion.
  - TeMPOraL 2y ago
    This argument only barely holds water for those big SOTA models like llama derivatives, and that's only because of practical costs involved. Or should I say, it held water until few days ago. Personally though, I never bought it. Saying that weights are the "preferred form of the work for making modifications to it" because a) approximately no one can afford to start with the training data, and b) fine-tuning and training LoRAs are cheap enough, is basically like saying binary blobs are "open source" as long as they provide an API (or ABI) for other programs to use. By this line of reasoning, NVIDIA GPU stack and Broadcom chipset firmware would qualify as open source, too.