3 ms·

Not really, those weights are the same ones that are there during training getting modified by backprop. You can take them and continue training the model in or

Not really, those weights are the same ones that are there during training getting modified by backprop. You can take them and continue training the model in order to produce different behaviour. You can also modify them directly like with LoRA or steering. You can also make bigger modifications like expanding the layers or removing nodes. It’s not a frozen binary, those numbers aren’t obfuscated or optimized into some special purpose form that prevents or makes modification difficult.

fragmede 2y ago
And I can edit binaries with binedit. Reverse engineering is a lost art, but reversing binaries with Ida pro and patching binaries is a time honored skill. It's a shame it's usually used to patch out license checks but there have been some truely impressive reverse engineering on video game ROMs for which the source is not available so we're left with modifying the binary. or as we're looking at it, the model.
- fragmede 2y ago
  https://github.com/WerWolv/ImHex https://github.com/WerWolv/ImHex (https://news.ycombinator.com/item?id=40828493 https://news.ycombinator.com/item?id=40828493) is a hex editor specially made for people that do exactly that kind of binary editing.
- throwthrowuknow 2y ago
  But you’re not reversing anything with weights. They aren’t altered at all. There’s no secret human readable code, just training/test data and hyperparameters. Even if they did give you access to the exact data set they used there’s not much chance you could do anything with it without millions of dollars worth of compute time and even then they’ve already trained the model on that data so the best you could do is start from scratch and see if you could get a better result in which case your time is better spent on building your own data set and RLHF process. Like what value do you think you’re going to get anyways?

fragmede 2y ago

And I can edit binaries with binedit. Reverse engineering is a lost art, but reversing binaries with Ida pro and patching binaries is a time honored skill. It's a shame it's usually used to patch out license checks but there have been some truely impressive reverse engineering on video game ROMs for which the source is not available so we're left with modifying the binary. or as we're looking at it, the model.

fragmede 2y ago
https://github.com/WerWolv/ImHex https://github.com/WerWolv/ImHex (https://news.ycombinator.com/item?id=40828493 https://news.ycombinator.com/item?id=40828493) is a hex editor specially made for people that do exactly that kind of binary editing.
throwthrowuknow 2y ago
But you’re not reversing anything with weights. They aren’t altered at all. There’s no secret human readable code, just training/test data and hyperparameters. Even if they did give you access to the exact data set they used there’s not much chance you could do anything with it without millions of dollars worth of compute time and even then they’ve already trained the model on that data so the best you could do is start from scratch and see if you could get a better result in which case your time is better spent on building your own data set and RLHF process. Like what value do you think you’re going to get anyways?