8 ms·

Unexpected Risks Found in Editing Binary Code to Fix Bugs in Legacy Codebase. Genetic engineering has exactly the same problems that modifying binary-only code

by gioele 10y ago

Unexpected Risks Found in Editing Binary Code to Fix Bugs in Legacy Codebase. Genetic engineering has exactly the same problems that modifying binary-only code. After some very complex observation of the system you may spot the point that is responsible for a certain problem, but you will never be sure (as in provably sure) that the "fix" you put in place: 1. is going to stop all the occurrences of the behavior you want to fix; 2. will only affect the behavior you want to fix; 3. will have no effect on other systems that you did not modify. (The article is mostly about 1 and a bit about 2 and 3.) Modifying legacy code is hard enough when the uncommented source code is around. Changing directly the binary is both a great engineering feat and something to be scared of.

pasbesoin 10y ago
http://www.catb.org/jargon/html/story-of-mel.html http://www.catb.org/jargon/html/story-of-mel.html It's not always fully apparent what's going on.
quotemstr 10y ago
People modify old binary-only console game ROMs all the time, usually with useful and reliable results.
- Natsu 10y ago
  Problem is that we might only get one try to get the edits right in a person. And we don't have much ability to 'debug' our modifications, either.
Moshe_Silnorin 10y ago
If we replace degenerate alleles with the most-common version of that allele, isn't that pretty safe? It would have been tested on billions of people naturally.
- static_noise 10y ago
  Not if the people with the degenerate version also have another mutation that interacts. If you only fix one, you run into a third kind of bug.
  - manmal 10y ago
    Exactly. For example, "fixing" MTHFR genes could lead to an increased risk of developing lymphoma: https://www.ncbi.nlm.nih.gov/m/pubmed/15551285/ https://www.ncbi.nlm.nih.gov/m/pubmed/15551285/
  - Moshe_Silnorin 10y ago
    >Not if the people with the degenerate version also have another mutation that interacts. Is this common? An uncommon detrimental variant that is slightly-mitigated by another uncommon detrimental variant, such that the removal of one makes the other worse isn't the type of thing I would expect to come about through natural selection.
  - Manglano 10y ago
    The degeneracy of a given allele is relative to the phenotype of the organism and also the organism's environment. Sickle-cell confers survival advantage in environments where malaria is endemic, but is considered a degenerate allele in environments without malaria.
- rpcope1 10y ago
  Because we know genetic loci do not affect the phenotype in purely orthogonal ways (this is known as epistasis), there is a huge risk of having unforeseen second order consequences. In state space even on simple models like NK model for low K, epistasis can have profound effects on fitness metrics and survivability as organisms attempt to hill climb.
drvdevd 10y ago
Arguably this is where the future of binary code is anyway: automatic static and dynamic analysis and simulation to automate patching. Machines will be much better at this than we can ever hope to be. One could imagine the only way to guarentee some level of saftey in genetic modification is via the same or similar means.
- sbierwagen 10y ago
  automatic static and dynamic analysis and simulation [...] in genetic modification That's going to be tough to do in humans. Mutations in the complement component 4 gene dramatically increase the risk of developing schizophrenia. The only way to test for that is to simulate a human growing from a fertilized egg, through childhood, into adolescence, to early adulthood, which is typically when schizophrenia manifests. You'll need to run a few thousand of these simulations, to have a sufficiently powered clinical trial. And, of course, these simulations would have to equivalent to real humans, otherwise it'd be pointless.
  - drvdevd 10y ago
    Right. I certainly wouldn't argue that we do/don't have the computational power to do this or that it's even going to be possible as I'm nowhere near qualified. This is simply my layman's view of "bioinformatics"... I'd like to imagine we'd get something like this eventually.
    - ordu 10y ago
      There is one more problem. Schizophrenia developing is a part of development of human. Not just biological object, but real human, who use his neural system widely in a lot of developing tasks. So to be sure, you'll need to simulate human development. You'll need to simulate for a such experimental object complete environment with loving mother, child games, school, some social relatives (human is sufficiently social creature and he is unable to develop without social environment). From technical point of view its just a computational complexity issue, but how about ethical point of view? How about human rights of simulated consious person?
micro_cam 10y ago
Now imagine the code arose originally by random chance, happened to be self replicating and evolved over billions of generations to include mechanisms that both allow it to viable after small changes and allow those small changes to have large effects (due to selective pressure since diversity from mutations is advantageous to a point, especially in single celled organisms). As an example, Some bacterial genomes are circular, code in both directions and have overlapping coding regions, and occasionally trade genetic material between species because why wouldn't they? I've worked applying machine learning and statistics to genetics and I generally think it's pretty hilarious that many self declared rationalists think we need to worry about AI safety but declare scepticism of genetic engineering anti-scientific.
- hinkley 10y ago
  Now imagine the code arose originally by random chance I could introduce you to some former coworkers, but I hoped never to encounter them again.
- naasking 10y ago
  > many self declared rationalists think we need to worry about AI safety but declare scepticism of genetic engineering anti-scientific. It is because we've already done genetic engineering. Certainly there are tractability problems with the scope of a change and predicting it's full success, but the fact that changes can be made seems undeniable, even if all we do is stick to direct engineering and exclude the engineering we've been doing for thousands of years. Protein folding is a computationally hard problem, so fully predicting a change's effect will remain imprecise, unless we find some new shortcuts, which is entirely possible.
  - micro_cam 10y ago
    My point is there are all sorts of hidden complexities and 2nd order interaction between things we don't yet understand. I'd argue that this goes beyond computational tractability and is still an issue of statistical power. There are millions of common variants in the human genome and we've only sequenced thousands of people so it's not exactly controversial to say genetics is poorly understood. Which as the AI-safety people say, isn't to say we should be researching it just that we should be devoting more energy to doing it safely as the potential consequences are so high even if the chances of things going wrong is low.
dom0 10y ago
Meh. Compilers generate pretty simple code, people have been patching complex binaries and ROMs for ages, with generally good and stable results. I think the genetic code generated by evolution is likely far more complex, convoluted and inter-dependent than any assembly spat out by a compiler, so I think the analogy goes too easy on genetic editing.
- fooker 10y ago
  Look at the code in games using Denuvo.
greenhatman 10y ago
>3. will have no effect on other systems that you did not modify. I think you can be sure of the opposite, without even testing.
Manglano 10y ago
Also, there's an array of environmentally-determined switches and gene-determined proteins which are being changed concurrently by a team with whom you cannot communicate. Also the previous team which worked on all patches had several billion years and the specification was exacting. Like trying to make a pretty patch on modular synth with access to only one module. This may be a bad idea.
emehrkay 10y ago
Someone should start writing some unit tests and we could possibly have that healing machine from the movie Elysium https://www.youtube.com/watch?v=ETreC62bitE https://www.youtube.com/watch?v=ETreC62bitE
- dTal 10y ago
  Diagnosis is the easy part...
LargoLasskhyfv 10y ago
Maybe it's a matter of tooling. I don't want to appear disrespectful, but after having seen Christopher Domas present "The Future of RE: Dynamic Binary Visualization" my mind was forever blown. Working with hexdumps, debuggers, disassemblers etc. seems so stone-agey compared to that. Like so much in IT in general. Anyways, caveman CAN more than crawl, obviously ;-) So, if you have up to an hour of time to waste, maybe you can have your mind blown to cantor dust too? https://sites.google.com/site/xxcantorxdustxx/home https://sites.google.com/site/xxcantorxdustxx/home https://www.youtube.com/watch?v=C8--cXwuuFQ https://www.youtube.com/watch?v=C8--cXwuuFQ Something very similar seems to be developed here: https://github.com/letoram/senseye/wiki https://github.com/letoram/senseye/wiki Happy New Cyber! edit: spelling

by gioele 10y ago

pasbesoin 10y ago

http://www.catb.org/jargon/html/story-of-mel.html http://www.catb.org/jargon/html/story-of-mel.html It's not always fully apparent what's going on.

quotemstr 10y ago

People modify old binary-only console game ROMs all the time, usually with useful and reliable results.

Natsu 10y ago
Problem is that we might only get one try to get the edits right in a person. And we don't have much ability to 'debug' our modifications, either.

Moshe_Silnorin 10y ago

If we replace degenerate alleles with the most-common version of that allele, isn't that pretty safe? It would have been tested on billions of people naturally.

static_noise 10y ago
Not if the people with the degenerate version also have another mutation that interacts. If you only fix one, you run into a third kind of bug.
- manmal 10y ago
  Exactly. For example, "fixing" MTHFR genes could lead to an increased risk of developing lymphoma: https://www.ncbi.nlm.nih.gov/m/pubmed/15551285/ https://www.ncbi.nlm.nih.gov/m/pubmed/15551285/
- Moshe_Silnorin 10y ago
  >Not if the people with the degenerate version also have another mutation that interacts. Is this common? An uncommon detrimental variant that is slightly-mitigated by another uncommon detrimental variant, such that the removal of one makes the other worse isn't the type of thing I would expect to come about through natural selection.
- Manglano 10y ago
  The degeneracy of a given allele is relative to the phenotype of the organism and also the organism's environment. Sickle-cell confers survival advantage in environments where malaria is endemic, but is considered a degenerate allele in environments without malaria.
rpcope1 10y ago
Because we know genetic loci do not affect the phenotype in purely orthogonal ways (this is known as epistasis), there is a huge risk of having unforeseen second order consequences. In state space even on simple models like NK model for low K, epistasis can have profound effects on fitness metrics and survivability as organisms attempt to hill climb.

drvdevd 10y ago

Arguably this is where the future of binary code is anyway: automatic static and dynamic analysis and simulation to automate patching. Machines will be much better at this than we can ever hope to be. One could imagine the only way to guarentee some level of saftey in genetic modification is via the same or similar means.

sbierwagen 10y ago
automatic static and dynamic analysis and simulation [...] in genetic modification That's going to be tough to do in humans. Mutations in the complement component 4 gene dramatically increase the risk of developing schizophrenia. The only way to test for that is to simulate a human growing from a fertilized egg, through childhood, into adolescence, to early adulthood, which is typically when schizophrenia manifests. You'll need to run a few thousand of these simulations, to have a sufficiently powered clinical trial. And, of course, these simulations would have to equivalent to real humans, otherwise it'd be pointless.
- drvdevd 10y ago
  Right. I certainly wouldn't argue that we do/don't have the computational power to do this or that it's even going to be possible as I'm nowhere near qualified. This is simply my layman's view of "bioinformatics"... I'd like to imagine we'd get something like this eventually.
  - ordu 10y ago
    There is one more problem. Schizophrenia developing is a part of development of human. Not just biological object, but real human, who use his neural system widely in a lot of developing tasks. So to be sure, you'll need to simulate human development. You'll need to simulate for a such experimental object complete environment with loving mother, child games, school, some social relatives (human is sufficiently social creature and he is unable to develop without social environment). From technical point of view its just a computational complexity issue, but how about ethical point of view? How about human rights of simulated consious person?

micro_cam 10y ago

Now imagine the code arose originally by random chance, happened to be self replicating and evolved over billions of generations to include mechanisms that both allow it to viable after small changes and allow those small changes to have large effects (due to selective pressure since diversity from mutations is advantageous to a point, especially in single celled organisms). As an example, Some bacterial genomes are circular, code in both directions and have overlapping coding regions, and occasionally trade genetic material between species because why wouldn't they? I've worked applying machine learning and statistics to genetics and I generally think it's pretty hilarious that many self declared rationalists think we need to worry about AI safety but declare scepticism of genetic engineering anti-scientific.

hinkley 10y ago
Now imagine the code arose originally by random chance I could introduce you to some former coworkers, but I hoped never to encounter them again.
naasking 10y ago
> many self declared rationalists think we need to worry about AI safety but declare scepticism of genetic engineering anti-scientific. It is because we've already done genetic engineering. Certainly there are tractability problems with the scope of a change and predicting it's full success, but the fact that changes can be made seems undeniable, even if all we do is stick to direct engineering and exclude the engineering we've been doing for thousands of years. Protein folding is a computationally hard problem, so fully predicting a change's effect will remain imprecise, unless we find some new shortcuts, which is entirely possible.
- micro_cam 10y ago
  My point is there are all sorts of hidden complexities and 2nd order interaction between things we don't yet understand. I'd argue that this goes beyond computational tractability and is still an issue of statistical power. There are millions of common variants in the human genome and we've only sequenced thousands of people so it's not exactly controversial to say genetics is poorly understood. Which as the AI-safety people say, isn't to say we should be researching it just that we should be devoting more energy to doing it safely as the potential consequences are so high even if the chances of things going wrong is low.

dom0 10y ago

Meh. Compilers generate pretty simple code, people have been patching complex binaries and ROMs for ages, with generally good and stable results. I think the genetic code generated by evolution is likely far more complex, convoluted and inter-dependent than any assembly spat out by a compiler, so I think the analogy goes too easy on genetic editing.

fooker 10y ago
Look at the code in games using Denuvo.

greenhatman 10y ago

>3. will have no effect on other systems that you did not modify. I think you can be sure of the opposite, without even testing.

Manglano 10y ago

Also, there's an array of environmentally-determined switches and gene-determined proteins which are being changed concurrently by a team with whom you cannot communicate. Also the previous team which worked on all patches had several billion years and the specification was exacting. Like trying to make a pretty patch on modular synth with access to only one module. This may be a bad idea.

emehrkay 10y ago

Someone should start writing some unit tests and we could possibly have that healing machine from the movie Elysium https://www.youtube.com/watch?v=ETreC62bitE https://www.youtube.com/watch?v=ETreC62bitE

dTal 10y ago
Diagnosis is the easy part...

LargoLasskhyfv 10y ago

Maybe it's a matter of tooling. I don't want to appear disrespectful, but after having seen Christopher Domas present "The Future of RE: Dynamic Binary Visualization" my mind was forever blown. Working with hexdumps, debuggers, disassemblers etc. seems so stone-agey compared to that. Like so much in IT in general. Anyways, caveman CAN more than crawl, obviously ;-) So, if you have up to an hour of time to waste, maybe you can have your mind blown to cantor dust too? https://sites.google.com/site/xxcantorxdustxx/home https://sites.google.com/site/xxcantorxdustxx/home https://www.youtube.com/watch?v=C8--cXwuuFQ https://www.youtube.com/watch?v=C8--cXwuuFQ Something very similar seems to be developed here: https://github.com/letoram/senseye/wiki https://github.com/letoram/senseye/wiki Happy New Cyber! edit: spelling