7 ms·

It's good as long self-termination is a nice-to-have feature. If you REALLY want them to self-terminate but you found out that they mutated and can now synthesi

by _yosefk 10y ago

It's good as long self-termination is a nice-to-have feature. If you REALLY want them to self-terminate but you found out that they mutated and can now synthesize X & Y, you're in trouble. Genetically modified organisms are a bit scary, but genetically modified organisms that replicate autonomously are more than a bit scary. As programmers - a profession that managed to spread Shellshock, the dumbest imaginable backdoor, so widely as to make much of human population vulnerable, we should understand this better than others. We have no idea what happens in our own code, code that we copy from place to place ourselves, written in languages we designed and running on machines we designed, and there's nothing like mutation and self-replication with this stuff, for the most part. I kinda doubt that biologists modifying "code" which is actually pretty large molecules existing and interacting in a 3D space with a lot of forces pulling and pushing things, and without the benefit of comments or design documents, can make something that can be trusted to autonomously replicate and gradually mutate.

jcoffland 10y ago
> We have no idea what happens in our own code That's an overstatement. Sure code can be very complex but we actually can see exactly what's going on whenever we really want to. It's just that we cannot see what's going on with all the code all the time.
- portent 10y ago
  code can be very simple, and we don't always know what it will do. Simple example: can you tell me if this snippet of (python) code will ever terminate or not? x=0.5 while x<0.6 or x>0.7: x=3.59*x*(1-x) print x ... and what if the 3.59 was replaced by a different number - maybe 3.60 ? or 3.84 ?
  - sbmassey 10y ago
    It will eventually terminate, if only because the machine it is running on will eventually be shut down.
    - portent 10y ago
      OK, well we can always say that the heat-death of the universe renders all problems irrelevent, but it's a bit of a cheat in my opinion. The original poster seems to imply that knowing the code means that you can know the behaviour of the system; I do not think that is the case, and my simple (chaotic) example tries to demonstrate this.
      - joshjje 10y ago
        If you have the compiled assembly obviously you can say exactly what will happen. I agree with your meaning somewhat though that more often than not many people have no clue, especially with large systems. Its not as bad as your statement though..
        portent 10y ago
        Well... the compiled assembly doesn't give you any more information than the code snippet. The issue is not so much how this code is translated from higher abstraction level to lower abstraction level... the issue is, that this code represents a simple chaotic function (the logistic map). As such, for a simple few lines of code the behaviour is very complex and virtually impossible to predict; for certain values of the controlling number it will (1) halt relatively quickly, (2) never halt, or (3) halt after a very long time... but good luck in distinguishing between cases 2 and 3!
        BraveNewCurency 10y ago
        No, the parent is actually talking about the Halting Problem. It has been proven impossible to predict when a program will exit without actually executing the program. (except in a few trivial edge cases) https://en.wikipedia.org/wiki/Halting_problem https://en.wikipedia.org/wiki/Halting_problem
      - sbmassey 10y ago
        Well indeed, as per Turing on the Entscheidungsproblem one cannot know what some abstract bit of code does without running it, but one can sometimes put limits on its behaviour - your snippet will never do anything but print out numbers, for example, or something running under seccomp might be prevented from making certain syscalls.
  - klibertp 10y ago
    I think that the argument is that I could know, pretty exactly, not only when (or if) this piece of code terminates, but also how much iterations it's going to take. Sure, it's much easier (in this case) to simply run the thing, experiment with some tweaks a bit and come to some conclusion (like biologists supposedly do?). But I could also go read CPython (assuming CPython) implementation of floating point arithmetic, eventually dropping down to assembly and the workings of an FPU unit, while at the same time I could also take an analytic approach, treating this as a well-defined mathematical problem of summing a series (I think? sorry, I'm personally not that good on that front... but the option is there!). I think biologists are constrained only to the first, experimental approach - or that's how I understand the argument, at least.
    - portent 10y ago
      What you are saying sounds intuitively true, but the mathematics of Chaos Theory showed that it is not actually the case. What my code snippet is doing is running a sample of a particular chaotic function that was originally inspired by biology (a simple predator/prey model). Ultimately, what happens is that you just cannot predict how the function will behave - it is chaotic. Ultimately most complex systems start to show some chaotic behaviour, which basically means that the behaviour of the system cannot be predicted in detail, even if virtually everything is known about the system in advance. https://en.wikipedia.org/wiki/Logistic_map https://en.wikipedia.org/wiki/Logistic_map
      - Retric 10y ago
        No, virtually is the important bit. In code we can know all the details. So, you can predict the code just fine. For example running it twice and getting the same output the second time. Chaos theory is based on real world systems that can't be measured accurately. However, internal computer simulations don't have that limitation. One example is if you try saving a simulation to disk you need to copy all internal state or you get a different output.
        BraveNewCurency 10y ago
  - taneq 10y ago
    And this is a great illustration of why arguments that "humans are special because computers can never solve the halting problem" are specious. We can't solve the halting problem (in the general case) either. That said, we can still see what that code does, even if we don't know precisely what code path it takes. Great example though!
- stargrazer 10y ago
  Then there is a subject called 'genetic programming', where programs are explicitly designed to mutate, reproduce, cross-pollinate, and randomly generate new code from a set of pre-defined operators. Which, when performed in a large scale, might be 'scary'.
- RugnirViking 10y ago
  Is the universe deterministic? we honestly don't know. So at some level, the same can be said of any natural process such as this.
colordrops 10y ago
No, GMOs are inherently safe and you are anti-science /s
- RHSeeger 10y ago
  GMOs are not inherently safe, they're just not inherently unsafe. It's possible to create something devastating to the environment, but it's also possible to create wonderful things. The trick is being careful to create the right things (which, given how companies work, generally means regulation).
  - colordrops 10y ago
    I fully agree, which is why I added the /s(arcasm) tag at the end. Not sure if that is commonly understood here on HN. I was just saying that frequently comments that imply even the slightest criticism about GMOs get attacked as anti-science, which is absurd.
    - TeMPOraL 10y ago
      I hope, though, that you understand that this is basically a reaction to GMOs being assumed inherently dangerous and evil by the general population, by the virtue of not being "natural" enough.
      - colordrops 10y ago
        I don't. I've given what I've thought to be well balanced and scientifically backed statements about GMOs being safe because of the all the testing, and have even used the same terminology that you have, saying they are neither inherently safe or unsafe, and have been viciously attacked by pro-GMO supporters for not blindly saying they are inherently safe. And in almost all cases the background of the posters that attacked me either worked in the industry or a mod of /r/GMOMyths/ or the like. The funny thing is that anywhere I post, they come out of the woodwork, as if they have bots that are constantly scanning various forums for keywords. It's not merely an attack of people with stupid uninformed opinions. It's a tobacco industry style propaganda campaign that attacks anyone that is not 100% patriotically behind GMOs.
SilasX 10y ago
>It's good as long self-termination is a nice-to-have feature. If you REALLY want them to self-terminate but you found out that they mutated and can now synthesize X & Y, you're in trouble. Also, if they get powerful enough that they can take over your synthesis facility...
stcredzero 10y ago
We have no idea what happens in our own code, code that we copy from place to place ourselves, written in languages we designed and running on machines we designed, and there's nothing like mutation and self-replication with this stuff, for the most part. Documented: Computer viruses mutating in the wild by two different viruses accidentally copying themselves into overlapping regions of memory.
- PhasmaFelis 10y ago
  Interesting. When has that happened? It seems awfully unlikely to create "viable offspring" that way, but I suppose most biological mutations are neutral or negative as well...
- mrkgnao 10y ago
  I'd love a reference! This sounds very interesting.
- RugnirViking 10y ago
  I'd love to read more about this, it sounds fascinating.
eivarv 10y ago
Genetically modified organisms are a bit scary... Why?
- astrodust 10y ago
  The biggest problem is they're not open source. What went into making that organism? Is there a standard for documenting this and reasonable public disclosure when relevant? Foods have ingredient lists for a reason.
  - AnkhMorporkian 10y ago
    I'd argue that any organism is inherently open source, just not necessarily free and open source.
    - astrodust 10y ago
      If it's been baked the source is wrecked, so not really.
      - AnkhMorporkian 10y ago
        Ending a bash script in rm $1 doesn't magically make it not open source, just inherently self-destructive.
    - an-honest-moose 10y ago
      By that logic you could argue that software is inherently open source.
      - zeta0134 10y ago
        As the reverse engineering communities will be glad to tell you, software basically is open source. It must eventually be decoded into instructions to run on a processor, and while it's not impossible to fully sign / encrypt the binary blobs before they are run, it is practically not worried about for most applications. Without using specialized hardware, you can at best obfuscate your code, but you cannot make it impossible to eventually disassemble and understand. Assume your code will be disassembled at some point by a bad actor. Don't assume that any code distributed to a client is safe or trustworthy, and expect that any data, methods, or secrets contained in that code are now public knowledge. To secure your environment systems under any other assumption is a dangerous falsehood.
taneq 10y ago
> If you REALLY want them to self-terminate but you found out that they mutated and can now synthesize X & Y, you're in trouble. There's a good hook for a sci-fi story. Some low-level manager in a biochem company starts siphoning off XY juice to sell on the black market, and doesn't realise that by slowly ramping up their thievery they're evolving the killswitch out of their engineered organisms...
clouddrover 10y ago
> If you REALLY want them to self-terminate but you found out that they mutated and can now synthesize X & Y, you're in trouble. And speaking of the Jem'Hadar, there was an episode of Deep Space Nine based on this idea: http://memory-alpha.wikia.com/wiki/Hippocratic_Oath_(episode) http://memory-alpha.wikia.com/wiki/Hippocratic_Oath_(episode...
fatboy93 10y ago
> It's good as long self-termination is a nice-to-have feature. If you REALLY want them to self-terminate but you found out that they mutated and can now synthesize X & Y, you're in trouble. That's the premise of Jurassic Park 1, except with an amino acid.
- arximboldi 10y ago
  There is this soviet science fiction story with a similar premise, but self-replicating machines: Crabs On The Island (Russian 1958, English 1968) (original: Kraby idut po ostrovu), by Anatoly Dneprov http://arvindguptatoys.com/arvindgupta/15r.pdf http://arvindguptatoys.com/arvindgupta/15r.pdf

by _yosefk 10y ago

jcoffland 10y ago

> We have no idea what happens in our own code That's an overstatement. Sure code can be very complex but we actually can see exactly what's going on whenever we really want to. It's just that we cannot see what's going on with all the code all the time.

portent 10y ago
code can be very simple, and we don't always know what it will do. Simple example: can you tell me if this snippet of (python) code will ever terminate or not? x=0.5 while x<0.6 or x>0.7: x=3.59*x*(1-x) print x ... and what if the 3.59 was replaced by a different number - maybe 3.60 ? or 3.84 ?
- sbmassey 10y ago
  It will eventually terminate, if only because the machine it is running on will eventually be shut down.
  - portent 10y ago
    OK, well we can always say that the heat-death of the universe renders all problems irrelevent, but it's a bit of a cheat in my opinion. The original poster seems to imply that knowing the code means that you can know the behaviour of the system; I do not think that is the case, and my simple (chaotic) example tries to demonstrate this.
    - joshjje 10y ago
      If you have the compiled assembly obviously you can say exactly what will happen. I agree with your meaning somewhat though that more often than not many people have no clue, especially with large systems. Its not as bad as your statement though..
      - portent 10y ago
        Well... the compiled assembly doesn't give you any more information than the code snippet. The issue is not so much how this code is translated from higher abstraction level to lower abstraction level... the issue is, that this code represents a simple chaotic function (the logistic map). As such, for a simple few lines of code the behaviour is very complex and virtually impossible to predict; for certain values of the controlling number it will (1) halt relatively quickly, (2) never halt, or (3) halt after a very long time... but good luck in distinguishing between cases 2 and 3!
      - BraveNewCurency 10y ago
        No, the parent is actually talking about the Halting Problem. It has been proven impossible to predict when a program will exit without actually executing the program. (except in a few trivial edge cases) https://en.wikipedia.org/wiki/Halting_problem https://en.wikipedia.org/wiki/Halting_problem
    - sbmassey 10y ago
      Well indeed, as per Turing on the Entscheidungsproblem one cannot know what some abstract bit of code does without running it, but one can sometimes put limits on its behaviour - your snippet will never do anything but print out numbers, for example, or something running under seccomp might be prevented from making certain syscalls.
- klibertp 10y ago
  I think that the argument is that I could know, pretty exactly, not only when (or if) this piece of code terminates, but also how much iterations it's going to take. Sure, it's much easier (in this case) to simply run the thing, experiment with some tweaks a bit and come to some conclusion (like biologists supposedly do?). But I could also go read CPython (assuming CPython) implementation of floating point arithmetic, eventually dropping down to assembly and the workings of an FPU unit, while at the same time I could also take an analytic approach, treating this as a well-defined mathematical problem of summing a series (I think? sorry, I'm personally not that good on that front... but the option is there!). I think biologists are constrained only to the first, experimental approach - or that's how I understand the argument, at least.
  - portent 10y ago
    What you are saying sounds intuitively true, but the mathematics of Chaos Theory showed that it is not actually the case. What my code snippet is doing is running a sample of a particular chaotic function that was originally inspired by biology (a simple predator/prey model). Ultimately, what happens is that you just cannot predict how the function will behave - it is chaotic. Ultimately most complex systems start to show some chaotic behaviour, which basically means that the behaviour of the system cannot be predicted in detail, even if virtually everything is known about the system in advance. https://en.wikipedia.org/wiki/Logistic_map https://en.wikipedia.org/wiki/Logistic_map
    - Retric 10y ago
      No, virtually is the important bit. In code we can know all the details. So, you can predict the code just fine. For example running it twice and getting the same output the second time. Chaos theory is based on real world systems that can't be measured accurately. However, internal computer simulations don't have that limitation. One example is if you try saving a simulation to disk you need to copy all internal state or you get a different output.
      - BraveNewCurency 10y ago
- taneq 10y ago
  And this is a great illustration of why arguments that "humans are special because computers can never solve the halting problem" are specious. We can't solve the halting problem (in the general case) either. That said, we can still see what that code does, even if we don't know precisely what code path it takes. Great example though!
stargrazer 10y ago
Then there is a subject called 'genetic programming', where programs are explicitly designed to mutate, reproduce, cross-pollinate, and randomly generate new code from a set of pre-defined operators. Which, when performed in a large scale, might be 'scary'.
RugnirViking 10y ago
Is the universe deterministic? we honestly don't know. So at some level, the same can be said of any natural process such as this.

colordrops 10y ago

No, GMOs are inherently safe and you are anti-science /s

RHSeeger 10y ago
GMOs are not inherently safe, they're just not inherently unsafe. It's possible to create something devastating to the environment, but it's also possible to create wonderful things. The trick is being careful to create the right things (which, given how companies work, generally means regulation).
- colordrops 10y ago
  I fully agree, which is why I added the /s(arcasm) tag at the end. Not sure if that is commonly understood here on HN. I was just saying that frequently comments that imply even the slightest criticism about GMOs get attacked as anti-science, which is absurd.
  - TeMPOraL 10y ago
    I hope, though, that you understand that this is basically a reaction to GMOs being assumed inherently dangerous and evil by the general population, by the virtue of not being "natural" enough.
    - colordrops 10y ago
      I don't. I've given what I've thought to be well balanced and scientifically backed statements about GMOs being safe because of the all the testing, and have even used the same terminology that you have, saying they are neither inherently safe or unsafe, and have been viciously attacked by pro-GMO supporters for not blindly saying they are inherently safe. And in almost all cases the background of the posters that attacked me either worked in the industry or a mod of /r/GMOMyths/ or the like. The funny thing is that anywhere I post, they come out of the woodwork, as if they have bots that are constantly scanning various forums for keywords. It's not merely an attack of people with stupid uninformed opinions. It's a tobacco industry style propaganda campaign that attacks anyone that is not 100% patriotically behind GMOs.

SilasX 10y ago

>It's good as long self-termination is a nice-to-have feature. If you REALLY want them to self-terminate but you found out that they mutated and can now synthesize X & Y, you're in trouble. Also, if they get powerful enough that they can take over your synthesis facility...

stcredzero 10y ago

We have no idea what happens in our own code, code that we copy from place to place ourselves, written in languages we designed and running on machines we designed, and there's nothing like mutation and self-replication with this stuff, for the most part. Documented: Computer viruses mutating in the wild by two different viruses accidentally copying themselves into overlapping regions of memory.

PhasmaFelis 10y ago
Interesting. When has that happened? It seems awfully unlikely to create "viable offspring" that way, but I suppose most biological mutations are neutral or negative as well...
mrkgnao 10y ago
I'd love a reference! This sounds very interesting.
RugnirViking 10y ago
I'd love to read more about this, it sounds fascinating.

eivarv 10y ago

Genetically modified organisms are a bit scary... Why?

astrodust 10y ago
The biggest problem is they're not open source. What went into making that organism? Is there a standard for documenting this and reasonable public disclosure when relevant? Foods have ingredient lists for a reason.
- AnkhMorporkian 10y ago
  I'd argue that any organism is inherently open source, just not necessarily free and open source.
  - astrodust 10y ago
    If it's been baked the source is wrecked, so not really.
    - AnkhMorporkian 10y ago
      Ending a bash script in rm $1 doesn't magically make it not open source, just inherently self-destructive.
  - an-honest-moose 10y ago
    By that logic you could argue that software is inherently open source.
    - zeta0134 10y ago
      As the reverse engineering communities will be glad to tell you, software basically is open source. It must eventually be decoded into instructions to run on a processor, and while it's not impossible to fully sign / encrypt the binary blobs before they are run, it is practically not worried about for most applications. Without using specialized hardware, you can at best obfuscate your code, but you cannot make it impossible to eventually disassemble and understand. Assume your code will be disassembled at some point by a bad actor. Don't assume that any code distributed to a client is safe or trustworthy, and expect that any data, methods, or secrets contained in that code are now public knowledge. To secure your environment systems under any other assumption is a dangerous falsehood.

taneq 10y ago

> If you REALLY want them to self-terminate but you found out that they mutated and can now synthesize X & Y, you're in trouble. There's a good hook for a sci-fi story. Some low-level manager in a biochem company starts siphoning off XY juice to sell on the black market, and doesn't realise that by slowly ramping up their thievery they're evolving the killswitch out of their engineered organisms...

clouddrover 10y ago

> If you REALLY want them to self-terminate but you found out that they mutated and can now synthesize X & Y, you're in trouble. And speaking of the Jem'Hadar, there was an episode of Deep Space Nine based on this idea: http://memory-alpha.wikia.com/wiki/Hippocratic_Oath_(episode) http://memory-alpha.wikia.com/wiki/Hippocratic_Oath_(episode...

fatboy93 10y ago

> It's good as long self-termination is a nice-to-have feature. If you REALLY want them to self-terminate but you found out that they mutated and can now synthesize X & Y, you're in trouble. That's the premise of Jurassic Park 1, except with an amino acid.

arximboldi 10y ago
There is this soviet science fiction story with a similar premise, but self-replicating machines: Crabs On The Island (Russian 1958, English 1968) (original: Kraby idut po ostrovu), by Anatoly Dneprov http://arvindguptatoys.com/arvindgupta/15r.pdf http://arvindguptatoys.com/arvindgupta/15r.pdf