3 ms·

The issue is I’ll have to vibecode all the laws and the laws could be wrong.

by garrisonj 11d ago

futurisold 11d ago
Words of wisdom.
LightMachine 11d ago
true
foota 11d ago
Jokes aside, I think the idea is that the law is simple to code, the proof that it holds is where the agent is responsible. This probably becomes less true though as you try to express more complicated laws.
- pixl97 11d ago
  Heh, it's like we all need to collectively read I, Robot yet again, and the myriad of SF books on the subjects. Black and white quickly dithers to grey.
  - burner420042 10d ago
    Indeed Robots are logical, but not rational.
    - Jolter 10d ago
      LLM driven agents aren’t even that.
    - hannasanarion 10d ago
      The point of those books is that robots can be perfectly rational, and for a useful robot we would expect them to be. What they aren't is moral, because of the orthogonality principle: you can't use facts and logic to discover correct moral beliefs. Morality is about values, goals, and the definition of "good". They must be provided to the robot by its creator, and those are things that are very hard to precisely describe in a way that is fully consistent with the speaker's intent in all possible scenarios, and agreeable by all other people.
drdrey 10d ago
writing specs is hard. For instance the easiest way to comply with "player should never reach the flag" is to disable movement completely, so then you have to specify "player should never reach the flag while still being allowed to move" and so on (liveness). It's not practical for most programs
- defmacr0 10d ago
  Yeah, in small examples it's pretty neat: Define a "sorted" property, write a law that checks that the list is sorted and contains the same elements and someone else implements the sort and together with a proof for correctness. However larger software systems have an exponentially larger surface for reasonable and unreasonable edge cases. When I hit ctrl+s in my editor I expect that my cursor does not change colour, that the window does not minimize, that the program does not crash if there is no disk space left and so on. I don't see how this does not devolve into "negative space programming" where the user would have to anticipate and constrain every possible unwanted behavior of his software.
  - YeGoblynQueenne 10d ago
    >> When I hit ctrl+s in my editor I expect that my cursor does not change colour, that the window does not minimize, that the program does not crash if there is no disk space left and so on. I take it you haven't used Microsoft Windows? Joking, joking...
- nottorp 10d ago
  That's an old "AI" joke. It went like that: They asked a computer how to minimize production expenses at a factory. The computer said: "Do not manufacture anything, expenses will be zero". I believe it's been around since expert systems at least.

The issue is I’ll have to vibecode all the laws and the laws could be wrong.

futurisold 11d ago

Words of wisdom.

LightMachine 11d ago

true

foota 11d ago

Jokes aside, I think the idea is that the law is simple to code, the proof that it holds is where the agent is responsible. This probably becomes less true though as you try to express more complicated laws.

pixl97 11d ago
Heh, it's like we all need to collectively read I, Robot yet again, and the myriad of SF books on the subjects. Black and white quickly dithers to grey.
- burner420042 10d ago
  Indeed Robots are logical, but not rational.
  - Jolter 10d ago
    LLM driven agents aren’t even that.
  - hannasanarion 10d ago
    The point of those books is that robots can be perfectly rational, and for a useful robot we would expect them to be. What they aren't is moral, because of the orthogonality principle: you can't use facts and logic to discover correct moral beliefs. Morality is about values, goals, and the definition of "good". They must be provided to the robot by its creator, and those are things that are very hard to precisely describe in a way that is fully consistent with the speaker's intent in all possible scenarios, and agreeable by all other people.

drdrey 10d ago

writing specs is hard. For instance the easiest way to comply with "player should never reach the flag" is to disable movement completely, so then you have to specify "player should never reach the flag while still being allowed to move" and so on (liveness). It's not practical for most programs

defmacr0 10d ago
Yeah, in small examples it's pretty neat: Define a "sorted" property, write a law that checks that the list is sorted and contains the same elements and someone else implements the sort and together with a proof for correctness. However larger software systems have an exponentially larger surface for reasonable and unreasonable edge cases. When I hit ctrl+s in my editor I expect that my cursor does not change colour, that the window does not minimize, that the program does not crash if there is no disk space left and so on. I don't see how this does not devolve into "negative space programming" where the user would have to anticipate and constrain every possible unwanted behavior of his software.
- YeGoblynQueenne 10d ago
  >> When I hit ctrl+s in my editor I expect that my cursor does not change colour, that the window does not minimize, that the program does not crash if there is no disk space left and so on. I take it you haven't used Microsoft Windows? Joking, joking...
nottorp 10d ago
That's an old "AI" joke. It went like that: They asked a computer how to minimize production expenses at a factory. The computer said: "Do not manufacture anything, expenses will be zero". I believe it's been around since expert systems at least.