2 ms·

It's because we can put responsibility on humans to be correct but we can't on computers. Humans given the appropriate incentives are very good at their jobs an

It's because we can put responsibility on humans to be correct but we can't on computers. Humans given the appropriate incentives are very good at their jobs and there is a path for compensation if they screw up. Computers have neither of these things.

sigmoid10 2y ago
Humans already put a lot of trust in computers not because they can take responsibility but because traditional software can be made very predictable or at least compliant. There are whole industries built around software standards to ensure that. The problem is we don't yet know enough about identifying and patching problems in these models. Once we get something equivalent to MISRA for LLMs to achieve the same level of compliance, there is very little that could still hold them back.
- pclmulqdq 2y ago
  Yes. Traditional software has an unbroken chain of hard responsibility back to a human. Once you introduce non-determinism, things get weird.
  - eru 2y ago
    Non-determinism is a well-understood tool, and does not diminish responsibility. Grep works just fine, despite implementing non-deterministic finite state machines. Monte Carlo simulations are behind nuclear weapons (where they were invented), weather forecasts, financial trading, etc. Las Vegas algorithms like randomised quicksort also diminish no one's responsibility. In principle, you can run training and inference on neural networks completely deterministically. But I don't think that makes any difference to responsibility. (To make them deterministic, you obviously have to use pseudo-random number generators with fixed seeds, but less obviously you also have to make sure that when you merge the results of parallel runs, the results 'merge' deterministically. Deterministic parallelism is an extremely interesting field of study! Or, since we are only talking about principles, not what's practical, you could just run everything in series.) The problem with LLMs is that they are complicated and their actions are hard for humans to predict or reason through. Complexity is the bane of responsibility: if you have a complicated enough system (and a complicated enough management structure involved in producing that system), that's where responsibility goes to die, unless you specifically work to establish it. In this case, employing LLMs is no worse than employing humans. If upper management gives bad instructions and incentives for lower level employees, we tend to pin the responsibility on upper management.
ndm000 2y ago
I think this is a key argument in how powerful AI can become. We may be able to create incredibly intelligent systems, but at the end of the day you can’t send a computer to jail. That inherently limits the power that will be given over to AI. If an AI accidentally kills a person, the worst that could be done to it is that it is turned off, whereas the owners of the AI would be held liable.

sigmoid10 2y ago

Humans already put a lot of trust in computers not because they can take responsibility but because traditional software can be made very predictable or at least compliant. There are whole industries built around software standards to ensure that. The problem is we don't yet know enough about identifying and patching problems in these models. Once we get something equivalent to MISRA for LLMs to achieve the same level of compliance, there is very little that could still hold them back.

pclmulqdq 2y ago
Yes. Traditional software has an unbroken chain of hard responsibility back to a human. Once you introduce non-determinism, things get weird.
- eru 2y ago
  Non-determinism is a well-understood tool, and does not diminish responsibility. Grep works just fine, despite implementing non-deterministic finite state machines. Monte Carlo simulations are behind nuclear weapons (where they were invented), weather forecasts, financial trading, etc. Las Vegas algorithms like randomised quicksort also diminish no one's responsibility. In principle, you can run training and inference on neural networks completely deterministically. But I don't think that makes any difference to responsibility. (To make them deterministic, you obviously have to use pseudo-random number generators with fixed seeds, but less obviously you also have to make sure that when you merge the results of parallel runs, the results 'merge' deterministically. Deterministic parallelism is an extremely interesting field of study! Or, since we are only talking about principles, not what's practical, you could just run everything in series.) The problem with LLMs is that they are complicated and their actions are hard for humans to predict or reason through. Complexity is the bane of responsibility: if you have a complicated enough system (and a complicated enough management structure involved in producing that system), that's where responsibility goes to die, unless you specifically work to establish it. In this case, employing LLMs is no worse than employing humans. If upper management gives bad instructions and incentives for lower level employees, we tend to pin the responsibility on upper management.

ndm000 2y ago

I think this is a key argument in how powerful AI can become. We may be able to create incredibly intelligent systems, but at the end of the day you can’t send a computer to jail. That inherently limits the power that will be given over to AI. If an AI accidentally kills a person, the worst that could be done to it is that it is turned off, whereas the owners of the AI would be held liable.