3 ms·

The interesting version of the argument isn't about substrate: it's about motivation. Present the trolley problem to GPT-4 and it gives you a philosophy survey

The interesting version of the argument isn't about substrate: it's about motivation. Present the trolley problem to GPT-4 and it gives you a philosophy survey answer. Present it to a human and their palms sweat. The gap isn't computation, it's that humans are value-making machines shaped by millions of years of selection pressure. Pollan lands on the wrong argument (biology vs. silicon) when the real one is: where do the values come from, and can they emerge without a reproductive lineage that stakes survival on getting them right?

ath3nd 8mo ago
[dead]
judahmeek 8mo ago
I refer to this as moral grounding. I'm not sure I would call it a requirement for consciousness, but knowing that most beings with general intelligence (humans) have a form of it similar to my own does make it easier to sleep at night.
- adamzwasserman 8mo ago
  To clarify: I'm not talking about morals specifically. I mean value in the broader sense of spontaneously assigning relative importance to things, producing a hierarchy that drives action. You're thirsty. There's pond water in the forest and clean well water in the town square, but you're an escaped prisoner. Suddenly the value hierarchy flips: safety trumps water quality. You do this instantly, with incomplete information, integrating survival, context, and preference in a way that no one programmed into you. Morality is one expression of this capacity, but so is aesthetic judgment, risk assessment, curiosity, and the decision to walk down a dark alley or not. The trolley problem is just a dramatic example. The mundane examples are actually more telling, because we do them thousands of times a day without noticing. No current AI has any form of this. It has no mechanism for deciding that anything matters more than anything else except through weightings that were derived from human-generated training data. It borrows our value hierarchies statistically. It doesn't have its own.

ath3nd 8mo ago

[dead]

judahmeek 8mo ago

I refer to this as moral grounding. I'm not sure I would call it a requirement for consciousness, but knowing that most beings with general intelligence (humans) have a form of it similar to my own does make it easier to sleep at night.

adamzwasserman 8mo ago
To clarify: I'm not talking about morals specifically. I mean value in the broader sense of spontaneously assigning relative importance to things, producing a hierarchy that drives action. You're thirsty. There's pond water in the forest and clean well water in the town square, but you're an escaped prisoner. Suddenly the value hierarchy flips: safety trumps water quality. You do this instantly, with incomplete information, integrating survival, context, and preference in a way that no one programmed into you. Morality is one expression of this capacity, but so is aesthetic judgment, risk assessment, curiosity, and the decision to walk down a dark alley or not. The trolley problem is just a dramatic example. The mundane examples are actually more telling, because we do them thousands of times a day without noticing. No current AI has any form of this. It has no mechanism for deciding that anything matters more than anything else except through weightings that were derived from human-generated training data. It borrows our value hierarchies statistically. It doesn't have its own.