6 ms·

Systems design 2: What we hope we know

181 points by Amorymeltzer 4y ago

danielvaughn 4y ago
"I've never seen anyone say ChatGPT feels like magic" This is a curious line - I have, and I personally have that reaction when I use it. It might even be the most magical technology I've ever encountered.
- jt2190 4y ago
  > This is a curious line… Literally in the next sentence the author also remarks that he finds it odd: > “That makes me curious.” He finds it curious because he also finds the technology is magical. Eventually he concludes: > “Let's say you have a Large Language Model AI, trained on all the text in the world. We understand the mechanisms (engineering) but the outcomes are surprising and emergent (magic). Its job is to repeatedly predict the next word that would come if a person wrote it – a form of negative latency, or non-causality. From this, for reasons we don't really understand, it can say an impressive variety of unexpectedly meaningful things.” > “When you have all the text in the world memorized, that means you have access to all the insights that have ever been written. You need only recognize them, and have a good idea of what the reader knows already, and you can produce insights – things the reader has never heard before – on demand. You can get an A+ from my English teacher. Can our AI do that? I don't know. Certainly sometimes, maybe not reliably yet. Any time it does, it will feel like magic.”
- MH15 4y ago
  While I personally don't think ChatGPT feels magical, I know many people who feel this way. It's a really common opinion, especially among non-programmers.
- amadeuspagel 4y ago
  I guess it doesn't feel like magic because it feels human?
carapace 4y ago
FWIW, apenwarr is one of the most intelligent humans I've ever met. When people bring up 10x programmers I think of him. I don't really have much to add, I just wanted to mention that for context.
tudorw 4y ago
thanks, a good read, I'm always curious to go back and look at early knowledge system innovators who were looking at the 'causes' of 'breakthrough hypothesis' and looking to then 'engineer' a surrounding system to support more frequent breakthroughs.
rektide 4y ago
Placing emerged complexity/magic in a bucket together & saying science despises them is a novel & revelatory point of view. Avery is such a treasure! Lovely idea towards the bottom too, > Or maybe we need to draw a distinction between producing insight and recognizing insight. I personally identify with this & see it as super core a distinction. I'm reasonably bright & can find some clever ways of doing things even if I haven't any learnings here. But like 90% of what I love & appreciate about self is a resourcefulness & ability to recognize & integrate insight.
kqr 4y ago
On the quote opening the article: > Any sufficiently advanced technology is indistinguishable from magic. I had a lecturer who used to counter "if something ever seems like magic it means you're not asking enough questions." That stuck with me, although the article goes on to counter it fairly well.
Sniffnoy 4y ago
> Since we now know what magic means, we know what magical thinking means: a tendency to interpret something as magic instead of trying to understand it. No, "magical thinking" has a particular meaning, and it's not that: https://en.wikipedia.org/wiki/Magical_thinking https://en.wikipedia.org/wiki/Magical_thinking Mind you, people often use the phrase in a pretty loose way as a way of dismissing an argument, but it does seem to be based on the technical use, and not the meaning Apenwarr puts forth.
- epgui 4y ago
  You’re allowed to define different meanings for words if it makes your argument go forward and if you make it clear what it is. However I still find the argument very unpersuasive, and the usual meaning of “magical thinking” makes me feel uncomfortable too.
  - Sniffnoy 4y ago
    Sure, but in this case he's explicitly discussing other people's use: > On the other hand, people who believe AIs and specifically LLMs possess "intelligence" are often accused of "magical thinking."
    - epgui 4y ago
      You’re very right about that.
- singpolyma3 4y ago
  That definition seems pretty closely related to his.
epgui 4y ago
I appreciate that the author is using very specific meaning for the term “magical thinking”, but even with this, I am uncomfortable with the assertion that somehow understanding emergence has anything to do with magic or magical thinking, as (re)defined in the article. Anyone who spends enough time studying biochem will have an intuitive understanding / acceptance that biology emerges from chemistry, which emerges from physics… But there’s not really any “magical thinking” (as defined in the article) involved. Emergence of cognitive processes and experiences in the brain is not really any different in this respect. If anything, it’s people who insist that human experiences are “special” and “just different, obviously” that are doing the magical thinking.
- analog31 4y ago
  My thought about emergence is that it is not a phenomenon in its own right, but a category of phenomena that are identified subjectively. A lot has been written describing the wonders of emergence. Synthetic model systems have been developed with seemingly emergent behaviors, without leading to novel, empirically testable predictions. So far the only empirically testable models leading to emergent behaviors such as we observe in nature, are in fact the laws of physics and chemistry, more or less as we know them today. To me, "magic" means that a phenomenon is inherently inaccessible to naturalistic inquiry, not merely that such inquiry is temporarily stalled due to inattention, lack of the right insight, or insufficient technology.
  - epgui 4y ago
    To that short list I would add some purely mathematical topics such as cellular automata (which show great promise towards helping formalize the concept of emergence IMO), but also biology, ecology, neuroscience, psychology, sociology, economics... There's layers upon layers of emergence all around us. Even just within "physics" there are probably multiple layers of emergence (a striking and popular example is particle-like behaviour arising from wave-like fields). There is "reasonable speculation" as to whether space, time, gravity might all be emergent rather than fundamental...
    - 4y ago
nyanpasu64 4y ago
> In short, in software engineering, we acknowledge that failures happen and we measure them, characterize them, and compensate for them. We don't aim for perfection. If software engineering revolves around writing software complex enough to have bugs not understood, or accepting memory safety errors as a natural byproduct of programming (rather than programming mistakes to be exterminated on sight, and made impossible by design unless it prevents writing useful programs), I'd be more effective and happier practicing applied math than engineering.
- skybrian 4y ago
  It's also accepting that performance is variable, that the network is not reliable, that machines can fail at any time, and that memory corruption happens. When you build systems at scale you need to worry about these things. Programming language API boundaries don't make real-time performance guarantees, not even in Rust. We hope to get typical performance. In a mathematical context (for example, proving programs correct), you often ignore these things and worry only about whether the program will give you the right answer, eventually, when the hardware works.
akkartik 4y ago
Minor quibble. "Summarization can be done with little to no insight at all. Perhaps to do it better requires some insight." By this standard caching requires little to no insight at all. Perhaps some predictors require insight.
- apenwarr 4y ago
  It’s possible to build a cache without any insight, but to make a cache that works well in a given domain requires knowledge from outside the world of caching itself. The same is true of summarization and compression.
ChatGTP 4y ago
Many people keep saying the brain isn’t magic so blah blah sentience etc, but if that’s the case, we cannot claim LLMs are magic either, or any “emergent properties”, else you do believe in magic. Which is it, magic or no magic? We can not have it both ways. Maybe the idea of magic shouldn’t keep being referenced, we should refrain from comparing and just apply equal respect to all beings and phenomena and just admit some things are, and always will be mysterious and move on with life ?
tuatoru 4y ago
I love the meta-joke about the "levels of indirection" quote.
- ignoramous 4y ago
  Can you point me to it? ;)

181 points by Amorymeltzer 4y ago

"I've never seen anyone say ChatGPT feels like magic" This is a curious line - I have, and I personally have that reaction when I use it. It might even be the most magical technology I've ever encountered.

jt2190 4y ago
> This is a curious line… Literally in the next sentence the author also remarks that he finds it odd: > “That makes me curious.” He finds it curious because he also finds the technology is magical. Eventually he concludes: > “Let's say you have a Large Language Model AI, trained on all the text in the world. We understand the mechanisms (engineering) but the outcomes are surprising and emergent (magic). Its job is to repeatedly predict the next word that would come if a person wrote it – a form of negative latency, or non-causality. From this, for reasons we don't really understand, it can say an impressive variety of unexpectedly meaningful things.” > “When you have all the text in the world memorized, that means you have access to all the insights that have ever been written. You need only recognize them, and have a good idea of what the reader knows already, and you can produce insights – things the reader has never heard before – on demand. You can get an A+ from my English teacher. Can our AI do that? I don't know. Certainly sometimes, maybe not reliably yet. Any time it does, it will feel like magic.”
MH15 4y ago
While I personally don't think ChatGPT feels magical, I know many people who feel this way. It's a really common opinion, especially among non-programmers.
amadeuspagel 4y ago
I guess it doesn't feel like magic because it feels human?

carapace 4y ago

FWIW, apenwarr is one of the most intelligent humans I've ever met. When people bring up 10x programmers I think of him. I don't really have much to add, I just wanted to mention that for context.

tudorw 4y ago

thanks, a good read, I'm always curious to go back and look at early knowledge system innovators who were looking at the 'causes' of 'breakthrough hypothesis' and looking to then 'engineer' a surrounding system to support more frequent breakthroughs.

rektide 4y ago

Placing emerged complexity/magic in a bucket together & saying science despises them is a novel & revelatory point of view. Avery is such a treasure! Lovely idea towards the bottom too, > Or maybe we need to draw a distinction between producing insight and recognizing insight. I personally identify with this & see it as super core a distinction. I'm reasonably bright & can find some clever ways of doing things even if I haven't any learnings here. But like 90% of what I love & appreciate about self is a resourcefulness & ability to recognize & integrate insight.

kqr 4y ago

On the quote opening the article: > Any sufficiently advanced technology is indistinguishable from magic. I had a lecturer who used to counter "if something ever seems like magic it means you're not asking enough questions." That stuck with me, although the article goes on to counter it fairly well.

Sniffnoy 4y ago

> Since we now know what magic means, we know what magical thinking means: a tendency to interpret something as magic instead of trying to understand it. No, "magical thinking" has a particular meaning, and it's not that: https://en.wikipedia.org/wiki/Magical_thinking https://en.wikipedia.org/wiki/Magical_thinking Mind you, people often use the phrase in a pretty loose way as a way of dismissing an argument, but it does seem to be based on the technical use, and not the meaning Apenwarr puts forth.

epgui 4y ago
You’re allowed to define different meanings for words if it makes your argument go forward and if you make it clear what it is. However I still find the argument very unpersuasive, and the usual meaning of “magical thinking” makes me feel uncomfortable too.
- Sniffnoy 4y ago
  Sure, but in this case he's explicitly discussing other people's use: > On the other hand, people who believe AIs and specifically LLMs possess "intelligence" are often accused of "magical thinking."
  - epgui 4y ago
    You’re very right about that.
singpolyma3 4y ago
That definition seems pretty closely related to his.

epgui 4y ago

I appreciate that the author is using very specific meaning for the term “magical thinking”, but even with this, I am uncomfortable with the assertion that somehow understanding emergence has anything to do with magic or magical thinking, as (re)defined in the article. Anyone who spends enough time studying biochem will have an intuitive understanding / acceptance that biology emerges from chemistry, which emerges from physics… But there’s not really any “magical thinking” (as defined in the article) involved. Emergence of cognitive processes and experiences in the brain is not really any different in this respect. If anything, it’s people who insist that human experiences are “special” and “just different, obviously” that are doing the magical thinking.

analog31 4y ago
My thought about emergence is that it is not a phenomenon in its own right, but a category of phenomena that are identified subjectively. A lot has been written describing the wonders of emergence. Synthetic model systems have been developed with seemingly emergent behaviors, without leading to novel, empirically testable predictions. So far the only empirically testable models leading to emergent behaviors such as we observe in nature, are in fact the laws of physics and chemistry, more or less as we know them today. To me, "magic" means that a phenomenon is inherently inaccessible to naturalistic inquiry, not merely that such inquiry is temporarily stalled due to inattention, lack of the right insight, or insufficient technology.
- epgui 4y ago
  To that short list I would add some purely mathematical topics such as cellular automata (which show great promise towards helping formalize the concept of emergence IMO), but also biology, ecology, neuroscience, psychology, sociology, economics... There's layers upon layers of emergence all around us. Even just within "physics" there are probably multiple layers of emergence (a striking and popular example is particle-like behaviour arising from wave-like fields). There is "reasonable speculation" as to whether space, time, gravity might all be emergent rather than fundamental...
  - 4y ago

nyanpasu64 4y ago

> In short, in software engineering, we acknowledge that failures happen and we measure them, characterize them, and compensate for them. We don't aim for perfection. If software engineering revolves around writing software complex enough to have bugs not understood, or accepting memory safety errors as a natural byproduct of programming (rather than programming mistakes to be exterminated on sight, and made impossible by design unless it prevents writing useful programs), I'd be more effective and happier practicing applied math than engineering.

skybrian 4y ago
It's also accepting that performance is variable, that the network is not reliable, that machines can fail at any time, and that memory corruption happens. When you build systems at scale you need to worry about these things. Programming language API boundaries don't make real-time performance guarantees, not even in Rust. We hope to get typical performance. In a mathematical context (for example, proving programs correct), you often ignore these things and worry only about whether the program will give you the right answer, eventually, when the hardware works.

akkartik 4y ago

Minor quibble. "Summarization can be done with little to no insight at all. Perhaps to do it better requires some insight." By this standard caching requires little to no insight at all. Perhaps some predictors require insight.

apenwarr 4y ago
It’s possible to build a cache without any insight, but to make a cache that works well in a given domain requires knowledge from outside the world of caching itself. The same is true of summarization and compression.

ChatGTP 4y ago

Many people keep saying the brain isn’t magic so blah blah sentience etc, but if that’s the case, we cannot claim LLMs are magic either, or any “emergent properties”, else you do believe in magic. Which is it, magic or no magic? We can not have it both ways. Maybe the idea of magic shouldn’t keep being referenced, we should refrain from comparing and just apply equal respect to all beings and phenomena and just admit some things are, and always will be mysterious and move on with life ?

tuatoru 4y ago

I love the meta-joke about the "levels of indirection" quote.

ignoramous 4y ago
Can you point me to it? ;)