3 ms·

> It's "statistics" in the same sense your own thoughts are. And you know this how? > we learn the language the same way LLMs do Based on? > So yeah, it's a

> It's "statistics" in the same sense your own thoughts are. And you know this how? > we learn the language the same way LLMs do Based on? > So yeah, it's all "thrown back at you via statistics". So is all of the human thought it's derivative of You say this based on what? Do you have any real research to support any of this? Because from what I see in the neuroscience community, LLMs are not even remotely an approximation of a real brain, despite being built of "neurons" (which are nothing like real neurons).

TeMPOraL 2y ago
> And you know this how? Probability theory, computability theory. There's no other way. > Based on? High-level description of the learning approach, and approximately the entirety of recorded human history? > Because from what I see in the neuroscience community, LLMs are not even remotely an approximation of a real brain, despite being built of "neurons" (which are nothing like real neurons). So what? Neurons in silica are to neurons in vivo as my Twitter clone in 15 lines of terse Perl is to the actual web-scale monstrosity a Twitter or other modern scalable SaaS is. I.e. it does the same core thing faster and better, but obviously lacks incidental, supporting complexity to work in the same environment as the real thing. Yes, a biological neuron is huge and complex. But a biological neuron is, first and foremost, an incrementally evolved, independent, self-replicating and self-maintaining nanomachine, tuned to be a part of a specialized self-assembling system, filling a role in a large and diverse ensemble of other specialized nanomachines. Incidentally, a neuron also computes - but it's not computing that uses this complexity, it's all the self-replicating self-assembling nanomachine business that needs it. There's absolutely no reason to believe that neurons in silica need to be comparably complex to crack intelligence - not when we're free to strip the problem out of all the Kubernetes/protein nanomachine incidental complexity business, and focus on the core computation.
- duhffahq 2y ago
  [dead]
- foobarqux 2y ago
  Everything you have said is completely baseless > Probability theory, computability theory. There's no other way. There is no credible evidence that humans learn via statistics and significant evidence against (poverty of stimulus, all languages have hierarchical structure). One other way was proposed by Chomsky, which is that you have built-in machinery to do language (which is probably intimately related to human intelligence) just like a foal doesn't learn to walk via "statistics" in any meaningful sense. > Neurons in silica are to neurons in vivo ... Again not true. Observations about things like inter-neuron communication time suggests that computation is being done within neurons which undermines the connectionist approach. You've just presented a bunch of your own intuitions about things which people who actually studied the field have falsified.
  - TeMPOraL 2y ago
    > There is no credible evidence that humans learn via statistics and significant evidence against Except living among humans on planet Earth, that is. Even the idea of a fixed language is an illusion, of the same kind like idk "natural balance", or "solid surface", or discrete objects. There's no platonic ideal of Polish or English that's the one language English speakers speak, that they speak more or less perfectly. No singular the English language that's fundamentally distinct from the German and the French. Claiming otherwise, or claiming built-in symbolic machinery for learning this thing, is confusing map for territory in a bad way - in the very way that gained academia a reputation of being completely out of touch with actual reality[0]. And the actual reality is, "language" is a purely statistical phenomenon, an aggregate of people trying to communicate with each other, individually adjusting themselves to meet the (perceived) expectations of others. At the scale of (population, years), it looks stable. At the scale of (population, decades), we can easily see those "stable" languages are in fact constantly changing, and blending with each other. At the scale of (individuals, days), everyone has their own language, slightly different from everyone else's. > poverty of stimulus More like poverty of imagination, dearth of idle minutes during the day, in which to ponder this. Well, to be fair, we kind of didn't have any good intuition or framework to think about this until information theory came along, and then computers became ubiquitous. So let me put this clear: a human brain is ingesting a continuous time stream of multisensory data 24/7 from the day they're born (and likely even earlier). That stream never stops, it's rich in information, and all that information is highly coherent and correlated with the physical reality. There's no poverty of language-related stimulus unless you literally throw a baby to the wolves, or have it grow up in a sensory deprivation chamber. > all languages have hierarchical structure Perhaps because hierarchy is a fundamental concept in itself. > you have built-in machinery to do language (which is probably intimately related to human intelligence) Another way of looking at it is: it co-evolved with language, i.e. languages reflect what's easiest for our brain to pick up on. Like with everything natural selection comes up with, it's a mix of fundamental mathematics of reality combined with jitter of the dried-up shit that stuck on the wall after being thrown at it by evolution. From that perspective, "built-in machinery" is an absolutely trivial observation - our languages look like whatever happened to work best with whatever idiosyncrasies our brains have. That is, whatever our statistical learning machinery managed to pick up on best. > like a foal doesn't learn to walk via "statistics" in any meaningful sense. How do they learn it then? Calculus? :). Developing closed-form analytical solutions for walking on arbitrary terrain? > Observations about things like inter-neuron communication time suggests that computation is being done within neurons So what? There's all kinds of "computation within CPUs" too. Cache management, hardware interrupts, etc., which don't change the results of the computation we're interested in, but might make it faster or more robust. > which undermines the connectionist approach. Wikipedia on connectionism: "The central connectionist principle is that mental phenomena can be described by interconnected networks of simple and often uniform units." Sure, whatever. It doesn't matter. Connectionist models are fine, because it's been mathematically proven that you can compute[1] any function with a finite network. We like them not because they're philosophically special, but because they're a simple and uniform computational structure - i.e. it's cheap to do in hardware. Even if you need a million artificial neurons to substitute for a single biological one, that's still a win, because making faster GPUs and ASICs is what we're good at; comprehending and replicating complex molecular nanotech, not so much. Computation is computation. Substrate doesn't matter, and methods don't matter - there are many ways of computing the same thing. Like, natural integration is done by just accumulating shit over time[2], but we find it easier to do it digitally by ADC-ing inputs into funny patterns of discrete bits, then flipping them back and forth according to some arcane rules, and then eventually DAC-ing some result back. Put enough bits into the ADC - digital math - DAC pipeline, and you get the same[1] result back anyway. -- [0] - I mean, what's next. Are you going to claim Earth is a sphere of a specific size and mass, orbiting the Sun in specific time, on an elliptical orbit of fixed parameters? Do you expect space probes will eventually discover the magical rails that hold Earth to its perfectly elliptical orbit? Of course you're not (I hope) - surely you're perfectly aware that Earth gains and loses mass, perfect elliptical orbits are neither perfect nor elliptical, etc. - all that is just approximations safe at the timescales usually consider. [1] - Approximate to an arbitrary high degree. Which is all we can hope for in a physical reality anyways. [2] - Or hey, did you know that one of the best ways to multiply two large numbers is to... do a Fourier transform on their digits and summing them up in the frequency domain? (Incidentally, being analog, nature really likes working in the frequency domain; between that, accumulation as addition, and not being designed to be driven by a shared clock signal it should be obvious that natural computers ain't gonna look like ours, and conversely, we don't need to replicate natural processes exactly to get the same results.)

TeMPOraL 2y ago

> And you know this how? Probability theory, computability theory. There's no other way. > Based on? High-level description of the learning approach, and approximately the entirety of recorded human history? > Because from what I see in the neuroscience community, LLMs are not even remotely an approximation of a real brain, despite being built of "neurons" (which are nothing like real neurons). So what? Neurons in silica are to neurons in vivo as my Twitter clone in 15 lines of terse Perl is to the actual web-scale monstrosity a Twitter or other modern scalable SaaS is. I.e. it does the same core thing faster and better, but obviously lacks incidental, supporting complexity to work in the same environment as the real thing. Yes, a biological neuron is huge and complex. But a biological neuron is, first and foremost, an incrementally evolved, independent, self-replicating and self-maintaining nanomachine, tuned to be a part of a specialized self-assembling system, filling a role in a large and diverse ensemble of other specialized nanomachines. Incidentally, a neuron also computes - but it's not computing that uses this complexity, it's all the self-replicating self-assembling nanomachine business that needs it. There's absolutely no reason to believe that neurons in silica need to be comparably complex to crack intelligence - not when we're free to strip the problem out of all the Kubernetes/protein nanomachine incidental complexity business, and focus on the core computation.

duhffahq 2y ago
[dead]
foobarqux 2y ago
Everything you have said is completely baseless > Probability theory, computability theory. There's no other way. There is no credible evidence that humans learn via statistics and significant evidence against (poverty of stimulus, all languages have hierarchical structure). One other way was proposed by Chomsky, which is that you have built-in machinery to do language (which is probably intimately related to human intelligence) just like a foal doesn't learn to walk via "statistics" in any meaningful sense. > Neurons in silica are to neurons in vivo ... Again not true. Observations about things like inter-neuron communication time suggests that computation is being done within neurons which undermines the connectionist approach. You've just presented a bunch of your own intuitions about things which people who actually studied the field have falsified.
- TeMPOraL 2y ago
  > There is no credible evidence that humans learn via statistics and significant evidence against Except living among humans on planet Earth, that is. Even the idea of a fixed language is an illusion, of the same kind like idk "natural balance", or "solid surface", or discrete objects. There's no platonic ideal of Polish or English that's the one language English speakers speak, that they speak more or less perfectly. No singular the English language that's fundamentally distinct from the German and the French. Claiming otherwise, or claiming built-in symbolic machinery for learning this thing, is confusing map for territory in a bad way - in the very way that gained academia a reputation of being completely out of touch with actual reality[0]. And the actual reality is, "language" is a purely statistical phenomenon, an aggregate of people trying to communicate with each other, individually adjusting themselves to meet the (perceived) expectations of others. At the scale of (population, years), it looks stable. At the scale of (population, decades), we can easily see those "stable" languages are in fact constantly changing, and blending with each other. At the scale of (individuals, days), everyone has their own language, slightly different from everyone else's. > poverty of stimulus More like poverty of imagination, dearth of idle minutes during the day, in which to ponder this. Well, to be fair, we kind of didn't have any good intuition or framework to think about this until information theory came along, and then computers became ubiquitous. So let me put this clear: a human brain is ingesting a continuous time stream of multisensory data 24/7 from the day they're born (and likely even earlier). That stream never stops, it's rich in information, and all that information is highly coherent and correlated with the physical reality. There's no poverty of language-related stimulus unless you literally throw a baby to the wolves, or have it grow up in a sensory deprivation chamber. > all languages have hierarchical structure Perhaps because hierarchy is a fundamental concept in itself. > you have built-in machinery to do language (which is probably intimately related to human intelligence) Another way of looking at it is: it co-evolved with language, i.e. languages reflect what's easiest for our brain to pick up on. Like with everything natural selection comes up with, it's a mix of fundamental mathematics of reality combined with jitter of the dried-up shit that stuck on the wall after being thrown at it by evolution. From that perspective, "built-in machinery" is an absolutely trivial observation - our languages look like whatever happened to work best with whatever idiosyncrasies our brains have. That is, whatever our statistical learning machinery managed to pick up on best. > like a foal doesn't learn to walk via "statistics" in any meaningful sense. How do they learn it then? Calculus? :). Developing closed-form analytical solutions for walking on arbitrary terrain? > Observations about things like inter-neuron communication time suggests that computation is being done within neurons So what? There's all kinds of "computation within CPUs" too. Cache management, hardware interrupts, etc., which don't change the results of the computation we're interested in, but might make it faster or more robust. > which undermines the connectionist approach. Wikipedia on connectionism: "The central connectionist principle is that mental phenomena can be described by interconnected networks of simple and often uniform units." Sure, whatever. It doesn't matter. Connectionist models are fine, because it's been mathematically proven that you can compute[1] any function with a finite network. We like them not because they're philosophically special, but because they're a simple and uniform computational structure - i.e. it's cheap to do in hardware. Even if you need a million artificial neurons to substitute for a single biological one, that's still a win, because making faster GPUs and ASICs is what we're good at; comprehending and replicating complex molecular nanotech, not so much. Computation is computation. Substrate doesn't matter, and methods don't matter - there are many ways of computing the same thing. Like, natural integration is done by just accumulating shit over time[2], but we find it easier to do it digitally by ADC-ing inputs into funny patterns of discrete bits, then flipping them back and forth according to some arcane rules, and then eventually DAC-ing some result back. Put enough bits into the ADC - digital math - DAC pipeline, and you get the same[1] result back anyway. -- [0] - I mean, what's next. Are you going to claim Earth is a sphere of a specific size and mass, orbiting the Sun in specific time, on an elliptical orbit of fixed parameters? Do you expect space probes will eventually discover the magical rails that hold Earth to its perfectly elliptical orbit? Of course you're not (I hope) - surely you're perfectly aware that Earth gains and loses mass, perfect elliptical orbits are neither perfect nor elliptical, etc. - all that is just approximations safe at the timescales usually consider. [1] - Approximate to an arbitrary high degree. Which is all we can hope for in a physical reality anyways. [2] - Or hey, did you know that one of the best ways to multiply two large numbers is to... do a Fourier transform on their digits and summing them up in the frequency domain? (Incidentally, being analog, nature really likes working in the frequency domain; between that, accumulation as addition, and not being designed to be driven by a shared clock signal it should be obvious that natural computers ain't gonna look like ours, and conversely, we don't need to replicate natural processes exactly to get the same results.)