3 ms·
> For one, humans can integrate absolutely gargantuan amounts of information extremely efficiently. What we can integrate, we seem to integrate efficiently*; b
by ben_w 2y ago
> For one, humans can integrate absolutely gargantuan amounts of information extremely efficiently.
What we can integrate, we seem to integrate efficiently*; but compared to the quantities used to train AI, we humans may as well be literally vegetables.
* though people do argue about exactly how much input we get from vision etc., personally I doubt vision input is important to general human intelligence, because if it was then people born blind would have intellectual development difficulties that I've never heard suggested exist — David Blunket's success says human intelligence isn't just fine-tuning on top of a massive vision-grounded model.
- Retric 2y agoHearing is also well into the terabytes worth of information per year. Add in touch, taste, smell, proprioception, etc and the brain gets a deluge. The difference is we’re really focused on moving around in 3D space and more abstract work, where an LLM etc is optimized for a very narrow domain.
- jdietrich 2y ago>Hearing is also well into the terabytes worth of information per year. If we assume that the human auditory system is equivalent to uncompressed digital recording, sure. Actual neural coding is much more efficient, so the amount of data that is meaningfully processed after multiple stages of filtering and compression is plausibly on the order of tens of gigabytes per year; the amount actually retained is plausibly in the tens of megabytes. Don't get me wrong, the human brain is hugely impressive, but we're heavily reliant on very lossy sensory mechanisms. A few rounds of Kim's Game will powerfully reveal just how much of what we perceive is instantly discarded, even when we're paying close attention.
- Retric 2y agoThe sensory information form individual hairs in the ear start off with a lot more data to process than simple digital encoding of two audio streams. Neural encoding isn’t particularly efficient from a pure data standpoint just an energy standpoint. A given neuron not firing is information and those nerve bundles contain a lot of neurons.
- PaulDavisThe1st 2y ago"That's a coyote" I say, based on auditory evidence. Pretty damn efficient coding.
- hammock 2y ago> Hearing is also well into the terabytes worth of information per year. Add in touch, taste, smell, proprioception, etc and the brain gets a deluge Is that supposed to be a lot? Only a small fraction of that is committed to permanent storage. A random server is today processing anywhere from tens of terabytes to hundreds of petabytes annually
- Retric 2y agoLow level details like that aren’t relevant to this discussion. Most human processing power is at the cellular level. The amount of processing power in a single finger literally dwarfs a modern data center, but we can’t leverage that to think only live. So it’s not a question of ‘a lot’ it’s a question of orders of magnitude vs “the quantities used to train AI” Library of congress has what 39 million books, tokenize every single one and you’re talking terabytes of training data for an LLM. We can toss blog posts etc to that pile but every word ever written by a person isn’t 20 orders of magnitude larger or anything.
- gf000 2y agoIs that a positive thing? If anything I would consider that as the reverse - LLMs have the "intelligence of vegetables" because even with literally the whole of human written knowledge they can at most regurgitate that back to us with no novelty whatsoever, even though a 2 years old with a not even matured brain can learn a human language from orderS of magnitude less and lower quality input from a couple of people only. But any Nobel-price winner has read significantly less than a basic LLM, and we see no LLM doing any tiny scientific achievement, let alone that high impact ones.
- ben_w 2y ago> Is that a positive thing? Neither. Both. Depends what you want to measure. It's perfectly legit to call these models "thick" because they *need* to read such a vast quantity of text that a human would literally spend two thousand lifetimes to go through it even if that was all the human did with their days. It also remains the case that, unlike us, they can go through all of that in a few months. > with no novelty whatsoever, even though a 2 years old with a not even matured brain can learn a human language from orderS of magnitude less and lower quality input from a couple of people only. You're either grossly underestimating AI or overestimating 2 year olds, possibly both. I just about remember being a toddler, somewhere between then and 5 was around the age I had the idea that everyone got an invisible extra brain floating next to them for every year they lived. Took me an embarrassingly long time (teens, IIRC) to realise that the witch-duck-weight-comparison scene in Monty Python and the Holy Grail wasn't a documentary, thanks to the part of the film captioned "Famous Historian". One time my dad fell ill, and he was talking to mum about "the tissue being damaged" while I was present, so I gave him a handkerchief (AKA "a tissue"). And while I don't remember this directly, my mum's anecdotes include me saying "fetrol fump", waving a spoon in a jam pan and calling this act "spelling", and when discovered running around with my pockets inside-out explaining myself as trying to fly because I apparently thought that the lining of a pocket was called a "wing". When it comes to human novelty, I also quite often find there's a lot of remixing going on that just isn't immediately apparent. As Steve Jobs apparently once said, “Good artists copy; great artists steal.”, except Jobs stole that quote from Picasso. It's easy to categorise different levels with AI, but which one of these counts as "novelty", and how often do humans ever achieve each of these grades? 0. Memorisation of the training set. Think: bunch of pictures, pick best fit. 1. Linear interpolation between any pair of elements in the training set. Think: simple cross-fade between any two pictures, but no tracking or distorting of features during that fade. 2. Let the training set form a basis vector space, and interpolate freely within the constraints of the examples. Think: if these pictures are faces, it would make any hair colour between the most extreme limits shown, etc. 3. Extrapolate beyond the examples. Think: Even if no black or white hair was visible, so long as several shades of grey were, it could reach the ideas of black or white hair. 4. Invent a new vector. Think: even if it had been trained only on black-and-white images, it could still invent green hair. > But any Nobel-price winner has read significantly less than a basic LLM, and we see no LLM doing any tiny scientific achievement, let alone that high impact ones. We do see them doing *tiny* scientific achievements, with extra emphasis on "tiny". Just like with using them in software, even the best "only" act like fresh graduates. When any AI gets to high-impact… the following (fictional) quote comes to mind: "as soon as we started thinking for you, it really became our civilization."