3 ms·
I think your question is backwards. Why should Google have some especially preferential advantage over any other company? The answer to your question, as you
by gumby 3y ago
I think your question is backwards. Why should Google have some especially preferential advantage over any other company?
The answer to your question, as you asked, is straightforward: they do have a lot of smart people and lots of money and computing resources, but they have exhibited serious structural problems moving technology forward since the departure of Schmidt in 2011. It is painfully obvious that de facto they haven't had CEO leadership since then. People can and do develop fully functioning example systems, and of course demos, but they then peter out. Gianndrea was pusing them forward on the AI front but after he left it feels to me like the impetus was not replaced.
But I don't think that's the real question. The real question is: what are the core functions needed to build leading LLMs, especially generative transformers. In these early days the key factor has been money for cycles. Personally I expect that advantage to diminish over the next few years -- IMHO it's one of those "with enough thrust you can get anything airborn" situations. There are a lot of smart opportunities to do more with less -- too much of the engineering is going into wrangling these hige systems but I see more effort going into wrangling what computation is done in the first place. Google, OpenAI et al are not preferentially positioned for such a transition.
I could be wrong: after all the human brain has 80 giganodes with 10^5 fanout. But on the other hand it only runs at less than 100 Hz.
- dharmab 3y agoIndividual primate neurons fire as frequently as >600Hz [1] and they're not synced on the same "clock". It's a poor analogue overall. 1: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5067378/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5067378/
- quietbritishjim 3y agoI'm not sure if this is exactly what that paper is referring to, but a real neutron firing is not like an artificial one firing. An artificial one always fires on a fixed clock, as you say, but some numeric output says how much it is firing. With a real neuron, it either fires or it doesn't, there's no numeric intensity. But how often it fires says how intense the response is. So that does agree with your overall statement that they're not very good analogues... At the very least, their mechanism is quite different. But it does mean that the 600Hz figure is quite misleading, because it's only after a period somewhat longer than 1/600th of a second that you can start to understand the intensity of the output.
- candiodari 3y agoThe average firing rate of a neocortex neuron (white or grey matter, non-specialized neurons), is only in the order of 1 Hz (for ease of remembering, in practice it is between 0.1 and 2 Hz, the further from the brainbridge a neuron is, generally speaking the slow it fires. With exceptions. Obviously, for example, this doesn't apply to the visual cortex). https://aiimpacts.org/rate-of-neuron-firing/ https://aiimpacts.org/rate-of-neuron-firing/ The very high firing rate neurons seem to be fixed function generators that don't actually process information, and some appear to have a function similar to the positional encoding in transformers, others encode the current "chemical" state of for example hormones, but there's many we don't know the function of.
- firecall 3y agoThe point about Eric Schmidt’s leadership seems to be lost in conversations about Google! The Google we all loved was Schmidt’s Google. Schmidt built the culture and set the tone. He is to my mind one of the greatest tech CEOs of all time in that he did exactly what he was hired to do beyond anyone’s expectations. Unlike Scully for instance who all but destroyed Apple. I’d love to see more validation of that claim from ex-Googlers, as I’m not on SC and have never worked at Google.