6 ms·
My theory is that architecture doesn't matter - convolutional, transformer or recurrent, as long as you can efficiently train models of the same size, what coun
My theory is that architecture doesn't matter - convolutional, transformer or recurrent, as long as you can efficiently train models of the same size, what counts is the dataset.
Similarly, humans achieve about the same results when they have the same training. Small variations. What matters is not the brain but the education they get.
Of course I am exaggerating a bit, just saying there are a multitude of architectures of brain and neural nets with similar abilities, and the differentiating factor is the data not the model.
For years we have seen hundreds of papers trying to propose sub-quadratic attention. They all failed to get traction, big labs still use almost vanilla transformer. At some point a paper declared "mixing is all you need" (MLP-Mixers) to replace "attention is all you need". Just mixing, the optimiser adapts to what it gets.
If you think about it, maybe language creates a virtual layer where language operations are performed. And this works similarly in humans and AIs. That's why the architecture doesn't matter, because it is running the language-OS on top. Similarly for vision.
I place 90% the merits of AI on language and 10% on the model architecture. Finding intelligence was inevitable, it was hiding in language, that's how we get to be intelligent as well. A human raised without language is even worse than a primitive. Intelligence is encoded in software, not hardware. Our language software has more breadth and depth than any one of us can create or contain.
Dataset counts, but also the number of total parameters in the network, i.e. capacity.
Agreed, it's in my first phrase "as long as you can efficiently train models of the same size, what counts is the dataset". But useful sizes are just a few. 7, 13, 35, 70, 120B - because they are targeted to various families of GPUs. A 2T model I can't run or too expensive to use on APIs is of no use. Not just dataset size, but data quality matters just as much, and diversity.
I believe LLMs will train mostly on synthetic data engineered to have extreme diversity and very high quality. This kind of data confers 5x gains in efficiency as demonstrated by Microsoft in the Phi-1.5 paper.
There are really 3 axes - the number of tokens, the number of parameters, and the number of flops per pass.
deleted 3y ago
[deleted]
I wish someone performed a large scale experiment to evaluate all these alternate architectures. I kind of feel that they get drowned out by new sota results from openai and others. What I wish is something that tries to see if emergent behaviors pop up with enough data and parameters.
Maybe vision is special enough that convnets and approache transformer level performance or it could be generalized to any modality. I haven't read enough papers to know if someone has already done something like this but everywhere I look on the application side of things, vanilla transformers seems to be dominating
> Similarly, humans achieve about the same results when they have the same training. Small variations. What matters is not the brain but the education they get.
That's misleading. Small average variation overall, but some outliers are dramatically better and dramatically worse. Some brains produce fields medals, some can barely count change. Unfortunate but true.
Yes but they all stood on the shoulders of giants. Alone they couldn't have done it. Intelligence is a collective affair.
Ideas are self replicators - usually by being useful to people, but their lifecycle is different from that of people, they evolve much faster. It's an evolutionary process. Modern man and that of 10,000 years ago are biologically not much different, the difference comes from cultural evolution.
> Ideas are self replicators
Maybe it's worth pointing out: That's a core idea of memetics, i.e. the study of memes, in the original sense of that word.
> Alone they couldn't have done it. Intelligence is a collective affair.
Yes, but the Gausses and Ramanujans of the world disprove the claim that architecture doesn't matter, and that only training data matters. The long tails that describe human variance I think show that architecture can matter a lot.
I think you might be right that architecture for some machine learning algorithms doesn't matter so much given enough data, but the fact that they need so much data to perform well I think implies that there's a better architecture awaiting discovery.
We showed this in our original scaling law paper:
https://arxiv.org/pdf/1712.00409.pdf https://arxiv.org/pdf/1712.00409.pdf
The slope of the power law is determined by the problem and dataset. Compute, parameter count, and data move you along the curve. Change in architecture/bias is a constant offset.
So architecture can give an advantage, but that advantage can be overcome by scale.
I kind of agree, architecture does seem to matter in broad strokes but neuron-local learning mechanics will take it from there and the rest is up to the dataset.
> same results when they have the same training
Genetics may play a huge role.
Part of what made the transformer so powerful when compared to RNNs and CNNs in NLP was the fact that they are highly parallelizable. That means you can optimize more parameters in the same amount of training time, which means you can model a more complex function.
> My theory is that architecture doesn't matter - convolutional, transformer or recurrent, as long as you can efficiently train models of the same size, what counts is the dataset
I assume you mean architecture doesn’t matter much, as long as it’s roughly sensibly designed.
The basics of architecture do matter though: a two-layer fully connected network won’t get very far no matter how many parameters it has. Architectural improvements like ResNets have definitely made a difference to the field.
deleted 3y ago
[deleted]
If by "basic architecture", you mean auditory, visual, and motor centres, sure, but we're clearly talking about higher cognitive abilities here. I'm not at all sure that they have the same architecture given Ramanujan had even less high school math than most American students, yet clearly took the same knowledge much further.
If you're saying Ramanujan had a drastically different brain architecture than other humans, it's an extraordinary claim which will require extraordinary evidence. Einstein's brain is well studied, and fundamentally human...
Not to defend the original argument (because comparing the difference between people, belonging to the exact same animal specie, and difference between radically different model architecture make zero sense).
> Small average variation overall, but some outliers are dramatically better and dramatically worse. Some brains produce fields medals, some can barely count change. Unfortunate but true.
We don't actually know which part of the “can barely count change” situation comes from the “brain” compared to the situation people were raises in. Sure you'd always have outliers (good or bad) for purely genetic reasons, but if you looked at sport performance for instance you'd find that the theoretical difference between people is much lower than the practical one for many environmental reasons: in theory, once you factor in rare medical conditions, almost every human between 18 and 40 should be able to run a marathon. The reason why most people cannot is entirely environmental (not enough physical activity, overweight, work related health issues, etc.).
And it's also true in terms of performance: almost any human ought to be able to run a 100m in less than 14 seconds, which is only 30% slower than the world record. Yet, for purely environmental reasons, only a small fraction of the population is actually able to do it.
I suspect that in terms of intelligence the results would be roughly in the same ballpark: almost everyone, except rare mental development disorders, ought to be 70% as intelligent as the most intelligent person in history (and the enormous majority fitting in a much narrower band), but most end up far worse for environmental reasons alone.
For some reason (mostly politico-philosophical), most people in the “nature” camp of the “nature vs nurture” debate are completely oblivious to the evidence for this.
> We don't actually know which part of the “can barely count change” situation comes from the “brain” compared to the situation people were raises i
I think this too is misleading. We've quantified the heritability of intelligence to quite a significant degree. Twin studies of twins who have been raised apart even shows that outcomes are largely the same despite different environments. Nurture is very much oversold. Basically, as long as you don't traumatize one of two twins they will develop very similarly.
> And it's also true in terms of performance: almost any human ought to be able to run a 100m in less than 14 seconds
But it's literally not true of all humans, and it won't be true if you drop that below 13 or 12 seconds. Which shows that the specific biomechanics you're born with will influence your natural physical abilities. By extension, we also have natural cognitive abilities with similar consequences.
> most people in the “nature” camp of the “nature vs nurture” debate are completely oblivious to the evidence for this
I think you're way overselling this claim. "It's all nurture" is a far more common opinion, and far less evidence supports it.
> We've quantified the heritability of intelligence to quite a significant degree
Not really. I can't find the link again but there was a survey of cognitive science experts, and the question of inheritability of intelligence is still an open question with lots of conflicting opinions in the academic community. Also, “heritability” doesn't mean what you think it means.
> Twin studies of twins who have been raised apart even shows that outcomes are largely the same despite different environments.
I'd love to know your source, because I doubt the number of “twins that have been raised apart ” is large enough to be a statically significant cohort even if you could have them all in your study.
> But it's literally not true of all humans
Yes, some people are born without legs. And some are born with Down syndrome, nobody disputes that.
> and it won't be true if you drop that below 13 or 12 seconds. Which shows that the specific biomechanics you're born with will influence your natural physical abilities. By extension, we also have natural cognitive abilities with similar consequences.
Nobody questions that either (although 13 secs is probably close to what's reachable by anyone, and 12 by any male), the question is about scale: if you take a random sample of 20 Americans of the right age, you'd have insane disparities, with a significant number not being able to run the distance at all and struggling to finish in less than a minute, despite their completely functional genome.
> I think you're way overselling this claim. "It's all nurture" is a far more common opinion, and far less evidence supports it.
That's a strawman, a very common one in the mouth of the folks of the ”nature” camp. Nobody argues that it's literally “all nurture”, everyone know that Down syndrome exist.
The actual argument is that when in my previous 100m example you have genetics variation with an impact of at best 30%, and a observed variation of more than 500%, it's fair to say nurture is the dominant factor.
Doesn't mean nature has no role at all, but it doesn't have an explanatory role: the reason for observed variation is essentially nurture.
Stop taking The Bell Curve seriously, this book has been written in bad faith by people with a political agenda (which even stated with a straight face in the book). The reason why these people argue for the genetic impact trumping everything is because they don't like the idea of state intervention to reduce poverty. When you say “these people are poor because they are genetically dumb” then you can claim there's no point in helping them, and without surprise that's exactly what Herrnstein and Murray are advocating in their book…
> I'd love to know your source, because I doubt the number of “twins that have been raised apart ” is large enough to be a statically significant cohort even if you could have them all in your study.
This, among others:
https://pubmed.ncbi.nlm.nih.gov/2218526/ https://pubmed.ncbi.nlm.nih.gov/2218526/
> Yes, some people are born without legs. And some are born with Down syndrome, nobody disputes that.
I see you haven't debated many blank slaters. The original post I responded to said, "What matters is not the brain but the education they get". That is literally false.
> if you take a random sample of 20 Americans of the right age, you'd have insane disparities, with a significant number not being able to run the distance at all and struggling to finish in less than a minute, despite their completely functional genome.
Yes, and you will find cases of people who ate basically the same diet and were just as active or sedentary who fall on both sides of that line of finishing/not finishing. Some people will just have a more efficient metabolism, or a more robust skeletal structure, or greater lung capacity, or more type II muscle fibers, and the positive confluence of all of these factors makes a great natural athlete, and a negative confluence on all of them makes for an unlucky individual who will likely never be athletic no matter how hard they try. Which doesn't reduce the importance of being fit, or reduce their human worth, but there's no point denying the obvious.
All of these factors fall within natural human variability, and all of these factors are influenced by environment of course, because genetics and environment can't be trivially separated, but the point is that it seems like these factors have multiplicative effects. Such factors exist for cognition as well, and the same conclusions follow. This counts as architecture.
> The actual argument is that when in my previous 100m example you have genetics variation with an impact of at best 30%, and a observed variation of more than 500%, it's fair to say nurture is the dominant factor.
Nurture is probably the dominant factor in almost all cases throughout the world because of malnutrition. It remains the case that even if you eliminate all environmental disparities, humans will still exhibit considerable variability on all traits. Some will still be natural gifted athletes and naturally gifted mathematicians, and some significant fraction will simply be below average on many categories, and some unlucky few will be below average on all categories, and not due to any obvious deficiency like Down's.
I'm not even going to bother addressing the irrelevant comments on Murray and his book, mostly because I haven't even read it. This is all standard statistics and genetics.
The original matter I disputed was literally false, as I said above, but the spirit of the comment was that data matters more than architecture. I think that's a very premature conclusion. It would be more correct to say that a) you can equate outcomes by varying data despite the architecture, but b) some architectures scale better (do more with less data), and I would add c) no known architecture scales as well as human cognition yet, which suggests better architectures remain to be found, and this suggests d) that architecture does matter.