6 ms·
It's curse and a blessing that discussion of topics happens in so many different places. I found this comment on Twitter/X interesting: https://x.com/fchollet/s
by xnx 2y ago
It's curse and a blessing that discussion of topics happens in so many different places. I found this comment on Twitter/X interesting: https://x.com/fchollet/status/1841902521717293273 https://x.com/fchollet/status/1841902521717293273
"Interesting work on reviving RNNs. https://arxiv.org/abs/2410.01201 https://arxiv.org/abs/2410.01201 -- in general the fact that there are many recent architectures coming from different directions that roughly match Transformers is proof that architectures aren't fundamentally important in the curve-fitting paradigm (aka deep learning)
Curve-fitting is about embedding a dataset on a curve. The critical factor is the dataset, not the specific hard-coded bells and whistles that constrain the curve's shape. As long as your curve is sufficiently expressive all architectures will converge to the same performance in the large-data regime."
- deleted 2y ago[deleted]
- WithinReason 2y agoIf you spent some time actually training networks you know that's not true, that's why batch norm, dropout, regularization is so successful. They don't increase the network's capacity (parameter count) but they increase its ability to learn.
- islewis 2y ago> "As long as your curve is sufficiently expressive all architectures will converge to the same performance in the large-data regime." I haven't fully ingested the paper yet, but it looks like it's focused more on compute optimization than the size of the dataset: > ... and (2) are fully parallelizable during training (175x faster for a sequence of length 512 Even if many types of architectures converge to the same loss over time, finding the one that converges the fastest is quite valuable given the cost of running GPU's at scale.
- teruakohatu 2y ago> Even if many types of architectures converge to the same loss over time, finding the one that converges the fastest is quite valuable given the cost of running GPU's at scale. This! Not just fastest but with the lowest resources in total. Fully connected neural networks are universal functions. Technically we don’t need anything but a FNN, but memory requirements and speed would be abysmal far beyond the realm of practicality.
- actionfromafar 2y agoUnless we could build chips in 3D?
- foota 2y agoNot even then, a truly fully connected network would have super exponential runtime (it would take N^N time to evaluate)
- ivan_gammel 2y agoWe need quantum computing there. I remember seeing a recent article about quantum processes in the brain. If that’s true, QC may be the missing part.
- tsimionescu 2y agoThis is just word salad. There is no known quantum algorithm that can compute the result of a fully-connected neural network exponentially faster than classical computers can. QCs have a known exponential advantage over classical computers only for a very limited class of problems, mostly related to the Quantum Fourier Transform. Animal brains have little to nothing in common to artifical neural networks. There is no reason whatsoever to think that there is any relation between the complexity class of brain functions and ANN inference. And the hypothesized (and still wildly speculative) quantum behaviors happening in the animal brain are at the level of the behavior of individual neurons, not of the network connections between neurons. So even if there is some kind of quantum computation happening, it's happening in individual neurons, not at the network level, and that would only go to show even more that animal brains are profoundly different from ANNs.
- eru 2y agoCompare and contrast https://www.smbc-comics.com/comic/the-talk-3 https://www.smbc-comics.com/comic/the-talk-3 (Summary: quantum computing is unlikely to help.)
- mvkel 2y ago
- byearthithatius 2y ago> finding the one that converges the fastest is quite valuable given the cost of running GPU's at scale Not to him, he runs the ARC challenge. He wants a new approach entirely. Something capable of few-shot learning out of distribution patterns .... somehow
- acchow 2y agoWhat it will come down to is computational efficiencies. We don’t want to retrain once a month - we want to retrain continuously. We don’t want one agent talking to 5 LLMs. We want thousands of LLMs all working in concert.
- ActorNightly 2y agoThis and also the way models are trained has to be rethought. BPP is good for figuring out complex function mappings, but not for storing information.
- pbhjpbhj 2y agoSounds like something that has unsustainable energy costs.
- deleted 2y ago[deleted]
- fsndz 2y ago[flagged]
- _cbsz 2y agopaper is paywalled; just logging into Medium won't do it
- fsndz 2y agosorry for the paywall, you can read the free version here: https://www.lycee.ai/blog/why-no-agi-openai https://www.lycee.ai/blog/why-no-agi-openai
- xpl 2y agoI would like to read it, but it's under a paywall.
- alwa 2y agohttps://archive.is/nGaiU https://archive.is/nGaiU
- ahzhou 2y agoAuthor: @fandzomga Username: fsndz Why try to funnel us to your paywalled article?
- josh-sematic 2y agoOne reason why I'm excited about o1 is that it seems like OpenAI have cracked the nut of effective RL during training time, which takes us out of the domain of just fitting to the curve of "what a human would have said next." I just finished writing a couple blog posts about this; the first [1] covers some problems with that approach and the second [2] talks about what alternatives might look like. [1] https://www.airtrain.ai/blog/how-openai-o1-changes-the-llm-training-picture-part-1 https://www.airtrain.ai/blog/how-openai-o1-changes-the-llm-t... [2] https://www.airtrain.ai/blog/how-openai-o1-changes-the-llm-training-picture-part-2 https://www.airtrain.ai/blog/how-openai-o1-changes-the-llm-t...
- 2y ago
- Lerc 2y agoI remember one of the initial transformer people saying in an interview that they didn't think this was the "one true architecture" but a lot of the performance came from people rallying around it and pushing in the one direction. On the other hand, while "As long as your curve is sufficiently expressive all architectures will converge to the same performance in the large-data regime." is true, a sufficiently expressive mechanism may not be computationally or memory efficient. As both are constraints on what you can actually build, it's not whether the architecture can produce the result, but whether a feasible/practical instantiation of that architecture can produce the result.
- viktor_von 2y ago> I remember one of the initial transformer people saying in an interview that they didn't think this was the "one true architecture" but a lot of the performance came from people rallying around it and pushing in the one direction. You may be referring to Aidan Gomez (CEO of Cohere and contributor to the transformer architecture) during his Machine Learning Street Talk podcast interview. I agree, if as much attention had been put towards the RNN during the initial transformer hype, we may have very well seen these advancements earlier.
- ants_everywhere 2y ago> is proof that architectures aren't fundamentally important in the curve-fitting paradigm (aka deep learning) (Somewhat) fun and (somewhat) related fact: there's a whole cottage industry of "is all you need" papers https://arxiv.org/search/?query=%22is+all+you+need%22&searchtype=all&abstracts=show&order=-announced_date_first&size=50 https://arxiv.org/search/?query=%22is+all+you+need%22&search...
- tsimionescu 2y agoStarting of course with the classic paper from Lennon and McCartney, 1967.
- TaurenHunter 2y agoReminds me of the "Considered Harmful" articles: https://meyerweb.com/eric/comment/chech.html https://meyerweb.com/eric/comment/chech.html
- jprete 2y agoI wonder if there's something about tech culture - or tech people - that encourages them to really, really like snowclones.
- observationist 2y agoYes. Do stuff that other people have been successful doing. Monkey see, monkey do - it's not a tech people thing, it's a human thing. Tech just happens to be most on display at the moment - because tech people are building the tools and the parameters and the infrastructure handling all our interactions.
- fennecfoxy 2y agoNot sure why people are surprised about this when it's the modus operandi of all life on the planet. I could spam we are the stochastic parrots after all, yet one more time.
- 2y ago
- wongarsu 2y agoOne big thing that bells and whistles do is limit the training space. For example when CNNs took over computer vision that wasn't because they were doing something that dense networks couldn't do. It was because they removed a lot of edges that didn't really matter, allowing us to spend our training budget on deeper networks. Similarly transformers are great because they allow us to train gigantic networks somewhat efficiently. And this paper finds that if we make RNNs a lot faster to train they are actually pretty good. Training speed and efficiency remains the big bottleneck, not the actual expressiveness of the architecture
- nutanc 2y agoThis is true. This is the reason, in many of our experiments we find that using a new algorithm, KESieve, we actually find the planes much faster than the traditional deep learning training approaches. The premise is, a neaural network builds planes which separate the data and adjusts these planes through an iterative learning process. What if we can find a non iterative method which can draw these same planes. We have been trying this and so far we have been able to replace most network layers using this approach. haven't tried for transformers though yet. Some links if interested: [1] https://gpt3experiments.substack.com/p/understanding-neural-networks-and https://gpt3experiments.substack.com/p/understanding-neural-... [2] https://gpt3experiments.substack.com/p/building-a-vector-database-in-2gb https://gpt3experiments.substack.com/p/building-a-vector-dat...
- deleted 2y ago[deleted]
- dheera 2y agoI mean, transformer-based LLMs are RNNs, just really really really big ones with very wide inputs that maintain large amounts of context.
- immibis 2y agoNo. An RNN has an arbitrarily-long path from old inputs to new outputs, even if in practice it can't exploit that path. Transformers have fixed-size input windows.
- WithinReason 2y agono, you can give as much context to a transformer as you want, you just run out of memory
- immibis 2y agoAn RNN doesn't run out of memory from that, so they are still fundamentally different. How do you encode arbitrarily long positions, anyway?
- WithinReason 2y agoThey are different but transformers don't have fixed windows, you can extend the context or make it smaller. I think you can extend a positional encoding if it's not a learned encoding.
- famouswaffles 2y agoYou can't have a fixed state and have arbitrarily-long path from input. Well you can but then it's just meaningless because you fundamentally cannot keep stuffing information of arbitrary length into a fixed state. RNNs effectively have fixed-size input windows.
- immibis 2y agoThe path is arbitrarily long, not wide. It is possible for an RNN to be made that remembers the first word of the input, no longer how long the input is. This is not possible with a transformer, so we know they are fundamentally different.
- quantadev 2y agoMost LLMs aren't even using a "curve" yet at all, right? All they're using is a series of linear equations because the model weights are a simple multiply and add (i.e. basic NN Perceptron). Sure there's a squashing function on the output to keep it in a range from 0 to 1 but that's done BECAUSE we're just adding up stuff. I think probably future NNs will be maybe more adaptive than this perhaps where some Perceptrons use sine wave functions, or other kinds of math functions, beyond just linear "y=mx+b" It's astounding that we DID get the emergent intelligence from just doing this "curve fitting" onto "lines" rather than actual "curves".
- OkayPhysicist 2y agoThe "squashing function" necessarily is nonlinear in multilayer nueral networks. A single layer of a neural network can be quite simply written a weight matrix, times an input vector, equalling an output vector, like so Ax = y Adding another layer is just multiplying a different set of weights times the output of the first, so B(Ax)= y If you remember your linear algebra course, you might see the problem: that can be simplified (BA)x = y Cx = y Completely indistinguishable from a single layer, thus only capable of modeling linear relationships. To prevent this collapse, a non linear function must be introduced between each layer.
- quantadev 2y agoRight. All the squashing is doing is keeping the output of any neuron in a range of below 1. But the entire NN itself (Perceptron ones, which most LLMs are) is still completely using nothing but linearity to store all the knowledge from the training process. All the weights are just an 'm' in the basic line equation 'y=m*x+b'. The entire training process does nothing but adjust a bunch of slopes of a bunch of lines. It's totally linear. No non-linearity at all.
- nazgul17 2y agoThe non linearities are fundamental. Without them, any arbitrarily deep NN is equivalent to a shallow NN (easily computable, as GP was saying), and we know those can't even solve the XOR problem. > nothing but linearity No, if you have non linearities, the NN itself is not linear. The non linearities are not there primarily to keep the outputs in a given range, though that's important, too.
- sakras 2y agoI figured this was pretty obvious given that MLPs are universal function approximators. A giant MLP could achieve the same results as a transformer. The problem is the scale - we can’t train a big enough MLP. Transformers are a performance optimization, and that’s why they’re useful.
- ctur 2y agoArchitecture matters because while deep learning can conceivably fit a curve with a single, huge layer (in theory... Universal approximation theorem), the amount of compute and data needed to get there is prohibitive. Having a good architecture means the theoretical possibility of deep learning finding the right N dimensional curve becomes a practical reality. Another thing about the architecture is we inherently bias it with the way we structure the data. For instance, take a dataset of (car) traffic patterns. If you only track the date as a feature, you miss that some events follow not just the day-of-year pattern but also holiday patterns. You could learn this with deep learning with enough data, but if we bake it into the dataset, you can build a model on it _much_ simpler and faster. So, architecture matters. Data/feature representation matters.
- mr_toad 2y ago> can conceivably fit a curve with a single, huge layer I think you need a hidden layer. I’ve never seen a universal approximation theorem for a single layer network.
- dongecko 2y agoI second that thought. There is a pretty well cited paper from the late eighties called "Multilayer Feedforward Networks are Universal Approximators". It shows that a feedforward network with a single hidden layer containing a finite number of neurons can approximate any continuous function. For non continous function additional layers are needed.
- ted_dunning 2y agoMinsky and Papert showed that single layer perceptrons suffer from exponentially bad scaling to reach a certain accuracy for certain problems. Multi-layer substantially changes the scaling.
- drodgers 2y ago> The critical factor is the dataset, not the specific hard-coded bells and whistles that constrain the curve's shape I have almost the opposite take. We've had a lot of datasets for ages, but all the progress in the last decade has come from advances how curves are architected and fit to the dataset (including applying more computing power). Maybe there's some theoretical sense in which older models could have solved newer problems just as well if only we applied 1000000x the computing power, so the new models are 'just' an optimisation, but that's like dismissing the importance of complexity analysis in algorithm design, and thus insisting that bogosort and quicksort are equivalent. When you start layering in normalisation techniques to minimise overfitting, and especially once you start thinking about more agentic architectures (eg. Deep Q Learning, some of the search space design going into OpenAI's o1), then I don't think the just-an-optimisation perspective can hold much water at all - more computing power simply couldn't solve those problems with older architectures.
- tsimionescu 2y agoI think by far the biggest advances are related to compute power. The amount of processing needed to run training algorithms on the amounts of data needed for the latest models was just not possible even five years ago, and definitely not ten years ago. I'm sure there are optimizations from the model shape as well, but I don't think that running the best algorithms we have today with hardware from five-ten years ago would have worked in any reasonable amount of time/money.
- freeqaz 2y agoA 30bn param model, hell even a 7bn param model, is still incredibly useful and I feel like that could have been doable a decade ago! We have GPT-4 (or at least 3.5) tier performance in these much smaller models now. If we teleported back in time it may have been possible to build
- tsimionescu 2y agoI think the size of the model is only one part of it. They're still training these 7bn parameter models on the whole data set, and just crunching through that takes enormous compute, that people just didn't have at the current price points until now. I should also mention that the idea itself of using GPUs for compute and then specifically for AI training was an innovation. And the idea that simply scaling up was going to be worth the investment is another major innovation. It's not just the existence of the compute power, it's the application to NN training tasks that got us here. Here[0] is an older OpenAI post about this very topic. They estimate that between 2012 and 2018, the compute power used for training the SotA models at those times increased roughly 300,000 times, doubling every ~3.5 months. [0] https://openai.com/index/ai-and-compute/ https://openai.com/index/ai-and-compute/
- _giorgio_ 2y agoChollet is just a philosopher. He also thinks that keras and tensorflow are important, when nobody uses those. And he punished false days about their usage.
- eru 2y agoWell, you also need an approach to 'curve fitting' where it's actually computationally feasible to fit the curve. The approach of mixing layers of matrix multiplication with a simple non-linearity like max(0, x) (ReLU) works really well for that. Earlier on they tried more complicated non-linearities, like sigmoids, or you could try an arbitrary curve that's not split into layers at all, you would probably find it harder. (But I'm fairly sure in the end you might end up in the same place, just after lots more computation spent on fitting.)
- tippytippytango 2y agoInductive bias matters. A lot.
- avereveard 2y agowell yes but actually no I guess: the transformers benefit at the time was that they were more stable while learning, enabling larger and larger network and dataset to be learnt.