5 ms·
> "As long as your curve is sufficiently expressive all architectures will converge to the same performance in the large-data regime." I haven't fully ingested
by islewis 2y ago
> "As long as your curve is sufficiently expressive all architectures will converge to the same performance in the large-data regime."
I haven't fully ingested the paper yet, but it looks like it's focused more on compute optimization than the size of the dataset:
> ... and (2) are fully parallelizable during training (175x faster for a sequence of length 512
Even if many types of architectures converge to the same loss over time, finding the one that converges the fastest is quite valuable given the cost of running GPU's at scale.
- teruakohatu 2y ago> Even if many types of architectures converge to the same loss over time, finding the one that converges the fastest is quite valuable given the cost of running GPU's at scale. This! Not just fastest but with the lowest resources in total. Fully connected neural networks are universal functions. Technically we don’t need anything but a FNN, but memory requirements and speed would be abysmal far beyond the realm of practicality.
- actionfromafar 2y agoUnless we could build chips in 3D?
- foota 2y agoNot even then, a truly fully connected network would have super exponential runtime (it would take N^N time to evaluate)
- ivan_gammel 2y agoWe need quantum computing there. I remember seeing a recent article about quantum processes in the brain. If that’s true, QC may be the missing part.
- tsimionescu 2y agoThis is just word salad. There is no known quantum algorithm that can compute the result of a fully-connected neural network exponentially faster than classical computers can. QCs have a known exponential advantage over classical computers only for a very limited class of problems, mostly related to the Quantum Fourier Transform. Animal brains have little to nothing in common to artifical neural networks. There is no reason whatsoever to think that there is any relation between the complexity class of brain functions and ANN inference. And the hypothesized (and still wildly speculative) quantum behaviors happening in the animal brain are at the level of the behavior of individual neurons, not of the network connections between neurons. So even if there is some kind of quantum computation happening, it's happening in individual neurons, not at the network level, and that would only go to show even more that animal brains are profoundly different from ANNs.
- eru 2y agoCompare and contrast https://www.smbc-comics.com/comic/the-talk-3 https://www.smbc-comics.com/comic/the-talk-3 (Summary: quantum computing is unlikely to help.)
- mvkel 2y agoWetware is the future.
- fennecfoxy 2y agoCan't wait to see this defiantly spray painted across a torn up brick wall while computronium brained super intelligences slowly disassemble our planet to make paperclips.
- mvkel 2y agohttps://imgur.com/zeBkh2P https://imgur.com/zeBkh2P
- ComputerGuru 2y agoHeat extraction.
- bob1029 2y agoWe are already doing this.
- byearthithatius 2y ago> finding the one that converges the fastest is quite valuable given the cost of running GPU's at scale Not to him, he runs the ARC challenge. He wants a new approach entirely. Something capable of few-shot learning out of distribution patterns .... somehow