4 ms·

We are currently mining just about all the internet data that's available. We are heading towards a limit and the AIs aren't getting much better. The entire re

by screye 4y ago

We are currently mining just about all the internet data that's available. We are heading towards a limit and the AIs aren't getting much better. The entire realm of video is under-explored. Think about the amount of content that lives in video. Image + text is already being solved, so video isn't the biggest leap. Embodied learning is underexplored. Constant surveillance is underexplored. There's a limit to the processing power that can be used to assemble the LLM's and the more that's used the more it will cost If the scaling law papers have shown us anything, it is that that the models don't need to get much bigger. More data is enough for now. People will guard their data more and will be less willing to share it. Fair. Though, companies might be able to prisoner's dilemma FOMO their way into everyone's data. was defined decades ago The core ideas around self-attention came about around 2015-2017. The ideas are as new as new ideas get. It's like saying that the ideas for the invention of calculus existed for decades before Newton because we could compute the area & volume of things. Yes, progress is incremental. There are new ideas out there, and we'll inevitably find something new in 20 years that some sad-phd is working on today, all while regretting not working on LLMs themselves. interesting times are ahead Yep

mnky9800n 4y ago
also most scientific data lives in silos somewhere unknown. i went to a really interesting talk last year where a scientist in uzbekistan saved a closet full of notebooks from old professors at his university. They had meticulously catalogued hydrology data, photos of glaciers, etc. and suddenly there was daily data going back to basically the formation of the soviet union for a region that had never had much data before. the department secretaries were going to burn all this as trash. stuff like this exists everywhere including on the internet.
- gcr 4y ago
  EXACTLY! there's so much research and citizen science being done, and even the field of CS is no good at identifying, preserving, and disseminating the good ideas that our lossy proxies miss. (proxies like "university prestige" or "went to grad school" or "was accepted at top-tier conference")
gcr 4y ago
I think OP was talking more about upending the universal dogma of "statistical optimization to minimize feedforward loss functions via gradient descent." We now have plenty more tricks of the trade (ReLUs were a big one, self-attention was another, making things convolutional was a third, and the smaller ones like dropout/batchnorm/LR schedules). However, in broad strokes, this paradigm remains relatively unchanged since LeCunn+Bengio+Hinton in the 80s. The fundamental idea informally stretches back to Rosenblatt's perceptrons in the 50s, which itself was built on McCulloch and Pitts' modeling of neurons as multiply-accumulate units from 1943. We've spent lots of time questioning these assumptions (e.g. branching off into Boltzmann machines or spiking neurons, which are really fascinating), but to my knowledge, there haven't been promising alternatives yet. We have "something that appears to work" but is this itself a local minimum? Biology found its own efficient processes that look a bit different after all. In my opinion, "fundamental breakthroughs" would answer questions like: - is there an inherent reason that humans learn through a process akin to statistical optimization, or is it a coincidence that feedforward-only gradient descent seems to work? - Are there other models for "neurons" that go beyond GEMM, like spikey activations? We know from biology that they're nonlinear; how might they be modeled better? - Are there ways of training effective agents beyond just the reinforcement Q-learning techniques that we've settled on? - What does it even mean for a model to be able to "reason;" does our current view of recursive models with hidden state really map to that philosophy one-to-one or is there more?
- adammarples 4y ago
  But he has a very good point. Transformers are such a quantum leap over anns, it's almost a whole new paradigm, Karpathy calls them a brand new way to do computation
  - gcr 4y ago
    Right, but transformers are artificial neural networks though, so it's not clear to me what you mean. The leap from RNNs to the Transformer architecture in 2017ish is similar to vision networks' leap from fully-connected layers to convolutional layers in 1987ish. In both cases, the key is to change the architecture to exploit some implicit structure in the data that wasn't quite used effectively before. For images, each pixel is strongly related to its close neighbors, so convolution is a natural way to simplify models while capturing that locality. For language, the opposite is true - almost any part of the text implicitly reference anything that's been said previously, so you need representations that are somewhat position-independent. Approaches that can model that structure naturally work better than an RNN munching tokens one by one. Each one was certainly a leap forward that unlocked its respective field, but at the end of the day, it's "just" an architecture tweak, you know?

by screye 4y ago

mnky9800n 4y ago

also most scientific data lives in silos somewhere unknown. i went to a really interesting talk last year where a scientist in uzbekistan saved a closet full of notebooks from old professors at his university. They had meticulously catalogued hydrology data, photos of glaciers, etc. and suddenly there was daily data going back to basically the formation of the soviet union for a region that had never had much data before. the department secretaries were going to burn all this as trash. stuff like this exists everywhere including on the internet.

gcr 4y ago
EXACTLY! there's so much research and citizen science being done, and even the field of CS is no good at identifying, preserving, and disseminating the good ideas that our lossy proxies miss. (proxies like "university prestige" or "went to grad school" or "was accepted at top-tier conference")

gcr 4y ago

I think OP was talking more about upending the universal dogma of "statistical optimization to minimize feedforward loss functions via gradient descent." We now have plenty more tricks of the trade (ReLUs were a big one, self-attention was another, making things convolutional was a third, and the smaller ones like dropout/batchnorm/LR schedules). However, in broad strokes, this paradigm remains relatively unchanged since LeCunn+Bengio+Hinton in the 80s. The fundamental idea informally stretches back to Rosenblatt's perceptrons in the 50s, which itself was built on McCulloch and Pitts' modeling of neurons as multiply-accumulate units from 1943. We've spent lots of time questioning these assumptions (e.g. branching off into Boltzmann machines or spiking neurons, which are really fascinating), but to my knowledge, there haven't been promising alternatives yet. We have "something that appears to work" but is this itself a local minimum? Biology found its own efficient processes that look a bit different after all. In my opinion, "fundamental breakthroughs" would answer questions like: - is there an inherent reason that humans learn through a process akin to statistical optimization, or is it a coincidence that feedforward-only gradient descent seems to work? - Are there other models for "neurons" that go beyond GEMM, like spikey activations? We know from biology that they're nonlinear; how might they be modeled better? - Are there ways of training effective agents beyond just the reinforcement Q-learning techniques that we've settled on? - What does it even mean for a model to be able to "reason;" does our current view of recursive models with hidden state really map to that philosophy one-to-one or is there more?

adammarples 4y ago
But he has a very good point. Transformers are such a quantum leap over anns, it's almost a whole new paradigm, Karpathy calls them a brand new way to do computation
- gcr 4y ago
  Right, but transformers are artificial neural networks though, so it's not clear to me what you mean. The leap from RNNs to the Transformer architecture in 2017ish is similar to vision networks' leap from fully-connected layers to convolutional layers in 1987ish. In both cases, the key is to change the architecture to exploit some implicit structure in the data that wasn't quite used effectively before. For images, each pixel is strongly related to its close neighbors, so convolution is a natural way to simplify models while capturing that locality. For language, the opposite is true - almost any part of the text implicitly reference anything that's been said previously, so you need representations that are somewhat position-independent. Approaches that can model that structure naturally work better than an RNN munching tokens one by one. Each one was certainly a leap forward that unlocked its respective field, but at the end of the day, it's "just" an architecture tweak, you know?