3 ms·

The breakthrough isn't the matrix multiplication you'd learn in high school math. The breakthroughs are the likes of transformers, deep learning, etc.. In short

The breakthrough isn't the matrix multiplication you'd learn in high school math. The breakthroughs are the likes of transformers, deep learning, etc.. In short the stuff that happens in a research lab, not what is thrown at you with flashy confetti by sales people.

roenxi 1y ago
I've read the transformers paper, it is still pretty simple math. And the architecture isn't particularly interesting from a theoretical sense, the concepts are basic - we're not talking Gauss, Newton or Euler here, we're talking 2-10% of software engineers [0]. The limit is most software engineers don't have enough hardware to test whether their ideas work or not. And it is actually the same matrix multiplication people might learn at high school. The standard definition of matrix multiplication is generally accepted. [0] We'd expect the figuring-things-out process to be faster for more cluey people, so the people who actually pick all this stuff up first probably are geniuses. But a more normal engineer with the hardware access would still figure it out independently sooner or later. It is just too neatly on the beaten path where we'd expect to be finding optimisations.
- _Algernon_ 1y ago
  My point is that a discovery always seems simple and inevitable once it has happened. Newton's laws are pretty simple, all things considered, but it still took most of human history for those to be discovered. An average high school student doing physics today could derive Newton's laws. That doesn't mean that that student put in the same situation as Newton, would have been capable of discovering newton's laws.
  - roenxi 1y ago
    Newton had to invent calculus before his laws made sense. There isn't anything equivalent to that in modern machine learning. It uses concepts that are centuries old except for maybe a few stats techniques that might be around WWII era since there was a big stats push around then. Plus back-propagation which is a pretty big deal, but it is hardly new to the boom we're seeing right now. It is an idea that was sitting around for a while before hardware changes made it effective.
  - amelius 1y ago
    But "invention" here is really different. They are basically trying out stuff and seeing what works, but they have no idea how it actually works. We're essentially watching a bunch of investor-backed alchemists here.

roenxi 1y ago

I've read the transformers paper, it is still pretty simple math. And the architecture isn't particularly interesting from a theoretical sense, the concepts are basic - we're not talking Gauss, Newton or Euler here, we're talking 2-10% of software engineers [0]. The limit is most software engineers don't have enough hardware to test whether their ideas work or not. And it is actually the same matrix multiplication people might learn at high school. The standard definition of matrix multiplication is generally accepted. [0] We'd expect the figuring-things-out process to be faster for more cluey people, so the people who actually pick all this stuff up first probably are geniuses. But a more normal engineer with the hardware access would still figure it out independently sooner or later. It is just too neatly on the beaten path where we'd expect to be finding optimisations.

_Algernon_ 1y ago
My point is that a discovery always seems simple and inevitable once it has happened. Newton's laws are pretty simple, all things considered, but it still took most of human history for those to be discovered. An average high school student doing physics today could derive Newton's laws. That doesn't mean that that student put in the same situation as Newton, would have been capable of discovering newton's laws.
- roenxi 1y ago
  Newton had to invent calculus before his laws made sense. There isn't anything equivalent to that in modern machine learning. It uses concepts that are centuries old except for maybe a few stats techniques that might be around WWII era since there was a big stats push around then. Plus back-propagation which is a pretty big deal, but it is hardly new to the boom we're seeing right now. It is an idea that was sitting around for a while before hardware changes made it effective.
- amelius 1y ago
  But "invention" here is really different. They are basically trying out stuff and seeing what works, but they have no idea how it actually works. We're essentially watching a bunch of investor-backed alchemists here.