3 ms·

>where we all know that mathematics and/or CS deserve the honor Or semiconductor manufacturers. All the math and CS needed for AI can fit on a napkin, and had

>where we all know that mathematics and/or CS deserve the honor Or semiconductor manufacturers. All the math and CS needed for AI can fit on a napkin, and had been known for 200+ years. It's the extreme scaling enabled by semiconductor science that really makes the difference.

jampekka 2y ago
TBF backpropagation was introduced only in the 1970's, although in hindsight it's a quite trivial application of the chain rule. There were also plenty of "hacks" involved to make the networks scale such as dropout regularization, batch normalization, semi-linear activation functions (e.g. ReLU) and adaptive stochastic gradient descent methods. The maths for basic NNs is really simple but the practice of them is really messy.
- miven 2y ago
  Residual connections are also worth mentioning as an extremely ubiquitous adaptation, one will be hard-pressed to find a modern architecture that doesn't use those at least to some extent, to the point where the original Resnet paper sits at over 200k citations according to google scholar[1]. [1] https://scholar.google.com/citations?view_op=view_citation&hl=en&user=DhtAFkwAAAAJ&citation_for_view=DhtAFkwAAAAJ:ALROH1vI_8AC https://scholar.google.com/citations?view_op=view_citation&h...
  - cma 2y ago
    Highway nets introduced them in the 90s
programjames 2y ago
> All the math and CS needed for AI can fit on a napkin, and had been known for 200+ years. This isn't really true. If you read a physics textbook from the early 1900s, they didn't really have multivariate calculus and linear algebra expressed as concisely as we do now. It would take several napkins. Plus, statistical mechanics was quite rudimentary, which is important for probability theory.
- 77pt77 2y ago
  > If you read a physics textbook from the early 1900s, they didn't really have multivariate calculus and linear algebra expressed as concisely as we do now. This is completely incorrect.
cubefox 2y ago
That's absurd. The computer science needed for AI has not been known for 200 years. For example, transformers were only invented in 2017, diffusion models in 2015. (When the required math was invented is a different question, but I doubt all of it was known 200 years ago.)
zmgsabst 2y ago
I don’t think calculus existed at sufficient rigor 200 years ago. Computer science wasn’t even a thing 100 years ago.
- KeplerBoy 2y ago
  Calculus has been around for quite some time. If Newton had the machinery to fit large models to data, he would have done so. No doubt.
  - zmgsabst 2y ago
    Cauchy’s main work was under 200 years ago; and there’s been quite a lot of work since. Again, I’m unsure that calculus existed at sufficient level 200 years ago — it didn’t appear in modern form from either Leibniz or Newton.
deleted 2y ago
[deleted]

jampekka 2y ago

TBF backpropagation was introduced only in the 1970's, although in hindsight it's a quite trivial application of the chain rule. There were also plenty of "hacks" involved to make the networks scale such as dropout regularization, batch normalization, semi-linear activation functions (e.g. ReLU) and adaptive stochastic gradient descent methods. The maths for basic NNs is really simple but the practice of them is really messy.

miven 2y ago
Residual connections are also worth mentioning as an extremely ubiquitous adaptation, one will be hard-pressed to find a modern architecture that doesn't use those at least to some extent, to the point where the original Resnet paper sits at over 200k citations according to google scholar[1]. [1] https://scholar.google.com/citations?view_op=view_citation&hl=en&user=DhtAFkwAAAAJ&citation_for_view=DhtAFkwAAAAJ:ALROH1vI_8AC https://scholar.google.com/citations?view_op=view_citation&h...
- cma 2y ago
  Highway nets introduced them in the 90s

programjames 2y ago

> All the math and CS needed for AI can fit on a napkin, and had been known for 200+ years. This isn't really true. If you read a physics textbook from the early 1900s, they didn't really have multivariate calculus and linear algebra expressed as concisely as we do now. It would take several napkins. Plus, statistical mechanics was quite rudimentary, which is important for probability theory.

77pt77 2y ago
> If you read a physics textbook from the early 1900s, they didn't really have multivariate calculus and linear algebra expressed as concisely as we do now. This is completely incorrect.

cubefox 2y ago

That's absurd. The computer science needed for AI has not been known for 200 years. For example, transformers were only invented in 2017, diffusion models in 2015. (When the required math was invented is a different question, but I doubt all of it was known 200 years ago.)

zmgsabst 2y ago

I don’t think calculus existed at sufficient rigor 200 years ago. Computer science wasn’t even a thing 100 years ago.

KeplerBoy 2y ago
Calculus has been around for quite some time. If Newton had the machinery to fit large models to data, he would have done so. No doubt.
- zmgsabst 2y ago
  Cauchy’s main work was under 200 years ago; and there’s been quite a lot of work since. Again, I’m unsure that calculus existed at sufficient level 200 years ago — it didn’t appear in modern form from either Leibniz or Newton.

deleted 2y ago

[deleted]