5 ms·

GradIEEEnt half decent

275 points by notmysql_ 3y ago

SAI_Peregrinus 3y ago
Tom7 back to his usual madness. This time, exploiting floating point rounding, first to allow a neural network activation function to be linear (except for rounding errors), then expanding until ultimately making a 6502 emulator and proving that linear operations + rounding errors are Turing complete.
chrisldgk 3y ago
Always delighted to see tom7 content. Dude just thinks out of the box like no one else.
- physicles 3y ago
  His videos also cover a super wide range of content areas. Just this latest one has a refresher on neural nets, ieee floating point (and explained the accuracy tradeoffs more clearly than I’ve ever seen), fractals, chess engines, and more.
  - newswasboring 3y ago
    Not just that, he is also very good at making videos. His story lines are good and everything flows like a deep river. It's mesmerizing to look at.
    - airstrike 3y ago
      and he's also great at naming things, at least based on this one submission
  - x86x87 3y ago
    Not only that but they are highly entertaining. This one is probably my all time favorite: https://youtu.be/LA_DrBwkiJA https://youtu.be/LA_DrBwkiJA
  - fugalfervor 3y ago
    And running the length of every street in Pittsburgh, of all things
- newswasboring 3y ago
  It has been a long standing dream to make videos like this. I have no idea where to start though. I think everyone has these out of the box ideas, but we (or at least I) disregard them as "who will be interested in this".
  - airstrike 3y ago
    In addition to picking a subject that interests people, it helps to be good at storytelling. Fortunately, that skill can also be learned
- waynecochran 3y ago
  Dude is way out of the box. Using IEEE floating point rounding error as his neural network transfer / activation function -- never thought to go there.
unlikelymordant 3y ago
Reminds of some earlier openai work https://openai.com/research/nonlinear-computation-in-deep-linear-networks https://openai.com/research/nonlinear-computation-in-deep-li... "Neural networks consist of stacks of a linear layer followed by a nonlinearity like tanh or rectified linear unit. Without the nonlinearity, consecutive linear layers would be in theory mathematically equivalent to a single linear layer. So it’s a surprise that floating point arithmetic is nonlinear enough to yield trainable deep networks."
- deleted 3y ago
  [deleted]
eur0pa 3y ago
Two videos so far this year and we're still in May. What a bliss.
- sethaurus 3y ago
  Tom7 our beloved. How might we make this creator publish more videos? Money? Is it money? Can we give him money somehow?
  - shadowgovt 3y ago
    Unfortunately, Tom's a lot more creativity and concept-driven than money driven. So the best strategy is probably to spend the money on some kind of near-C sublight vehicle and take a one-hour jaunt in it, then return to Earth in a year and see what's tickled Tom's fancy since you last saw him. this solution may have side-effects that are left as an exercise for the Rocketeer.
  - UncleMeat 3y ago
    Tom has a pretty good day job, so I suspect that money isn't the thing. The sigbovik conference seems to be the motivating goal behind these videos. I think the key thing would be to make sure that sigbovik continues and that people are excited about the papers.
ironbound 3y ago
Best start to a video this year, really wish more people would use a hook to get people engaged in learning.
sva_ 3y ago
> I think this is a fractal in the sense that it is chaotic, has a color gradient, and could be on the cover of an electronic music album I didn't know of this criterion
- pjc50 3y ago
  It's on my album "Hausdorff-Besicovitch Criterion"
VikingCoder 3y ago
Tom7 is kind of like the electronic version of the Primitive Technology Youtuber. It's fascinating watching someone using some of the worst tools ever to make something in about the most labor-intensive way imaginable - and it's just beautiful. It's practically meditative.
- hoosieree 3y ago
  I know what you mean by Primitive Technology Youtuber, but I would encourage you to not dismiss the underlying concepts out of hand. Building a chair from green wood with hand tools may not work for Ikea, but it is much more energy-efficient than power tools [1]. There's also something incredibly empowering about working with tools you understand, both in software and woodworking. [1]: https://www.mortiseandtenonmag.com/blogs/blog/podcast-12-the-radical-efficiency-of-green-woodworking https://www.mortiseandtenonmag.com/blogs/blog/podcast-12-the...
  - VikingCoder 3y ago
    Oh absolutely. I'm going to confess that I read this book, "Replay" by Ken Grimwood, which is roughly about a guy who keeps having a heart attack on the same day, and wakes up in his younger body, with all of his memories. Again and again and again. Kind of like Groundhog Day, but for an entire adult life. I kind of became obsessed for a while with the idea that I would like to live my life over again. It was a really unhealthy obsession. It took a while for me to live in the moment, try to not hold on to my regrets, etc. Back to the topic... I kind of watch things like Primitive Technology out of a hope/fear that I'll either get thrown back in time to when these skills were relevant, or that I'll live in a post-apocalyptic world where they're relevant again. (Which was probably inspired by reading Stephen King's "The Stand," and worrying about how my life would be if there was no electricity.) But yeah, I love watching people show their enthusiasm... I love watching people demonstrate mastery over tools I don't know how to use... I love watching someone build something... Tom7 and Primitive Technology check all those boxes. Adam Savage's Tested often does, too.
    - taneq 3y ago
alexb_ 3y ago
This is one of the few tom7 videos that I see and am utterly confused by in every way. It feels like he's speaking a different language than I am.
- shadowgovt 3y ago
  I lucked out a bit in having only recently learned about the need for a nonlinear transfer function (although it was taught to me as a "That's just how we do it," not with the associated underlying explanation that you can't represent all functions with only linear functions).
tysam_and 3y ago
I believe that this floating point imprecision was something that David Page used (I think it may have been accidental as he was originally doing something else with it?) to achieve a world record training speed on CIFAR10 by summing a bunch of loss values together instead of taking the average. The sum effectively reduced the precision of the loss value and seemed to have a regularizing impact on network training as best as I personally understand. :) <3
dirtyid 3y ago
two tom7 videos in as many months, what a treat.

275 points by notmysql_ 3y ago

Tom7 back to his usual madness. This time, exploiting floating point rounding, first to allow a neural network activation function to be linear (except for rounding errors), then expanding until ultimately making a 6502 emulator and proving that linear operations + rounding errors are Turing complete.

chrisldgk 3y ago

Always delighted to see tom7 content. Dude just thinks out of the box like no one else.

physicles 3y ago
His videos also cover a super wide range of content areas. Just this latest one has a refresher on neural nets, ieee floating point (and explained the accuracy tradeoffs more clearly than I’ve ever seen), fractals, chess engines, and more.
- newswasboring 3y ago
  Not just that, he is also very good at making videos. His story lines are good and everything flows like a deep river. It's mesmerizing to look at.
  - airstrike 3y ago
    and he's also great at naming things, at least based on this one submission
- x86x87 3y ago
  Not only that but they are highly entertaining. This one is probably my all time favorite: https://youtu.be/LA_DrBwkiJA https://youtu.be/LA_DrBwkiJA
- fugalfervor 3y ago
  And running the length of every street in Pittsburgh, of all things
newswasboring 3y ago
It has been a long standing dream to make videos like this. I have no idea where to start though. I think everyone has these out of the box ideas, but we (or at least I) disregard them as "who will be interested in this".
- airstrike 3y ago
  In addition to picking a subject that interests people, it helps to be good at storytelling. Fortunately, that skill can also be learned
waynecochran 3y ago
Dude is way out of the box. Using IEEE floating point rounding error as his neural network transfer / activation function -- never thought to go there.

unlikelymordant 3y ago

Reminds of some earlier openai work https://openai.com/research/nonlinear-computation-in-deep-linear-networks https://openai.com/research/nonlinear-computation-in-deep-li... "Neural networks consist of stacks of a linear layer followed by a nonlinearity like tanh or rectified linear unit. Without the nonlinearity, consecutive linear layers would be in theory mathematically equivalent to a single linear layer. So it’s a surprise that floating point arithmetic is nonlinear enough to yield trainable deep networks."

deleted 3y ago
[deleted]

eur0pa 3y ago

Two videos so far this year and we're still in May. What a bliss.

sethaurus 3y ago
Tom7 our beloved. How might we make this creator publish more videos? Money? Is it money? Can we give him money somehow?
- shadowgovt 3y ago
  Unfortunately, Tom's a lot more creativity and concept-driven than money driven. So the best strategy is probably to spend the money on some kind of near-C sublight vehicle and take a one-hour jaunt in it, then return to Earth in a year and see what's tickled Tom's fancy since you last saw him. this solution may have side-effects that are left as an exercise for the Rocketeer.
- UncleMeat 3y ago
  Tom has a pretty good day job, so I suspect that money isn't the thing. The sigbovik conference seems to be the motivating goal behind these videos. I think the key thing would be to make sure that sigbovik continues and that people are excited about the papers.

ironbound 3y ago

Best start to a video this year, really wish more people would use a hook to get people engaged in learning.

sva_ 3y ago

> I think this is a fractal in the sense that it is chaotic, has a color gradient, and could be on the cover of an electronic music album I didn't know of this criterion

pjc50 3y ago
It's on my album "Hausdorff-Besicovitch Criterion"

VikingCoder 3y ago

Tom7 is kind of like the electronic version of the Primitive Technology Youtuber. It's fascinating watching someone using some of the worst tools ever to make something in about the most labor-intensive way imaginable - and it's just beautiful. It's practically meditative.

hoosieree 3y ago
I know what you mean by Primitive Technology Youtuber, but I would encourage you to not dismiss the underlying concepts out of hand. Building a chair from green wood with hand tools may not work for Ikea, but it is much more energy-efficient than power tools [1]. There's also something incredibly empowering about working with tools you understand, both in software and woodworking. [1]: https://www.mortiseandtenonmag.com/blogs/blog/podcast-12-the-radical-efficiency-of-green-woodworking https://www.mortiseandtenonmag.com/blogs/blog/podcast-12-the...
- VikingCoder 3y ago
  Oh absolutely. I'm going to confess that I read this book, "Replay" by Ken Grimwood, which is roughly about a guy who keeps having a heart attack on the same day, and wakes up in his younger body, with all of his memories. Again and again and again. Kind of like Groundhog Day, but for an entire adult life. I kind of became obsessed for a while with the idea that I would like to live my life over again. It was a really unhealthy obsession. It took a while for me to live in the moment, try to not hold on to my regrets, etc. Back to the topic... I kind of watch things like Primitive Technology out of a hope/fear that I'll either get thrown back in time to when these skills were relevant, or that I'll live in a post-apocalyptic world where they're relevant again. (Which was probably inspired by reading Stephen King's "The Stand," and worrying about how my life would be if there was no electricity.) But yeah, I love watching people show their enthusiasm... I love watching people demonstrate mastery over tools I don't know how to use... I love watching someone build something... Tom7 and Primitive Technology check all those boxes. Adam Savage's Tested often does, too.
  - taneq 3y ago

alexb_ 3y ago

This is one of the few tom7 videos that I see and am utterly confused by in every way. It feels like he's speaking a different language than I am.

shadowgovt 3y ago
I lucked out a bit in having only recently learned about the need for a nonlinear transfer function (although it was taught to me as a "That's just how we do it," not with the associated underlying explanation that you can't represent all functions with only linear functions).

tysam_and 3y ago

I believe that this floating point imprecision was something that David Page used (I think it may have been accidental as he was originally doing something else with it?) to achieve a world record training speed on CIFAR10 by summing a bunch of loss values together instead of taking the average. The sum effectively reduced the precision of the loss value and seemed to have a regularizing impact on network training as best as I personally understand. :) <3

dirtyid 3y ago

two tom7 videos in as many months, what a treat.