31 ms·

I've not met Mister Hotz, but from what I've read and heard from him on podcasts, I'm not sure there's a lot of smart in the crazy smart.

benreesman 3y ago
If what you’ve read and listened to is a lot, please elaborate? If it isn’t a lot, I encourage you to look more closely. Let’s skip all the disputes about his achievements prior to this year (and you really need to take with a grain of salt anything bad you hear about someone who has been actively fucking up the afternoon of powerful people their entire life). Tinygrad is so obvious in retrospect, but hindsight is 20/20 and I missed it. Why the hell do we have kernels for any composite tensor operation? The machine economics are that starting a training run is who cares. Doing a training run is machine hours, days, centuries, millennia. Spend like a drunken sailor customizing the kernel, exploit the aforementioned “who cares” blank check to just dump the friggin CUDA or PTX or OpenCL or Metal or whatever into /tmp/ and hit it with a fork/execve to nvcc or whatever. You can fuse and columnize and whatever to your hears content in the IR. We spent how much time rigging up a trouś convolutions via IM2Col by hand with template instantiation in ATen? Well, we’re never getting that back. And while dating NVIDIA is fun, being chained in their soundproof basement not so fun. “It lives with the same 24Gb of GDDR6 on it paid for last year on its skin or it gets the p4d.24xlarge pricing again”. Nobody has jacked anyone so usuriously via API lock-in since Gates and Win32. Anyone who is doing a YouTube video called “Get in Losers, We’re building a Chatbot” and then live codes for 5 hours to turn 2700 lines of Python that can’t run LLaMA into 3200 lines of Python that can run it on three times the number of platforms that PyTorch? I’m struggling here.
- threeseed 3y ago
  I think many people's first introduction to George Hotz was with Twitter. Where his arrogance, ignorance and complete lack of experience with how teams and companies function effectively was on full display. You can spout off technical terms all day long but I've worked at two of the FAANG companies and met hundreds of incredible engineers and it wasn't their grasp of the technology that made them great. It was their ability to lead, deliver, communicate, relate etc which are skills that sound far less impressive but are infinitely more important.
  - benreesman 3y ago
    This meme isn't completely wrong, but it's not completely right either. Before you listen to anything I have to say about it I'd refer you to the guy who co-designed a Lisp variant and the site we're having this conversation on, who is also probably the most successful tech investor in history (who presumably knows a thing or two about what's important if your goal is building successful technology businesses) [0]: "Hackers like to work for people with high standards. But it's not enough just to be exacting. You have to insist on the right things. Which usually means that you have to be a hacker yourself. I've seen occasional articles about how to manage programmers. Really there should be two articles: one about what to do if you are yourself a programmer, and one about what to do if you're not. And the second could probably be condensed into two words: give up." I've only worked for one FAANG, but it was for like 7-8 years out of a 20+ year career, and it was mostly on stuff where even small mistakes cost a lot of money so I'll counter your anecdote with one of my own: at the beginning of the century the software business was really weak on demographic diversity (a problem we still have and still needs to be addressed) but it was unparalleled in neurodiversity. In particular there's a longitudinal component to my observation. I'm not here to diagnose anyone via the Internet, but it's pretty clear that George (like a lot of us) is pretty friggin "different". During the period of time (say 1995-2015 +/- 3 years) when most of the current empires of tech were built, staggeringly successful managers in both open-source (e.g. Linus) and industry (e.g. Sheryl) created a lot of space for aspy nerds who "spouted off" about tech stuff in blunt "this is stupid" kinds of ways, and all kinds of other stuff that's now called "being toxic" rather than "being a weird nerd". It was only during that latter half of this time (and really the end of it) that software became a sufficiently high-status occupation to draw in a bunch of people who decided that it was a good idea to outmaneuver those folks on the org chart and re-brand an entire industry of people from "probably on the spectrum at least a bit" to "toxic asshole". I don't know if packing Linus off to charm school because he was simply too important to write off was a better or worse move than just letting him be Linus on a mailing list he founded, I kinda liked the original Linus. But even if we decided that old-school hacker aspiness couldn't get a blank check anymore (on the Internet we fucking built), it's just a strictly better idea to strike some sort of compromise with the aspy nerds than to redefine the vibe that had prevailed for decades as basically up there with racism, misogyny, and homophobia (you might notice that aspy tech assholes are on average some of the least racist, misogynistic, and homophobic people in any industry: they on average accept anyone who kicks ass at code or is trying hard and are short with anyone who is dismissive of code, irrespective of demo). I wrote the `im2col` and dilated convolution kernels on NVIDIA and Intel that went into Caffe2 originally, and AFAIK that stuff migrated over to like ATen or whatever (though I imagine someone has rewritten it since), so I don't exactly love the "spout off technical terms all day log" characterization, but mostly I'll admonish you that calling leadership and communication skills "infinitely" more important than technical expertise is dangerously close to "arrogance and ignorance", and that overlooking that someone has been in the public eye on e.g. Twitter since they were 16 doesn't seem like a particularly serious effort to "relate". In a way rather reminiscent of early Linus, George has organized a few thousand iconoclastic accelerated computing pros and/or people aspiring to become one into a small but surprisingly credible threat to the current round of Bond Villain monopolists. Maybe we cut him some slack on being a bit of an edgelord on his own Twitter account, his own Twitch stream, his own GitHub Issues page, and to set the culture at his own startup? [0] http://www.paulgraham.com/gh.html http://www.paulgraham.com/gh.html

benreesman 3y ago

If what you’ve read and listened to is a lot, please elaborate? If it isn’t a lot, I encourage you to look more closely. Let’s skip all the disputes about his achievements prior to this year (and you really need to take with a grain of salt anything bad you hear about someone who has been actively fucking up the afternoon of powerful people their entire life). Tinygrad is so obvious in retrospect, but hindsight is 20/20 and I missed it. Why the hell do we have kernels for any composite tensor operation? The machine economics are that starting a training run is who cares. Doing a training run is machine hours, days, centuries, millennia. Spend like a drunken sailor customizing the kernel, exploit the aforementioned “who cares” blank check to just dump the friggin CUDA or PTX or OpenCL or Metal or whatever into /tmp/ and hit it with a fork/execve to nvcc or whatever. You can fuse and columnize and whatever to your hears content in the IR. We spent how much time rigging up a trouś convolutions via IM2Col by hand with template instantiation in ATen? Well, we’re never getting that back. And while dating NVIDIA is fun, being chained in their soundproof basement not so fun. “It lives with the same 24Gb of GDDR6 on it paid for last year on its skin or it gets the p4d.24xlarge pricing again”. Nobody has jacked anyone so usuriously via API lock-in since Gates and Win32. Anyone who is doing a YouTube video called “Get in Losers, We’re building a Chatbot” and then live codes for 5 hours to turn 2700 lines of Python that can’t run LLaMA into 3200 lines of Python that can run it on three times the number of platforms that PyTorch? I’m struggling here.

threeseed 3y ago
I think many people's first introduction to George Hotz was with Twitter. Where his arrogance, ignorance and complete lack of experience with how teams and companies function effectively was on full display. You can spout off technical terms all day long but I've worked at two of the FAANG companies and met hundreds of incredible engineers and it wasn't their grasp of the technology that made them great. It was their ability to lead, deliver, communicate, relate etc which are skills that sound far less impressive but are infinitely more important.
- benreesman 3y ago
  This meme isn't completely wrong, but it's not completely right either. Before you listen to anything I have to say about it I'd refer you to the guy who co-designed a Lisp variant and the site we're having this conversation on, who is also probably the most successful tech investor in history (who presumably knows a thing or two about what's important if your goal is building successful technology businesses) [0]: "Hackers like to work for people with high standards. But it's not enough just to be exacting. You have to insist on the right things. Which usually means that you have to be a hacker yourself. I've seen occasional articles about how to manage programmers. Really there should be two articles: one about what to do if you are yourself a programmer, and one about what to do if you're not. And the second could probably be condensed into two words: give up." I've only worked for one FAANG, but it was for like 7-8 years out of a 20+ year career, and it was mostly on stuff where even small mistakes cost a lot of money so I'll counter your anecdote with one of my own: at the beginning of the century the software business was really weak on demographic diversity (a problem we still have and still needs to be addressed) but it was unparalleled in neurodiversity. In particular there's a longitudinal component to my observation. I'm not here to diagnose anyone via the Internet, but it's pretty clear that George (like a lot of us) is pretty friggin "different". During the period of time (say 1995-2015 +/- 3 years) when most of the current empires of tech were built, staggeringly successful managers in both open-source (e.g. Linus) and industry (e.g. Sheryl) created a lot of space for aspy nerds who "spouted off" about tech stuff in blunt "this is stupid" kinds of ways, and all kinds of other stuff that's now called "being toxic" rather than "being a weird nerd". It was only during that latter half of this time (and really the end of it) that software became a sufficiently high-status occupation to draw in a bunch of people who decided that it was a good idea to outmaneuver those folks on the org chart and re-brand an entire industry of people from "probably on the spectrum at least a bit" to "toxic asshole". I don't know if packing Linus off to charm school because he was simply too important to write off was a better or worse move than just letting him be Linus on a mailing list he founded, I kinda liked the original Linus. But even if we decided that old-school hacker aspiness couldn't get a blank check anymore (on the Internet we fucking built), it's just a strictly better idea to strike some sort of compromise with the aspy nerds than to redefine the vibe that had prevailed for decades as basically up there with racism, misogyny, and homophobia (you might notice that aspy tech assholes are on average some of the least racist, misogynistic, and homophobic people in any industry: they on average accept anyone who kicks ass at code or is trying hard and are short with anyone who is dismissive of code, irrespective of demo). I wrote the `im2col` and dilated convolution kernels on NVIDIA and Intel that went into Caffe2 originally, and AFAIK that stuff migrated over to like ATen or whatever (though I imagine someone has rewritten it since), so I don't exactly love the "spout off technical terms all day log" characterization, but mostly I'll admonish you that calling leadership and communication skills "infinitely" more important than technical expertise is dangerously close to "arrogance and ignorance", and that overlooking that someone has been in the public eye on e.g. Twitter since they were 16 doesn't seem like a particularly serious effort to "relate". In a way rather reminiscent of early Linus, George has organized a few thousand iconoclastic accelerated computing pros and/or people aspiring to become one into a small but surprisingly credible threat to the current round of Bond Villain monopolists. Maybe we cut him some slack on being a bit of an edgelord on his own Twitter account, his own Twitch stream, his own GitHub Issues page, and to set the culture at his own startup? [0] http://www.paulgraham.com/gh.html http://www.paulgraham.com/gh.html