6 ms·
I find it so cool that technology created to make games like Quake look pretty has ended up becoming a core foundation of high performance computing and AI.
by hesdeadjim 9y ago
I find it so cool that technology created to make games like Quake look pretty has ended up becoming a core foundation of high performance computing and AI.
- pgodzin 9y agoMatrix multiplication is important for graphics and important for finding the weights of a neural network
- hesdeadjim 9y agoYep, hard to imagine though that the original creators of the Nvidia TNT or Voodoo had any idea that GPUs would become fully programmable computing hardware used for non-graphical applications.
- abritinthebay 9y agoI don't think they'd have been super surprised. Just pleasantly happy. AI Accelerators have been a thing for decades - DSPs were used as neural network accelerators in the early 90s - and Cell processors were a thing by 2001. GPUs just became vastly more accessible to general purpose program in the last decade. People were doing it back in the 90s but it was seriously hard. We finally hit a tipping point where it's just kinda hard.
- cr0sh 9y agoThere were also the various custom "systolic array" processor designs in the 1980s (the ALVINN vehicle, and earlier projects which led to it, used these for early neural-network based self-driving experiments).
- scott_s 9y agoI remember back in 2004 when I heard a fellow grad student was working on using GPUs as a co-processor for scientific computing, I though "Wow, that's esoteric and niche."
- abricot 9y agoThis reminds me of a comment i read here ages ago about a scientist using the "processors" of the univercity's postscript printers because they did the work much faster than their scientific workstations.
- ChickeNES 9y agoReminds me of some Commodore 64 programs running code on the 1541 disk drive to offload computation from the main CPU (both the C64 and the 1541 had 6502s (well, the C64 had a 6510 which had an I/O port) running @ 1Mhz). The original Apple Laserwriter had a 68k running at 12Mhz, while the Mac Plus, which came out almost a year later, had its 68k running at 8Mhz.
- rasz 9y agoCreators of Voodoo (3dfx = Gary Tarolli, Scott Sellers) came from the world of fully programmable GPUs. Silicon Graphics workstations had full T&L since ~1988 (http://www.sgistuff.net/hardware/systems/iris3000.html http://www.sgistuff.net/hardware/systems/iris3000.html). The whole point of Voodoo 1 was making it as simple and cheap as possible by removing all the advanced features and calculating geometry/lighting on the CPU. https://www.youtube.com/watch?v=3MghYhf-GhU https://www.youtube.com/watch?v=3MghYhf-GhU
- kijiki 9y agoIris Graphics Geometry Engines weren't programmable in the modern sense. There was a fixed pipeline of Matrix units, clippers and so on that fed the fixed function Raster Engines. You could change various state that went into the various stages, but the pipeline's operations were fixed. Later SGI Geometry Engines used custom, very specialized DSP-like processors, but the microcode for those were written by SGI, and not end-user programmable. There were probably research systems before it, but AFAIK the Geforce 3 was the first (highly limited) programmable geometry processor that was generally commercially available.
- Florin_Andrei 9y agoI think it's even cooler how matrix multiplication dominates both the universe at large, and the systems that understand it (neural networks).
- nostrebored 9y agoWell a large portion of that is desiring the data to be in that form. BLAS operations are ruthlessly efficient and use the system hardware so well.
- halflings 9y agoRelevant article: https://www.technologyreview.com/s/602344/the-extraordinary-link-between-deep-neural-networks-and-the-nature-of-the-universe/ https://www.technologyreview.com/s/602344/the-extraordinary-...
- Florin_Andrei 9y agoMax Tegmark keeps saying interesting things lately. Controversial for sure, but interesting. His book 'Our Mathematical Universe' (related to the article you've linked) is thought-provoking, and I would label it a must read if you're interested in what's going on at the outer edges of fundamental science. The chapters are clearly separated into: fact, hypothesis, and far-out speculation, so there's no need to criticize the whole thing indiscriminately. There was a series of attempts by Lee Smolin and others to come up with a theory of quantum gravity by assuming that the universe, at the bottom, is essentially simple and discrete (not in the fixed-grid sense, but in the sense of a discrete web of relations). That model also exhibits a remarkable similarity between the structure of the universe, and the structure of the neural networks that understand it. The future of fundamental science is sure to be fascinating.
- barrkel 9y agoI think that's a dreadful article. We do know how neural networks work; they're a bunch of hierarchical probabilistic calculations that are pipelined. I don't really see how that couldn't work well; it's just hard to find the right probabilities. The difficulty is far more in the training than the working, and that's where the deep learning advances come in - inferring more parameters in a deeper hierarchy. There's no relationship between a hierarchy of probabilistic estimations and a hierarchical decomposition of the cosmos. The cosmos forms an apparent hierarchy because of the rules that govern matter and the initial expansion of the universe. That a small number of parameters might be listed in describing both is neither here nor there. A small number of parameters describe the vectors in a font file. It doesn't follow that a typeface then has any relationship with my brain or the universe. The article reads, to me, like this: neural networks are this cool hierarchy thing, the cosmos is this cool hierarchy thing, and both of these things have low Kolmogorov complexity, isn't it amazing that our brains are like this and can understand the universe, wow.
- anonymfus 9y agoIt backwards: first common application of the technology gave it the name. Like in one of the Stanisław Lem's stories about Ijon Tichy people call intelligent anthropomorphic robots washing machines.
- matt4077 9y agoYou'll be delighted to hear that traffic signals are called "robots" in South Africa.
- malkia 9y agoThat itself reminded me, that the word "robot" comes from "robota, rabota" which means "to work, worker" in a lot of slavic languages (My native language is bulgarian). From the dictionary: robot, origin: from Czech, from robota ‘forced labor.’ The term was coined in K. Čapek's play R.U.R. ‘Rossum's Universal Robots’ (1920). https://www.google.com/search?q=robot+definition https://www.google.com/search?q=robot+definition
- contras1970 9y agoRossum is a riff on "rozum" which means reason (as in to reason about). https://en.wikipedia.org/wiki/R.U.R.#Origin_of_the_word https://en.wikipedia.org/wiki/R.U.R.#Origin_of_the_word
- mrkgnao 9y ago"Thinker's Universal Workers"?
- fulafel 9y agoYet another step in the progression in which mass market GPU silicon kills traditional vector and memory bandwidth rich4 HPC/supercomputing hardware. Cray-on-a-chip. Edit: traditional vector machines like the nec sx still hold the programmability crown because you get a usable single system image, right?
- dom0 9y agohttps://www.youtube.com/watch?v=ooLO2xeyJZA https://www.youtube.com/watch?v=ooLO2xeyJZA