4 ms·

I feel the same way. But beyond these concerns, I must admit I also feel a sense of sadness due to the inevitable rise of application-specific hardware. When

by Athas 4y ago

I feel the same way. But beyond these concerns, I must admit I also feel a sense of sadness due to the inevitable rise of application-specific hardware. When I hear about these super-fast specialised chips, all I can think about is whether I'd have enjoyed sitting in my bedroom as a teenager hacking on them. And the answer is probably no. They can do very specific things, but they don't have the kind of freedom of programming I enjoy. Maybe I just need to look at them differently, or maybe my perception of their capabilities is just wrong. I do like GPUs, for example. While they are ornery and limited in many ways, they are still flexible enough that you can do enjoyable general-purpose programming on them.

eru 4y ago
A type-writer or a piano is an application specific piece of hardware, yet people can still do an amazing variety of things with them. In the last few decades we got really lucky that general purpose CPUs kept getting faster and faster. So many of us could get away with ignoring application specific computer hardware. Btw, as far as I can tell, TPUs are really just a variant of GPUs with an emphasis on lower precision? Have a look also at the kind of special purpose hardware people make to mine crypto-currencies. I think the challenge for your teenage hacker is not so much to make them do interesting things, but to design them? (Assuming the tools and simulators become cheap enough?) FPGAs seem like they are cheap enough for a teenager?
- rrss 4y ago
  > Btw, as far as I can tell, TPUs are really just a variant of GPUs with an emphasis on lower precision? GPUs and TPUs have substantially different architectures, it’s not accurate to consider a TPU an a variant of a GPU. See https://dl.acm.org/doi/pdf/10.1145/3360307 https://dl.acm.org/doi/pdf/10.1145/3360307, particularly section “Contrasting GPU and TPU Architectures” looking only at the matrix multiply units (and ignoring the hardware multithreading in GPUs that was deliberately not part of the TPU architecture): TPUv3: 2 cores, 2 128x128 matrix multiply units per core V100: 80 cores, 8 4x4 matrix multiply units per core
  - eru 4y ago
    Thanks for the background information!
primordialsoup 4y ago
It's funny you say that GPUs are general purpose. It's almost like a universal law that generic stuff is helpful only to some extent, after which you need specialization. That's why we do PhDs I suppose. And that's why you have cardiologists.
dchftcs 4y ago
You can still enjoy hacking with a consumer FPGA card. For ML one major problem would be memory (for a FPGA that costs the same as a GPU), there's no free lunch. But there are applications where FPGAs of the same price work better.
Fordec 4y ago
The sooner specialized hardware comes out the better in my books. The sooner companies deliver it, the sooner patents will inevitably expire and we can get some open source implementations. The golden age of ASICs for everyone is a decade or two away yet. But the sooner that day comes, the better for those of us who want the ownership of it. I'd love to integrate an embedded GPU for an onboard AI into my hobby electronics projects with a low power overhead. We're just not quite there yet.

by Athas 4y ago

eru 4y ago

A type-writer or a piano is an application specific piece of hardware, yet people can still do an amazing variety of things with them. In the last few decades we got really lucky that general purpose CPUs kept getting faster and faster. So many of us could get away with ignoring application specific computer hardware. Btw, as far as I can tell, TPUs are really just a variant of GPUs with an emphasis on lower precision? Have a look also at the kind of special purpose hardware people make to mine crypto-currencies. I think the challenge for your teenage hacker is not so much to make them do interesting things, but to design them? (Assuming the tools and simulators become cheap enough?) FPGAs seem like they are cheap enough for a teenager?

rrss 4y ago
> Btw, as far as I can tell, TPUs are really just a variant of GPUs with an emphasis on lower precision? GPUs and TPUs have substantially different architectures, it’s not accurate to consider a TPU an a variant of a GPU. See https://dl.acm.org/doi/pdf/10.1145/3360307 https://dl.acm.org/doi/pdf/10.1145/3360307, particularly section “Contrasting GPU and TPU Architectures” looking only at the matrix multiply units (and ignoring the hardware multithreading in GPUs that was deliberately not part of the TPU architecture): TPUv3: 2 cores, 2 128x128 matrix multiply units per core V100: 80 cores, 8 4x4 matrix multiply units per core
- eru 4y ago
  Thanks for the background information!

primordialsoup 4y ago

It's funny you say that GPUs are general purpose. It's almost like a universal law that generic stuff is helpful only to some extent, after which you need specialization. That's why we do PhDs I suppose. And that's why you have cardiologists.

dchftcs 4y ago

You can still enjoy hacking with a consumer FPGA card. For ML one major problem would be memory (for a FPGA that costs the same as a GPU), there's no free lunch. But there are applications where FPGAs of the same price work better.

Fordec 4y ago

The sooner specialized hardware comes out the better in my books. The sooner companies deliver it, the sooner patents will inevitably expire and we can get some open source implementations. The golden age of ASICs for everyone is a decade or two away yet. But the sooner that day comes, the better for those of us who want the ownership of it. I'd love to integrate an embedded GPU for an onboard AI into my hobby electronics projects with a low power overhead. We're just not quite there yet.