6 ms·

I’m bearish on new hardware for AI training. The most important thing is the software stack, and thus far everyone has failed to support pytorch in a drop-in wa

I’m bearish on new hardware for AI training. The most important thing is the software stack, and thus far everyone has failed to support pytorch in a drop-in way. The philosophy here seems to be “if we build it, they’ll buy it.” But suppose you wanted to train a gpt model with this specialized hardware. That means you’re looking at two months of R&D minimum to get everything rewritten, running, tested, trained, and with an inferencing pipeline to generate samples. And that’s just for gpt — you lose all the other libraries people have written. This matters more in GAN training, since for example you can find someone else’s FID implementation and drop it in without too much hassle. But with this specialized chip, you’d have to write it from scratch. We had a similar situation in gamedev circa 2003-2009. Practically every year there was a new GPU, which boasted similar architectural improvements. But, for all its flaws, GL made these improvements “drop-in” —- just opt in to the new extension, and keep writing your gl code as you have been. Ditto for direct3d, except they took the attitude of “limit to a specific API, not arbitrary extensions.” (Pixel shader 2.0 was an awesome upgrade from 1.1.) AI has no such standards, and it hurts. The M1 GPU in my new Air is supposedly ready to do AI training. Imagine my surprise when I loaded up tensorflow and saw that it doesn’t support any GPU devices whatsoever. They seem to transparently rewrite the cpu ops to run on the gpu automatically, which isn’t the expected behavior. So I dig into Apple’s actual api for doing training, and holy cow, that looks miserable to write in swift. I like how much control it gives you over allocation patterns, but I can’t imagine trying to do serious work in it on a daily basis. What we need is a unified API that can easily support multiple backends — something like “pytorch, but just enough pytorch to trick everybody” since supporting the full api seems to be beyond hardware vendors’ capabilities at the moment. (Lookin’ at you, google. Love ya though.)

whimsicalism 6y ago
I'm on board with you that there should be a "drop-in" cross support of these chips, but pytorch is at a way higher abstraction level than what should be commonly supported.
socialdemocrat 6y ago
Maybe people will get to their senses and switch to Julia instead of having to waste all these time on Python bindings.
- habibur 6y ago
  Python here basically works as a binder for C, which is what everything is written in.
  - eigenspace 6y ago
    That is in fact the problem, yes.
- dnautics 6y ago
  Honestly I think julia missed the boat. There was an opportunity for julia to treat the gpu as an abstract distributed node (which is a first class concept in julia) and pay respect to the reality of data gravity. But they chose to instead basically treat the gpu as a synchronous entity. I'm now bullish on elixir-nx because I think there's an outside shot they will get it right. Python is a lost cause.
  - BadInformatics 6y ago
    I'm not sure that's necessarily the domain of a low-level package like CUDA.jl though (which I assume you're referring to). That kind of interface is more the domain of higher-level packages like https://github.com/JuliaGPU/DaggerGPU.jl https://github.com/JuliaGPU/DaggerGPU.jl and to a lesser extent https://juliagpu.github.io/KernelAbstractions.jl/stable/ https://juliagpu.github.io/KernelAbstractions.jl/stable/. Moreover, the jury is still out on whether the built-in Distributed module is an ideal abstraction for every use-case (clusters, heterogeneous compute, etc.) WRT Nx, my biggest question is how they'll crack the problem of still needing big balls of C++ and the shims everywhere to get acceleration. Creating a compiler that generates efficient GPU or other accelerator code is a massive research project with no clear winners, never mind the challenge of reconciling the very mutation-heavy needs of GPU compute with a mostly immutable language model.
zucker42 6y ago
> The philosophy here seems to be “if we build it, they’ll buy it.” Supposedly Cerebras is already profitable, so it's hardly a situation where they are building something and hoping people buy it eventually. > That means you’re looking at two months of R&D minimum to get everything rewritten, running, tested, trained, and with an inferencing pipeline to generate samples. Again, based on the companies representations, Cebebras transparently supports Pytorch and Tensorflow, only requiring a few lines of changed code. Source: https://www.anandtech.com/show/16626/cerebras-unveils-wafer-scale-engine-two-wse2-26-trillion-transistors-100-yield https://www.anandtech.com/show/16626/cerebras-unveils-wafer-... (Dr. Cutress's video on TechTechPotato is also good).
- sillysaurusx 6y ago
  I’ve learned to be skeptical of such claims. TPUs made that same claim, but it’s hard to use it for real work, since you don’t have access to the 300GB TPU host memory / cpu, where you infeed training examples. There is a very specific test for “supports pytorch / tendorflow”: show an MLPerf imagenet resnet benchmark. It’s impossible to fake that. If you come anywhere close to TPUs in tensorflow, people (like me) will leap: https://mlcommons.org/en/training-normal-06/ https://mlcommons.org/en/training-normal-06/ Till then, it’s a “proof, please” type of situation.
  - sanxiyn 6y ago
    I agree. I have high hope for Graphcore. They stated "We will be participating in MLPerf in 2021, starting with the first training submission in the spring". https://www.graphcore.ai/posts/graphcore-sets-new-ai-performance-standards-with-mk2-ipu-systems https://www.graphcore.ai/posts/graphcore-sets-new-ai-perform... (I would ignore all benchmarks in that post except the sentence quoted above however.)
  - algo_trader 6y ago
    Software will lag new hardware. Duh. The question is whether this is solvable (given a couple of master devs) Cerebras itself commented that DL is not their first use case. They mentioned fluid dynamics instead. (I can find the reference if someone is interested). Cerebras also has a weird memory/compute ratio, so very hard for some uses. Somewhat off-topic. If u have standard keras/TF training code, the GPU to TPU is a smooth-ish transition. No? A re-write will squeeze another 20% but not 200%?
habibur 6y ago
So we are looking for OpenCUDA for AI. Like OpenGL in the 90s.
- DesiLurker 6y ago
  So basically OpenCL?
  - byronyi 6y ago
    I'd bet on SPIR-V.
zitterbewegung 6y ago
I think apple expects you to use pretrained models and convert them to CoreML. There is also webGPU which is experimental on Safari and Chrome. But that doesn’t touch the ML accelerators on the M1.
systemvoltage 6y ago
You're bearish on new hardware for AI training because it takes 2 months to setup? That's like saying you're bearish on the invention of transistor because there are no computers yet. This chip is for research organizations such as LLNL and ANL. Tooling and commoditization will happen. > “if we build it, they’ll buy it.” This is literally how any new thing is invented and commercialized.
- sillysaurusx 6y ago
  Well, two months has a way of turning into two years. June 2019 was right around when I fell in love with TPUs, and we’re still happily TPUing today. Sometimes, though, it would be nice to simply focus on doing interesting ML work rather than wrestling with the intricacies of tensor slicing and memory access pattern optimization (to say nothing of the cursed inability to manually manage memory, resulting in explosions of allocations in the forward pass for unclear reasons). Let’s put it this way. If you contract me to implement a full gpt model on this new hardware, two months full time work would be my minimum estimate. That’s 40 hours a week of focused effort, with no breaks and no other projects. I don’t know about you, but two months with nothing to show can demoralize most teams I’ve worked with. The point was, we’re early in ML’s lifecycle. And, like React for webdev, people are slow to change their habits — for better or worse, pytorch and tensorflow are the APIs people think in. So if you want your hardware to be widely adopted, you need tooling that supports the workflows people have spent months learning. Jax is on the horizon too. Supposedly they’re launching something soon that might tip the scale in their favor. Perhaps an ambitious hardware vendor could capture the future market by preemptively implementing Jax support. But perhaps not: at that point you’d be competing toe to toe with Google’s TPU offering, since Jax is Google’s dogfood. Right now my money is on TPUs, partly because of their fantastic support staff. But maybe some other company will come along and offer a better integrated cloud experience.
  - durst 6y ago
    Do you have a longer writeup on your experience performance tuning for the TPU? Is https://cloud.google.com/tpu/docs/performance-guide https://cloud.google.com/tpu/docs/performance-guide the best resource in this area?
oivey 6y ago
Graphics APIs and deep learning frameworks are really nothing alike. In graphics, save for a few new features, the goal every year is the same: draw more and faster. Deep learning is evolving much faster than that. The answer isn’t an API; it’s powerful and mostly transparent automatic differentiation. Anything that is just an API will forever be behind and is much less useful for active research. There are a few projects around improving AD support, but it’s a relatively difficult problem because to do it best you need deep compiler integration.
- sanxiyn 6y ago
  The answer is an API, like NNAPI. AD is a frontend concern and doesn't really matter to accelerators. For AD, I am bullish for Enzyme, which does AD on LLVM IR, avoiding deep compiler integration: https://enzyme.mit.edu/ https://enzyme.mit.edu/
  - oivey 6y ago
    I would say that integrating AD at the level of LLVM IR is deep compiler integration. The AD has to consume IR and then emit more IR to actually take the derivatives. If you’re using a Python wrapper for this you’ll need codegen for interacting with the generated IR, too. The upside with that approach is that it might more easily work across languages. The downside is that LLVM IR can’t rely on the rich invariants that even C exposes, which makes codegen and optimization harder. An API is always going to be necessarily behind state-of-the-art because often research depends on inventing things that don’t fit within an existing API.
    - sanxiyn 6y ago
      I think we are talking past each other. sillysaurusx is talking about OpenGL/Direct3D as a portability layer for hardwares. API vs AD is not a hardware portability layer concern. The proposal is to standardize OpenGL/Direct3D-like hardware portability layer defined by API. API vs AD you are talking about happens far above that and it is irrelevant.
      - 6y ago
justaguy88 6y ago
You might like this talk that describes parts of the compiler ecosystem that are lacking standardization for accelerators https://www.youtube.com/watch?v=4HgShra-KnY https://www.youtube.com/watch?v=4HgShra-KnY
sanxiyn 6y ago
> What we need is a unified API that can easily support multiple backends Android standardized NNAPI and on-device inference is in pretty good shape. As you said, training is different matter.
de6u99er 6y ago
Thr Anandtech article from a other comment here indicates that both PyTorch and Tensorflow are already running on it. >A key to the design is the custom graph compiler, that takes pyTorch or TensorFlow and maps each layer to a physical part of the chip, allowing for asynchronous compute as the data flows through. https://www.anandtech.com/show/16626/cerebras-unveils-wafer-scale-engine-two-wse2-26-trillion-transistors-100-yield https://www.anandtech.com/show/16626/cerebras-unveils-wafer-...
jquery 6y ago
> We had a similar situation in gamedev circa 2003-2009. Practically every year there was a new GPU, which boasted similar architectural improvements. You mean one of the most rapid periods of graphical improvement in history? Growth was just too breakneck to standardize for a bit.
- MaxBarraclough 6y ago
  sillysaurusx's comment goes on to acknowledge that the major graphics libraries (OpenGL and Direct3D) did a pretty good job of abstracting away/indirecting away the architectural changes. As you indicate, the architectural changes were made for good reason, and translated to real improvements in performance/capabilities. If AI acceleration chips aren't able to offer similar API standardisation/stability then it's not the same as what happened with GPUs, which was sillysaurusx's point.
TFortunato 6y ago
The article mentions that developers use standard PyTorch and Tensorflow.
docfort 6y ago
You might be interested in this for your M1 MBA: https://github.com/apple/tensorflow_macos https://github.com/apple/tensorflow_macos
- sillysaurusx 6y ago
  That was actually what I was referring to! It's super weird. I like it, and it's what I use primarily, but only because no other version of tensorflow will install. Their compiler is closed source, so it's almost impossible to tell what's going on. But, when I connect using tf.Session(), then .list_devices() shows only CPU available. However, when I enable the TF_MLC_LOGGING=1 magic variable, it does seem to be printing out messages that indicates it's doing some kind of graph substitution under the hood. Therefore, I assume that this is the intended usage mode. In other words, there seems to be zero difference between the "CPU" and the "GPU". Normally you can say "Do this on the CPU" while "do that on the GPU." But not with this. Hopefully they'll open source the code sometime this century so that it's clearer what the heck it's doing. For now, though, it's reasonably fast in whatever this "CPU" mode is -- I only need to run unit tests on my laptop anyway, since all training happens on TPUs. So I ended up happy. (For the first day or so, I was panicking that I was going to have no working tensorflow whatsoever on my M1 laptop, which would've necessitated a swift return + substitution.)
  - buildbot 6y ago
    That seems very apple honestly, they’d rather you emit pure metal primitives (is that the right term? Idk) and then let their backend schedule on their soc- because for all we know, they have some special asic ip that they added/or will add and can take advantage.
  - markonen 6y ago
    With the disclaimer that I know nothing about this: doesn't the M1 have a separate "Neural Engine" for this? So it's using neither the CPU nor the GPU cores for TensorFlow?
blihp 6y ago
The philosophy is working as it often does in the early days/years of a technology. The customers for this have problems at an entirely different scale. It's worth it for them to drop millions on hardware and potentially millions more to (re)write the software as needed. Sure, they'd prefer to be buying commodity hardware with standardized APIs. But since that doesn't currently exist for this class of performance, they'll pay up. People did the same back in the days of Cray/SGI/Lisp Machines and so on. These custom solutions can be quite profitable until the competition gets to the 'good enough' stage and then orders start drying up. If you're looking for standardized hardware and software for AI, Apple is the wrong platform to be on. I'm fairly confident it won't be happening there. (I don't say that to be a hater: see the direction they've gone re: GPU APIs... it's just Apple being Apple)
dmingod666 6y ago
If you look at it historically and put it in scale with the AI winter, 2 months is next to nothing..
solidasparagus 6y ago
I don't really agree, at least from a longer term perspective. It's early days yet, but XLA seems to be a promising intermediate representation for allowing the DL frameworks to run on a wider array of hardware without user-facing software changes. It has traction with Google, NVIDIA, IIRC Intel and maybe more (others are definitely using the same approach of compute graph splitting and subgraph scheduling, but I'm not certain if they are using XLA specifically - I know some aren't like Mindspore). XLA has already proven its value by allowing PyTorch to run on TPUs (shittily, but that appears to be more of a VM/GCP infra problem than an XLA problem). The work done for TPUs (and to a lesser extent for GPU optimization) has started to expose some of the major issues and so work can start on addressing them (the cost of dynamic XLA compilation as tensor shapes change and how lots of important code assumes that accelerator-to-CPU communication isn't tooo expensive, but it's is a huge issue when trying to compile the graph into machine-specific code with XLA or similar to because it forces you to only be able to compile small subgraphs). It's early, but the rise of a really effective IR in XLA combined with the huge amount of resources that Google/NVIDIA can pour into XLA makes me very bullish on purpose-built hardware for AI training. It will take a while I admit.
- sillysaurusx 6y ago
  Mr/mrs anonymous HN person, please put some info in your profile. You clearly have some deep knowledge of TPUs that I didn’t expect to pop up offhandedly on HN. You’re correct on all counts: dynamic tensor shapes are more or less impossible with XLA, making it more or less impossible to train a model with arbitrary image size inputs, even though the math would allow for that; the pytorch XLA work on TPUs is indeed kind of shitty, and I’m surprised as heck that literally anyone said this except me; and XLA as an IR is promising for portability. Now I’m curious what you’ve been doing to have experienced these things, since there didn’t seem to be many others who have (or at least, who are vocal about it). I agree with you, but I think we differ on our timetables. I am bearish for the next two years, at which point I’ll awaken from my slumber and become a flaming bull. (It helps to remember that “we overestimate the impact of years, but underestimate the impact of decades.” I try to plan accordingly.) In other words, if you’re bullish that two years from now we’ll start seeing portability implemented in the field across various HPC chips, then we fully agree. But that’s also a glacial pace; GPT-2 changed the world almost two years ago now, and DALL-E seems to be the next frontier for doing interesting generative work. So, we’ll split the difference and say that the bears and bulls will meet in two years for a deep learning hackathon. As a bonus, the pandemic will be over by then, so it can be an in-person meetup.
fromaj 6y ago
But they do have PyTorch support??? They ship a 'cerebras graph compiler' extension for pytorch that allows you to replace the default optimizer with theirs.
londons_explore 6y ago
> That means you’re looking at two months of R&D minimum to get everything rewritten, For a hobbiest, sure, that's a problem. But for a big company with a big ML research team already, that isn't an issue - they just assign a few people to work on it, and in a few months it's done. If you're running any model at scale anyway you probably want to rewrite everything to make it run efficiently on your hardware.

whimsicalism 6y ago

I'm on board with you that there should be a "drop-in" cross support of these chips, but pytorch is at a way higher abstraction level than what should be commonly supported.

socialdemocrat 6y ago

Maybe people will get to their senses and switch to Julia instead of having to waste all these time on Python bindings.

habibur 6y ago
Python here basically works as a binder for C, which is what everything is written in.
- eigenspace 6y ago
  That is in fact the problem, yes.
dnautics 6y ago
Honestly I think julia missed the boat. There was an opportunity for julia to treat the gpu as an abstract distributed node (which is a first class concept in julia) and pay respect to the reality of data gravity. But they chose to instead basically treat the gpu as a synchronous entity. I'm now bullish on elixir-nx because I think there's an outside shot they will get it right. Python is a lost cause.
- BadInformatics 6y ago
  I'm not sure that's necessarily the domain of a low-level package like CUDA.jl though (which I assume you're referring to). That kind of interface is more the domain of higher-level packages like https://github.com/JuliaGPU/DaggerGPU.jl https://github.com/JuliaGPU/DaggerGPU.jl and to a lesser extent https://juliagpu.github.io/KernelAbstractions.jl/stable/ https://juliagpu.github.io/KernelAbstractions.jl/stable/. Moreover, the jury is still out on whether the built-in Distributed module is an ideal abstraction for every use-case (clusters, heterogeneous compute, etc.) WRT Nx, my biggest question is how they'll crack the problem of still needing big balls of C++ and the shims everywhere to get acceleration. Creating a compiler that generates efficient GPU or other accelerator code is a massive research project with no clear winners, never mind the challenge of reconciling the very mutation-heavy needs of GPU compute with a mostly immutable language model.

zucker42 6y ago

> The philosophy here seems to be “if we build it, they’ll buy it.” Supposedly Cerebras is already profitable, so it's hardly a situation where they are building something and hoping people buy it eventually. > That means you’re looking at two months of R&D minimum to get everything rewritten, running, tested, trained, and with an inferencing pipeline to generate samples. Again, based on the companies representations, Cebebras transparently supports Pytorch and Tensorflow, only requiring a few lines of changed code. Source: https://www.anandtech.com/show/16626/cerebras-unveils-wafer-scale-engine-two-wse2-26-trillion-transistors-100-yield https://www.anandtech.com/show/16626/cerebras-unveils-wafer-... (Dr. Cutress's video on TechTechPotato is also good).

sillysaurusx 6y ago
I’ve learned to be skeptical of such claims. TPUs made that same claim, but it’s hard to use it for real work, since you don’t have access to the 300GB TPU host memory / cpu, where you infeed training examples. There is a very specific test for “supports pytorch / tendorflow”: show an MLPerf imagenet resnet benchmark. It’s impossible to fake that. If you come anywhere close to TPUs in tensorflow, people (like me) will leap: https://mlcommons.org/en/training-normal-06/ https://mlcommons.org/en/training-normal-06/ Till then, it’s a “proof, please” type of situation.
- sanxiyn 6y ago
  I agree. I have high hope for Graphcore. They stated "We will be participating in MLPerf in 2021, starting with the first training submission in the spring". https://www.graphcore.ai/posts/graphcore-sets-new-ai-performance-standards-with-mk2-ipu-systems https://www.graphcore.ai/posts/graphcore-sets-new-ai-perform... (I would ignore all benchmarks in that post except the sentence quoted above however.)
- algo_trader 6y ago
  Software will lag new hardware. Duh. The question is whether this is solvable (given a couple of master devs) Cerebras itself commented that DL is not their first use case. They mentioned fluid dynamics instead. (I can find the reference if someone is interested). Cerebras also has a weird memory/compute ratio, so very hard for some uses. Somewhat off-topic. If u have standard keras/TF training code, the GPU to TPU is a smooth-ish transition. No? A re-write will squeeze another 20% but not 200%?

habibur 6y ago

So we are looking for OpenCUDA for AI. Like OpenGL in the 90s.

DesiLurker 6y ago
So basically OpenCL?
- byronyi 6y ago
  I'd bet on SPIR-V.

zitterbewegung 6y ago

I think apple expects you to use pretrained models and convert them to CoreML. There is also webGPU which is experimental on Safari and Chrome. But that doesn’t touch the ML accelerators on the M1.

systemvoltage 6y ago

You're bearish on new hardware for AI training because it takes 2 months to setup? That's like saying you're bearish on the invention of transistor because there are no computers yet. This chip is for research organizations such as LLNL and ANL. Tooling and commoditization will happen. > “if we build it, they’ll buy it.” This is literally how any new thing is invented and commercialized.

sillysaurusx 6y ago
Well, two months has a way of turning into two years. June 2019 was right around when I fell in love with TPUs, and we’re still happily TPUing today. Sometimes, though, it would be nice to simply focus on doing interesting ML work rather than wrestling with the intricacies of tensor slicing and memory access pattern optimization (to say nothing of the cursed inability to manually manage memory, resulting in explosions of allocations in the forward pass for unclear reasons). Let’s put it this way. If you contract me to implement a full gpt model on this new hardware, two months full time work would be my minimum estimate. That’s 40 hours a week of focused effort, with no breaks and no other projects. I don’t know about you, but two months with nothing to show can demoralize most teams I’ve worked with. The point was, we’re early in ML’s lifecycle. And, like React for webdev, people are slow to change their habits — for better or worse, pytorch and tensorflow are the APIs people think in. So if you want your hardware to be widely adopted, you need tooling that supports the workflows people have spent months learning. Jax is on the horizon too. Supposedly they’re launching something soon that might tip the scale in their favor. Perhaps an ambitious hardware vendor could capture the future market by preemptively implementing Jax support. But perhaps not: at that point you’d be competing toe to toe with Google’s TPU offering, since Jax is Google’s dogfood. Right now my money is on TPUs, partly because of their fantastic support staff. But maybe some other company will come along and offer a better integrated cloud experience.
- durst 6y ago
  Do you have a longer writeup on your experience performance tuning for the TPU? Is https://cloud.google.com/tpu/docs/performance-guide https://cloud.google.com/tpu/docs/performance-guide the best resource in this area?

oivey 6y ago

Graphics APIs and deep learning frameworks are really nothing alike. In graphics, save for a few new features, the goal every year is the same: draw more and faster. Deep learning is evolving much faster than that. The answer isn’t an API; it’s powerful and mostly transparent automatic differentiation. Anything that is just an API will forever be behind and is much less useful for active research. There are a few projects around improving AD support, but it’s a relatively difficult problem because to do it best you need deep compiler integration.

sanxiyn 6y ago
The answer is an API, like NNAPI. AD is a frontend concern and doesn't really matter to accelerators. For AD, I am bullish for Enzyme, which does AD on LLVM IR, avoiding deep compiler integration: https://enzyme.mit.edu/ https://enzyme.mit.edu/
- oivey 6y ago
  I would say that integrating AD at the level of LLVM IR is deep compiler integration. The AD has to consume IR and then emit more IR to actually take the derivatives. If you’re using a Python wrapper for this you’ll need codegen for interacting with the generated IR, too. The upside with that approach is that it might more easily work across languages. The downside is that LLVM IR can’t rely on the rich invariants that even C exposes, which makes codegen and optimization harder. An API is always going to be necessarily behind state-of-the-art because often research depends on inventing things that don’t fit within an existing API.
  - sanxiyn 6y ago
    I think we are talking past each other. sillysaurusx is talking about OpenGL/Direct3D as a portability layer for hardwares. API vs AD is not a hardware portability layer concern. The proposal is to standardize OpenGL/Direct3D-like hardware portability layer defined by API. API vs AD you are talking about happens far above that and it is irrelevant.
    - 6y ago

justaguy88 6y ago

You might like this talk that describes parts of the compiler ecosystem that are lacking standardization for accelerators https://www.youtube.com/watch?v=4HgShra-KnY https://www.youtube.com/watch?v=4HgShra-KnY

sanxiyn 6y ago

> What we need is a unified API that can easily support multiple backends Android standardized NNAPI and on-device inference is in pretty good shape. As you said, training is different matter.

de6u99er 6y ago

Thr Anandtech article from a other comment here indicates that both PyTorch and Tensorflow are already running on it. >A key to the design is the custom graph compiler, that takes pyTorch or TensorFlow and maps each layer to a physical part of the chip, allowing for asynchronous compute as the data flows through. https://www.anandtech.com/show/16626/cerebras-unveils-wafer-scale-engine-two-wse2-26-trillion-transistors-100-yield https://www.anandtech.com/show/16626/cerebras-unveils-wafer-...

jquery 6y ago

> We had a similar situation in gamedev circa 2003-2009. Practically every year there was a new GPU, which boasted similar architectural improvements. You mean one of the most rapid periods of graphical improvement in history? Growth was just too breakneck to standardize for a bit.

MaxBarraclough 6y ago
sillysaurusx's comment goes on to acknowledge that the major graphics libraries (OpenGL and Direct3D) did a pretty good job of abstracting away/indirecting away the architectural changes. As you indicate, the architectural changes were made for good reason, and translated to real improvements in performance/capabilities. If AI acceleration chips aren't able to offer similar API standardisation/stability then it's not the same as what happened with GPUs, which was sillysaurusx's point.

TFortunato 6y ago

The article mentions that developers use standard PyTorch and Tensorflow.

docfort 6y ago

You might be interested in this for your M1 MBA: https://github.com/apple/tensorflow_macos https://github.com/apple/tensorflow_macos

sillysaurusx 6y ago
That was actually what I was referring to! It's super weird. I like it, and it's what I use primarily, but only because no other version of tensorflow will install. Their compiler is closed source, so it's almost impossible to tell what's going on. But, when I connect using tf.Session(), then .list_devices() shows only CPU available. However, when I enable the TF_MLC_LOGGING=1 magic variable, it does seem to be printing out messages that indicates it's doing some kind of graph substitution under the hood. Therefore, I assume that this is the intended usage mode. In other words, there seems to be zero difference between the "CPU" and the "GPU". Normally you can say "Do this on the CPU" while "do that on the GPU." But not with this. Hopefully they'll open source the code sometime this century so that it's clearer what the heck it's doing. For now, though, it's reasonably fast in whatever this "CPU" mode is -- I only need to run unit tests on my laptop anyway, since all training happens on TPUs. So I ended up happy. (For the first day or so, I was panicking that I was going to have no working tensorflow whatsoever on my M1 laptop, which would've necessitated a swift return + substitution.)
- buildbot 6y ago
  That seems very apple honestly, they’d rather you emit pure metal primitives (is that the right term? Idk) and then let their backend schedule on their soc- because for all we know, they have some special asic ip that they added/or will add and can take advantage.
- markonen 6y ago
  With the disclaimer that I know nothing about this: doesn't the M1 have a separate "Neural Engine" for this? So it's using neither the CPU nor the GPU cores for TensorFlow?

blihp 6y ago

The philosophy is working as it often does in the early days/years of a technology. The customers for this have problems at an entirely different scale. It's worth it for them to drop millions on hardware and potentially millions more to (re)write the software as needed. Sure, they'd prefer to be buying commodity hardware with standardized APIs. But since that doesn't currently exist for this class of performance, they'll pay up. People did the same back in the days of Cray/SGI/Lisp Machines and so on. These custom solutions can be quite profitable until the competition gets to the 'good enough' stage and then orders start drying up. If you're looking for standardized hardware and software for AI, Apple is the wrong platform to be on. I'm fairly confident it won't be happening there. (I don't say that to be a hater: see the direction they've gone re: GPU APIs... it's just Apple being Apple)

dmingod666 6y ago

If you look at it historically and put it in scale with the AI winter, 2 months is next to nothing..

solidasparagus 6y ago

I don't really agree, at least from a longer term perspective. It's early days yet, but XLA seems to be a promising intermediate representation for allowing the DL frameworks to run on a wider array of hardware without user-facing software changes. It has traction with Google, NVIDIA, IIRC Intel and maybe more (others are definitely using the same approach of compute graph splitting and subgraph scheduling, but I'm not certain if they are using XLA specifically - I know some aren't like Mindspore). XLA has already proven its value by allowing PyTorch to run on TPUs (shittily, but that appears to be more of a VM/GCP infra problem than an XLA problem). The work done for TPUs (and to a lesser extent for GPU optimization) has started to expose some of the major issues and so work can start on addressing them (the cost of dynamic XLA compilation as tensor shapes change and how lots of important code assumes that accelerator-to-CPU communication isn't tooo expensive, but it's is a huge issue when trying to compile the graph into machine-specific code with XLA or similar to because it forces you to only be able to compile small subgraphs). It's early, but the rise of a really effective IR in XLA combined with the huge amount of resources that Google/NVIDIA can pour into XLA makes me very bullish on purpose-built hardware for AI training. It will take a while I admit.

sillysaurusx 6y ago
Mr/mrs anonymous HN person, please put some info in your profile. You clearly have some deep knowledge of TPUs that I didn’t expect to pop up offhandedly on HN. You’re correct on all counts: dynamic tensor shapes are more or less impossible with XLA, making it more or less impossible to train a model with arbitrary image size inputs, even though the math would allow for that; the pytorch XLA work on TPUs is indeed kind of shitty, and I’m surprised as heck that literally anyone said this except me; and XLA as an IR is promising for portability. Now I’m curious what you’ve been doing to have experienced these things, since there didn’t seem to be many others who have (or at least, who are vocal about it). I agree with you, but I think we differ on our timetables. I am bearish for the next two years, at which point I’ll awaken from my slumber and become a flaming bull. (It helps to remember that “we overestimate the impact of years, but underestimate the impact of decades.” I try to plan accordingly.) In other words, if you’re bullish that two years from now we’ll start seeing portability implemented in the field across various HPC chips, then we fully agree. But that’s also a glacial pace; GPT-2 changed the world almost two years ago now, and DALL-E seems to be the next frontier for doing interesting generative work. So, we’ll split the difference and say that the bears and bulls will meet in two years for a deep learning hackathon. As a bonus, the pandemic will be over by then, so it can be an in-person meetup.

fromaj 6y ago

But they do have PyTorch support??? They ship a 'cerebras graph compiler' extension for pytorch that allows you to replace the default optimizer with theirs.

londons_explore 6y ago

> That means you’re looking at two months of R&D minimum to get everything rewritten, For a hobbiest, sure, that's a problem. But for a big company with a big ML research team already, that isn't an issue - they just assign a few people to work on it, and in a few months it's done. If you're running any model at scale anyway you probably want to rewrite everything to make it run efficiently on your hardware.