9 ms·

I'm happy to hear that this is finally public so I can actually talk about the work I did when I was at Google :-). I'm a bit surprised they announced this, th

• mike_hearn 10y ago
  Google has always been strategic about announcing what it was doing, even since the early days (I used to work there too). Think about the impact the first MapReduce and GFS/BigTable papers had. My guess as to why they're announcing the TPU is that they are feeling the pressure from Facebook and other AI labs, and want to reinforce their reputation as being the best place to do AI research. By revealing that AlphaGo was based on this hardware, they indicate to researchers around the world that if you want to build the most advanced ML models you need to be at Google. Same reason they talked about MapReduce/GFS back in the day.
  • arcanus 10y ago
    > By revealing that AlphaGo was based on this hardware Interesting, as the nature/science paper made no mention of this, it was exclusively trained on GPUs.
    • honkhonkpants 10y ago
      Training and inference are not the same thing.
      • arcanus 10y ago
        Sure... I think you are missing the point. In Nov., the Nature paper which contains the AlphaGo algorithm, the hardware detailed was exclusively CPU+GPUs. Between that time and the Lee Sedol match, the hardware running AlphaGo was switched, to these TPU. From the paper: "The final version of AlphaGo used 40 search threads, 48 CPUs, and 8 GPUs. We also implemented a distributed version of AlphaGo that exploited multiple machines, 40 search threads, 1,202 CPUs and 176 GPUs."
        • dsl 10y ago
          Or they consider the TPU to be something similar to a math co-processor, it just offloads work from the CPU. They also didn't discuss the custom built power modules, or the custom built network switches used for interconnects.
    • Merovius 10y ago
      Note that the version of AlphaGo that beat Fan Hui and was presented in Nature is significantly different from the Version that played Lee Sedol. Unless you believe that they didn't work on it for half a year.
      • indolering 10y ago
        This explains a lot. Going into the match, Sedol thought he could beat DeepMind but thought it might only be a couple of years until the technology outpaced him. We knew Fan Hui and other Go professionals were helping the team, but a massive speedup is always nice too. It's a bit underhanded, however. IMHO, the player should be able to study recent games before the match. But this is pretty typical, there were similar late-stage improvements with Chinook (checkers) and Deep Blue (chess).
  • Krakulf 10y ago
    It really boosts Google Cloud Platform with image and prestige, they're coming out with what is pretty much tensorflow as a service with their machine learning product. Coming out saying they can give you a service no one else can right down to a custom chip may sway a few buyers in the market. https://cloud.google.com/ml/ https://cloud.google.com/ml/
    • hendler 10y ago
      For me this is true. GPUs on Amazon are not cheap, and if google passes the savings on from performance power consumption I'd certainly offload some work to Google cloud.
    • engizeer 10y ago
      This is true - I've heard rumors that GCE has been trying hard to poach cloud consumers from other providers, this could be another stroke in that effort.
    • manav 10y ago
      Well this also shows their commitment to Tensorflow if they willing to pay to fab custom chips for it. Then again, in the Google scheme of things that's probably not a huge cost. They've probably carefully modeled the energy savings over GPU/FPGA and found it to be substantial enough, even taking into account costs of design changes.
      • vidarh 10y ago
        If you need enough chips it doesn't take particularly careful modelling to say that you'll beat an FPGA quite substantially in most cases.
    • vintermann 10y ago
      It might also give Nvidia some sorely needed worries.
• gsnedders 10y ago
  My vague impression is that there had started to be more and more research on use of ASICs for ML (though I can't say I follow it that closely). Enough of a changing tide that the competitive advantage was going?
• zymhan 10y ago
  Wouldn't you typically prototype the design on an FPGA and then manufacture an ASIC once you'd worked out the kinks?
  • amelius 10y ago
    I'm suspecting that the design contains a lot of repetitive units, each of which can easily be tested in software. So perhaps testing on an FPGA isn't even needed.
    • zhemao 10y ago
      I think zymhan meant prototyping on an FPGA to see if you could actually see an improvement by mapping parts of your algorithm to custom hardware. This would be a sensible first step before committing to a full ASIC run. I'm sure Google did do this initially. As for testing for correctness, ASIC designs are almost always tested in software simulation instead of on FPGAs. Mainly because RTL written for FPGAs can look quite different from RTL written for ASIC synthesis. There's also a lot incidental engineering work needed to get something working on an FPGA.
      • gchadwick 10y ago
        > ASIC designs are almost always tested in software simulation instead of on FPGAs. Mainly because RTL written for FPGAs can look quite different from RTL written for ASIC synthesis. There's also a lot incidental engineering work needed to get something working on an FPGA. It's actually very common to test ASIC RTL on FPGA. You are right that if you are targeting an FPGA you may write things differently, but this doesn't mean RTL written for an ASIC won't work it's just it might not clock as fast as it could and may not make the most efficient use of resources. There is a lot of engineering work to take an ASIC targeted IP and run it in an FPGA, but thorough verification of an ASIC design is extremely important so it's usually worth the time to bring up an FPGA that can run it (sometimes as it's too large you have to split it across multiple FPGAs).
    • raverbashing 10y ago
      Ah I'd state the opposite. Your design will have new problems showing up when going from FPGA to ASIC Especially if you try to make it go faster
• amelius 10y ago
  There were no references to this TPU in the TensorFlow source code?
  • hedgehog 10y ago
    No but if you look at the code it's clearly designed to support pluggable hardware.
    • zenlikethat 10y ago
      Very interesting observation -- this seems so much more obvious in retrospect given the meticulous mapping of graph to devices outlined in the paper.
• rdtsc 10y ago
  "We have an FPGA, oh but need to go faster, ok, build an ASIC then" is a natural thing to come up with. That's kind of what bitcoin farms did. Obviously details and plans how it was done is where all the good stuff is, so that being hidden is understandable.
  • randyrand 10y ago
    I'm not even a hardware or AI person, and even I could have told you ASICs would make way more sense than GPUs or FPGAs for machine learning. It's all about data locality. Fetching memory is the most costly thing a GPU does, and for ML (DNNs) there no big need for global access memory 99% of the time. Anyone the casually follows AI knows that people have been talking about making DNN ASICS for some time. It was all a matter of time and $$$$$$ There is no doubt FB is working on them too. Which is why Google is finally publicaly saying that "we did it first ;)"
    • rdtsc 10y ago
      > Google is finally publicaly saying that "we did it first ;)" Makes sense. In that respect yeah, they probably wanted to keep it under wraps to avoid Facebook/others from getting a timeline estimate out of it and jump ahead.
    • mtgx 10y ago
      > It's all about data locality. That's kind of what Movidius says, too, about its Myriad 2 VPU, which is kind of a GPU (SIMD-VLIW) with larger amounts of local memory combined with hardware accelerators.
• blue11 10y ago
  OK, so what is it? The announcement neither says what a TPU actually is nor what it can do. It's a magic black box. No specs. No price.
  • Florin_Andrei 10y ago
    Well, it's a first announcement on a blog. They say it accelerates TensorFlow by 10x. They say it fits in an HDD slot. And the whole announcement must stay within a page or two. It's a "more details to follow" type of thing. Pretty standard actually.
    • CydeWeys 10y ago
      The cool thing here is that it's already widely deployed in production, so we know it actually works and that it's not just vaporware. The "more details to follow" is thus a lot more convincing than for a lot of similar-seeming announcements of things that don't actually exist yet (and often don't ever).
    • lightcatcher 10y ago
      > They say it accelerates TensorFlow by 10x They say 10x performance / watt, nothing about performance per unit time.
      • DigitalJack 10y ago
        You can make some assumptions though. If the power consumption was equal, the performance is 10x. The speed at which an ASIC will run is constrained by temperature (power dissipation) and and logic timing, which itself has a dependency on temperature. So we could call that vertical scaling, to some power ceiling which may not take us all the way to 10x, but it's not impossible. Then there is horizontal, which I assume is applicable to these problems... running more in parallel. In both cases, I think it's safe to assume they are getting a performance increase in the instantaneous sense.
        • zenlikethat 10y ago
• panic 10y ago
  How different are TPUs from GPUs? From the article, it sounds like TPUs use lower-precision arithmetic: are there any other differences?
  • sitkack 10y ago
    Not just lower precision but probably less error correction in the lower bits. -Or- ... they convert the floating point values to analog values and do all the math in the analog domain and convert it back, just like old analog computers. But by going fully analog, I think the gains would be over 10x better, so probably not. See 'low-power (inexact|approximate) computing'
    • redcalx 10y ago
      They likely use half precision (i.e. 16 bit) floats in some of the computing stages. They've also discussed using even fewer bits of precision in some of their research output. When training and following an error gradient you kind of need lots of precision, but when running a learned model you often don't need much at all, 8 or even 4 bit numbers are sufficient in some calcs where rounding error propagation isn't an issue.
• rcpt 10y ago
  > it's impossible to deploy GPUs at scale, let alone ASICs A bit old but cf. DE Shaw and Anton https://en.m.wikipedia.org/wiki/Anton_(computer) https://en.m.wikipedia.org/wiki/Anton_(computer)
  • dekhn 10y ago
    I think the jury is truly still out on whether the anton designs are worth it (compared to commodity clusters/GPUs, etc). and Anton wasn't really "scale" in this sense. It was a vertically scaled single (or several) machines.
• joshhart 10y ago
  I've only dabbled in this area, so it's entirely possible my question doesn't make sense. Machine learning algos in general seems to have a bunch of iterative components, but within each iteration there is massive parallelism. Did you consider using a whole bunch of DSPs? They can also do parallel processing but might be cheaper in bulk than GPUs.
• chlestakoff 10y ago
  Now that you are free to talk about it, could you explain how ASICs stack up against GPUs, in practical terms?
  • DigitalJack 10y ago
    ASIC stands for "application specific integrated circuit" and the name conveys a special purpose design of an integrated circuit, like for example bit coin mining. But that's just terminology/convention. One could argue a GPU is an ASIC, and a CPU is an ASIC. The only thing to argue is how specific does an application have to be to call it an ASIC instead of some other made up name. The prime differentiators are the process, which is the term used for the many steps of fabrication of an IC (processes are often referred to as nodes, distinguished by the smallest feature size of a transistor they create), and the ability of the design engineers to create an optimal design, in terms of boolean logic and semiconductor physics. Given the right team of engineers, and a top notch foundry, and a great deal of experience in the problem domain (machine learning in this case), a custom IC could very likely trounce a GPU. GPUs were designed for the domain of graphics processing, which happens to have some commonality with the processing in machine learning. But, at least until recently, GPUs weren't focused on machine learning. Just graphics. Now the GPU vendors are trying to leverage their knowledge of graphics processing and building of graphics processors to create machine learning processors, but the thing they are leveraging could also be what handcuffs them. Which gives opportunity for a company like Google to do a fresh take on the problem domain without the baggage of the knowledge of graphics processing.
• sandGorgon 10y ago
  What was your design flow? Did you use the standard cadence, Synopsys toolset to build this? How did you achieve basic design and JIT(or whatever it is you call the machine code part of the Python VM ) together? Rrally interested to know how this was managed...
• jldugger 10y ago
  It makes a lot more sense now why they open sourced Tensor Flow -- they have an ASIC design that supports it, and if they either want to sell the chips, or sell access to them via Google Cloud Platform, they'll need customers. And it's not like HP is about to roll out a server line sporting these any time soon, so Google will continue to have their advantage over the outside world.
• auvi 10y ago
  any idea which foundry google have used for the fabrication?
• rolandog 10y ago
  I wonder if this is a response as well to Nvidia's announced Tesla P100 chip?
• sowhatquestion 10y ago
  Hold up, you mean teams at Microsoft are "mailing zip files around!?"