20 ms·
Google supercharges machine learning tasks with TPU custom chip
- isseu 10y agoTensor Processing Unit (TPU) Using it for over a year? Wow
- semisight 10y agoThis is huge. If they really do offer such a perf/watt advantage, they're serious trouble for NVIDIA. Google is one of only a handful of companies with the upfront cash to make a move like this. I hope we can at least see some white papers soon about the architecture--I wonder how programmable it is.
- dharma1 10y agoit's ASIC tuned for specific calculations, I'm sure it's better power consumption than general purpose GPUs. Same as crypto mining ASIC's crush GPU's in terms of power efficiency. There isn't much data yet but I'm also guessing they probably have access to much more RAM than NVidia cards and can process much bigger data sets
- codecamper 10y agoI don't see any mention of offering these chips for sale. You can rent them it seems via cloud offerings & that's it.
- semisight 10y agoYup, I assume they're gonna keep them in house as a competitive advantage for a time. I doubt they'll do it forever; the most valuable part of NVIDIA's CUDA is the ecosystem, and I think Google knows that.
- knorker 10y agoI would assume that the API to use these is tensorflow. So... just use Google's machine learning cloud thingy. The software can build the community, where the supercharging is only available when you run it on Google cloud. (although GPU performance isn't bad either, so you don't have to, thus community)
- babo 10y agoSure, but that's the deal. I'll buy the latest nVidia 1080 card as soon as I can but renting these custom chips per minute would be a way better option for me.
- retox 10y agoGPUs also have this nice side effect of being great at playing games on. Purely as a guess I'd think that the gaming market is bigger than the AI researcher market.
- omarforgotpwd 10y agoIn a future where AI is everywhere, Nvidia hopes it can sell GPUs by the hundreds and thousands to large data centers. You can make a lot more money a lot faster selling your hardware this way, and Nvidia is very interested in it judging from how much they talked about it at their recent conference.
- retox 10y agoI would be surprised if they weren't working on their own specialised chips then, though Google have the advantage of already having the software specs to build for.
- CydeWeys 10y ago> Purely as a guess I'd think that the gaming market is bigger than the AI researcher market. Machine learning isn't just targeting the AI researcher market though -- it's widely used by a huge number of companies, and of course, by many of Google's most important products. I would argue that those markets combined are larger than gaming.
- nickysielicki 10y agoThere's no way Google lets this leave their datacenters. Chip fabrication is a race to the bottom at this point. [1] Google is doubling down on hosting as a source of future revenue, and they're doing that by building an ecosystem around Tensorflow. What I think is interesting is how weak Apple looks. Amazon has the talent and money to be able to compete with Google on this playing field. Microsoft is late, but they can, too. Where's Apple? In the corner dreaming about mythical self-driving luxury cars? [1]: http://spectrum.ieee.org/semiconductors/design/the-death-of-moores-law-will-spur-innovation http://spectrum.ieee.org/semiconductors/design/the-death-of-...
- honkhonkpants 10y agoApple designs their own CPUs. I think they'd be able to field a massively parallel FMAC chip if they thought that was a good idea. Where Apple really looks weak is in datacenters, networking, and cloud services.
- nickysielicki 10y agoWhat does the iPhone of 2021 look like? I get the feeling from today's announcements that Google sees the 2021 version of Google Now as the selling point for their 2021 Nexus line. I don't think Apple is preparing to compete on that.
- habitue 10y agoI would say they're already not competing on the assistant side. Siri is considerably worse than Google Now, even though it came out first
- Razengan 10y agoApple's strength is in consumer (and to a lesser extent, developer) ecosystems; the cozy comfortable bubble you get when you're surrounded by everything Apple. Getting access to your stuff across multiple devices is virtually effortless and continually seamless, with almost no configuration required. Whether that's good or not may be arguable, but it's certainly a selling point for many and I don't see Google or any other company's offerings approaching the same experience, and I suspect that's by design; they have to be more open and support all devices but that kinda dilutes everything. Apple will only get stronger in that aspect IMO.
- deleted 10y ago[deleted]
- mtgx 10y agoQuantum computers, OpenPower, RISC-V, and now this - I'm really liking Google's recent focus on designing new types of chips and bringing some real competition into the chip market.
- DSingularity 10y agoWhat are they doing with RISC-V?
- PeCaN 10y agoThey dumped a bunch of money into it, so presumably they're at least interested.
- agumonkey 10y agoI'm surprised by the perf claims. Nvidia isn't doing kids play. The graph implied they were untouchable in terms of perf...
- azinman2 10y agoNdvidia has to be general purpose. This is not and thus can be better optimized.
- fiatmoney 10y ago"General purpose" isn't that general, if you look at the actual operations they support and their threading model. It's already fairly optimized for these sorts of operations, and this amount of claimed headroom makes me suspicious.
- azinman2 10y agoThe fact that I can compile arbitrary programs for the GPGPU means it is general purpose. NVIDIA isn't writing softmax or backprop into silicon as a CPU instruction. Look at how much faster ASICs for bitcoin mining are than the GPU... orders of magnitude.
- fiatmoney 10y ago"Backprop" isn't even close to something that would be a "CPU instruction", it's an entire class of algorithm. It's like saying "calculus" should be a CPU instruction. Matrix multiplication & other operations, on the other hand, do neatly decompose into such instructions, which have been implemented by NVidia et al., since that's the core set of functionality they've been pushing for like a decade now. Additional die space on additional functionality might hurt the power envelope (which is where the focus on performance / watt rather than performance kicks in) but it doesn't make your chips slower per se.
- agumonkey 10y ago
- nkw 10y agoI guess this explains why Google Cloud Compute hasn't offered GPU instances.
- hyperopt 10y agoThat's what I'm thinking. I was anticipating the release of GPU instances, but now I'm thinking that they will simply leapfrog over GPU instances straight to this.
- harigov 10y agoHow is this different from - say - synthetic neurons that IBM is working on, or what nvidia is building?
- DannyBee 10y agoWithout sounding crass: 1. It works already (IE it's already in use) 2. It works really well (or else they wouldn't be using it so broadly) 3. Considering how long this was said to be in development, it also likely means they are working on the next big improvement before these guys have even gotten the current one working.
- daveguy 10y agoThey are already using it in production and have been for over a year? That is a huge difference compared to what people are working on and building. They beat Lee Sedol using this hardware. Over 100 teams at Google are using machine learning. This hardware already accelerates their projects.
- bgalbraith 10y agoIBM's TrueNorth chip is taking a much more neuromorphic design approach by trying to approximate networks of biological neurons. They are investigating a new form of computer architecture away from the classic Von Neumann model. TPUs are custom ASICs that speed up math on tensors i.e. high-dimensional matrices. Tensors feature prominently in artificial neural networks, especially the deep learning architectures. While GPUs help accelerate these operations, they are optimized first and foremost for video rendering/gaming applications -- compute-specific features are mostly tacked on. TPUs are optimized solely for doing ML-related computations.
- jhartmann 10y ago3 generations ahead of moore law??? I really wonder how they are accomplishing this beyond implementing the kernels in hardware. I suspect they are using specialized memory and an extremely wide architecture. Sounds they also used this for AlphaGo. I wonder how badly we were off on AlphaGo's power estimates. Seems everyone assumed they were using GPU's, sounds like they were not. At least partially. I would really LOVE for them to market these for general use.
- reitzensteinm 10y agoBut isn't 3 generations ahead just 8x? Which doesn't sound at all unreasonable for a custom hardware.
- dnautics 10y agoThis is about right! 64-bit IEEE fp -> 16-bit IEEE-style fp[0] is a 4x bit size reduction, and multiplication is O(n^2) is silicon transistor count. [0] If google is smart, they'd ditch +/- infinity and if they were ballsy, they'd ditch zero in their FP implementation.
- afsina 10y agoActually for many neural network apps 8, 4 even 1 bit is sufficient for representing weights.
- kevinnk 10y agoI doubt FP hardware size is the limiting factor in their implementation (it's not in GPUs and definitely not in high perf CPUs). More likely they came up with higher level architectural tricks that let them specialize for machine learning (i.e. taking better advantage of locality in the application, etc).
- Symmetry 10y agoGenerally speaking GPUs are already very good at running with float32s, usually much better than they are at using float64s in fact. The big advantages of using an ASIC are mostly on the storage side but they also allow you to get away with non IEEE floating point numbers that don't necessarily implement subnormals, NaN, etc.
- abritishguy 10y agoI think this shows a fundamental difference between Amazon (AWS) and Google Cloud. AWSs offerings seem fairly vanilla and boring. Google are offering more and more really useful stuff: - cloud machine learning - custom hardware - live migration of hosts without downtime - Cold storage with access in seconds - bigquery - dataflow
- obulpathi 10y ago- Google Container Engine - Cloud Shell to name a couple more.
- eva1984 10y agoMeaning even deeper level of vendor lock -- now you cannot even find the chips to run your application elsewhere!
- dgacmu 10y agoNo - tensorflow is open source and you can run it on many platforms. TPUs are about efficiency. You might not be able to do image recognition as efficiently without one, but you can still perform exactly the same tasks. (I work on TF this year.)
- eva1984 10y agoWell, that is good to know. Thanks for clarification.
- Retric 10y agoI would be shocked if tensorflow optimizations where useful 1:1 for stock Intel chips or GPU's. So, there is still plenty of lock-in even if your process runs. GPU vendors love to play this game by helping optimize games.
- dgacmu 10y agoIt's possible, but I think that the majority of ML optimization as seen by a programmer using tensorflow is more about optimizing the balance of accuracy, training & inference speed, and memory use, and a lot of the solutions in this space are pretty hardware independent. There's an entire other type of optimization about, e.g., making conv2d insanely fast, but that's not something that a typical data scientist-type user deals with. (To elaborate -- it's questions like "how deep should I make this convolution? Should I use tf.relu or tf.sigmoid? How many fully-connected layers should I put here, and how big should I make them?". These are really knotty deep learning design questions, but they're often h/w independent. Not always - we certainly have some ops on TF that we only support in CPUs and not on GPUs, for example - but often.)
- Bromskloss 10y agoWhat is the capabilities that a piece of hardware like this needs to have to be suitable for machine learning (and not just one specific machine learning problem)?
- wmf 10y agoAFAIK 16-bit "half-precision" floating point.
- HappyTypist 10y ago8 bit is enough and I suspect it's what the TPU is using: https://petewarden.com/2016/05/03/how-to-quantize-neural-networks-with-tensorflow/ https://petewarden.com/2016/05/03/how-to-quantize-neural-net...
- visarga 10y agoIt is interesting how malleable are neural networks. - you can drop half the connections and it still works, in fact it works even better, during training - you can represent the weights on as little as one bit, but still use real numbers for computing activations - you can insert layers and extend the network - you can "distill" a network into a smaller, almost as efficient network or an ensemble of heavy networks into a single one with higher accuracy - you can add a fixed weights random layer and sometimes it works even better - you can enforce sparsity of activations and then precompute a hash function to only activate those neurons that will respond to the input signal, thus making the network much faster It seems the neural network is a malleable entity with great potential for making it faster on the algorithmic side. They got 10x speedup mainly on exploiting a few of these ideas, instead of making the hardware 10x faster. Otherwise, they wouldn't have made it the size of a HDD - because they would need much more ventilation in order to dissipate the heat. It's just a specialized hardware taking advantage of the latest algorithmic optimizations.
- PeterisP 10y agoSacrifice generality, accuracy and ability to randomly access a lot of memory so that you can implement fast and power-efficient matrix operations with a single, low accuracy datatype thus requiring less memory, bandwidth and transistors.
- cschmidt 10y agoThis seems very similar to the "Fathom Neural Compute Stick" from Movidius: http://www.movidius.com/solutions/machine-vision-algorithms/machine-learning http://www.movidius.com/solutions/machine-vision-algorithms/... TensorFlow on a chip....
- azinman2 10y agoAlthough that stuff seems to be more about on the fly reasoning at low wattage so you can embed a drone with neural nets... This is more for servers.
- asimuvPR 10y agoNow this is really interesting. I've been asking myself why this hadn't happened before. Its been all software, software, software for the last decade or so. But now I get it. We are at a point in time where it makes sense to adjust the hardware to the software. Funny how things work. It used to be the other way around.
- kens 10y agoThis is known as the Wheel of Reincarnation. Functionality moves to special-purpose hardware then back to software, and the cycle repeats. (The computer term is from 1968 so this has been happening for a long time.) http://www.catb.org/jargon/html/W/wheel-of-reincarnation.html http://www.catb.org/jargon/html/W/wheel-of-reincarnation.htm...
- asimuvPR 10y agoWonder what it will do the the industry term Full Stack Developer. Will people who call themselves that will now need to know about chip design? OT: Cool blog! :)
- dekhn 10y agoAt some point in the past, Rob Pike mentioned that when we was working on Voyager (that spaceship that almost 40 years after launch, has left the solar system and continues to send back valuable science data), he had a relatively good understanding of the system from the quantum level (transistors are based on quantum theory) to the solar system. He wasn't kidding, either.
- asimuvPR 10y agoVery interesting fact. But the average programmer is not Rob Pike. How do you see this panning out for the average programmer? Will people need to learn a bit about chips to build more efficient CRUD apps?
- 10y ago
- niels_olson 10y agoI like that the images are mislabeled :)
- fiatmoney 10y agoI'm guessing that the performance / watt claims are heavily predicated on relatively low throughput, kind of similar to ARM vs Intel CPUs - particularly because they're only powering it & supplying bandwidth via what looks like a 1X PCIE slot. IOW, taking their claims at face value, a Nvidia card or Xeon Phi would be expected to smoke one of these, although you might be able to run N of these in the same power envelope. But those bandwidth & throughput / card limitations would make certain classes of algorithms not really worthwhile to run on these.
- arcanus 10y ago> I'm guessing that the performance / watt claims are heavily predicated on relatively low throughput, kind of similar to ARM vs Intel CPUs - particularly because they're only powering it & supplying bandwidth via what looks like a 1X PCIE slot. Agreed. Also tells you that they don't need to communicate with the CPU much, given that it only has a PCIE. Reminds me of Knights Ferry, in this respect. > a Nvidia card or Xeon Phi would be expected to smoke one of these Will be very interesting to see some head-to-head benchmarks between these guys (on tensorflow and other libraries) in the next few months. Especially as Knights Landing starts to appear, and the new Nvidia card.
- visarga 10y agoI am so happy that Nvidia got a real competitor in the ML hardware market. Maybe now they will be two times more creative.
- mng2 10y agoThere are 8 high-speed serial links on the front side of that board and probably 8 on the back. That's quite a bit of bandwidth.
- wyldfire 10y ago> IOW, taking their claims at face value, a Nvidia card or Xeon Phi would be expected to smoke one of these, although you might be able to run N of these in the same power envelope. That seems unlikely, right? GPGPU software beating an ASIC? I guess it depends on just how abstract and adaptable the TPU IC is. Sounds like they use a custom number representation -- if they can squeeze it into less than half precision (IEEE FP16) then it would be super hard for a Phi or GPU to beat it. But ultimately it all comes down to specific applications' bottlenecks. If they don't have to go off-card ever and their workset fits into the on-die memory, then they'd have no real advantage by using a PCIe GPU with GDDRx on it.
- saganus 10y agoIs that a Go board stick to the side of the rack? Maybe they play one move every time someone gets to go there to fix something? or could it be just a way of numbering the racks or something eccentric like that?
- cven714 10y agoLikely it was one of the racks used by AlphaGo
- triangleman 10y agoIt appears to be a commemorative plaque from when they defeated world champion Lee Sodol.
- saganus 10y agoAh. That makes sense.
- Others 10y agoThe pictures have swapped captions.
- bd 10y agoSo now open sourcing of "crown jewels" AI software makes sense. Competitive advantage is protected by custom hardware (and huge proprietary datasets). Everything else can be shared. In fact it is now advantageous to share as much as you can, the bottleneck is a number of people who know how to use new tech.
- eva1984 10y agoAgree. To attract devs to using your spec, then eventually go to your platform. But since the spec are open, AMZN/MS can make their own too. So it is still a good thing.
- mtgx 10y agoIndeed. I kind of realized this more recently when I saw some new chips promote that Tensorflow works on them. This is really what it was all about - getting chips everywhere to support Google's version of AI, which means that in the long term they can get those chips themselves in volume - chips that are optimized for their AI out of the box.
- tptacek 10y agohttp://www.joelonsoftware.com/articles/StrategyLetterV.html http://www.joelonsoftware.com/articles/StrategyLetterV.html
- kristianp 10y agoCool how he foreshadows the end of Sun (takeover by Oracle in 2010) in that article from 2002: "Sun's two strategies are (a) make software a commodity by promoting and developing free software (Star Office, Linux, Apache, Gnome, etc), and (b) make hardware a commodity by promoting Java, with its bytecode architecture and WORA. OK, Sun, pop quiz: when the music stops, where are you going to sit down? Without proprietary advantages in hardware or software, you're going to have to take the commodity price, which barely covers the cost of cheap factories in Guadalajara, not your cushy offices in Silicon Valley."
- 10y ago
- dharma1 10y agohasn't made a dent on Nvidia's share price yet
- camkego 10y agoDoes anyone have links to the talk or the graphs?
- honkhonkpants 10y agoLooks to be approximately here https://youtu.be/862r3XS2YB0?t=7320 https://youtu.be/862r3XS2YB0?t=7320
- j1vms 10y agoI wouldn't be surprised if Google is looking to build (or done so already) a highly dense and parallel analog computer with limited precision ADC/DACs. I mean that's simplifying things quite a bit, but it would probably map pretty well to the Tensorflow application.
- skaevola 10y agoWhat are the advantages of analog computing for this application?
- read_only 10y agoIf you have an application that can tolerate error (like classification), then analog computing can give enormous gains in terms of speed _and_ power efficiency. Essentially, the savings come from using physics to perform the math (see Kirchhoff's current law) vs. using discrete time steps vs. fully-unrolling the logic. Google may not be using analog processing for this version, but I read an analog neural network researcher's page who said he moved to Google last year. (Sorry, I can't find the page again, but I think he was from the UK.)
- dharma1 10y agoWhat do you think about http://optalysys.com/ http://optalysys.com/ or http://lighton.io http://lighton.io?
- read_only 10y agoI think Optalysys looks interesting! For the curious, Optalysys has built a general purpose optics-based correlation/pattern matching machine. From some of their predecessor-company marketing material: The correlator performs pattern matching on large data sets such as high-resolution images, providing a measure of similarity and relative position between objects within the input scene. This allows large images [and general data converted to images] to be analysed far faster than electronic equivalents. Going back to the topic of NN-based computing, I found this talk to be intriguing: https://www.youtube.com/watch?v=dkIuIIp6bl0 https://www.youtube.com/watch?v=dkIuIIp6bl0. The main argument is that because Moore's law may no longer be in effect, it will become increasingly important to explore alternate computing solutions. (Google's TPU could be supporting evidence for this argument.) The speaker also co-authored a paper which I liked "General-Purpose Code Acceleration with Limited-Precision Analog Computation".
- swalsh 10y agoI wonder if opening this up as a cloud offering is a way to get a whole bunch of excess capacity (if it needs it for something big?) but have it paid for.
- luu 10y agoI'm happy to hear that this is finally public so I can actually talk about the work I did when I was at Google :-). I'm a bit surprised they announced this, though. When I was there, there was this pervasive attitude that if "we" had some kind of advantage over the outside world, we shouldn't talk about it lest other people get the same idea. To be clear, I think that's pretty bad for the world and I really wished that they'd change, but it was the prevailing attitude. Currently, if you look at what's being hyped up at a couple of large companies that could conceivably build a competing chip, it's all FPGAs all the time, so announcing that we built an ASIC could change what other companies do, which is exactly what Google was trying to avoid back when I was there. If this signals that Google is going to be less secretive about infrastructure, that's great news. When I joined Microsoft, I tried to gently bring up the possibility of doing either GPUs or ASICs and was told, very confidentially by multiple people, that it's impossible to deploy GPUs at scale, let alone ASICs. Since I couldn't point to actual work I'd done elsewhere, it seemed impossible to convince folks, and my job was in another area, I gave up on it, but I imagine someone is having that discussion again right now. Just as an aside, I'm being fast and loose with language when I use the word impossible. It's more than my feeling is that you have a limited number of influence points and I was spending mine on things like convincing my team to use version control instead of mailing zip files around.
- mike_hearn 10y agoGoogle has always been strategic about announcing what it was doing, even since the early days (I used to work there too). Think about the impact the first MapReduce and GFS/BigTable papers had. My guess as to why they're announcing the TPU is that they are feeling the pressure from Facebook and other AI labs, and want to reinforce their reputation as being the best place to do AI research. By revealing that AlphaGo was based on this hardware, they indicate to researchers around the world that if you want to build the most advanced ML models you need to be at Google. Same reason they talked about MapReduce/GFS back in the day.
- arcanus 10y ago> By revealing that AlphaGo was based on this hardware Interesting, as the nature/science paper made no mention of this, it was exclusively trained on GPUs.
- danielvf 10y agoFor the curious, that's a plaque on the side if the rack showing the Go board at the end of AlphaGo vs Lee Sedol Game 3, at the moment Lee Sedol resigned and AlphaGo won the tournament (of five games).
- visarga 10y agoBy the way, more merit for Lee Sedol now that we know what he played against.
- LogicFailsMe 10y agoPerf/W, the official metric of slow but efficient processors. How many times must we go down this road? Let's see this sucker train AlexNet...
- dgacmu 10y agoWearing my CMU hat for a moment (but keeping in mind Google's paying me this year): Google's always been cautious about the balance of speed and efficiency, out of concerns about programmer productivity, parallelization, and generality. See, for example, Urs's article in response to my and a few other people's crazy-academic research on using "Wimpy" nodes: http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36448.pdf http://static.googleusercontent.com/media/research.google.co... - vs http://www.cs.cmu.edu/~fawnproj/ http://www.cs.cmu.edu/~fawnproj/ There's a big difference between just cranking down the GHz and going for ASIC specialization. GPUs, for example, already represent a point on this spectrum -- it's true that they run at reduced GHz compared to high-end CPUs, but they're arithmetic monsters. The blog post notes, in fact, that the use of TPUs in AlphaGo let them do more searching. So why would you assume automatically that they're slow?
- LogicFailsMe 10y agoBecause there's an absolute sippy straw of bandwidth to the thing if that's a 1x pci-e connection. For if it were delivering performance on par with a $1000 Maxwell class GPU, why wouldn't you guys crow about it? That would be a really big deal wouldn't it? TitanX for 20W? That'd be awesome. And having suffered through multiple pitches for us to buy various FPGA and boutique processors, I have yet to see someone who produced perf per watt numbers first, subsequently produce an impressive performance number. In fact, it took nearly yelling at one vendor for them to finally admit perf was abysmal. Finally, training does not equal inference. Training requires strong scaling, but inference need only weak scale. So I suspect that Urs had to bite his tongue and buy a bunch of gpus for training networks. Am I missing something?
- dgacmu 10y ago
- protomok 10y agoI'd be interested to know more technical details. I wonder if they're using 8-bit multipliers, how many MACs running in parallel, power consumption, etc.
- mistobaan 10y agoAnother point is that they will be able to provide much higher computing capabilities at a much lower price point that any competitors. I really like the direction that the company is taking.
- deleted 10y ago[deleted]
- revelation 10y agoThere is not a single number in this article. Now these heatsinks can be deceiving for boards that are meant to be in a server rack unit with massive fans throwing a hurricane over them, but even then that is not very much power we're looking at there.
- __jal 10y agoMy favorite part is what looks like flush-head sheet metal screws holding the heat sink on. No wondering where you left the Torx drivers with this one.
- amelius 10y agoOne question: what has this got to do with tensors?
- vintermann 10y agoThat they're used a lot in machine learning? If you're processing video for instance, you might have a 5-dimensional tensor: x, y, color channel, time index and batch index.
- PeterisP 10y agoThe (only?) calculations that this specialized device can do well are tensor calculations.
- ungzd 10y agoDoes it use approximate computing technology?
- paulsutter 10y ago> Our goal is to lead the industry on machine learning and make that innovation available to our customers. Are they saying Google Cloud customers will get access to TPUs eventually? Or that general users will see service improvements?
- taliesinb 10y agoI don't know much about this sort of thing but I wonder if the ultimate performance would come with co-locating specialized compute with memory, so that the spatial layout of the computation on silicon ends up mirroring the abstract dataflow dag, with fairly low-bandwidth and energy efficient links between static register arrays that represent individual weight and grad tensors. Minimize the need for caches and power hungry high bandwidth lanes, ideally the only data moving around is your minibatch data going one way and your grads going the other way. I wonder if they're doing that, and to what degree.
- mrpippy 10y agoBah, SGI made a Tensor Processing Unit XIO card 15 years ago. evidence suggests they were mostly for defense customers: http://forums.nekochan.net/viewtopic.php?t=16728751 http://forums.nekochan.net/viewtopic.php?t=16728751 http://manx.classiccmp.org/mirror/techpubs.sgi.com/library/manuals/4000/007-4222-002/pdf/007-4222-002.pdf http://manx.classiccmp.org/mirror/techpubs.sgi.com/library/m...
- hedgehog 10y agoThe term is pretty generic, Nervana also calls their chip a "tensor processing unit".
- Keyframe 10y agoDamn, what a riveting read would be to find out what IT toys defense has now! Even an informed speculation would be nice to read.
- vt240 10y agoI remember going to see the SGI Origin 2000 installation at University of Alaska, Fairbanks. Pretty neat, I seem to remember they had a Cray T3E 900 and some J vector systems there too.
- rando3826 10y agoWhy use an ANKY in the title? Using an ANKY(Acronym no one knows yet) is bad writing, makes readers feel dumb, etc. Google JUST NOW invented that acronym, sticking it in the title like just another word we should understand is absolutely ridiculous.
- placeybordeaux 10y agoI honestly didn't find it that hard to understand as they recently released a machine learning library that started with a T.
- hristov 10y agoIt is interesting that they would make this into an ASIC, provided how notoriously high the development costs for ASICs are. Are those costs coming down? If so life will get very hard for the FPGA makers of the world soon. It would be interesting to see what the economics of this project are. I.e., what are the development costs and costs per chip. Of course it is very doubtful I will ever get to see the economics of this project, it would be interesting.
- zhemao 10y agoIt's mainly a question of volume. ASICs have a big economy of scale, so the cost-per-chip goes down considerably once you go over a certain number of chips. Plus, there's all the NRE costs of an ASIC design over an FPGA design. Google probably figured they could use enough chips to make the cost of manufacturing ASICs worthwhile. I don't think FPGAs are going to be beat out by ASICs for low volume applications anytime soon.
- indolering 10y agoBut even a custom run of silicon (on yesterday's technology) will only set you back $30K. That's one of the reasons there is so much interest in RISC-V.
- visarga 10y agoI think the initial investment was cheaper than the projected GPU energy cost and if it is actually faster, then there are advantages for faster iteration.
- TorKlingberg 10y agoMaking a custom ASIC is expensive for a startup, but not that expensive. Many companies are making their own high-end chips for things like signal processing.
- breatheoften 10y agoA podcast I listen to posted an interview with an expert last week saying that he perceived that much of the interest in custom hardware for machine learning tasks died when people realized how effective GPUs were at the (still-evolving-set-of) tasks. http://www.thetalkingmachines.com/blog/2016/5/5/sparse-coding-and-madbits http://www.thetalkingmachines.com/blog/2016/5/5/sparse-codin... I wonder how general the gains from these ASIC's are and whether the performance/power efficiency wins will keep up with the pace of software/algorithm-du-jour advancements.
- hyperopt 10y agoI listen to the Talking Machines as well. Great podcast. Another question would be are the gains worth the cost of an ML-specific ASIC. GPUs have the entire, massive gaming industry driving the cost down. I suppose that as adoption of gradient-descent-based neural networks increases, it may be worth the cost in a similar way that GPUs are worth the cost. Then again, I have never implemented SGD on a GPU so I'm not aware if there are any bottlenecks that could be solved with an ML-specific ASIC. Can anyone else shed some light on this?
- indolering 10y ago> massive gaming industry driving the cost down. Per-unit manufacturing cost scales logarithmically. Even a single batch of custom silicon on yesterday's technology is only $30K. This is one of the reasons there is so much interest in RISC-V; hardware costs are not the barrier-to-entry that they used to be. So yeah, the gaming market pushes the per-unit price of GPUs down, but even an additional 2x reduction in rackspace and power will pay for itself at the right scale.
- RIMR 10y agoSomewhat off topic, but if you look at the lower-left hand corner of the heatsink in the first image, there's two red lines and some sort of image artifact. https://2.bp.blogspot.com/-z1ynWkQlBc8/VzzPToH362I/AAAAAAAACp0/2QBREGUEikoHrML1nh9h3SEKQVzm8NV7QCLcB/s1600/tpu-2.png https://2.bp.blogspot.com/-z1ynWkQlBc8/VzzPToH362I/AAAAAAAAC... They probably didn't mean to use this version of the image for their blog - but I wonder what they were trying to indicate/measure there.
- cm3 10y agoDid the image change? I can't seem to find what you're describing.
- davidlakata 10y agoIt's still there: the red lines outline the lower left hand corner of the heatsink (the big metallic structure).
- cm3 10y agoGot it. That looks like a bad digital image stitching job. Especially the misaligned fins. But the red lines are odd indeed.
- jhayward 10y agoIt kind of looks like the original image had a cutaway section there and they didn't want that to be seen so pasted a top surface back on.
- manav 10y agoInteresting. Plenty of work has been done with FPGAs, and a few have developed ASICs like DaDianNao in China [1]. Google though actually has the resources to deploy them in their datacenters. Microsoft explored something similar to accelerate search with FPGAs [2]. The results show that the Arria 10 (20nm latest from Altera) had about 1/4th the processing ability at 10% of the power usage of the Nvidia Tesla K40 (25w vs 235w). Nvidia Pascal has something like 2/3x the performance with a similar power profile. That really bridges the gap for performance/watt. All of that also doesn't take into account the ease of working with CUDA versus the complicated development, toolchains, and cost of FPGAs. However, the ~50x+ efficiency increase of an ASIC though could be worthwhile in the long run. The only problem I see is that there might be limitations on model size because of the limited embedded memory of the ASIC. Does anyone have more information or a whitepaper? I wonder if they are using eAsic. [1]: http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=7011421&url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D7011421 http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=701142... [2]: http://research.microsoft.com/pubs/240715/CNN%20Whitepaper.pdf http://research.microsoft.com/pubs/240715/CNN%20Whitepaper.p...
- hedgehog 10y agoYou can build an ASIC with fast external memory, it adds to the cost but then you can handle larger models similar to a GPU. Software support is an issue but for deep learning applications there's no reason in principle you couldn't add support to TensorFlow etc for new hardware to make it simple for application developers to adopt. Movidius has announced that they're doing this and it's likely that other ML chip vendors will do the same.
- emcq 10y agoExternal memory kills your power budget. Efficient processors, whether they be GPUs, cpus, or asics all gain efficiencies by keeping data local to computation. Having your chip focused on off chip data is difficult to optimize power for if you don't have compute bound problems.
- petra 10y ago>> I wonder if they are using eAsic. If you read the academic literature, they talk about 100x-3000x energy savings with asic vs GPU. So in that light Google's 10x improvement sounds low, and could certainly fit an eAsic story. Furthermore, eAsic has fixed wires, but logic is defined via sram configuration(AFAIK) and not fixed, so it could offer a level of programmability .
- aaronsnoswell 10y agoI'm curious to know; is this announcement something that an expert in these sorts of areas could have (or did?) predict months or years ago, given Google's recent jumps forwards in Machine learning products? Can someone with more knowledge about this comment?
- p1esk 10y agoYes. Back in 1989, Intel built one of the first custom chips for ML (ETANN). Since then, there have been hundreds of such designs, and dozens of working chips. So, people have been building them long before "deep learning" arrived in all its glory. Now that it did, it was expected that efforts in custom hardware should intensify. However, such initiatives still face the same problems as 30 years ago: custom hardware is expensive, inflexible, hard to program, and quickly becomes obsolete. It's still worth it if there's no other way to speed things up, but Moore's law is still alive and kicking, as evidenced by 15B transistor GP100 chip form Nvidia, so we can still just wait a little bit for the next gen GPUs. Google is certainly in a good position, having developed a very popular ML framework, and having enough resources to develop good hardware (the blog post was written by Norm Jouppi - one of the best computer architects in history). It remains to be seen, however, how well these TPUs are supported in TensorFlow. What kind of models will get the advertised speed up?
- hellameta 10y agoSure. I don't have anything to link on the spot but this was/is/has been foreseeable for some time. Although it's all very cool and shiny - most business applications of machine learning remain squarely in the territory of classic algos like GLM & forests (random, boosted trees etc. etc.). As a fun note, advances like these highlight that data scientists etc. will not be beaten by more complex automated methods, but simply by speed. Much like the filing system that 'runs' whatever you're using to see these words (https://www.youtube.com/watch?v=EKWGGDXe5MA https://www.youtube.com/watch?v=EKWGGDXe5MA). Edit: to elaborate... single model training runs are possible to do quite fast now, but knowing how to tune hyper parameters remains the 'voodoo' of the field. But the best hyper params are also possible to discover through brute force: try every combination you can! Today, you can use various heuristics to improve this process, but either way, being able to train whatever X times faster just means we can search hyper parameter space that much faster. The robots are coming :)
- nxzero 10y agoIs there anyway to detect what hardware to being used by the cloud service if you're using the cloud service? (yes, realize this question is a bit of a paradox, but figured I'd ask.)
- nxzero 10y agoIs there anyway to detect what hardware to being used by the cloud service if you're using the cloud service? (yes, realize this question is a bit of a paradox, but figured I'd ask.)
- bravo22 10y agoGiven the insane mask costs for lower geometries, the ASIC is most likely an Xilinx EasyPath or Altera Hardcopy. Otherwise the amortization of the mask and dev costs -- even for a structured cell ASIC -- over 1K unit wouldn't make much sense versus the extra cooling/power costs for a GPU.
- nickpsecurity 10y agoDon't forget shuttle runs. Adepteva used those and otherwise good engineering practice to develop two products, latest in 65nm, with no more than $2mil. This one might be simpler and cheaper given its requirements. http://www.adapteva.com/andreas-blog/a-lean-fabless-semiconductor-startup-model/ http://www.adapteva.com/andreas-blog/a-lean-fabless-semicond...
- bravo22 10y agoTrue. That's another possibility. I'd imagine since they'd want to squeeze every performance per want out of the chip they'd want to go for smallest node possible. Virtex7 EasyPath is 16nm! It is pin to pin compatible with the FPGA version -- because they just change the mask layer and you get it in about 6 weeks. Hard to beat that.
- nickpsecurity 10y agoI didn't know they were still doing EasyPath. And I surely didn't know they did pin-for-pin on 16nm. Holy crap! :)
- bravo22 10y agoYes. It is very popular for high-end FPGA use cases. Essentially instead of relying on the built-in routing matrix which makes FPGAs what they are, they modify the metal layer and connect the chip per your design as an ASIC. You get much lower power consumption and much faster clock rates. You also get it in about 6-8 weeks and is guaranteed to match your original design in functionality. It is 100% pin compatible because it is the same base silicon and packaging.
- phsilva 10y agoI wonder if this architecture is the same Lanai architecture that was recently introduced by Google on LLVM. http://lists.llvm.org/pipermail/llvm-dev/2016-February/095118.html http://lists.llvm.org/pipermail/llvm-dev/2016-February/09511...
- startling 10y agoNo, this is an ASIC. It's not general-purpose.
- hyperopt 10y agoThe Cloud Machine Learning service is one that I'm highly anticipating. Setting up arbitrary cloud machines for training models is a mess right now. I think if Google sets it up correctly, it could be a game changer for ML research for the rest of us. Especially if they can undercut AWS's GPU instances on cost per unit of performance through specialized hardware. I don't think the coinciding releases/announcements of TensorFlow, Cloud ML, and now this are an accident. There is something brewing and I think it's going to be big.
- visarga 10y agoI'd love to have a place to experiment cheaply but still not be required to invest 2-3K$ in it.
- hyperopt 10y agoIt doesn't just need to be a place to experiment cheaply. Many companies building software around ML techniques still rent time on EC2. Unless you are training models 24/7 and have your machines located in a very cost efficient location in terms of power/cooling, It's probably better for your training to be done in the cloud. It think very few use cases fall into the latter category.
- eggy 10y agoPretty quick implementation. On the energy savings and space savings front, this type of implementation coupled with the space-saving, energy-saving claims of going to unums vs. float should get it to the next order of magnitude. Come on, Google, make unums happen!
- manyama66 10y agohttp://halonewstza.blogspot.com/2016/05/nato-risks-nuclear-war-with-russia.html?m=1 http://halonewstza.blogspot.com/2016/05/nato-risks-nuclear-w...
- j-dr 10y agoThis is great, but can google stop putting tensor in the name of everything when nothing they do really has anything to do with tensors?
- Difwif 10y agoI'm unsure if Tensorflow actually uses this but I know in literature some models use tensor decompositions for learning.
- simunaga 10y agoIn what sense in this a great news? Yes, it's a progress, so what? After all, you - programmers - earn money for your jobs and pretty soon you might not have one. Because of these kinds of great news -- "Whayyy, this is really interesting, AI, maching learning. Aaaaa!". "I'll get fired, won't have money for living and AI will take my place, but the world will be better! Yes! Progress!" Who will benefit from this? Surely not you. Why are you so ecstatic then?
- visarga 10y agoI thought long and hard about the future when 95% of the jobs as they exist today will be taken by robots. Initially, it would seem that people would be reduced to beggars, as they depend on BHI or other forms of welfare to subsist. But then it struck me: You don't need jobs as long as you have land, renewable energy sources and robots (and 3d printers). You can live in a community that is self sufficient. You will be employed by your land, as it always was up until 100 years ago. We will also have robots, maybe not the latest generation, but we don't need to go back to the 19th century agriculture. It is you who will benefit in the end, if you can use AI to improve your life. As long as AI doesn't remain locked in the hands of one entity and we all share into the benefits, it will work out ok. In the short run we need some sort of social welfare though, and to invest in renewables and self-sufficiency technologies. How much self-sufficient a country, city, village or small farm could be? There is a lot of potential to migrate back to small community agrarian economy with robotics and 3d printing and solar panels. <speculation>People could trade using a different currency than that used for robotic produced goods. This currency will have to enforce differentiation of economic agents (diversity) and integration (low barriers of entry). A currency that will automatically disable the accumulation of power in a few hands and work for humans. We have to build an economy that functions more like the brain. In the brain there is no master neuron. They all share in the activity. So should be an enlightened human society.</speculation>
- simunaga 10y agoyes, could be. but not necessarily will be and not necessarily that you and other people of the middle or low class will be given a luxury to have a robot and not to work. maybe low and middle class will have to serve people from high class who will have robots.
- nathan_f77 10y agoI'm thinking that this has the potential to change the context of many debates about the "technological singularity", or AI taking over the world. Because it all seems to be based on FUD. While reading this article, one of my first reactions was "holy shit, Google might actually build a general AI with these, and they've probably already been working on it for years". But really, nothing about these chips is unknown or scary. They use algorithms that are carefully engineered and understood. They can be scaled up horizontally to crunch numbers, and they have a very specific purpose. They improve search results and maps. What I'm trying to say is that general artificial intelligence is such a lofty goal, that we're going to have to understand every single piece of the puzzle before we get anywhere close. Including building custom ASICs, and writing all of the software by hand. We're not going to accidentally leave any loopholes open where AI secretly becomes conscious and decided to take over the world.
- Houshalter 10y agoJust because you understand a machine doesn't mean it can't be dangerous. I could completely understand every aspect of a nuclear bomb, and I could still make a mistake and cause quite a bit of damage with a real one. Complex systems are notorious for having all sorts of unexpected problems, and mistakes happen all the time. How much complex software is entirely bug free? The danger of AI is more than just a random bug though. It's that an intelligent AI is inherently not good. If you give it a goal, like to make as many paperclips as possible, it will do everything in it's power to convert the world to paperclips. If you give it the goal of self preservation, it try to destroy anything that has a 0.0001% chance of hurting it, and make as many redundant copies as possible. Etc. Very, very few goals actually result in an AI that wants to do exactly what you want it to do. And if the AI is incredibly powerful, that will be a very bad outcome for humanity.
- visarga 10y agoYou can condition the AI on the well being and freedom of human population. Hard to define precisely what that means, but it can be approximated with indirect measures. This is just what Asimov thought of in his novels. Another way to protect against catastrophe would be to launch multiple AI agents that optimize for the goal of nurturing humanity. They can keep each other in check. Also, humans will evolve as well. Genetics is advancing very fast. We will be able to design bigger/better brains for ourselves, perhaps also with the help of AI. Human learning could be assisted by AIs to achieve much higher levels than today. We will also be able to link directly to computers and become part of their ecosystem, thus, creating an incentive for it to keep us around. Taking this path would enable uploading and immortality for humans as well. My guess is that we will all become united with the AI. We already are united by the internet and we spend a lot of time querying the search engine (AI), learning its quirks and, by feedback, helping improving it. This trend will continue up to the point where humans and AI become one thing. Killing humans would be for the AI like cutting out a part of your brain. Maybe it will want a biological brain of its own and come over to the other side, of biological intelligence.
- eggy 10y agoI think the confluence of new technologies, and the re-emergence / rediscovery of older technologies is going to be the best combination. Whether it goes that way is not certain, since the best technology doesn't always win out. Here, though, the money should, since all would greatly reduce time and energy in mining and validating: * Vector processing computers - not von Neumann machines [1]. * Array languages new, or like J, K, or Q in the APL family [2,3] * The replacement of floating point units with unum processors [4] Neural networks are inherently arrays or matrices, and would do better on a designed vector array machine, not a re-purposed GPU, or even a TPU in the article in a standard von Neumann machine. Maybe non-von Neumann architectire like the old Lisp Machines, but for arrays, not lists (and no, this is not a modern GPU. The data has to stay on the processor, not offloaded to external memory). I started with neural networks in late 80s early 1990s, and I was mainly programming in C. matrices and FOR loops. I found J, the array language many years later, unfortunately. Businesses have been making enough money off of the advantage of the array processing language A+, then K, that the per-seat cost of KDB+/Q (database/language) is easily justifiable. Other software like RiakTS are looking to get in the game using Spark/shark and other pieces of kit, but a K4 query is 230 times faster than Spark/shark, and uses 0.2GB of memory vs. 50GB. The similar technologies just don't fit the problem space as good as a vector language. I am partial to J being a more mathematically pure array language in that it is based on arrays. K4 (soon to be K5/K6) is list-based at the lower level, and is honed for tick-data or time series data. J is a bit more general purpose or academic in my opinion. Unums are theoretically more energy efficient and compact than floating point, and take away the error-guessing game. They are being tested with several different language implementations to validate their creator's claims, and practicality. The Mathematica notebook that John Gustafson modeled his work on is available free to download from the book publisher's site. People have already done some type of explorator investigations in Python, Julia and even J already. I believe the J one is a 4-bit implementation of enums based on unums 1.0. John Gustafson just presented unums 2.0 in February 2016. [1] http://conceptualorigami.blogspot.co.id/2010/12/vector-processing-languages-future-of.html http://conceptualorigami.blogspot.co.id/2010/12/vector-proce... [2] jsoftware.com [3] http://kxcommunity.com/an-introduction-to-neural-networks-with-kdb.php http://kxcommunity.com/an-introduction-to-neural-networks-wi... [4] https://www.crcpress.com/The-End-of-Error-Unum-Computing/Gustafson/p/book/9781482239867 https://www.crcpress.com/The-End-of-Error-Unum-Computing/Gus...
- Coding_Cat 10y agoI wonder if we will be seeing more of this in the (near) future. I expect so, and from more people then just Google. Why? Look at the problems the fab labs have had with the latest generation of chips and as they grow smaller the problems will probably rise. We are already close to the physical limit of transistor size. So, it is fair to assume that Moore's law will (hopefully) not outlive me. So what then? I certainly hope the tech sector will not just leave it at that. If you want to continue to improve performance (per-watt) there is only one way you can go then: improve the design at an ASIC level. ASIC design will probably stay relatively hard, although there will probably be some technological solutions to make it easier with time, but if fabrication stalls at a certain nm level, production costs will probably start to drop with time as well. I've been thinking about this quite a bit recently because I hope to start my PhD in ~1 year, and I'm torn between HPC or Computer Architecture. This seems to be quite a pro for Comp. Arch ;).