7 ms·
MTIA v1: Meta’s first-generation AI inference accelerator
- tartavull 3y agoHow do they compare to TPUs?
- sebzim4500 3y agoWhy does the headline just mention inference when the acronym also mentions training? Is it primarily for inference and the training is just an after thought?
- ZiiS 3y agoThese seem power and density optimized. This sort of custom hardware is all about supply chains and getting a lot of them everywhere. This flavors the inference use-case. For large training jobs it is more about turn around time; running hideously expensive GPUs sucking down huge amounts of power is fine.
- layer8 3y agoIt looks rather general-purpose (for ML tasks) to me: Each PE is equipped with two processor cores (one of them equipped with the vector extension) and a number of fixed-function units that are optimized for performing critical operations, such as matrix multiplication, accumulation, data movement, and nonlinear function calculation. The processor cores are based on the RISC-V open instruction set architecture (ISA) and are heavily customized to perform necessary compute and control tasks.
- ramshanker 3y ago>>>> fabricated in TSMC 7nm process and runs at 800 MHz, providing 102.4 TOPS at INT8 precision and 51.2 TFLOPS at FP16 precision. It has a thermal design power (TDP) of 25 W. So 2 generation of immediate improvement available.
- bhouston 3y agoComparing MTIA v1 vs Google Cloud TPU v4: MTIA v1's specs: The accelerator is fabricated in TSMC 7nm process and runs at 800 MHz, providing 102.4 TOPS at INT8 precision and 51.2 TFLOPS at FP16 precision. It has a thermal design power (TDP) of 25 W. Up to 128 GB of ram LPDDR5. Googles Cloud TPU v4: 275 teraflops (bf16 or int8), 90/170/192 W. 32 GiB of HBM2 RAM, 1200 GBps. From here: https://cloud.google.com/tpu/docs/system-architecture-tpu-vm#tpu_v4 https://cloud.google.com/tpu/docs/system-architecture-tpu-vm... So it seems that the Google Cloud TPU v4 has an advantage in terms of compute per chip and ram speed, but the Meta one is much more efficient (2x to 4x, it is hard to tell) and has more ram but it is slower ram?
- innagadadavida 3y agoIs there something that compares these to more consumer offering like Apple’s ANE?
- u017937387 3y ago[flagged]
- benstrumental 3y agoFWIW, you're comparing a training-specialized chip to an inference-specialized chip. It'd be more apples to apples to compare to TPU v4 lite, but I can't find that chip's details anywhere beyond some mentions in the TPU v4 paper: https://arxiv.org/abs/2304.01433 https://arxiv.org/abs/2304.01433
- KRAKRISMOTT 3y agoHow does a training specialized chip function? Forward mode is simple, just a dot product machine. But how do you accelerate backprop on hardware? Does it have the vector Jacobian transformation lookup logic and table baked into hardware?
- m00x 3y ago
- notfried 3y agoHas there been any rumors or statements from Facebook on them eventually stepping into selling cloud compute? I'd be surprised if they are investing in building hardware accelerators just for their own services.
- mgdev 3y agoTheir footprint for just their own services rivals some other public clouds.
- bradleyjg 3y agoI think they’d be bad at it for the same reason google is bad it. Enterprise sales is not in their dna.
- sebzim4500 3y agoThe AI inference/training market is so competitive that I doubt enterprise sales is going to be the problem. A company planning on spending $50M training a model is not going to be convinced by some smooth talking sales guy over a golf game. They will look at the actual price/performance.
- bradleyjg 3y agoYou’d be surprised. Azure is growing like gangbusters on the backs of smooth talking salesmen taking CTOs golfing.
- foverzar 3y agoGiven that these chips seem to be power optimised and Facebook's recently released sensory model, I wouldn't be surprised to see them in their next iteration of VR devices.
- villgax 3y agoJust missed FP8 implementation on hardware
- seydor 3y agoIt's curious why nobody is selling these systems yet
- m3kw9 3y agoProbably the software needs to be optimized for the hw and also the hw may not be general purpose enough even if offered. People demand nvidia because cuda is very optimized for their gpus and many AI software use cuda
- bee_rider 3y agoCompeting against NVIDIA must be exhausting. You come up with a clever ASIC that is better than their current GPU for your workload… and by the time it comes out they’ve released the next year’s chip that just has like 50% more memory bandwidth or something ridiculous like that, and beats you by pure grunt. “No replacement for displacement” actually seems to be true in compute.
- jnwatson 3y agoFor the same reason why it took a long time for crypto mining accelerators to actually ship. It is more profitable to keep them for yourself.
- latchkey 3y agoThis is a popular myth. Bitcoin asic's were 'shipping' in 2012/2013. Some companies definitely played games and mined with the asic's themselves (and then shipped those used asic's)... but in general, it was always a lot more profitable to sell the shovels than it was to mine the gold.
- kccqzy 3y agoCheck out https://coral.ai/products/ https://coral.ai/products/ accelerators you can actually buy.
- deleted 3y ago[deleted]
- htrp 3y agoThis looks like a customized ASIC specializing solely in recommendation systems possibly focused on ads ranking >We found that GPUs were not always optimal for running Meta’s specific recommendation workloads at the levels of efficiency required at our scale. Our solution to this challenge was to design a family of recommendation-specific Meta Training and Inference Accelerator (MTIA) ASICs.
- flangola7 3y agoWhat a tragic waste of human effort and potential.
- nikhilsimha 3y agoI hope they take comfort in the fact that this is open-source.
- throwuwu 3y agoSame thing was said about GPUs when they were just for games
- flangola7 3y agoI don't remember that. Games are art and offer at least some benefit to the world. Optimizing Facebook attention algorithms harms society.
- gmm1990 3y agoThey designed it in 2020 does that mean it is likely to have been in use for a while or is the design lag a few years?
- bhouston 3y agoIt is ambiguous on that front. If you designed it in 2020, getting through test runs at TSMC and then to a final production run would take a while. So when they had it deployed at scale at FB is unclear.
- 0zemp2c 3y agoJust as incredible is the corresponding announcement of their RSC which is purportedly one of the world's most powerful clusters Amazing times! Private companies now have compute resources previously only showing up in government labs, and in many cases using novel components like MTIA This feels like the start of a golden age and in a few years we will have incredible results and breakthroughs
- brooksbp 3y agoWhy are there so many Mini SMP (?) connectors on the board? (video time 1:21)
- rektide 3y agoCan OpenXLA/IREE target it? Supposedly PyTorch 2.0's big shift was a switch to these new systems. Curiosity to know if that's actually happened here. Side note, the chip says Korea on it & I this expected it was Samsung... But it's TSMC made chips? What's up with that?
- deepnotderp 3y agoProbably a Korean packaging company
- two_in_one 3y agoI want one. This thing can run LLaMA 64b int8 easily. Meta is going to use it in datacenters, Much more efficient than NVidia generic GPUs. They are serious about putting AI everywhere.
- u017937387 3y ago[flagged]
- u017937387 3y ago[flagged]