11 ms·
Nvidia will build 700-petaflop supercomputer for University of Florida
- deleted 6y ago[deleted]
- fluffything 6y agoI'm a bit baffled. How high are the tuitions at the university of florida? This supercomputer is more powerful than many at the national labs and must cost a fortune (multiple 100 millions of dollars).
- adventured 6y ago> How high are the tuitions at the university of florida? In-state tuition at the University of Florida is inexpensive compared to most major US universities and is generally considered one of the better values among universities. ~$6,400 per year in-state. Florida State is also comparable in cost. https://www.sfa.ufl.edu/cost/ https://www.sfa.ufl.edu/cost/
- deleted 6y ago[deleted]
- ipsum2 6y agoThe article mentions that the money/hardware was donated by Nvidia co-founders.
- Bootvis 6y agoOne of the founders, Chris Malachowsky, is an alum: https://en.wikipedia.org/wiki/Chris_Malachowsky https://en.wikipedia.org/wiki/Chris_Malachowsky
- 01100011 6y ago> The initiatives and HiPerGator enhancements are anchored by a $50 million gift — $25 million from UF alumnus and Nvidia fellow Chris Malachowsky and $25 million in hardware, software, training, and services from Nvidia. UF will also make investments around its new machine “well beyond” $20 million, targeted at upgrading its on-campus datacenter.
- Blammar 6y agoIt's $50 million. That's all. See https://blogs.nvidia.com/blog/2020/07/21/university-of-florida-nvidia-ai-supercomputer/ https://blogs.nvidia.com/blog/2020/07/21/university-of-flori... for the numbers. It's probable the hardware is being offered at a deep discount. The building and infrastructure will cost an additional $20m, covered by the University.
- mhh__ 6y agohttps://lambdalabs.com/blog/demystifying-gpt-3/ https://lambdalabs.com/blog/demystifying-gpt-3/ According to ^, that's about 11 GPT-3s trainings worth in the cloud.
- nl 6y agoNo one who's training GPT-3 pays those prices.
- autokad 6y agoI'm in favor of getting high powered compute in everyone's hands, so this is great to me.
- throwaway_pdp09 6y agoWhy? I mean, we already have (my old 2013 desktop would - I guess - compete with a cray 1 from the 1980s). What do you expect will happen if people get more CPU?
- eecc 6y agoBetter simulations? Successfully working on previously intractable problems?
- throwaway_pdp09 6y agoBut most people ie. not 'everyone' as the post talked about, is doing that. Most just want web browsing + LibreOffice stuff. Oh, and games I suppose.
- deleted 6y ago[deleted]
- jhj 6y agoNo, those national lab supercomputers typically quote double-precision flops, whereas this appears to be about float16 tensor ops for inner product (they call it "AI flops"). The A100 is 9.7 general purpose Tflops of float64, versus 624 Tflops of float16 / bfloat16 (perhaps also just tensor cores not general purpose)? So quoting it the same way, this Florida machine would only be a (700 * (9.7 / 624)) = 10.9 Pflop supercomputer, not counting host CPUs. Granted though, the hot new HPC approach is to approximate certain expensive high-precision calculations with ML inference instead, but that only works for certain algorithms.
- boulos 6y agoJinx :). I really am surprised the marketing folks didn’t claim this was an ~3 exa-op supercomputer based on int4 w/ sparsity.
- fluffything 6y agoThe FLOPs quoted by national lab supercomputers are usually measured double-precision FLOPs from HPL (High-performance Linpack - at least on the Top500 reports). IIUC, HPL can use double-precision tensor operations. It might also be able to use mixed-precision for some computations. Are the performance numbers for this HPC system at University of Florida also from HPL ?
- jabl 6y ago> IIUC, HPL can use double-precision tensor operations. Lets put it this way: As of the A100 GPU generation, the tensor cores are able to do double precision calculations too. By using the tensor core the peak FP64 flops double to 19.5 Tflops. And yes, the CUDA BLAS library (that the HPL benchmark uses) uses the tensor cores on the A100. > It might also be able to use mixed-precision for some computations. There is currently a lot of interest in mixed precision algorithms, but I'm not sure that the algorithm used for HPL is amenable to that approach, nor whether it's allowed per the benchmark rules.
- jabl 6y agoThe machine will be 140 nodes, which is half the size of the current #7 on the top500 list which is otherwise identical: https://top500.org/system/179842/ https://top500.org/system/179842/ That #7 machine got 27.6 Pflops on HPL, so I would guess the Florida machine will get roughly half of that. (Please keep in mind the tensor cores on the A100 can do FP64, so the theoretical peak flops per GPU is 19.5)
- deleted 6y ago[deleted]
- ponker 6y agoResearch at most US major universities is funded from federal grants, not from tuition. A place like the University of Florida, MIT, Michigan State, whatever, is conducting research with government money in a manner very similar to a national lab.
- jkhdigital 6y agoIndeed, and to paraphrase my PhD adviser, we currently have more research dollars than good students to spend them on.
- auslegung 6y agoIn addition to what everyone else has said, UF has over 50,000 students, making it one of the five largest universities in the USA, iirc.
- wil421 6y agoCorrect. Florida has 3 of the top 10 by enrollment. UCF, UF, FIU, and USF. Georgia made the list, go dawgs.
- gavinray 6y agoI moved to Miami from Boulder, and was working out of an office that was on FIU campus for a good while. The first time I walked on the campus, it blew my mind. I'd never heard about Florida International University as an American, but it's easily the biggest and most beautiful I've seen.
- xvilka 6y agoPrecisely explains why NVIDIA did that - to hook up all these 50k students to CUDA. Much more profitable in the long run.
- fma 6y agoAll 50k Students aren't engineers...
- QuotedForTruth 6y agoAs a UF CS grad, I bet less than 100 students will get hands on this computer each year. Maybe way less than 100. Although I wasn't a graduate student so maybe way more of them have access than I realize.
- easton 6y agoDo you know how the CS program is doing these days? I heard it was on the ropes a few years ago but this looks like they are investing in it again?
- etaioinshrdlu 6y agoIt seems like supercomputers are still generally many times more powerful than what AI researchers at top universities or orgs are using. Has any well known AI research been done on supercomputers? It seems like the case that literally just throwing money at the problem is a solid idea nowadays.
- llukas 6y agoCmon. Just look at requirements to train GPT-3 for example... https://lambdalabs.com/blog/demystifying-gpt-3/ https://lambdalabs.com/blog/demystifying-gpt-3/
- etaioinshrdlu 6y agoI guess I had it in my head that they used something like 500 V100's -- which is still a far cry from a big supercomputer these days -- but they haven't published what hardware they used and for how long...
- ydau 6y agoGPT-3 was another buckets worth of evidence in favor of the scaling hypothesis. Performance kept improving (and cost to train kept increasing) as more parameters were added. Even with 175 billion parameters, the performance had not yet plateaued. One take-away is that throwing a lot of compute at the problem helps tremendously :). You can read more about GPT-3 here: https://lambdalabs.com/blog/gpt-3/ https://lambdalabs.com/blog/gpt-3/
- sitkack 6y agoAre you alluding to "The Bitter Lesson" [1] by Rich Sutton [2]? [1] http://incompleteideas.net/IncIdeas/BitterLesson.html http://incompleteideas.net/IncIdeas/BitterLesson.html [2] http://incompleteideas.net/ http://incompleteideas.net/
- deleted 6y ago[deleted]
- shmerl 6y ago> It will also benefit from Nvidia’s suite of AI application frameworks Sounds like lock-in.
- HeWhoLurksLate 6y agoIs it lock-in when it's the only thing or the best thing on the market?
- shmerl 6y agoIt is lock-in when it makes it very hard to move to anything else even if it's better. That's Nvidia's point. They don't play fair. They give this "for free" with hard strings attached to Nvidia.
- boulos 6y agoSigh, I wish people wouldn’t say “petaflop” for these. https://www.nvidia.com/en-us/data-center/a100/ https://www.nvidia.com/en-us/data-center/a100/ is the most official reference. If you scroll to the bottom, you’ll see that an A100 part can do ~20 Teraflops (either FP32 or FP64 in little-matrix aka tensor mode). When they say “each A100 can do 5 petaflops”, they mean each DGX which has 8 such cards and thus they mean 600-ish something-ops per card. The generous assumption is that’s FP16 or bfloat16 for sparse ops, and therefore they are “flops”. The reality is that if someone says “supercomputer” in the general sense, they mean scientific computing and so mean a double-precision LINPACK benchmark. The 1120 A100 parts (8x140) doing 20 “real” teraflops each has an absolute peak of about 22 Petaflops (and older code without tensor mode would be half that on FP64). tl;dr: ML isn’t scientific computing, and those are different flops, but “10 petaflops” just doesn’t sound as impressive.
- deleted 6y ago[deleted]
- rrss 6y agoYou're the "disclaimer: I work for google cloud" person, right? Maybe you can help solve this problem (and it's a real one, it massively sucks that everyone counts whatever "flops" they want). Start here: https://cloud.google.com/tpu https://cloud.google.com/tpu > Cloud TPU v3 Pod > 100+ petaflops Is the Cloud TPU v3 even functionally capable of computing in fp64 at any performance?
- m0zg 6y agoThis being NVIDIA, University of Florida would be wise to hire an independent company to benchmark their "700-petaflop".
- abainbridge 6y agoDoes anyone know why everyone is still buying Nvidia instead of custom AI accelerators from other vendors? For example, on paper the new Graphcore machines look like an easy win, or at least a risk worth taking. (I see this particular supercomputer was funded by Nvidia but my question is about the general trend).
- freeone3000 6y agoDo those custom AI accelerators work with PyTorch and Tensorflow? The graphcore website says that support with PyTorch aten is available in 2020, but it's not -- and Aten is inference only, no autograd, and no extensions. Without those, we can't train on it.
- abainbridge 6y agoGraphcore claims they do. I don't know true that is though.
- JBorrow 6y agoMany people use HPC for things that are not AI/ML, such as CFD. Those applications are typically written using MPI/CUDA
- Cthulhu_ 6y agoRisk mitigation, maintenance, resell value (?), support, reliability. The custom AI accelerators you mentioned, how long have those been in business for? How many units have they moved? How many generations of hardware have they produced? Will they still be around to replace or upgrade units in 5-10 years? How flexible are they in their workload? That's a lot of factors to keep in mind when you're spending millions on a supercomputer. I'd go for an established hardware provider as well. I'm not knocking the custom AI chips, but I wouldn't try and max out my budget with those - keep them for smaller applications for now.
- eecc 6y agoYour supercomputer won’t last more than a couple years, 5 at max. Then it will simply be unjustifiably expensive to run when newer, more power efficient hardware becomes available. What matters is the software platform and compatibility: you don’t want to retool and port all your existing programs. That’s where having a platform - CUDA - becomes a deal-maker. What I don’t understand is how consumers - big, institutional consumers- don’t insist on viable vendor neutral platforms like OpenCL. Obviously they’re the only ones that have an interest in cross-vendor compatibility.
- VadimPR 6y agoHow bad is this for the environment?
- abainbridge 6y agoSeems like it has about 700/5 = 140 GPUs. And they are about 0.4kW each. Let's say it is 50% utilized for a year. That is 140 * 0.4 * 0.5 * 365 * 24 = 245280 kWh. Let's say electricity generation creates 0.5 kg CO2e per kWh. So 122640 kg CO2e per year. For comparison, driving a cars creates about 0.2kg CO2 per km. Or 0.32 kg per mile. So about the same as driving 122640 / 0.32 = 383250 miles per year. Call it 38 cars. Edit: Corrected the above CO2 per km figure to fix a factor of 100 error. Obviously there are a million things wrong with this analysis. For example, we don't know if they're going to use the machine to do climate modelling that will lead to a headline in the media that causes the green party to be elected ;-)
- lorenzhs 6y agoHPC system utilization is typically upwards of 90%, these things are not idle. There's almost always a queue of jobs waiting to run, at least in my experience.
- rubatuga 6y agoDefinitely, also it makes you hate those who hog up the resources. A good scheduler will take into account the past usage however.
- jcims 6y agoCars generate closer to 122g of CO2 per km [1]. 122,000kg would be ~1 million km or ~50 cars. 1 - https://ec.europa.eu/clima/policies/transport/vehicles/cars_en https://ec.europa.eu/clima/policies/transport/vehicles/cars_...
- abainbridge 6y agoDamn. I was confused by the fuel consumption being in per 100 km units. I've edited my post. Thanks.
- codecamper 6y agoI predict this supercomputer will be a flop.
- unnouinceput 6y agoYup, all almost 11 of them :)
- 29athrowaway 6y agoA supercomputer for flat earth research, and other subjects relevant to Floridians.
- OffensiveTomato 6y agoOne of Nvidia's confounders is actually a UF alumnus. Good way to give back!
- swiley 6y agoAll these big companies giving free stuff to university students to hook them into their proprietary technology. I remember some of my friends graduating and realizing marlin wasn’t free.
- fortran77 6y agoMaybe they can use it to simulate load and stress on pedestrian bridges for some of the other Florida Universities: https://en.wikipedia.org/wiki/Florida_International_University_pedestrian_bridge_collapse https://en.wikipedia.org/wiki/Florida_International_Universi...
- xvilka 6y agoThey should build proper Linux driver first without enormous signed blobs. One of the worst companies for FOSS community.
- deleted 6y ago[deleted]
- evancox100 6y agoGreat name for the computer: “HiPerGator“
- svalto 6y agoThat's an awesome name. Just like hipermercado or KluckKluckBell.
- throwgeorge 6y agoHiPerGator is the existing HPC cluster https://www.rc.ufl.edu/services/hipergator/ https://www.rc.ufl.edu/services/hipergator/
- ja27 6y agoJust baffling that they were on the verge of closing their CS department 8 years ago. https://www.forbes.com/sites/stevensalzberg/2012/04/22/university-of-florida-eliminates-computer-science-department-increases-athletic-budgets-hmm/#78eb901b56a1 https://www.forbes.com/sites/stevensalzberg/2012/04/22/unive...
- bubblethink 6y agoI think this was a bit of a clickbait headline, and the real story was bit more nuanced. They were trying to do some reorgs (kind of like merging/splittling different CS, ECE, Information Systems etc. depts.).