4 ms·
Just listened to Lex Fridman's three hour interview with George Hotz this weekend. He spoke about his new company, Tiny Corp. Tiny Corp. will be producing the
by rmason 3y ago
Just listened to Lex Fridman's three hour interview with George Hotz this weekend. He spoke about his new company, Tiny Corp.
Tiny Corp. will be producing the Tiny Box that lets you host your own LLM at home using TinyGrad software.
The tinybox
738 FP16 TFLOPS
144 GB GPU RAM
5.76 TB/s RAM bandwidth
30 GB/s model load bandwidth (big llama loads in around 4 seconds)
AMD EPYC CPU
1600W (one 120V outlet)
Runs 65B FP16 LLaMA out of the box (using tinygrad, subject to software development risks)
$15,000
Hardware startups are extremely difficult. But Hotz's other company Comma.ai is already profitable so it is possible. I find the guy extremely encouraging and he is always doing interesting stuff.
- bananapub 3y agowhy are you valuing George Hotz (or Lex Fridman's) opinions in this at all?
- outside1234 3y agoWhat podcast (or otherwise) was this? Was it this one? https://www.youtube.com/watch?v=dNrTrx42DGQ https://www.youtube.com/watch?v=dNrTrx42DGQ
- rmason 3y agoYes, that's the podcast I watched. He also talks a lot about comma.ai which is now profitable on $5 million of VC funding. I think that he has raised a similar amount of funding for Tiny Corp.
- LoganDark 3y agoIs $15,000 really an "at home" sort of price? (If money is no object, why not grab an oxide.computer rack? Assuming you have three-phase power, of course...)
- phone8675309 3y agoA new car, especially electric, costs significantly more than $15,000, yet people buy them and bring them home daily.
- paulryanrogers 3y agoTransportation is a universal need, and the time savings over other forms are often worth the premium. Doubtful the same can be said for these.
- MuffinFlavored 3y ago> Is $15,000 really an "at home" sort of price? For something that is... going to hallucinate? I don't get it?
- bavell 3y agoI mean, people spend way more than that on random art paintings and sculptures right? With this, it could always be repurposed if something new and shiny comes along
- LoganDark 3y ago> With this, it could always be repurposed if something new and shiny comes along I mean, the oxide.computer specs make me go "this would be the most killer gaming machine in all of existence" ... though it's a server rack meant to host an entire private cloud for an entire corporation, I still would love to glom together as much resources as can fit into a single VM for gaming purposes.
- natsucks 3y agoRight, especially when it's not clear if such a machine will be needed a year or two from now to run a model with the equivalent performance.
- hereonout2 3y agoAt this rate it may not be needed in a few months to run a model of equivalent performance. At work I've been successfully running llama2 variants on Mac M2 systems with more than adequate throughput for small experiments and examples. Can also boot a 4x Nvidia T4 instance on AWS (g4dn.12xlarge) to larger models. I feel you'd need to be doing inference on a 24/7 basis for at least a year to justify the price of this machine.
- Me1000 3y agoI don't think the $15k price tag is intended for regular consumers. I didn't listen to the Lex Fridman interview (my patience starts to wane after the first two hours), but I did listen to another of Hotz's interview on another podcast, and I believe he referred to this their "Tiny Box" as more of a dev box. It's for the engineers and tinkerers who want a capable machine for AI. So while expensive, it fills a niche for those who don't want to deal with linking multiple Nvidia GPUs together and deal with the power and cooling setup that come with that.
- Havoc 3y agoHe's definitely thinking bigger than just dev boxes. Lately he's repeatedly mentioned building own chips to be somewhere in the future. How plausible that is is another matter
- Me1000 3y agoRight, that’s the long term plan. I’m speaking specifically about the $15k first release of the tiny box.
- Havoc 3y agoDoes oxide do GPU builds? I thought they're classic server racks (CPU) > three-phase power, Think for the tiny grad it's run off single, but for the US the GPUs are power limited given the 110V grid.
- steveklabnik 3y agoWe (oxide) aren't doing a GPU focused product yet, that's right.
- d_sem 3y agoThat's roughly the price of two original Macintosh computers adjusted for inflation.
- naillo 3y agoImagine how many 4090s you could buy and run in a cluster for $15,000 though
- J_Shelby_J 3y agoI hate to burst your bubble, but more than two 4090s is going to put you at BoM + labor costs around $15k. Especially if you have to upgrade your electrical and hvac.
- elorant 3y agoThey seem to have six, either 4090s or 3090s judging by the total amount of VRAM. How many more do you think you could get with $15k considering all the other hardware costs? I doubt you could make it more efficient at this price point.
- ColonelPhantom 3y agoThe Tinybox is planned to use AMD, not NVIDIA. The GPU they're using to build Tinybox is the 7900 XTX.
- heyitsguay 3y agoIt might be tough to make more efficient, but $15k seems exactly about the price of "stick 6 4090s in a decent box and throw in a couple grand for my troubles", versus any revolutionary hardware configuration. The way it advertises running fp16 Llama 70B feels a bit contrived too, given the prevelance of quantizing to 8 bit at minimum.
- lawn 3y agoWhy buy this instead of building your own setup with X 4090s?
- gmerc 3y ago144GB Vram vs 48?
- thatcherthorn 3y agoI think he mentions this in the podcast. One of the challenging problems they're attempting to tackle is enabling the build to work with standard 120 volt wall sockets. He mentions throttling the system (GPUs specifically) to use less power at cost to performance. That and setting up the software to work w/ an array of GPUs rather than one seems difficult.
- Aerroon 3y agoI don't know if he actually said this, but power limits on gpus do work. I've been limiting my gpu to about 60% max power during long AI workloads and I lost about 10% of performance. A 40% reduction in power usage seems worthwhile. It probably depends on your gpu though. Undervolting would probably be even better.
- robotnikman 3y agosame here with the 3090 I use. Power limited to 60% and I barely notice a difference when experimenting around with AI or playing games. Really good to use in a SFF PC
- psyclobe 3y agoThink of it like an ac unit, they are expensive but indispensable. I have always envisioned locally trained ai and now it is happening!
- Aerroon 3y agoYou probably need an AC unit too if you're going to be dumping 2000W of heat into your living space.
- doctorpangloss 3y agoGeorge Hotz misread the NVIDIA Inception pricing as per unit instead of as a rebate, believing the GPUs are 80% cheaper than they actually are.