4 ms·
Petaflops to the People: From Personal Compute Cluster to Person of Compute
- croes 3y agoI see George Hotz and I think another interesting idea that gets another half-baked implementation.
- dang 3y agoPlease don't cross into personal attack on HN.
- swyx 3y agothanks for posting tosh! we were really honored that George granted us the first interview on tiny corp. We even set up a dedicated podcast youtube for the first time (Alessio and I are relative youtube newbs) and George was kind enough to come back to Twitter just to help us kick it off: https://twitter.com/realGeorgeHotz/status/1671207867045806080 https://twitter.com/realGeorgeHotz/status/167120786704580608... Personal highlights for the HN crowd: - the three theses of tiny are all interesting: the CISC/RISC analogy, why you should start with software for commodity GPUs rather than taping out chip design, why to avoid Turing complete layers in the ML stack - I loved comparing his natural debugging and CI style to John Carmack - developers who assume geohot is a cowboy coder might be surprised at how seriously he takes developer experience - Dishing on tinygrad vs ggml and Mojo (i'll just leave it there, its good) - how everyone is in this mad quantization rush but nobody's putting up benchmarks to show that it works (tinybox is resolutely supporting non quantized LLaMA) - hardware design constraint choices for tinybox (<60 decibels, single power plug, etc) and then we of course indulged in the array of offtopic convos for fan service: - GPT4 is just a mixture of experts - 8 GPT's in a trenchcoat - Elon vs e/acc - Rich Sutton's Bitter Lesson - The Goddess of Everything Else (go listen to him read it https://mixtape.swyx.io/episodes/the-goddess-of-everything-else-george-hotz-scott-alexander https://mixtape.swyx.io/episodes/the-goddess-of-everything-e...) and of course, he's doing all this so that you'll consider working on/for tiny. job application details on https://tinygrad.org/ https://tinygrad.org/
- swyx 3y agopersonal reflections from interviewing him: - Idk how to say this but surprisingly not arrogant. we both prepped by watching his lex fridman and prior livestreams, so its not like we were dealing with an unknown quantity, but I've never seen him interact with "normal" (technical, but not his level of technical) people like us before. he was pretty patient I think, except I definitely got roasted for suggesting you could jailbreak the tinybox. instantly regretted it but he only joked once aboout it which was super cool :) - quite reasonable expectations for his AMD frustrations. the fact that Lisa Su got personally involved and they listened was also fantastic. this isn't a "tech diva flipping the table because they stubbed their toe" story - AMD was found wanting in basic documentation requirements, he raised the issue, they took a while but eventually fixed it. Great partnership all around imo - hiring in the age of AI is really going to get strange. he thinks we're ~20 years out from developer jobs being gone. Yes he is essentially asking people to do work for small bounties. in some circles, that's a no go. but it basically guarantees that the candidate will be a fit, and is fine if you want to keep the team tiny anyway. he also changed his mind on remote work which is cool. - this was the first time i heard anything materially different about GPT4 architecture being a mixture of experts rather than a 1 trillion super model. does raise the question about what GPT5 will be built on if we have indeed run into the limits of scaling model size. - notes to future interviewers: george is ultra open (he identifies strongly as a streamer, is very comfortable with what he is saying and doesnt need a ton of prep or post recording review) - just be super attentive to what he does and does not want to talk about and let him riff and he will indulge. i think we were maybe like a 7 out of 10 on getting the best out of him, but hopefully we will be a lot better by the time we go down to San Diego for the tinybox launch :fingerscrossed: - He gave us advice on whether or not we wanted to go big and be the next Lex Fridman or stay niche and I think we were pretty convicted on wanting to stay niche and technical. To my knowledge there's no other podcast in the world that could have had this deep-but-still-kinda-accessible conversation with George and I strongly feel like something like this has to exist.
- lhl 3y ago> how everyone is in this mad quantization rush but nobody's putting up benchmarks to show that it works (tinybox is resolutely supporting non quantized LLaMA) I don't think this is true. llama.cpp has historically been very conscientious about benchmarking perplexity. Here's a detailed chart of baseline FP16 vs the new k-quants: https://github.com/ggerganov/llama.cpp/pull/1684 https://github.com/ggerganov/llama.cpp/pull/1684 While most evals aren't currently evaluating performance between quantized models, there are two evals that are: * Gotzmann LLM Score: https://docs.google.com/spreadsheets/d/1ikqqIaptv2P4_15Ytzro46YysCldKY7Ub2wcX5H1jCQ/edit#gid=0 https://docs.google.com/spreadsheets/d/1ikqqIaptv2P4_15Ytzro... * llm-jeopardy: https://github.com/aigoopy/llm-jeopardy https://github.com/aigoopy/llm-jeopardy - You can see that the same Airoboros 65B model goes from a score of 81.62% to 80.00% going from an 8_0 to 5_1 quant, and 5_1 solidly beats out the 33B 8_0, as expected. Also, GPTQ, SPQR, AWQ, SqueezeLLM all have arXiv papers and every single team is running their own perplexity tests. Now, that being said, every code base seems to be calculating perplexity slightly differently. I recently have been working on trying to decode them all for apples-to-apples comparisons between implementations.
- jxy 3y agoThey are clearly scared of ggml.