29 ms·
Training LLMs from ground zero as a startup
- mleonhard 3y ago> I think this could be more about the competency of the hardware team that manages your accelerators rather than the underlying chip. Google's systems are reliable because of the tens of billions of dollars that Google has invested into developing datacenter hardware, software, and processes over 25 years. Highly-competent teams at smaller and less-mature organizations will always deliver a much worse product. Another thing to consider is priorities. Google prioritizes reliability. They retire parts that fail repeatedly, even if the failures are relatively infrequent. Smaller and less-sophisticated datacenters keep parts in service even with frequent failures, or don't even monitor failure rates of certain parts. Smaller datacenters buy and use Google's old parts and unreliable parts. Therefore unreliable machines does not imply anything about the competency of the hardware team. If the low reliability of the hardware is making your work slow, then how about improving the software so it can tolerate the unreliable hardware, or switching to a more reliable (more expensive) hardware provider?
- swyx 3y agofor context Yi Tay was Tech Lead on Google PaLM, UL2, Flan, Bard, etc and now is cofoudner at Reka (which has shipped some v interesting small multimodal models that have featured on here). I prompted him for this post as an ex-Googler now training LLMs as an independent startup https://twitter.com/YiTayML/status/1765105066263052718 https://twitter.com/YiTayML/status/1765105066263052718 our conversation was recorded here https://sub.thursdai.news/p/thursdai-feb-15-2024-openai-changes https://sub.thursdai.news/p/thursdai-feb-15-2024-openai-chan...
- swyx 3y ago(update: i submitted this yesterday and it didnt get traction, i guess @dang must’ve merged the old submission in here. you really didnt have to, but its a nice gesture. thanks dang!!)
- axpy906 3y agoGreat too see you on here. Love Latent Space podcast.
- swyx 3y agoaw thank you for listening. some weeks its very much a labor of love lol. no events planned near term but come to the big shindig in june https://ti.to/software-3/ai-engineer-worlds-fair https://ti.to/software-3/ai-engineer-worlds-fair . last year's summit was the first time i really understood how much of a reach we have and how many good AI people we've managed to gather as friends.
- dwaltrip 3y agoI love it as well, it’s a fantastic resource :)
- 3abiton 3y agoIs he the person after the Yi LLM model?
- bigcat12345678 3y agoNo Yi LLM models are from [0], Kaifu Li's LLM startup. [0] https://www.lingyiwanwu.com/ https://www.lingyiwanwu.com/
- pama 3y agoTraining LLM from scratch is a super important issue that affects the pace and breadth of iteration of AI almost as much as the raw hardware improvements do. The blog is fun but somewhat shallow and not technical or very surprising if you’ve worked with clusters of GPUs in any capacity over the years. (I liked the perspective of a former googler, but I’m not sure why past colleagues would recommend Jax over pytorch for LLMs outside of Google.) I hope this newco eventually releases a more technical report about their training adventures, like the PDF file here: https://github.com/facebookresearch/metaseq/tree/main/projects/OPT/chronicles https://github.com/facebookresearch/metaseq/tree/main/projec...
- axpy906 3y agoIf you’re doing research JAX makes some sense. Probably some Google bias in there too.
- lyapunova 3y agoTo be honest, most researchers in applied ML in the bay say the opposite. If you are trying to be nimble and prototype, use pytorch. If you're trying to gain some optimizations as you near deployment, rewrite in Jax.
- abeppu 3y agoIt's worth taking a second to note that the author just assumes that readers understand "the wilderness" to mean "not Google". This post gives a lot of credit to Google's infra and hardware teams, and I'd love to read a perspective from one of those insiders who then went on to do related work elsewhere.
- joe_the_user 3y agoI took the phrase to mean "outside any large company". It seems like a fairly obvious metaphor; if you have a starup working on a large scale infrastructure project, you have to set your own logistics just a camp in the literal wildness.
- choppaface 3y agoReally telling quote: > I was completely taken aback by the failure rate of GPUs as opposed to my experiences on TPUs at Google Should be "I was completely unaware of the failure modes of GPUs, because all my career I've been inside Google and used Google TPUs and was well-acquainted with those failure modes." I've used GPUs mostly, and when I tried TPUs the jobs failed all the time for really hard-to-debug reasons. Often the indirection between the x86 chip and the TPU device caused hours of hair-pulling, stuff you never get with x86+nvidia+pytorch. 10-15 years ago, Google minted many $10m+ data scientists (aka Sawzall engineers) who also ventured "into the wilderness" and had very similar reactions. This blog post is much more about the OP hyping his company and personal brand than contributing useful notes to the community.
- StarCyan 3y agoWhen was this? I use JAX+TPUs to train LLMs and haven't experienced many issues. IMO it was way easier to set up distributed training, sharding, etc compared to Pytorch+GPUs.
- quadrature 3y agoI think the OP is referring to hardware failures rather than software not playing well together.
- 3y ago
- yalok 3y ago> All in all, this is only a small part of the story of how we started a company, raised some money, bought some chips and matched Gemini pro/GPT 3.5 and outperformed many others in less than a year having to build everything from scratch. I wonder what was the budget spent for the chips/cloud GPUs to achieve GPT 3.5 level LLM - at least in the order to magnitude - 2-5 millions?
- joe_the_user 3y agoSo essentially a startup in this context has a small number of people and a large amount of money for training clusters. The article describes many operation leasing servers - that you assume to go many startups (or existing firms). So it seems like you have the various LLM creators all doing roughly the same sort of thing (training with text and image data) with similar hardware and similar data. Each of these naturally has their own brand of "secret sauce" for distinguishing their venture. The various secret sauces can make a difference in the quality of an LLM's output. Yet overall, this seems like a massive, energy intensive exercise in redundancy.
- dauertewigkeit 3y agoI don't think most of them have any kind of secret sauce. I think the founders hope to get bought out simply for being able to train "near-SOTA" LLMs. I guess achieving that level of skill and infra could be valuable enough to build upon.
- imtringued 3y agoThere was a guy who followed a tutorial about how to fine tune mistral with DPO, who has zero computer science skills and his model ended up at the top of the hugging face leader board among the opensource models with 7 billion parameters. Some random guy managed to outdo the creators of the LLM.
- joe_the_user 3y agoSure, that's also a factor but I'd say it reinforces my main point.
- DeepChill 3y agoGood point, so the only real differentiator would be the size & quality of the data being fed and the fine tuning done on the model? I wonder what else differentiates LLMs from each other
- 3y ago
- barkingcat 3y ago[flagged]
- omeze 3y agoWell, GPU failure modes (from what Ive heard from ML infra people) are often subtle, eg incorrect multiplication results. So its not as simple as the usual ‘treat resources like cattle not pets’ because tou dont know which cows have mad cow disease before committing to an expensive training run
- esafak 3y agoI think it is a bit more nuanced than that. The author is experienced.
- deleted 3y ago[deleted]
- twelfthnight 3y ago> To be very frank, I would have to say the quality of codebases externally significantly lag behind those I’ve been used to at Google Haven't worked at Google, anyone else share this sentiment? I always feel like working with Google code is typically not idiomatic and super difficult to go "under the hood" if anything isn't precisely on the happy path.
- winwang 3y ago(not googler) Google's codebase is idiomatic to Google due to their strict language tooling. e.g. their C++ code stays away from advanced features. The tooling teams at Google have very strong say.
- twelfthnight 3y agoI get that sense too. Probably does work awesome if you're inside. But man it's a mess when they externalize stuff. Just one example: their cloud platform CLI includes an entire python installation and takes 1.7G on disk, just to make API calls...
- jen20 3y agoI have never understood why cloud providers seem to think it is OK to write their CLIs in Python. The AWS one is too, and the Azure one went from Node.js to Python some time ago.
- anonymous-panda 3y agoPackaging and stability reasons. Same for why it’s a 1.7gb install - probably where they landed after having tons of support issues on some random Python version they didn’t test or some issue with a dependency that had that issue. Freezing the entire set of artifacts is more stable and Python lets you move pretty quick. I can’t speak to why nodejs vs Python though - maybe Python is easier to embed?
- 3y ago
- observationist 3y ago[flagged]
- anonymous-panda 3y agoIt has many meanings. Terrorism, weaponry and destruction are one meaning. Here are two others: > The center of rapid or intense development or change. noun > The starting point or most basic level. The author used it correctly even though you got triggered by the word. https://www.wordnik.com/words/ground%20zero https://www.wordnik.com/words/ground%20zero
- observationist 3y agoNot necessarily. In the context you reference, you could say Silicon Valley is ground zero for technological revolutions, or the explosive spread of social media. Deliberately building a thing from the ground up doesn't match the nuances of the many and varied meanings of the phrase; it's not quite right. The primary semantic connection is the misuse of the phrase "from the ground up" and conflating it with "ground zero." There's no direct or indirect semantic relationship between something explosive or chaotic or destructive, and deliberately and methodically building a language model, in a calm and rational process. The meaning doesn't map to the phrase in the context of this article's title. If you were building an etymology website from the ground up, you'd hope it would be ground zero for an explosion of awareness about how different words, phrases, and sayings could be used correctly in different contexts. Something like that - I was trying to be considerate about the Ground Zero memorial, and the many ways in which "ground zero" can refer to things that people are deeply invested in. My personal stake is just an observation that it's not quite the right turn of phrase for the use to which it was put. It seems likely that "from the ground up" was the intended phrase, and it might have gotten awkwardly revised, since the word startup is also in the title?
- throw310822 3y agoI agree with the gp- while ground zero can have neutral or even positive vibes, the meaning is always that of point of origin of an explosion, real or metaphoric. And I think that the meaning "most basic level" is wrong- the dictionaries might be simply recording a rare and wrong usage of the expression.
- bo1024 3y agoThis is very interesting, but I really want to hear about the training data process!
- stealthcat 3y agoShould list most of the technical debt accumulated so far and rank them. At this stage, lots of corners have been cut.
- LZ_Khan 3y agoI wish I knew how to do yolo runs. - signed, a compute resource hog at FAANG
- planet_y 3y agoI’m wondering if the title should read “from the ground up” instead of “ground zero”? https://en.wikipedia.org/wiki/Hypocenter https://en.wikipedia.org/wiki/Hypocenter
- zer00eyz 3y agohttps://www.merriam-webster.com/dictionary/ground%20zero https://www.merriam-webster.com/dictionary/ground%20zero It is a perfectly acceptable use of the idiom.
- davidmurdoch 3y agoAcceptable, but maybe not perfectly.
- dotancohen 3y agoYes, the title sounds like somebody confused two idioms. That's not the type of author from whom I want to learn.
- frozenseven 3y ago1. As others have pointed out, it's a perfectly valid idiom. Check a dictionary. 2. How do you think idioms are created in the first place? 3. What exactly forces you to act like this?
- makoto12 3y agocould be intentional. Implying LLMs are a proverbial nuclear bomb to the tech landscape. but honestly it threw me as well
- julianh65 3y agoSo which compute providers have folks had a good experience with?
- rokkitmensch 3y ago[flagged]
- deleted 3y ago[deleted]
- hackerlight 3y ago> In the end it took us only a very small number of smaller scale & shorter ablation runs to get to the strong 21B Reka Flash and 7B edge model (and also our upcoming largest core model). Finding a solid recipe with a very limited number of runs is challenging and requires changing many variables at once given the ridiculously enormous search space. In order to do this, one has to abandon the systematicity of Bigtech and rely a lot on “Yolo”, gut feeling and instinct. > Thankfully, I (and many of us in the team) have built up this intuition quite a bit in our ML careers to get it right within a substantially short amount of tries. While we’ve trained really good models before in our previous jobs, differences in training infrastructure, data, incorporation of new ideas and other environmental issues can still cause non-trivial differences in outcomes. That said, a strong prior helps to significantly cut down the search space and is probably one of the easiest explanations to why we were able to train really strong models with so few trials, resources and experimentation.
- a_bonobo 3y agoBut what is the product they're selling? The main Reka.AI page looks like a regular ChatGPT clone, an LLM you pay for by the token. How is this different from all these other companies? Pricing seems to be comparable to ChatGPT 3.5-Turbo.
- polygamous_bat 3y agoPerhaps a cure for venture capitalist FOMO for not having invested in AI?
- classified 3y agoAbsorbing the risk of copyright and license violations en masse for the training data as a service?
- TrackerFF 3y agoBig question is, how do small startups manage to get funding for LLM products if they don’t have the “correct” background / pedigree? The world of LLM startups is beginning to look like the world of hedge funds and private equity firms - where the prerequisites for seed/funding are: A) Prestigious employment history / correct pedigree. B) Solid network of investors ready to jump before any product has even begun.
- nlpnerd 3y agoThey don't. This is probably one reason why VCs invest in these companies. There is a natural moat since there is only a very finite number of people in the world has the right experience to raise, and only those who can raise can ever have the experience. At least until compute cost drop to a cheap enough level...
- h100 3y ago[dead]
- rvz 3y agoThen what happens when the LLM or AI performs worse than expected? Spend more money fine tuning? By the time you get it all working, not only you've spend lots of your VC capital on training alone, your competitors (Google, Meta, etc) already released a more powerful model much better and quicker than you before you could your run the second training epoch. Another example of a startup incinerating the VC pump and dump scheme for vaporware AI snake-oil.
- tkgally 3y agoI learned about reka.ai from this post; their LLMs don’t seem to have been discussed much on HN yet [1]. So, out of curiosity, I spent the last hour testing prompts with their chat interface [2] in comparison with ChatGPT 4, Gemini Advanced, Claude 3, and Mistral Large. I put the results at [3]. Overall, Reka Flash doesn’t seem significantly worse or better than the others. A lot more testing would be necessary to be sure, of course. [1] https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=reka&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... [2] https://chat.reka.ai/chat https://chat.reka.ai/chat [3] https://gally.net/temp/20240307llmcomparison.html https://gally.net/temp/20240307llmcomparison.html
- egberts1 3y agoTL:DR: LLM training is highly susceptible to GIGO. (GIGO is what one gets when feeding LLM with "G"arbage "I"n, "G"arbage "O"ut.) This is the current problem about making a vaccine signature fitting like a glove ... as tight as possible ... when populating the anti-malware (i.e. IDS/IPS/NDS/XNS) search pattern engine for use by Aho-Corasick-variant algorithms (such as Parallel-Failureless Aho Corasick). However, LLM as a binary code-based detector for malware detection has a very limited benefit (it is there but only as a backend topical add-on after all other conditionals have been identified). LLM lacks qualifying conditionals surrounding a premise data, and I have my doubts of using LLM for medical diagnosis as well: until we start having LLM denote the much-needed weighted combo-conditionals by "percentages".