11 ms·
Long Live the 'GPU Poor' – Open-Source AI Grants
- politelemon 3y agoVery happy to see oobabooga on the list. Everyone's contributing in different ways, and ooba serves as a fantastic portal to try out various models which I appreciate. Its readme even says it wants to be like the automatic 1111 of text generation. For beginners like me who are curious about this new world, ooba is a great entry point.
- satvikpendem 3y agoWell, at least this time a16z is giving money to actually useful technologies instead of bloviating about the world changing potential of cryptocurrencies. I imagine they've written down most of their cryptocurrency investments at this point. That being said, these are some great grants for local LLM builders, local computation for such powerful tools is needed, and will continue to be needed, as industry incumbents like OpenAI continue to not release their models.
- zitterbewegung 3y agoDon't VCs basically chase everything because they only rely on 10% or less to succeed but, that 10% needs to have at least 10x or more profit and then they do another round?
- espadrine 3y agoI had become honestly cynical of their actions in the past, as they felt hype-driven and externalities-blind. Yet I don’t see how this grant could become rapacious. Hopefully it is an attempt at cleaning the brand which can work as making amends.
- deleted 3y ago[deleted]
- rajko_rad 3y agoHi HN, we are super excited to announce this new initiative!! Please note, this program was designed to support individuals, teams, and hackers who are not pursuing commercial companies. Nevertheless, these projects push the state of the art in open source AI and help provide us with a more robust and comprehensive understanding of the technology as it is developed. We are really proud to be contributing in this small fashion and grateful to the first cohort and all others contributing in this space!
- latchkey 3y agoI'm curious, where are you getting the GPUs?
- rajko_rad 3y agoin most cases the teams already had their own source or platform of choice!
- samstave 3y agoMaybe build a FAQ for others on how to accomplish, or resources you can point people at?
- rajko_rad 3y agoInteresting, hadn’t thought of that, thank you! If you want to host end points Replicate is a great option, they also have a newer fine tuning api and solution! for raw VMs with GPUs right now it’s a bit situational and you have to try multiple different vendors tbh, also really depends on the capacity you need and which machines!!
- kcorbitt 3y agoA lot of people are using RunPod for experimental/small-scale workloads. They have good network and disk speeds and you can generally find availability for a latest-gen GPU like an L40 or 4090 if your workload can fit on a single GPU. One GPU is plenty for fine-tuning Llama 2 7B or 13B with LoRA or QLoRA. They also sometimes have availability for multi-GPU servers like 8xA100s, but that's more hit-or-miss. If you want to go even cheaper vast.ai is a popular option. It's a P2P marketplace for individuals to rent out their GPUs. You can generally get a ~20-30% discount vs RunPod prices by using Vast, but network speeds and perf are much more variable and there's always the possibility that the host will just shut you off without warning. I also wouldn't recommend using it if you're training with proprietary data since they can't guarantee the host isn't logging it, but most of the OSS fine-tuning community publishes their datasets anyway.
- Michelangelo11 3y agoThis is great, but I'm not seeing anything about how to apply, or really anything about the application and selection process at all. Is this by design?
- deleted 3y ago[deleted]
- zekrioca 3y agoIt seems so.
- tikkun 3y agoMy guess is that they'll prefer to get nominations. Having once done a grant application process - I think their approach is smart. If they opened grant applications, they'd get thousands of (sadly mostly very low quality) applications. It's extremely hard to vet. It's a much safer bet to wait until projects are already popular in the community, already being relied on by people, and where those devs would love to spend more time on their open source projects but they're limited by money. Yes, the downside of this is that some people could make things happen if they were given money first. But, if I was them, I'd still prefer to do what they're doing and pick projects that are already successful and give grants to those. (This isn't a comment about your project specifically, for all I know your project might be a popular open source LLM project that lots of people already rely on, in which case they probably already know about your project and will probably reach out to you for the second round, is my guess)
- Havoc 3y agoAmazing. Those are really solid 1st picks. A huge chunk of the organic experimentation relies directly on them.
- rajko_rad 3y agothanks so much - we agree!
- minimaxir 3y agoIt's interesting to note that there's particular emphasis on "removing censorship from LLM outputs" given a16z's investments outside the AI space in some ideological social media companies.
- skilled 3y ago> We believe artificial intelligence has the power to save the world—and that a thriving open source ecosystem is essential to building this future. Thank you for understanding the fundamental flaws of the World and human consciousness, and thank you for spending multiple lifetimes as Buddhist monks (not to mention lamas) to understand the intricacies of the human mind from a scientific perspective. This is exactly what we need right now and this initiative looks to be the answer to all of our problems.
- deleted 3y ago[deleted]
- waihtis 3y agohail e/acc and the thermodynamic god
- svnt 3y ago“We are disguising this round of seed funding as a grant, accomplishing all of the below: 1) brand-washing for people who don’t do the math 2) avoiding angering our LPs and/or violating our fund theses and 3) getting in on the ground floor of things that might follow OpenAI through the non-profit-to-VC-darling door Thanks for reading our press release, feel free to try to apply using your personal network (somehow) because we don’t care as we’ve already landed our positions.”
- deleted 3y ago[deleted]
- notatoad 3y agounless i'm missing something, they're not taking equity. that's what makes it a grant, and not seed funding. what math should i be doing here?
- svnt 3y agoProbably the grants have clauses that convert them to debt or equity in case of a transition of the company. [0] The bigger math is significance_of_contribution = $size_ai_grant_program/$size_ai_vc_program (or maybe $size_ai_grant_program - $size_ai_vc_program) [ 0 https://link.springer.com/chapter/10.1007/978-3-030-72428-3_17 https://link.springer.com/chapter/10.1007/978-3-030-72428-3_... ]
- wslh 3y agoWe work in source code security auditing with semi-automated tools using AI and other techniques. Is this approach covered in the grants?
- rajko_rad 3y agoif the work is OSS/public and you aren't trying to start a VC-backed company, it could be! We usually want to see some early work already published, community reception and validation, but feel free to DM Matt or myself and we can take a look!
- wslh 3y ago> if the work is OSS/public and you aren't trying to start a VC-backed company, it could be! Yes, it is a company that often applies and receives grants. For example, we are working on adding AI to security tools such as https://github.com/CoinFabrik/scout https://github.com/CoinFabrik/scout
- redox99 3y agoI want to congratulate you based on your current grant recipients. They are all very impactful individuals/teams. It shows that you clearly did your research and put the money in the right places.
- rajko_rad 3y agothank you!! Indeed, we have been interacting in space very closely and doing our best to keep up and contribute where possible
- malux85 3y agoIs there a way we can apply? I am building https://atomictessellator.com https://atomictessellator.com, and since I am 100% technical I am not sure about commercialising or open sourcing it, I definitly would like to open source some of the components (e.g. turnkey dockerized scientific computing environments, the molecular force viewer, and also open up a hub for the environments and models) - when I was in Stanford meeting with some comp chemists a few months ago they got very excited for this. I negotiated with my day job to work 2AM to 10AM so that I can work on this passion project in the afternoons, some grant funding would really help me focus on this more. There's quite a bit of interesting stuff not on the website in catalyst discovery and molecular dynamics. Contact me if you'd like to chat / demo. In my ideal world I would work on this full time and totally open source it.
- rajko_rad 3y agofeel free to DM on twitter or linkedin. In a nutshell, if you want to pursue a company now in or in the near future, we would likely need to consider you through our normal investing process for equity funding. If you are considering just working on it OSS, we would encourage taking first steps in that direct, and testing community-project fit validation before quite going full or near-full time!
- intalentive 3y agoTheBloke has had this sponsorship in his byline for a little while now.
- UncleOxidant 3y ago"We believe artificial intelligence has the power to save the world" I don't agree with the side that says AI is about to destroy everything, it seems very hyperbolic... but neither do I agree with this sentiment that it's going to save the world.
- deleted 3y ago[deleted]
- hgsgm 3y agoI don't even understand. A16Z's crypto already saved the world. Why is the world in trouble again?
- geodel 3y agoIt's to make it even more savier. Think if someone could combine LLAMA with blockchain and create Lockchain. This is kind of saviness we all need.
- deleted 3y ago[deleted]
- andrewprock 3y agoAgree. It's not at all obvious the world is in need of saving. That said, software of all kinds (including machine learning) does make the world more interesting and exciting, so happy to see more investment.
- roschdal 3y ago[flagged]
- jxf 3y agoQ: What are the grant amounts?
- mirekrusin 3y agoThey should call this initiative Open AI.
- capableweb 3y agoLibreAI for extra fun :)
- Kon-Peki 3y agoI'm glad you guys are sharing some money around, but I really have a hard time thinking of a person with "Dual 4090s" as "GPU Poor". LOL
- capableweb 3y agoIt's all relative. Compared to the huge companies spending thousands per month/week/day on GPU usage, having two 4090s (which you could acquire for like ~$3000) could be considered on the lower end for sure.
- jacquesm 3y agoDepending on where you live what you pay for power may well be a bigger factor than what you've paid for the hardware.
- deleted 3y ago[deleted]
- qntty 3y agoI interpret "GPU poor" the same way as "house poor", i.e. you're poor (lack cash) because you've use it to purchase GPUs, not that you don't have enough GPUs.
- KaoruAoiShiho 3y agoNo, GPU poor comes from this definition: https://www.businessinsider.com/gpu-rich-vs-gpu-poor-tech-companies-in-each-group-2023-8 https://www.businessinsider.com/gpu-rich-vs-gpu-poor-tech-co... Less than 10k H100s makes you GPU poor. The author is presumably using it as a proxy for "possible to achieve AGI".
- ForHackernews 3y agoIt's like how you're house poor if you only have one brownstone in Brooklyn and a pied-à-terre in Palo Alto.
- htrp 3y agoehartford and thebloke are definitely great picks advancing SOTA in underserved fields. Curious why there are no image ai projects receiving grants here?
- retrocryptid 3y agoI'm still not sure what I'm supposed to be using AI for. I already know how to code, tried for a decade to make ACR (auto content recognition) work with CNNs, and as best I can tell, chatGPT can write a news article for you if you write it first and then hire someone else to edit it.
- artninja1988 3y agoThank you for doing this. Although I'm sure it's not purely for altruistic reasons, open source ai is important:)
- thinktown 3y agoWhat was actually being offered here? Money? How much? This seems like such vague VC-marketing speak.
- deleted 3y ago[deleted]
- bodecker 3y agoCongrats to the folks involved. Even knowing this is partly motivated by branding/marketing, it's great to see a16z getting more aligned with solving real pain points (vs crypto and churning out shallow media in recent years). Hope they can keep it up and hopefully more "thought leaders"/VCs follow suit. Best of luck.
- jacquesm 3y agoNo, they're still saving the world, only now not through crypto but through AI... https://a16z.com/2023/06/06/ai-will-save-the-world/ https://a16z.com/2023/06/06/ai-will-save-the-world/
- skeeter2020 3y agothis just seems like a cost-effective attempt to buy credibility and stay relevant, but I'm a crusty, old curmudgeon so what do I know.
- bodecker 3y agoAgreed, but IMO it's their job to do stuff like this (and prob unrealistic to expect a world where it doesn't exist). I'd rather see a higher percentage of marketing for things that have more real value vs. less
- bodecker 3y agoOf course - there will always be investors/influencers pushing narratives to hype their bets. IMO everyone is better off when the hype is grounded in solving real pain points, which the AI projects I've seen seem to be closer to than most of the crypto projects I've seen. Certainly there are plenty of grifters in AI too (as with any gold rush) and many AI efforts will fizzle out. But it seems there is more real value being created here than in crypto, which is the main thing I'm excited about and hope to see more of
- calderwoodra 3y agoI agree this initiative helps solve real pain points, and I'm not trying to defend "Crypto" with my question here but isn't solving a "real pain" kinda subjective? For example, many people/companies have the real pain/need to improve ad click-through-rates and reduce crypto/gas fees but maybe these aren't problems you care about.
- ricardo81 3y agoLeftfield thought, imagining a tax (vaguely like carbon credits) on energy/cpu cycles/modelling (just my vague idea). With the proceeds going to other projects to diversify the outcomes. Always seems to me our economy is being less based on money and more on energy and productivity, basically being rounded down to physics as a healthy market economy should.
- deleted 3y ago[deleted]
- itissid 3y agoNoob Q: If I have 25000$ can I buy some GPUs and support the effort for supporting GPU Poor? Like is there a DIYer's guide to do this? I don't want to donate the cash but instead set up a handful of GPUs(Somehow?) and let people pay at cost to use them.
- itissid 3y agoAssume location is East Coast(NYC).
- capableweb 3y agovast.ai can help you offer this and has bunch of docs on how you can start hosting: https://vast.ai/docs/hosting/hosting https://vast.ai/docs/hosting/hosting You won't be able to limit it to a particular set of clients who want to use it, but you're basically supporting the "GPU Poor" by doing this, as I think it's mostly smaller companies, individuals and researchers using vast.ai, rather than huge companies.
- redox99 3y ago25K will either get you money for renting servers, or allow you to jerry rig yourself something like a 8x4090 machine. The servers that most people would like to run (8xA100, 8xH100) are over 200k, and even if you had the money they're probably very hard to get right now.
- naillo 3y ago8x4090 machines should be a lot cheaper than that. 1599*8=12,792. I.e. closer to 16x4090 machines. Sure labor is worth something but he has it in cash, and also there's barely a lot of people (yet) paying high prices for research (non ML ops) labor hours for this yet so the salary wouldn't be that high anyway. Just a pet peeve when ppl under exaggerate how many gpus/flops a dollar can buy (if you're slighly smart with your money).
- junipertea 3y agoWhat about PSU to power all of this, GPU motherboards, SSDs, CPU to not bottleneck this and sufficient RAM? You could argue it's more than 8, but definitely not 16. To utilize those GPUs to the max you need memory bandwidth to even feed it enough data.
- RcouF1uZ4gsC 3y ago> Eric Hartford: fine-tuning uncensored LLMs Makes me very happy that this is being undertaken. LLMs will be a very valuable tool. Having censorship built into these tools makes me very nervous. Imagine if Word refused to allow you to type any “bad word” in the name of safety and ethics.
- naillo 3y agoA16z is getting 'mad respect' from me for taking this line of the 'ai wars' and generally their attitude on all this. Really cool that they're going straight into 2010 era startup energy on this new wave. Idk I just find them really cool and inspirational.
- outside1234 3y agoIf nothing else, it is good to see them moving off of the Bitcoin grift
- deleted 3y ago[deleted]
- zekrioca 3y agoI'm not sure why an application link wasn't provided. How is one suppose to reach them to apply?
- deleted 3y ago[deleted]
- seism 3y agoHi Robin Hood. I wonder about the "continue their work without the pressure to generate financial returns" part. I don't see an expectation to work on non-commercial, or socially conscious, applications, just to commit to open source (what level of commitment also being unclear). It would actually be great if support programs along these lines were being offered by governments and public service institutions, with some serious governance and transparency around the process. Surely there are also some interesting econometrics on the "GPU poverty line" - someone has literature to share?
- evanwolf 3y agoYeah, this is moneyed interests looking out for their own investments instead of government acting on behalf of the public good.
- GaggiX 3y agoThis reminds when a random chinese user finetuned the leaked NovelAI Stable Diffusion model and released it online, the Anything V3 model, tech company like Tencent started using it in their products.
- tikkun 3y agoWhat are other open source LLM projects that people rely on? As a way to come up with ideas for future grant recipients. This is a good initiative, and an excellent initial batch. A great use of management fees! I'm struggling to think of additional ones because I feel like this covers everyone I would've suggested (particularly TheBloke, vLLM and oobabooga). Edit, some ideas for future grant recipients would be: * MLC LLM * ExLlama * An open source fork of text-generation-inference * AutoGPTQ
- rajko_rad 3y agoAwesome suggestions, thank you!
- syntaxing 3y agoWhoa this is honestly awesome, a lot of tools I see on the list are ones I’ve been using so it’s great to see they’re getting financial support. Really hoping Ollama joins the list soon. I have some fun ideas I’ve been wanting to test out but probably requires a couple hours on 8X A100s + EvolveInstruct style data that cost upward of $1K+. Gonna start preparing some code to see if I can join grant #2!
- rajko_rad 3y agoLove it!! Good luck with the mvps! And thank you so much for the input - we are meeting the folks at ollama!!
- stuaxo 3y agoThis is great, but "dual 4090s" bring this stuff down to the Ryzen APU level.
- jph00 3y agoI'm one of the recipients of the AI grants, to support my work at fast.ai. I'm extremely grateful to a16z for their support. Here's some additional details based on questions I see in the comments: My grant covered the purchase of a Scalar server from Lambda Labs, which allowed me to configure a system with 8 previous-gen A6000 GPUs, partly also thanks to NVIDIA who has recently given me 3 of those cards, and Lambda Labs who offered to provide everything at cost. a16z didn't ask for any equity or any kind of quid pro quo, other than to let folks know that they provided this grant. They couldn't have been more helpful through the process - I didn't have to fill out any forms (other than sign a single one page agreement), the contract was clear and totally fair (even explicitly saying that a16z wasn't going to receive any kind of rights to anything), and they wired me the money for buying the server promptly.
- Mascobot 3y agoThanks for all the work Jeremy! There's a large community of folks that have been learning from you, the Fast.ai community and your work. Super excited for this grant!
- rajko_rad 3y agoHonored to have you in the inaugural cohort @jph00!!
- jasrys 3y agoCouldn't go to a better home. Thanks for making this field approachable.
- jdshaffer 3y agoIt makes me very happy to read about things like this -- support where it's needed and appreciated AND absolutely no strings attached. Wow. Wish there were more cases like this. Best wishes in your work, jph00!
- 0xcde4c3db 3y ago> previous-gen A6000 GPUs Just in case anyone hasn't surveyed the landscape lately, the distinction being made here is between "RTX A6000" (Ampere architecture; GeForce 30-series equivalent) and "RTX 6000 Ada Generation" (Ada architecture; GeForce 40-series equivalent) products. Nvidia is apparently convinced that it can get away with being almost as terrible at naming technology generations as the USB Implementers Forum.
- lee101 3y ago[dead]
- JonChesterfield 3y agoIs this a venture capital fund donating money to nvidia in order to support open source? It seems like some wires have got crossed along the way here