5 ms·

What I learned from looking at 900 most popular open source AI tools

413 points by swyx 3y ago

destroyer9000 3y ago
[flagged]
swyx 3y ago
author highlight tweet thread: https://twitter.com/chipro/status/1768388213008445837 https://twitter.com/chipro/status/1768388213008445837
gerdesj 3y ago
This section invites some discussion (says British bloke): "https://huyenchip.com/2024/03/14/ai-oss.html#the_growing_china_open_source_ecosystem https://huyenchip.com/2024/03/14/ai-oss.html#the_growing_chi..."
- swyx 3y ago
  gotta be careful with china "open source". a lot of them look like this https://github.com/dreamoving/dreamoving-project https://github.com/dreamoving/dreamoving-project which is a very nice Apache 2.0 readme file
brianjking 3y ago
This is a fantastic read.
- swyx 3y ago
  elaborate? wondering what stood out to you/what you wanted more detail on. (am OP but not the author). i always feel like looking at github trends is kinda cool but fail to get deeper insight i can use to inform my work.. its more of a trailing indicator right?
  - beauzero 3y ago
    This saved me probably 3 months work digging up a good summary of the current state of specifically LLMs. This is a highly dense, easily consumable summary for someone like me. Even if it's only 80% encompassing super valuable. Thank you. Context: Business apps. Bleeding edge curious and supportable day-to-day pragmatic developer. 25+ years web stack, 10% client desktop apps and small business sysadmin. Spending 6+ hours a day currently retooling concepts and where they can fit into day-to-day business stacks. "Retired" from SMB market after spending 25 years there and moved to government hoping for another 20-30. I love what I do.
chiphuyen 3y ago
Hi, I'm Chip, the author of the post. I spent waaaay too much time doing research for this. The AI engineering layer was especially hard because so many tools have similar and/or overlapping features. It was also a lot of pain trying to understand the repos that only have Chinese in the README do. The full list of the repos is published here: https://huyenchip.com/llama-police https://huyenchip.com/llama-police
- swyx 3y ago
  great list! as someone who's also trying to map the ai engineering landscape... i wonder what u think of adding other parts of the AI stack (https://www.latent.space/p/dec-2023 https://www.latent.space/p/dec-2023). right now you have 4 categories and those are all in the text/code-heavy RAG/Agent world, but i think the space has broadened out a bit as i see it. for example, you could add: - finetuning/other post-pretrain model tools (axolotl, mergekit <- all made and used by people without traditional ML engineer/researcher background) - multimodal models/frameworks like vocode and comfyui - AI UX tools like vercel ai sdk - synthetic data generation tooling? whatever the nous pple have made open question whether inference frameworks like llama.cpp/ollama or vllm and tgi count as AI Eng tools? again given the background of ggeranov and the students behind the other projects, arguably yes but ofc it starts to bleed into classical mlops here. (update: i see u have them in the "model development" category, ok fair)
  - chiphuyen 3y ago
    IMO, the classical mlops is closer to the genai stack than most people think. E.g. experiment tracking is the same: with classical mlops, you experiment with hyperparams, with genai, you experiment with prompts. Similarly, finetuning is just an extension of training. Even vector databases for RAG is just vector search + databases, both of which have been around forever. The post-train world is what I find to be the most fun. Techniques like model merging, constrained sampling, and all the new creative techniques for inference optimization and faster decoding are super cool!
- braza 3y ago
  Hey Chip, thanks for your contributions along the years and the amazing book. 2 questions: From what you researched, how many of those solutions are ready for production? and Regarding this mortality, what are the let's say top 5 things someone needs to think even before to do a PoC over those tools?
nixlim 3y ago
Thank you for this. A good read and useful information.
neom 3y ago
Chips book is really good FWIW: Designing Machine Learning System - https://www.oreilly.com/library/view/designing-machine-learning/9781098107956/ https://www.oreilly.com/library/view/designing-machine-learn...
hintymad 3y ago
> So many cool ideas are being developed by the community. Here are some of my favorites. > Batch inference optimization: FlexGen, llama.cpp > Faster decoder with techniques such as Medusa, LookaheadDecoding > Model merging: mergekit > Constrained sampling: outlines, guidance, SGLang So essentially a handful of people are doing God's work. They have deep knowledge on modeling and optimization, and they build amazing libraries for millions of mortals. On the other hand, it'll be hard for an engineer to work on training frameworks or building models with new knowledge or new capabilities (except some small-scale finetuning) or optimization in general -- the hardware cost for doing such work is prohibitive to such engineers.
- pama 3y ago
  Isn’t that the same with most engineering disciplines though? A nuclear engineer cannot build a nuclear power station at home, a chemical engineer or process chemist doesn’t have access to the industrial grade infrastructure outside of their job, a computer hardware architect cannot hope to design hardware at home and fabricate it at 3nm or better at TSMC. I guess that software engineering was more of an exception to this rule for a while because home computers were amazing enough to help build useful software. Even throughout all these years many people worked on parallel code that run on large clusters or infrastructure that was not appropriate for operating at home and now with deep learning a subset of that skill set is very desirable. I agree that additional public contributions to the training process on large clusters would be fantastic; eventually these people will be trained in all the right systems courses and will figure out their way to the jobs where they can apply their skills and grow.
  - hintymad 3y ago
    But not necessarily true for software engineers. The so-called three romances of CS can all be attempted by anyone: compiler, operating systems, and computer graphics. And in the recent years, databases, distributed systems, and machine learning algorithms can also be attempted by anyone at home. Only the large models and the associated optimizations are really beyond most people's reach.
userbinator 3y ago
I find it quite astounding that there are already over 900 open source AI tools (and from the article, it doesn't sound like these are all mainly clones/forks of each other.)
Der_Einzige 3y ago
Wish that Automatic1111/Ooobabooga/Comfy would be specifically talked about since they are such unique examples of popular open source AI tools.
- porkbeer 3y ago
  Well, talk about them! How are they unique?
  - washadjeffmad 3y ago
    Well, for one, their popularity eclipses the examples given: AUTOMATIC1111/stable-diffusion-webui - 126K stars oobabooga/text-generation-webui - 34.4K stars comfyanonymous/ComfyUI - 28.1K stars Some people want to avoid drawing attention to chan sites, where a lot of those projects' developers are active, but the omissions are still glaring (a bit like leaving John Prine out of the CMA). These were the first and most famous projects in inference and generative AI, so they tend to be more advanced frameworks, making them less accessible as introductory tools to low-depth end users, which the author doesn't seem to be?
    - diggan 3y ago
      > Well, for one, their popularity eclipses the examples given: That's like the least interesting comparison, especially if we want to talk about "unique examples". I'd say out of those, ComfyUI is probably the most interesting one, as it's naturally extensible and has a node-UI so you can basically reorganize the image generation pipeline and your workflow to however you want.
visitor4712 3y ago
Thank you for this outstanding work!
zone411 3y ago
Einops and safetensors are not niche! They are just more technical and well-known by people who do more than GPT wrappers starting in 2022 ;)
timrogers 3y ago
Super helpful article - it’s so great to have a zoomed out view of the space. One question that stands out to me is where evaluation at the application level should be its own category, rather than folded in to bigger groups.
sevagh 3y ago
Something feels off about sticking Einops (a great utility library for tensor reshaping, been around for years) as a footnote in a "LLM 2023-2024 hype" list - it's an anachronistic inclusion. Feels like the author creates a narrative that starts with LLMs and acts like everything else is a building block for LLMs.
grbsh 3y ago
The graph of cumulative repos over time is really interesting — it looks like we may be approaching the end of the S-curve for AI hype. I wonder if the graph will continue to flatten, or if there will be a much higher, longer, slower, and more enduring S curve. I imagine a similar double S curve pattern occurred after 2001 in web. Anyone have ideas for how I could measure this for 1995-2015 web?
- samstave 3y ago
  not very granular, but looking up, if available Alexa rankings (and whatever the other competitors were) for traffic... --- Sure, here's what I found: *Physical Size of the Internet (1995-2000)*: - In 1995, the Internet had a worldwide user base of less than 40 million⁸. - By 2000, there were 361 million users worldwide⁸. - In terms of websites, there were 9,950,491 websites in 2000⁹. *Internet Bandwidth (1995-2000)*: - The average internet access speed in 1995 was 24 kbps². - By 2000, the average internet access speed had increased to 1,116 kbps². - In terms of telecommunications capacity, it was 2.2 optimally compressed exabytes in 2000⁶. *Fastest Internet Links (1995-2000)*: - In the early 1990s, the fastest available modem was capable of transferring data at a maximum speed of 14.4 kilobytes per second (kbps)³. - By the late 1990s, broadband had emerged, offering a maximum theoretical data transfer speed of 512k per second³, which was over nine times as fast as a 56k modem.
MattyRad 3y ago
You can actively see a fresh "hype curve" in the transformer-debugger repo that was posted a couple days ago (https://github.com/openai/transformer-debugger https://github.com/openai/transformer-debugger) (star history https://star-history.com/#openai/transformer-debugger&Date https://star-history.com/#openai/transformer-debugger&Date). At the time I saw the repo link posted on HN, it had 1.6k stars/16 hours. What channel/platform are people subscribed to to star it so quickly? Discord? I'm not implying any nefariousness, mind you, I'm only wondering where all the stargazers were referred from so fast and in such volume.
- Diris 3y ago
  Personally, I saw it on the tweet from Jan Leike. https://x.com/janleike/status/1767347608065106387?s=20 https://x.com/janleike/status/1767347608065106387?s=20

413 points by swyx 3y ago

[flagged]

swyx 3y ago

author highlight tweet thread: https://twitter.com/chipro/status/1768388213008445837 https://twitter.com/chipro/status/1768388213008445837

gerdesj 3y ago

This section invites some discussion (says British bloke): "https://huyenchip.com/2024/03/14/ai-oss.html#the_growing_china_open_source_ecosystem https://huyenchip.com/2024/03/14/ai-oss.html#the_growing_chi..."

swyx 3y ago
gotta be careful with china "open source". a lot of them look like this https://github.com/dreamoving/dreamoving-project https://github.com/dreamoving/dreamoving-project which is a very nice Apache 2.0 readme file

brianjking 3y ago

This is a fantastic read.

swyx 3y ago
elaborate? wondering what stood out to you/what you wanted more detail on. (am OP but not the author). i always feel like looking at github trends is kinda cool but fail to get deeper insight i can use to inform my work.. its more of a trailing indicator right?
- beauzero 3y ago
  This saved me probably 3 months work digging up a good summary of the current state of specifically LLMs. This is a highly dense, easily consumable summary for someone like me. Even if it's only 80% encompassing super valuable. Thank you. Context: Business apps. Bleeding edge curious and supportable day-to-day pragmatic developer. 25+ years web stack, 10% client desktop apps and small business sysadmin. Spending 6+ hours a day currently retooling concepts and where they can fit into day-to-day business stacks. "Retired" from SMB market after spending 25 years there and moved to government hoping for another 20-30. I love what I do.

chiphuyen 3y ago

Hi, I'm Chip, the author of the post. I spent waaaay too much time doing research for this. The AI engineering layer was especially hard because so many tools have similar and/or overlapping features. It was also a lot of pain trying to understand the repos that only have Chinese in the README do. The full list of the repos is published here: https://huyenchip.com/llama-police https://huyenchip.com/llama-police

swyx 3y ago
great list! as someone who's also trying to map the ai engineering landscape... i wonder what u think of adding other parts of the AI stack (https://www.latent.space/p/dec-2023 https://www.latent.space/p/dec-2023). right now you have 4 categories and those are all in the text/code-heavy RAG/Agent world, but i think the space has broadened out a bit as i see it. for example, you could add: - finetuning/other post-pretrain model tools (axolotl, mergekit <- all made and used by people without traditional ML engineer/researcher background) - multimodal models/frameworks like vocode and comfyui - AI UX tools like vercel ai sdk - synthetic data generation tooling? whatever the nous pple have made open question whether inference frameworks like llama.cpp/ollama or vllm and tgi count as AI Eng tools? again given the background of ggeranov and the students behind the other projects, arguably yes but ofc it starts to bleed into classical mlops here. (update: i see u have them in the "model development" category, ok fair)
- chiphuyen 3y ago
  IMO, the classical mlops is closer to the genai stack than most people think. E.g. experiment tracking is the same: with classical mlops, you experiment with hyperparams, with genai, you experiment with prompts. Similarly, finetuning is just an extension of training. Even vector databases for RAG is just vector search + databases, both of which have been around forever. The post-train world is what I find to be the most fun. Techniques like model merging, constrained sampling, and all the new creative techniques for inference optimization and faster decoding are super cool!
braza 3y ago
Hey Chip, thanks for your contributions along the years and the amazing book. 2 questions: From what you researched, how many of those solutions are ready for production? and Regarding this mortality, what are the let's say top 5 things someone needs to think even before to do a PoC over those tools?

nixlim 3y ago

Thank you for this. A good read and useful information.

neom 3y ago

Chips book is really good FWIW: Designing Machine Learning System - https://www.oreilly.com/library/view/designing-machine-learning/9781098107956/ https://www.oreilly.com/library/view/designing-machine-learn...

hintymad 3y ago

> So many cool ideas are being developed by the community. Here are some of my favorites. > Batch inference optimization: FlexGen, llama.cpp > Faster decoder with techniques such as Medusa, LookaheadDecoding > Model merging: mergekit > Constrained sampling: outlines, guidance, SGLang So essentially a handful of people are doing God's work. They have deep knowledge on modeling and optimization, and they build amazing libraries for millions of mortals. On the other hand, it'll be hard for an engineer to work on training frameworks or building models with new knowledge or new capabilities (except some small-scale finetuning) or optimization in general -- the hardware cost for doing such work is prohibitive to such engineers.

pama 3y ago
Isn’t that the same with most engineering disciplines though? A nuclear engineer cannot build a nuclear power station at home, a chemical engineer or process chemist doesn’t have access to the industrial grade infrastructure outside of their job, a computer hardware architect cannot hope to design hardware at home and fabricate it at 3nm or better at TSMC. I guess that software engineering was more of an exception to this rule for a while because home computers were amazing enough to help build useful software. Even throughout all these years many people worked on parallel code that run on large clusters or infrastructure that was not appropriate for operating at home and now with deep learning a subset of that skill set is very desirable. I agree that additional public contributions to the training process on large clusters would be fantastic; eventually these people will be trained in all the right systems courses and will figure out their way to the jobs where they can apply their skills and grow.
- hintymad 3y ago
  But not necessarily true for software engineers. The so-called three romances of CS can all be attempted by anyone: compiler, operating systems, and computer graphics. And in the recent years, databases, distributed systems, and machine learning algorithms can also be attempted by anyone at home. Only the large models and the associated optimizations are really beyond most people's reach.

userbinator 3y ago

I find it quite astounding that there are already over 900 open source AI tools (and from the article, it doesn't sound like these are all mainly clones/forks of each other.)

Der_Einzige 3y ago

Wish that Automatic1111/Ooobabooga/Comfy would be specifically talked about since they are such unique examples of popular open source AI tools.

porkbeer 3y ago
Well, talk about them! How are they unique?
- washadjeffmad 3y ago
  Well, for one, their popularity eclipses the examples given: AUTOMATIC1111/stable-diffusion-webui - 126K stars oobabooga/text-generation-webui - 34.4K stars comfyanonymous/ComfyUI - 28.1K stars Some people want to avoid drawing attention to chan sites, where a lot of those projects' developers are active, but the omissions are still glaring (a bit like leaving John Prine out of the CMA). These were the first and most famous projects in inference and generative AI, so they tend to be more advanced frameworks, making them less accessible as introductory tools to low-depth end users, which the author doesn't seem to be?
  - diggan 3y ago
    > Well, for one, their popularity eclipses the examples given: That's like the least interesting comparison, especially if we want to talk about "unique examples". I'd say out of those, ComfyUI is probably the most interesting one, as it's naturally extensible and has a node-UI so you can basically reorganize the image generation pipeline and your workflow to however you want.

visitor4712 3y ago

Thank you for this outstanding work!

zone411 3y ago

Einops and safetensors are not niche! They are just more technical and well-known by people who do more than GPT wrappers starting in 2022 ;)

timrogers 3y ago

Super helpful article - it’s so great to have a zoomed out view of the space. One question that stands out to me is where evaluation at the application level should be its own category, rather than folded in to bigger groups.

sevagh 3y ago

Something feels off about sticking Einops (a great utility library for tensor reshaping, been around for years) as a footnote in a "LLM 2023-2024 hype" list - it's an anachronistic inclusion. Feels like the author creates a narrative that starts with LLMs and acts like everything else is a building block for LLMs.

grbsh 3y ago

The graph of cumulative repos over time is really interesting — it looks like we may be approaching the end of the S-curve for AI hype. I wonder if the graph will continue to flatten, or if there will be a much higher, longer, slower, and more enduring S curve. I imagine a similar double S curve pattern occurred after 2001 in web. Anyone have ideas for how I could measure this for 1995-2015 web?

samstave 3y ago
not very granular, but looking up, if available Alexa rankings (and whatever the other competitors were) for traffic... --- Sure, here's what I found: *Physical Size of the Internet (1995-2000)*: - In 1995, the Internet had a worldwide user base of less than 40 million⁸. - By 2000, there were 361 million users worldwide⁸. - In terms of websites, there were 9,950,491 websites in 2000⁹. *Internet Bandwidth (1995-2000)*: - The average internet access speed in 1995 was 24 kbps². - By 2000, the average internet access speed had increased to 1,116 kbps². - In terms of telecommunications capacity, it was 2.2 optimally compressed exabytes in 2000⁶. *Fastest Internet Links (1995-2000)*: - In the early 1990s, the fastest available modem was capable of transferring data at a maximum speed of 14.4 kilobytes per second (kbps)³. - By the late 1990s, broadband had emerged, offering a maximum theoretical data transfer speed of 512k per second³, which was over nine times as fast as a 56k modem.

MattyRad 3y ago

You can actively see a fresh "hype curve" in the transformer-debugger repo that was posted a couple days ago (https://github.com/openai/transformer-debugger https://github.com/openai/transformer-debugger) (star history https://star-history.com/#openai/transformer-debugger&Date https://star-history.com/#openai/transformer-debugger&Date). At the time I saw the repo link posted on HN, it had 1.6k stars/16 hours. What channel/platform are people subscribed to to star it so quickly? Discord? I'm not implying any nefariousness, mind you, I'm only wondering where all the stargazers were referred from so fast and in such volume.

Diris 3y ago
Personally, I saw it on the tweet from Jan Leike. https://x.com/janleike/status/1767347608065106387?s=20 https://x.com/janleike/status/1767347608065106387?s=20