4 ms·

theres always a lot of openTHING clones of THING after THING is announced. they all usually (not always[1]!) disappoint/dont get traction. i think the causes ar

by swyx 2y ago

theres always a lot of openTHING clones of THING after THING is announced. they all usually (not always[1]!) disappoint/dont get traction. i think the causes are 1. running things in production/self hosting is more annoying than just paying like 20-200/month 2. openTHING makers often overhype their superficial repros ("I cloned Perplexity in a weekend! haha! these VCs are clowns!") and trivializing the last mile, most particularly in this case... 3. long horizon planning trained with RL in a tight loop that is not available in the open (yes, even with deepseek). the thing that makes OAI work as a product+research company is that products are never launched without first establishing a "prompted baseline" and then finetuning the model from there (we covered this process in https://latent.space/p/karina https://latent.space/p/karina recently) - which becomes an evals/dataset suite that eventually gets merged in once performance impacts stabilize 4. that said, smolagents and HF are awesome and I like that they are always this on the ball. how does this make money for HF? --- [1]: i think opendevin/allhands is a pretty decent competitor to devin now

tshepom 2y ago
Maybe these open projects start to get more attention when we have a distribution system/App Store for AI projects. I know YC is looking to fund this https://www.ycombinator.com/rfs https://www.ycombinator.com/rfs
- transpute 2y ago
  Is "AI appstore" envisioned on Linux edge inference hardware, e.g. PC+NPU, PC+GPU, Nvidia Project Digits? Or only in the cloud? Apple probably wouldn't accept a 3rd-party AI app store on MacOS and iOS, except possibly in the EU. If antitrust regulation leads to Android becoming a standalone company, that could support AI competition.
- deleted 2y ago
  [deleted]
deleted 2y ago
[deleted]
littlestymaar 2y ago
Except that in this particular case (like in many others as far as AI goes, actually), the open version came before, by three full months: https://www.reddit.com/r/LocalLLaMA/comments/1gvlzug/i_created_an_ai_research_assistant_that_actually/ https://www.reddit.com/r/LocalLLaMA/comments/1gvlzug/i_creat... OpenAI pretty much never acknowledge prior art in their marketing material because they want you to believe they are the true innovators, but you should not take their marketing claims for granted.
- transpute 2y ago
  Thanks for the pointer, https://github.com/TheBlewish/Automated-AI-Web-Researcher-Ollama https://github.com/TheBlewish/Automated-AI-Web-Researcher-Ol...
janalsncm 2y ago
> RL in a tight loop that is not available in the open Completely agree that a real RL pipeline is needed here, not just some clever prompting in a loop. That being said, it wouldn’t be impossible to create a “gym” for this task. You are essentially creating a simulated internet. And hiding a needle is a lot easier than finding it.
- pixl97 2y ago
  I think models will have to some kind of internal training to teach them they are agents that can come back and work on things. Working on complex problems tends to explode in to a web of things that needs done. You need to be able to separate these in to subtasks and work on them semi-independently. In addition when a subtask gets stuck in a loop, you need to work on another task or line of thought, and then come back and 're-run' your thinking to see if anything changed.
  - janalsncm 2y ago
    The idea of reinforcement learning is that for some things it is hard to give an explicit plan for how to do something. For example, many games. Recently, DeepSeek showed that it worked for certain reasoning problems too, like leetcode problems. Instead, RL just rewards the model when it accomplishes some measurable goal (like winning the game). This works for certain types of problems but it’s pretty inefficient because the model wastes a lot of time doing stuff that doesn’t work.
shekhargulati 2y ago
> running things in production/self hosting is more annoying than just paying like 20-200/month This is an important point. As people rely more on AI/LLM tools, reliability will become even more critical. In the last two weeks, I've heavily used Claude and DeepSeek Chat. ChatGPT is much more reliable compared to both. Claude struggles with long-context chats and often shifts to concise responses. DeepSeek often has its "fail whale" moment.
- GTP 2y ago
  > In the last two weeks, I've heavily used Claude and DeepSeek Chat. ChatGPT is much more reliable compared to both Which reliability problems did you face? I heard about connection issues due to too much traffic with Deepseek, but those would go away if you self-host the model.
  - edude03 2y ago
    Obviously the reliability problems would go away if you self-host but the "point" is most people rely on external providers because they can't locally run models of similar quality. So what do you do if deepseek cuts you off? For most getting 12? H100s (for the 671b) is essentially impossible.
    - creamyhorror 2y ago
      You use a Deepseek model not hosted by Deepseek, but another provider (e.g. DeepInfra currently). Hopefully a robust provider market will emerge and thrive, even if open models start thinning out.
kolinko 2y ago
As for them not getting traction - many projects just don’t advertise which tech they use. Also, having an open source alternative - even if slightly worse - gives projects an alternative if OpenAI decides to cut them off or sth. What I tell startups is to start off with whatever OpenAI/Anthropic have to offer, if they can, and consider switching into Open Source only if it allows for something they can’t do (often in gen graphics), or when they reach product market fit and they can finetune/train a smaller model that handles their specific case better and cheaper.
- rfoo 2y ago
  > if it allows for something they can’t do (often in gen graphics) What makes gen graphics stand out?
  - alumic 2y ago
    I’d venture to say it’s the range: What style? Is it a texture? If so, I’ll need a model that can generate a large tiled image. Is it a logo? What kind? Is it badged, vintage, corporate, etc
  - edude03 2y ago
    Porn. When I first looked into stable diffusion I wanted to see what people were making with it and there was a few sites that showed stuff people had generated and it was 70% porn 29% high fantasy wallpapers (numbers are illustrative). Recently I've been looking at different text generation inference platforms like tgi and vllm on reddit, and 1 in 5 posts in the localllm subreddits are "What's the best model for erotic role play?"
    - rfoo 2y ago
      Definitely. My question was, porn (or other censored thing) could be in text, too. I don't understand why those who want censored content are primarily interested in graphics.

by swyx 2y ago

tshepom 2y ago

Maybe these open projects start to get more attention when we have a distribution system/App Store for AI projects. I know YC is looking to fund this https://www.ycombinator.com/rfs https://www.ycombinator.com/rfs

transpute 2y ago
Is "AI appstore" envisioned on Linux edge inference hardware, e.g. PC+NPU, PC+GPU, Nvidia Project Digits? Or only in the cloud? Apple probably wouldn't accept a 3rd-party AI app store on MacOS and iOS, except possibly in the EU. If antitrust regulation leads to Android becoming a standalone company, that could support AI competition.
deleted 2y ago
[deleted]

deleted 2y ago

[deleted]

littlestymaar 2y ago

Except that in this particular case (like in many others as far as AI goes, actually), the open version came before, by three full months: https://www.reddit.com/r/LocalLLaMA/comments/1gvlzug/i_created_an_ai_research_assistant_that_actually/ https://www.reddit.com/r/LocalLLaMA/comments/1gvlzug/i_creat... OpenAI pretty much never acknowledge prior art in their marketing material because they want you to believe they are the true innovators, but you should not take their marketing claims for granted.

transpute 2y ago
Thanks for the pointer, https://github.com/TheBlewish/Automated-AI-Web-Researcher-Ollama https://github.com/TheBlewish/Automated-AI-Web-Researcher-Ol...

janalsncm 2y ago

> RL in a tight loop that is not available in the open Completely agree that a real RL pipeline is needed here, not just some clever prompting in a loop. That being said, it wouldn’t be impossible to create a “gym” for this task. You are essentially creating a simulated internet. And hiding a needle is a lot easier than finding it.

pixl97 2y ago
I think models will have to some kind of internal training to teach them they are agents that can come back and work on things. Working on complex problems tends to explode in to a web of things that needs done. You need to be able to separate these in to subtasks and work on them semi-independently. In addition when a subtask gets stuck in a loop, you need to work on another task or line of thought, and then come back and 're-run' your thinking to see if anything changed.
- janalsncm 2y ago
  The idea of reinforcement learning is that for some things it is hard to give an explicit plan for how to do something. For example, many games. Recently, DeepSeek showed that it worked for certain reasoning problems too, like leetcode problems. Instead, RL just rewards the model when it accomplishes some measurable goal (like winning the game). This works for certain types of problems but it’s pretty inefficient because the model wastes a lot of time doing stuff that doesn’t work.

shekhargulati 2y ago

> running things in production/self hosting is more annoying than just paying like 20-200/month This is an important point. As people rely more on AI/LLM tools, reliability will become even more critical. In the last two weeks, I've heavily used Claude and DeepSeek Chat. ChatGPT is much more reliable compared to both. Claude struggles with long-context chats and often shifts to concise responses. DeepSeek often has its "fail whale" moment.

GTP 2y ago
> In the last two weeks, I've heavily used Claude and DeepSeek Chat. ChatGPT is much more reliable compared to both Which reliability problems did you face? I heard about connection issues due to too much traffic with Deepseek, but those would go away if you self-host the model.
- edude03 2y ago
  Obviously the reliability problems would go away if you self-host but the "point" is most people rely on external providers because they can't locally run models of similar quality. So what do you do if deepseek cuts you off? For most getting 12? H100s (for the 671b) is essentially impossible.
  - creamyhorror 2y ago
    You use a Deepseek model not hosted by Deepseek, but another provider (e.g. DeepInfra currently). Hopefully a robust provider market will emerge and thrive, even if open models start thinning out.

kolinko 2y ago

As for them not getting traction - many projects just don’t advertise which tech they use. Also, having an open source alternative - even if slightly worse - gives projects an alternative if OpenAI decides to cut them off or sth. What I tell startups is to start off with whatever OpenAI/Anthropic have to offer, if they can, and consider switching into Open Source only if it allows for something they can’t do (often in gen graphics), or when they reach product market fit and they can finetune/train a smaller model that handles their specific case better and cheaper.

rfoo 2y ago
> if it allows for something they can’t do (often in gen graphics) What makes gen graphics stand out?
- alumic 2y ago
  I’d venture to say it’s the range: What style? Is it a texture? If so, I’ll need a model that can generate a large tiled image. Is it a logo? What kind? Is it badged, vintage, corporate, etc
- edude03 2y ago
  Porn. When I first looked into stable diffusion I wanted to see what people were making with it and there was a few sites that showed stuff people had generated and it was 70% porn 29% high fantasy wallpapers (numbers are illustrative). Recently I've been looking at different text generation inference platforms like tgi and vllm on reddit, and 1 in 5 posts in the localllm subreddits are "What's the best model for erotic role play?"
  - rfoo 2y ago
    Definitely. My question was, porn (or other censored thing) could be in text, too. I don't understand why those who want censored content are primarily interested in graphics.