3 ms·

This is gross It feels like we’ve been in the golden age and the window is coming to a close Let the enshitification begin, I guess

by gxs 6mo ago

2ndorderthought 6mo ago
In the past month local models have been ramping up in major way meanwhile the namesake providers have upped prices, went offline randomly, and started doing slimier and slimier things. I really think the future is local compute. Or at least self hosted models.
- SchemaLoad 6mo ago
  The hosted ones still have the advantage of being able to search the internet for live info rather than being limited to a knowledge cut off date.
  - darepublic 6mo ago
    Local ones that support tool use can do the same
  - gbear605 6mo ago
    I’m not sure why a model needs to be hosted in order to make network calls?
    - hansvm 6mo ago
      Is there a library of good tools for LLMs to call? I have to imagine the bot-detection avoidance mechanisms are a major engineering effort and not likely to work out of the box with a simple harness and random local LLM.
      - ossa-ma 6mo ago
        Even the hosted ones are blocked from searching certain sites, for example Claude is banned from searching Reddit: `Error: "The following domains are not accessible to our user agent: ['reddit.com']."`
      - wyre 6mo ago
        Tavily, Exa, Firecrawl, Perplexity, and Linkup are all tools for agents to search the web. I’ve been building a harness the past few months and supports them all out of the box with an API key.
        goosejuice 6mo ago
        Kagi also has an API. People who hate ads are probably the same folk that should be paying for Kagi. That's the sane alternative world where companies respect their users.
        wyre 6mo ago
        Oh, you got me so excited. I've had a Kagi sub for 3 years, but their API is still in closed beta. I guess I could (and should reach out and ask for access).
        lukewarm707 6mo ago
        be warned though: firecrawl: "if you post content or intellectual property within the Services or give us Feedback about the Services, you hereby grant to us a worldwide, irrevocable, non-exclusive, royalty-free license to use, reproduce, modify, publish, translate and distribute any content that you submit in any form [...] You also grant to us the right to sub-license these rights" exa: "Query Data is used to improve our products and technology, including by training and fine-tuning models that power our Services" perplexity: "Perplexity may retain, copy, distribute and otherwise use Search Data for its lawful business purposes, including the improvement and development of products and services." linkup: "Client grants Linkup a worldwide right to use, reproduce and modify the Client Data, including prompts, for the purposes of providing, maintaining, developing, training" tavily: "we may use certain portions of your query data to improve our responses to future queries"..."We may share your query data with third-party search index providers (e.g., Google)"
      - gbear605 6mo ago
        If your volume is low enough, it should be pretty fine. It can just piggy back onto your personal browser cookies for Cloudflare.
  - eightysixfour 6mo ago
    You can do that locally too!
  - chrisweekly 6mo ago
    That's not how it works. Whether local or hosted, every modern model has a cutoff date for its training data, and can be leveraged by agents / harnesses / tools to fetch context from the internet or wherever.
- CSMastermind 6mo ago
  What's the rough equivalent of a local model? Are we talking GPT-4?
  - Terretta 6mo ago
    Depends on your VRAM or "unified" memory for how smart it is, and CPU/GPU for how quick it is. 128GB of RAM? Sure, the early to mid 4s releases, except maybe 4o. And on an M5 Max, about the same speed. I wouldn't really bother under 64GB (meaning 32GB or less) except for entertainment value (chats, summaries, tasky read-only agent things).
  - kay_o 6mo ago
    GLM 5.1 and DeepSeek 4 are acceptable, but the cost of hardware and energy cost that depending on your use case you may as well purchase a Tokens. They get useless and stupid rapidilty if you quant enough to run on single 16-24GB GPU style.
  - 2ndorderthought 6mo ago
    Qwen 3.6 which was released this month is a large but still smaller model. Supposedly it's at about sonnet level when configured correctly. It can be run on commodity hardware without purchasing a data center. https://www.reddit.com/r/LocalLLaMA/comments/1so1533/qwen36_this_is_it/ https://www.reddit.com/r/LocalLLaMA/comments/1so1533/qwen36_... Then there are middle size ones which require multiple gpus which are like gpts latest flagships. Then there is kimi 2.6 which is a monster that is beating opus in some benchmarks. https://www.reddit.com/r/LocalLLaMA/comments/1sr8p49/kimi_k26_is_a_legit_opus_47_replacement/ https://www.reddit.com/r/LocalLLaMA/comments/1sr8p49/kimi_k2... It's basically whatever you can afford. Any trash heap laptop can run code auto complete models locally no problem. The rest require some level of investment, an idle gaming pc, or a serious investment
rnxrx 6mo ago
The arc of the technological universe is short, but it bends toward enshitification.
dannyw 6mo ago
How do you expect the spend & COGS for free LLM inference to be funded? For users who don't want to pay, or maybe can't pay?
- infinite_spin 6mo ago
  From things like defense/private contracts e.g. colleges pay for institutional subscriptions
  - 2ndorderthought 6mo ago
    The average person doesn't benefit from defense contracts ... Like ever.
    - IX-103 6mo ago
      The average person is slightly more female than male and has 2.1 children, but they do benefit from defense contracts since it makes up a small percentage of their salary.
      - 2ndorderthought 6mo ago
        You are a fun person. We should be friends
- derektank 6mo ago
  Perhaps it’s a glib and easy thing to say, but after a teaser period, I would simply not offer free LLM inference. Agreeing to serve ads just completely re-aligns your interests away from providing the best possible user experience to something else entirely.
iammrpayments 6mo ago
It has begun ever since they nerfed chatgpt4 before releasing 4o

This is gross It feels like we’ve been in the golden age and the window is coming to a close Let the enshitification begin, I guess

by gxs 6mo ago

This is gross It feels like we’ve been in the golden age and the window is coming to a close Let the enshitification begin, I guess

2ndorderthought 6mo ago

In the past month local models have been ramping up in major way meanwhile the namesake providers have upped prices, went offline randomly, and started doing slimier and slimier things. I really think the future is local compute. Or at least self hosted models.

SchemaLoad 6mo ago
The hosted ones still have the advantage of being able to search the internet for live info rather than being limited to a knowledge cut off date.
- darepublic 6mo ago
  Local ones that support tool use can do the same
- gbear605 6mo ago
  I’m not sure why a model needs to be hosted in order to make network calls?
  - hansvm 6mo ago
    Is there a library of good tools for LLMs to call? I have to imagine the bot-detection avoidance mechanisms are a major engineering effort and not likely to work out of the box with a simple harness and random local LLM.
    - ossa-ma 6mo ago
      Even the hosted ones are blocked from searching certain sites, for example Claude is banned from searching Reddit: `Error: "The following domains are not accessible to our user agent: ['reddit.com']."`
    - wyre 6mo ago
      Tavily, Exa, Firecrawl, Perplexity, and Linkup are all tools for agents to search the web. I’ve been building a harness the past few months and supports them all out of the box with an API key.
      - goosejuice 6mo ago
        Kagi also has an API. People who hate ads are probably the same folk that should be paying for Kagi. That's the sane alternative world where companies respect their users.
        wyre 6mo ago
        Oh, you got me so excited. I've had a Kagi sub for 3 years, but their API is still in closed beta. I guess I could (and should reach out and ask for access).
      - lukewarm707 6mo ago
        be warned though: firecrawl: "if you post content or intellectual property within the Services or give us Feedback about the Services, you hereby grant to us a worldwide, irrevocable, non-exclusive, royalty-free license to use, reproduce, modify, publish, translate and distribute any content that you submit in any form [...] You also grant to us the right to sub-license these rights" exa: "Query Data is used to improve our products and technology, including by training and fine-tuning models that power our Services" perplexity: "Perplexity may retain, copy, distribute and otherwise use Search Data for its lawful business purposes, including the improvement and development of products and services." linkup: "Client grants Linkup a worldwide right to use, reproduce and modify the Client Data, including prompts, for the purposes of providing, maintaining, developing, training" tavily: "we may use certain portions of your query data to improve our responses to future queries"..."We may share your query data with third-party search index providers (e.g., Google)"
    - gbear605 6mo ago
      If your volume is low enough, it should be pretty fine. It can just piggy back onto your personal browser cookies for Cloudflare.
- eightysixfour 6mo ago
  You can do that locally too!
- chrisweekly 6mo ago
  That's not how it works. Whether local or hosted, every modern model has a cutoff date for its training data, and can be leveraged by agents / harnesses / tools to fetch context from the internet or wherever.
CSMastermind 6mo ago
What's the rough equivalent of a local model? Are we talking GPT-4?
- Terretta 6mo ago
  Depends on your VRAM or "unified" memory for how smart it is, and CPU/GPU for how quick it is. 128GB of RAM? Sure, the early to mid 4s releases, except maybe 4o. And on an M5 Max, about the same speed. I wouldn't really bother under 64GB (meaning 32GB or less) except for entertainment value (chats, summaries, tasky read-only agent things).
- kay_o 6mo ago
  GLM 5.1 and DeepSeek 4 are acceptable, but the cost of hardware and energy cost that depending on your use case you may as well purchase a Tokens. They get useless and stupid rapidilty if you quant enough to run on single 16-24GB GPU style.
- 2ndorderthought 6mo ago
  Qwen 3.6 which was released this month is a large but still smaller model. Supposedly it's at about sonnet level when configured correctly. It can be run on commodity hardware without purchasing a data center. https://www.reddit.com/r/LocalLLaMA/comments/1so1533/qwen36_this_is_it/ https://www.reddit.com/r/LocalLLaMA/comments/1so1533/qwen36_... Then there are middle size ones which require multiple gpus which are like gpts latest flagships. Then there is kimi 2.6 which is a monster that is beating opus in some benchmarks. https://www.reddit.com/r/LocalLLaMA/comments/1sr8p49/kimi_k26_is_a_legit_opus_47_replacement/ https://www.reddit.com/r/LocalLLaMA/comments/1sr8p49/kimi_k2... It's basically whatever you can afford. Any trash heap laptop can run code auto complete models locally no problem. The rest require some level of investment, an idle gaming pc, or a serious investment

rnxrx 6mo ago

The arc of the technological universe is short, but it bends toward enshitification.

dannyw 6mo ago

How do you expect the spend & COGS for free LLM inference to be funded? For users who don't want to pay, or maybe can't pay?

infinite_spin 6mo ago
From things like defense/private contracts e.g. colleges pay for institutional subscriptions
- 2ndorderthought 6mo ago
  The average person doesn't benefit from defense contracts ... Like ever.
  - IX-103 6mo ago
    The average person is slightly more female than male and has 2.1 children, but they do benefit from defense contracts since it makes up a small percentage of their salary.
    - 2ndorderthought 6mo ago
      You are a fun person. We should be friends
derektank 6mo ago
Perhaps it’s a glib and easy thing to say, but after a teaser period, I would simply not offer free LLM inference. Agreeing to serve ads just completely re-aligns your interests away from providing the best possible user experience to something else entirely.

iammrpayments 6mo ago

It has begun ever since they nerfed chatgpt4 before releasing 4o