5 ms·

Notes on gotchas while migrating 35kb preprompts from Opus to self-hosted Ollama

SyneRyder 21d ago
TLDR: Local models have a smaller context window, so your 35kB prompts that worked fine against a hosted 1 Million token window, crash out when you only have a 65K (!) token window locally. I dislike being negative, but I was really hoping for more substance when reading this. It would have been an interesting topic.
- 0o_MrPatrick_o0 21d ago
  Thanks for the feedback. I wanted to get into more detail, but I spent the whole weekend working these problems and then constructing this post. Dario’s behavior this weekend made me feel like this just needed to get out quick. In the future, I’ll be sharing more details about some other things in the process and some ways I found to use automation to accelerate splitting prompts for use on local inference.
  - monegator 21d ago
    Yes, please!
  - SyneRyder 21d ago
    Understood, and I realized you'd posted this to HN yourself, so I felt a bit bad making the comment. I think maybe for me, this might have worked better if the motivation had been one separate post, and the details of the gotchas as a post of its own. But I also think, if you're going to be limited to 65k token windows, you're going to have a really difficult time. Even 250k windows were cramped for me when that's all we had on Anthropic models. I just don't think a 65k window is going to be big enough for proper cyberdefence work, even if I totally agree with going local wherever you can. It feels like if you're defending against swarms of 1-10M context windows, you need to get as close as you can to similar. I've had to reach for Chinese 1m models instead because the American models just refuse me here in Australia.
  - vardalab 21d ago
    Well, if you are serious about it and you have Strix Halo, there are better ways of getting more context and capability and speed. Lookup halogen for Strix The most cost-effective local option right now, I think, is dual R9700. You can run a 27B dense Qwen at FP8 around with a full context and 2-3 concurrent sessions of 260K context. If you go down to an MXFP4, you get 4 to 5 concurrent sessions. And speed is on par with anything you'll get from hosted providers. You're getting between 60-80 for FP8 and 150+ tokens per second speed for MXFP4 and pp is 4K+. Lookup vllm radiance There is also a lot of progress in running Qwen 3.8 next flash with dual R9700. Obviously one gets less context and speed is a little bit less, but it's still very acceptable. Better than what you're getting with Llama on Strix Halo, that's for sure. These are small dense models,meaning Qwen, have gotten capable and fast. And there's been a lot of progress in the area. So sticking with Llama you are not taking advantage of the hardware you have. And yeah, for Strix Halo, you should just look into halogen and you shouldn't be just using 96 gigs for the VRAM. You should give it most of the VRAM to the inference and connect to it from your laptop or something. People ar egetting 1000+ pp with qwen 3.8 Next Flash
    - 0o_MrPatrick_o0 21d ago
      Thank you. This is the hackernews I miss.
  - hermitShell 21d ago
    Your hardware can do way more than 64k tokens context window, can't it? And with Ollama it's very easy, superficially you just drag the slider. I'm now reading "Friends Don't Let Friends Use Ollama" linked in another comment so a lot of problems with that approach are surfacing for me right now. So yeah. Along with others, I think you should come up with some empirical means of understanding if your preprompt is doing anything good since I doubt that it's all necessary and helpful. Second maybe you and I need to fix our runtimes.
    - 0o_MrPatrick_o0 21d ago
      Yes. I have to observe: Seeing the raw CoT is the best signal for evaluating prompt efficiency. The fact that frontier providers don’t share this feels criminal after this last weekend. They’ve played us all for fools. Thanks for the token setting observation. I’m running ollama headless and from the command line. I didn’t even know there was a web ui. Conf file is pretty straightforward though. I settled on 64k tokens because of a recommendation from an LLM. I now see that that guidance is associated with similar sounding (but not the same) configuration constraints. So this weekend I’ll try turning it to 11 (250k tokens?). Anticipating some hard locks over here.
      - deleted 21d ago
        [deleted]
      - jurgenburgen 20d ago
        CoT is performative and doesn’t reveal how reasoning happens. If you look at those traces locally it’s just gibberish, especially if the model falls into a loop. https://arxiv.org/abs/2605.11746 https://arxiv.org/abs/2605.11746
- zozbot234 21d ago
  The new DeepSeek models address this issue very cleanly. DeepSeek Flash V4.1 requires less than 1 GB memory for a full 1M context, down from about ~10 GB in DeepSeek Flash V4.0. This is a significant step towards making near-frontier models usable even on low-end consumer hardware, though of course with significant tradeoffs in overall performance.
dell2024 21d ago
I had hoped to get some new information out of this topic, but unfortunately found the same local "dead-ends" that I explored myself. It unfortunately feels like we will be stuck waiting for a burst bubble before local hardware can be reasonably acquired for personal LLM usage.
- 0o_MrPatrick_o0 21d ago
  FWIW, I’m not dead ended yet. Using frontier providers to manage prompt tuning experiments has been very exciting. Might be something there.
DiabloD3 21d ago
The article doesn't really describe the problem: if your prompt is 35kb, your prompt is confusing, unfocused, and doesn't work right on any LLM, and is needlessly bloating your context. At this point in time, due to how most people and companies run their inference engine, regardless of the model (yes, this includes the newest from OpenAI and Anthropic and the Chinese Tigers and Dragons), you run out of useful context that the model can accurately attend to around the 250k mark no matter how much they advertise their context size is. You need to cut your prompt up. If you believe LLMs work, have the LLM help you shape the overall plan, and then have multiple sessions run each step in the plan without being bloated with the context of previous successful steps. I don't see LLMs being production-ready until the context rot and sampling problem is fixed forever. This has not occurred, and the big inference providers aren't even bothering to integrate any of the research on that subject. If anything, many of the bigger companies are actively making inference quality worse just to extend their runway a tiny bit farther before they go bankrupt. The only thing the article gets right is this: if you're serious about LLMs, abandon Big AI and infer locally only. This is the only way you have control over the quality of the output.
- andai 21d ago
  The Claude Code system prompt was >50KB, though I think they trimmed it down heavily recently. (The newer models don't need as much hand-holding.)
  - embedding-shape 21d ago
    Yeah, strong evidence for what parent says is correct. Been my experience as well, especially with local (smaller) models but also SOTA. The less instructions you have, the better they get at following them. Conflicting instructions is like poison, and it's harder to find those conflicting parts the longer the prompt is too.
    - cyanydeez 21d ago
      the longer the context grows, the greater the probability that it generates ambiguity, and the probability that it makes mistakes approaches 1. A system prompt should be looked at like a starting point and a direction, but not a giant atlas map of everything it may be asked to do. I've been playing around with Qwen3.8-Next-Flash that has great logic, recall, roleplaying, etc. From what I can tell, it's definitely on part with the SOTA a few months ago, and my opinion, it likely is around the pinnacle of advancement without high inefficiency in preparing with the current LLM recipes. Further, I'm of the opinion America's SOTA is hitting the real cost-sigmoid and there's no singularity in site. These things will hack the planet if you put them in a group of agents and tell them to do it; but as context grows, the probability that they can answer "how many r's are in strawberry" goes down. No amount of parameters it going to erradicate that. But I digress, my new stage of working with LLMs is figuring out how to use Qwen3.6-35B-A3B as the entry point to collect the context, and then use the big boys Qwen3.8 to make the edits, then degrade back and forth to complete changes. There's no harness yet for this, but there's clearly an intelligent way to setup a engineering harness. And I do understand people have codebases that simply can't easily live in the smaller (~128k-256k) context windows, but instead of porting your codebase to another novel language, breaking it into context-aware components would make it closer to what these things can do. And I'll repeat: I don't think we're approaching the singularity of self-recursion primarily because the LLMs will duplicate errors, context poison, and whatever else they encounter and there's no human who can sit around correcting it constantly. The American AI apparatus should cut their models like the Chinese down and work on real problems and stop imbibing the singularity-watts-are-all-we-need drugs.
andai 21d ago
> Everyone who begins learning exploitation hits a phase of exploitability grief about 3 month into dedicated, practiced study. They hack something they didn’t think they had the skill to break into and it terrifies them. They’re smart enough to know that, relatively speaking, they are an idiot, and if an idiot can do this then nothing is safe. That feeling is correct.
- bitexploder 21d ago
  20 years later… I don’t get joy from hacking things. But the 20 year wisdom is a lot of the time it doesn’t matter if it is safe. Just know when it does matter and worry about that :)
stackedinserter 21d ago
The main gotcha for local models is insane hardware requirements. Even for $10K you get mediocre performance.
- Octoth0rpe 21d ago
  I think this is worth revisiting once we get some solid 3rd party numbers from the new mac studio ultras, which admittedly are a bit over $10k with a 2tb ssd + 256gb ram. I think I'd be seriously considering it if I had a $200/month subscription of some kind.
- hedgehog 21d ago
  This situation has improved quite a bit recently, Qwen Flash Next will run on a $4000 PC and can reliably implement small features on its own (feels comparable to Opus 4.5). It's a bit slow but pretty effective.
  - bitexploder 21d ago
    Less. Probably 2-3K if you build right. Qwen 3.8 27B on constrained tasks is Opus 4.6-ish to me, it just doesn’t know enough, but when task is laid out just gets it done. Comes down to how much of the ambiguity we expect out of the model.
    - hedgehog 21d ago
      What would you build? Flash Next on 128GB works ok, 3.8 27B might technicially work on less but it's just too slow at least on current APU style chips. I'm curious about the non-NVIDIA 32GB discrete GPU options but I haven't tried yet.
      - bitexploder 21d ago
        Flash next only needs 64GB for core model inference. If you really wanted it. Need 64GB vram, 900+ GB/s speed, and a lot of system ram (128GB). Seems feasible. Hmm. Maybe older GPUs work.
deleted 21d ago
[deleted]
cube00 21d ago
Friends Don't Let Friends Use Ollama https://news.ycombinator.com/item?id=47788385 https://news.ycombinator.com/item?id=47788385
- homarp 21d ago
  and tell your friends about https://llama.app/ https://llama.app/ by Nvidia
- fl0id 21d ago
  just use unsloth or jan.ai or...
- bitexploder 21d ago
  Things are different now on a Mac. Many small improvements make Ollama genuinely decent for many models now.
  - circularfoyers 21d ago
    The fact that those issues existed for so long while the entire time where not issues with other inference runtimes is more the point. And I think are indicative of future incompetence.
wonkofatlatl 21d ago
[dead]
robotswantdata 21d ago
Why are you using Ollama? Just use llama.cpp
- bitexploder 21d ago
  MLX?
  - kovrik 21d ago
    Then use oMLX or MTPLX or Rapid MLX.
roschdal 21d ago
Self-hosted Ollama is the best.
YavenTeam 21d ago
[flagged]
novaapi 21d ago
[flagged]
fghorow 21d ago
I've been using Claude Code Extension in VSCode (no phone-home configured), backed by DwarfStar on a LAN local MBPro 128GB M5. The context bloat is horrendous, leading to 5-10 minute prefills. I've recently been exploring tools like headroom to help manage context, with some limited "success" (for some definition of success). What do others with similar setups do? (I kind of hate to abandon Claude Code, as it seems to be the most capable coding assistant of the limited set of tools I've tried. But that horrendous context bloat is really painful!)
- bitexploder 21d ago
  OMP. Opinionated but completely configurable. Probably the beat to have a lot of batteries and let you uninstall what you don’t want. Sadly Anthropic forbids its use on their subscriptions.
  - fghorow 21d ago
    Thanks. I'll kick the tires on OMP a little harder...
  - crossroadsguy 20d ago
    Isn't OMP sort of Claude in Pi's clothing? I tried it and it seemed like I was using Claude. But if tweaking is needed there then why not stick to Pi and add/strip as needed? > Anthropic forbids its use on their subscriptions This! How are these companies even allowed to do this while they anyway charge for either API access or limit usage in the generic plans. It's blatantly just "I don't want you to spend less per generic task!".
- ImL1s 21d ago
  [flagged]
krttherealest 21d ago
attention is the key
Havoc 21d ago
Can a 27b model even do meaningful security tasks? I thought the interesting cyber stuff is really at the edge of frontier
- prettyblocks 21d ago
  It depends what you mean by meaningful, but yes, qwen3.8 27b is pretty mind blowing to me. It can easily solve portswigger labs for example at q4_k_m.
sdafa 21d ago
[flagged]
onesandofgrain 20d ago
What model are you using?
airspresso 20d ago
From what I'm seeing elsewhere, context size up to 128k should be possible on this hardware. It really matters for agentic workloads to push that context size headroom up. Anthropic are spoiling us with models that do 500k context and beyond.
Woodi 20d ago
Funny times! How many years after "public clouds" and non-local "disks" and "drives" we are ? :) And you still need to tell peoples that other have access to your private data :) Wait, no... They even have access to a thingie you just about to think about! ;) That is a superpower, no less :> And managers are firing peoples just to outsource "thinking" to some not owned by them cloud computer :>

TLDR: Local models have a smaller context window, so your 35kB prompts that worked fine against a hosted 1 Million token window, crash out when you only have a 65K (!) token window locally. I dislike being negative, but I was really hoping for more substance when reading this. It would have been an interesting topic.

0o_MrPatrick_o0 21d ago
Thanks for the feedback. I wanted to get into more detail, but I spent the whole weekend working these problems and then constructing this post. Dario’s behavior this weekend made me feel like this just needed to get out quick. In the future, I’ll be sharing more details about some other things in the process and some ways I found to use automation to accelerate splitting prompts for use on local inference.
- monegator 21d ago
  Yes, please!
- SyneRyder 21d ago
  Understood, and I realized you'd posted this to HN yourself, so I felt a bit bad making the comment. I think maybe for me, this might have worked better if the motivation had been one separate post, and the details of the gotchas as a post of its own. But I also think, if you're going to be limited to 65k token windows, you're going to have a really difficult time. Even 250k windows were cramped for me when that's all we had on Anthropic models. I just don't think a 65k window is going to be big enough for proper cyberdefence work, even if I totally agree with going local wherever you can. It feels like if you're defending against swarms of 1-10M context windows, you need to get as close as you can to similar. I've had to reach for Chinese 1m models instead because the American models just refuse me here in Australia.
- vardalab 21d ago
  Well, if you are serious about it and you have Strix Halo, there are better ways of getting more context and capability and speed. Lookup halogen for Strix The most cost-effective local option right now, I think, is dual R9700. You can run a 27B dense Qwen at FP8 around with a full context and 2-3 concurrent sessions of 260K context. If you go down to an MXFP4, you get 4 to 5 concurrent sessions. And speed is on par with anything you'll get from hosted providers. You're getting between 60-80 for FP8 and 150+ tokens per second speed for MXFP4 and pp is 4K+. Lookup vllm radiance There is also a lot of progress in running Qwen 3.8 next flash with dual R9700. Obviously one gets less context and speed is a little bit less, but it's still very acceptable. Better than what you're getting with Llama on Strix Halo, that's for sure. These are small dense models,meaning Qwen, have gotten capable and fast. And there's been a lot of progress in the area. So sticking with Llama you are not taking advantage of the hardware you have. And yeah, for Strix Halo, you should just look into halogen and you shouldn't be just using 96 gigs for the VRAM. You should give it most of the VRAM to the inference and connect to it from your laptop or something. People ar egetting 1000+ pp with qwen 3.8 Next Flash
  - 0o_MrPatrick_o0 21d ago
    Thank you. This is the hackernews I miss.
- hermitShell 21d ago
  Your hardware can do way more than 64k tokens context window, can't it? And with Ollama it's very easy, superficially you just drag the slider. I'm now reading "Friends Don't Let Friends Use Ollama" linked in another comment so a lot of problems with that approach are surfacing for me right now. So yeah. Along with others, I think you should come up with some empirical means of understanding if your preprompt is doing anything good since I doubt that it's all necessary and helpful. Second maybe you and I need to fix our runtimes.
  - 0o_MrPatrick_o0 21d ago
    Yes. I have to observe: Seeing the raw CoT is the best signal for evaluating prompt efficiency. The fact that frontier providers don’t share this feels criminal after this last weekend. They’ve played us all for fools. Thanks for the token setting observation. I’m running ollama headless and from the command line. I didn’t even know there was a web ui. Conf file is pretty straightforward though. I settled on 64k tokens because of a recommendation from an LLM. I now see that that guidance is associated with similar sounding (but not the same) configuration constraints. So this weekend I’ll try turning it to 11 (250k tokens?). Anticipating some hard locks over here.
    - deleted 21d ago
      [deleted]
    - jurgenburgen 20d ago
      CoT is performative and doesn’t reveal how reasoning happens. If you look at those traces locally it’s just gibberish, especially if the model falls into a loop. https://arxiv.org/abs/2605.11746 https://arxiv.org/abs/2605.11746
zozbot234 21d ago
The new DeepSeek models address this issue very cleanly. DeepSeek Flash V4.1 requires less than 1 GB memory for a full 1M context, down from about ~10 GB in DeepSeek Flash V4.0. This is a significant step towards making near-frontier models usable even on low-end consumer hardware, though of course with significant tradeoffs in overall performance.

dell2024 21d ago

I had hoped to get some new information out of this topic, but unfortunately found the same local "dead-ends" that I explored myself. It unfortunately feels like we will be stuck waiting for a burst bubble before local hardware can be reasonably acquired for personal LLM usage.

0o_MrPatrick_o0 21d ago
FWIW, I’m not dead ended yet. Using frontier providers to manage prompt tuning experiments has been very exciting. Might be something there.

DiabloD3 21d ago

The article doesn't really describe the problem: if your prompt is 35kb, your prompt is confusing, unfocused, and doesn't work right on any LLM, and is needlessly bloating your context. At this point in time, due to how most people and companies run their inference engine, regardless of the model (yes, this includes the newest from OpenAI and Anthropic and the Chinese Tigers and Dragons), you run out of useful context that the model can accurately attend to around the 250k mark no matter how much they advertise their context size is. You need to cut your prompt up. If you believe LLMs work, have the LLM help you shape the overall plan, and then have multiple sessions run each step in the plan without being bloated with the context of previous successful steps. I don't see LLMs being production-ready until the context rot and sampling problem is fixed forever. This has not occurred, and the big inference providers aren't even bothering to integrate any of the research on that subject. If anything, many of the bigger companies are actively making inference quality worse just to extend their runway a tiny bit farther before they go bankrupt. The only thing the article gets right is this: if you're serious about LLMs, abandon Big AI and infer locally only. This is the only way you have control over the quality of the output.

andai 21d ago
The Claude Code system prompt was >50KB, though I think they trimmed it down heavily recently. (The newer models don't need as much hand-holding.)
- embedding-shape 21d ago
  Yeah, strong evidence for what parent says is correct. Been my experience as well, especially with local (smaller) models but also SOTA. The less instructions you have, the better they get at following them. Conflicting instructions is like poison, and it's harder to find those conflicting parts the longer the prompt is too.
  - cyanydeez 21d ago
    the longer the context grows, the greater the probability that it generates ambiguity, and the probability that it makes mistakes approaches 1. A system prompt should be looked at like a starting point and a direction, but not a giant atlas map of everything it may be asked to do. I've been playing around with Qwen3.8-Next-Flash that has great logic, recall, roleplaying, etc. From what I can tell, it's definitely on part with the SOTA a few months ago, and my opinion, it likely is around the pinnacle of advancement without high inefficiency in preparing with the current LLM recipes. Further, I'm of the opinion America's SOTA is hitting the real cost-sigmoid and there's no singularity in site. These things will hack the planet if you put them in a group of agents and tell them to do it; but as context grows, the probability that they can answer "how many r's are in strawberry" goes down. No amount of parameters it going to erradicate that. But I digress, my new stage of working with LLMs is figuring out how to use Qwen3.6-35B-A3B as the entry point to collect the context, and then use the big boys Qwen3.8 to make the edits, then degrade back and forth to complete changes. There's no harness yet for this, but there's clearly an intelligent way to setup a engineering harness. And I do understand people have codebases that simply can't easily live in the smaller (~128k-256k) context windows, but instead of porting your codebase to another novel language, breaking it into context-aware components would make it closer to what these things can do. And I'll repeat: I don't think we're approaching the singularity of self-recursion primarily because the LLMs will duplicate errors, context poison, and whatever else they encounter and there's no human who can sit around correcting it constantly. The American AI apparatus should cut their models like the Chinese down and work on real problems and stop imbibing the singularity-watts-are-all-we-need drugs.

andai 21d ago

> Everyone who begins learning exploitation hits a phase of exploitability grief about 3 month into dedicated, practiced study. They hack something they didn’t think they had the skill to break into and it terrifies them. They’re smart enough to know that, relatively speaking, they are an idiot, and if an idiot can do this then nothing is safe. That feeling is correct.

bitexploder 21d ago
20 years later… I don’t get joy from hacking things. But the 20 year wisdom is a lot of the time it doesn’t matter if it is safe. Just know when it does matter and worry about that :)

stackedinserter 21d ago

The main gotcha for local models is insane hardware requirements. Even for $10K you get mediocre performance.

Octoth0rpe 21d ago
I think this is worth revisiting once we get some solid 3rd party numbers from the new mac studio ultras, which admittedly are a bit over $10k with a 2tb ssd + 256gb ram. I think I'd be seriously considering it if I had a $200/month subscription of some kind.
hedgehog 21d ago
This situation has improved quite a bit recently, Qwen Flash Next will run on a $4000 PC and can reliably implement small features on its own (feels comparable to Opus 4.5). It's a bit slow but pretty effective.
- bitexploder 21d ago
  Less. Probably 2-3K if you build right. Qwen 3.8 27B on constrained tasks is Opus 4.6-ish to me, it just doesn’t know enough, but when task is laid out just gets it done. Comes down to how much of the ambiguity we expect out of the model.
  - hedgehog 21d ago
    What would you build? Flash Next on 128GB works ok, 3.8 27B might technicially work on less but it's just too slow at least on current APU style chips. I'm curious about the non-NVIDIA 32GB discrete GPU options but I haven't tried yet.
    - bitexploder 21d ago
      Flash next only needs 64GB for core model inference. If you really wanted it. Need 64GB vram, 900+ GB/s speed, and a lot of system ram (128GB). Seems feasible. Hmm. Maybe older GPUs work.

deleted 21d ago

[deleted]

cube00 21d ago

Friends Don't Let Friends Use Ollama https://news.ycombinator.com/item?id=47788385 https://news.ycombinator.com/item?id=47788385

homarp 21d ago
and tell your friends about https://llama.app/ https://llama.app/ by Nvidia
fl0id 21d ago
just use unsloth or jan.ai or...
bitexploder 21d ago
Things are different now on a Mac. Many small improvements make Ollama genuinely decent for many models now.
- circularfoyers 21d ago
  The fact that those issues existed for so long while the entire time where not issues with other inference runtimes is more the point. And I think are indicative of future incompetence.

wonkofatlatl 21d ago

[dead]

robotswantdata 21d ago

Why are you using Ollama? Just use llama.cpp

bitexploder 21d ago
MLX?
- kovrik 21d ago
  Then use oMLX or MTPLX or Rapid MLX.

roschdal 21d ago

Self-hosted Ollama is the best.

YavenTeam 21d ago

[flagged]

novaapi 21d ago

[flagged]

fghorow 21d ago

I've been using Claude Code Extension in VSCode (no phone-home configured), backed by DwarfStar on a LAN local MBPro 128GB M5. The context bloat is horrendous, leading to 5-10 minute prefills. I've recently been exploring tools like headroom to help manage context, with some limited "success" (for some definition of success). What do others with similar setups do? (I kind of hate to abandon Claude Code, as it seems to be the most capable coding assistant of the limited set of tools I've tried. But that horrendous context bloat is really painful!)

bitexploder 21d ago
OMP. Opinionated but completely configurable. Probably the beat to have a lot of batteries and let you uninstall what you don’t want. Sadly Anthropic forbids its use on their subscriptions.
- fghorow 21d ago
  Thanks. I'll kick the tires on OMP a little harder...
- crossroadsguy 20d ago
  Isn't OMP sort of Claude in Pi's clothing? I tried it and it seemed like I was using Claude. But if tweaking is needed there then why not stick to Pi and add/strip as needed? > Anthropic forbids its use on their subscriptions This! How are these companies even allowed to do this while they anyway charge for either API access or limit usage in the generic plans. It's blatantly just "I don't want you to spend less per generic task!".
ImL1s 21d ago
[flagged]

krttherealest 21d ago

attention is the key

Havoc 21d ago

Can a 27b model even do meaningful security tasks? I thought the interesting cyber stuff is really at the edge of frontier

prettyblocks 21d ago
It depends what you mean by meaningful, but yes, qwen3.8 27b is pretty mind blowing to me. It can easily solve portswigger labs for example at q4_k_m.

sdafa 21d ago

[flagged]

onesandofgrain 20d ago

What model are you using?

airspresso 20d ago

From what I'm seeing elsewhere, context size up to 128k should be possible on this hardware. It really matters for agentic workloads to push that context size headroom up. Anthropic are spoiling us with models that do 500k context and beyond.

Woodi 20d ago

Funny times! How many years after "public clouds" and non-local "disks" and "drives" we are ? :) And you still need to tell peoples that other have access to your private data :) Wait, no... They even have access to a thingie you just about to think about! ;) That is a superpower, no less :> And managers are firing peoples just to outsource "thinking" to some not owned by them cloud computer :>