3 ms·

Are you not worried about where your data will end up? By now I‘m feeding things to Codex that I‘d rather not have in a leak.

by manmal 4mo ago

epolanski 4mo ago
Yes, that's exactly why I avoid OpenAI and Anthropic products. Besides the (quite true) joke, if sending data to DeepSeek is a concern the good thing is that the models are open weight, you can self host them or use third party providers.
- SwellJoe 4mo ago
  You can theoretically self-host. DeepSeek is big. DS4 (the 2-bit quantization of DeepSeek Flash) runs on my Strix Halo with 128GB, but it's slow as hell. Completely unusable for interactive work. But, I guess a company that cared about data privacy and wanted a Good Enough local model could spend $100,000 or more on hardware to run it properly.
  - epolanski 4mo ago
    DS4 flash runs okay on MacBook Pro though: https://github.com/antirez/ds4#speed https://github.com/antirez/ds4#speed
  - zozbot234 4mo ago
    The DS4 author has demoed upcoming work on Strix Halo that makes it roughly competitive with the Apple Silicon equivalent (i.e. Pro models with similar memory bandwidth figures, not Max or Ultra). Maybe even a bit faster for prefill, and with further potential for running small batches in parallel (since the GPU clearly has some amount of compute headroom during decode).
    - SwellJoe 4mo ago
      As far as I can tell you'll have a context limit of about 64k, which is also prohibitive for serious work. (My benchmark maxes out at 90k in context when running, so I'm giving the self-hosted models 128k to leave plenty of wiggle room.) But, still, it's cool that the work is happening. For some classes of problem it might be an option, and when the 192GB Strix Halo comes out, DS4 will probably become a real contender for self-hosting champ, as that leaves enough memory for a big context.
      - zozbot234 4mo ago
        > As far as I can tell you'll have a context limit of about 64k Source? The author has demoed a 100k ctx already, and I can't think of a reason why more wouldn't be supported. RAM is a bit tight but that only matters with really long contexts on DeepSeek V4, and proper support for SSD streaming would address this anyway. BTW, the official support is now merged too.
        SwellJoe 4mo ago
        No source, just back of the envelope math. 100k seems optimistic, but I guess I'll try it and see. That would be usable for at least a few use cases, including the security scanning work I'm focused on at the moment (at least, so far, the peak token usage has been 90k, which would make 100k tight but probably fine).
        SwellJoe 4mo ago
        OK, I just tried it with the new mainline ROCm and MTP support, and it is faster, but still uncomfortably slow for interactive coding agent use. It does about 14-15 t/s, which is faster than the 10-11 t/s I was seeing before, but still a crawl. I set it loose on a small 300-line Perl file, and it's still chewing several minutes later. So, it's super cool that such a solid model can run locally and it's probably useful for batched work overnight. But, I'm not going to sit around twiddling my thumbs while working. I think I can write code by hand faster than this. I'll gladly pay for a cloud model so I don't have to wait (especially since DeepSeek models are so cheap).
        zozbot234 4mo ago
        Well, that performance figure seems consistent with memory bandwidth on that machine (and its upcoming successor Gorgon Halo; Medusa Halo is projected to be faster) and even on DGX/RTX Spark. You'd get the same outcome on Apple Silicon Mn Pro (not Max or Ultra) if there was one with enough memory capacity. It's likely possible to raise aggregate tok/s on Strix Halo or DGX/RTX Spark (not realistically on Apple Silicon, at least not on a single machine) by batching multiple inference flows together, but that's admittedly a bit fiddly to implement and not what you're interested in anyway. It seems that you'll want either top-of-the-line Apple Silicon (Max/Ultra) or cloud inference, which will always be super competitive if your focus is on low latency.
axus 4mo ago
It might be a while before DeepSeek shows up on GovCloud
SwellJoe 4mo ago
These days I'm also worried about US companies having my data. I hate that we're at that point, but with Trump talking about taking an ownership stake in AI companies, and tech companies, including the leading AI companies, lining up to participate in the war crime of the day, I don't have a lot of faith my data is any safer with US companies than those in China. Though, I added Mistral's latest model to the mix in the hope that some European model could be a contender, but it failed completely. I don't know if it hit safety guardrails or is just not competent at security work, but it scored 0/9. No errors, it returned the empty JSON set it was supposed to return if it didn't find anything. But, there were plenty of real bugs to find, and some very small self-hosted models found at least some of them.
- epolanski 4mo ago
  I think it is a bit naive to assume that companies that have built their moats on violating copyright, scraping and ddosing all of the internet, and distilling each other's models will not leverage our data if they can have financial benefits out of it. I don't think that the country matters, whoever you send data to among these AI labs you are at security risk and data risk.
  - SwellJoe 4mo ago
    I hope that someday there are AI companies for whom ethical behavior is a selling point. We're certainly not there for the current leaders, though vibes vary a little bit between them. Some seem scarier than others.
fc417fc802 4mo ago
What is there to worry about? OpenRouter currently lists 13 alternate providers for V4 Pro, many of them in the US. https://openrouter.ai/deepseek/deepseek-v4-pro/providers https://openrouter.ai/deepseek/deepseek-v4-pro/providers Unless you meant being concerned about hosted AI in general, not specifically DeepSeek. In which case yeah that's a huge concern to me but I can't reasonably afford a half million dollar appliance to self host a large model at reasonable performance and don't have anywhere to put one even if I could.

Are you not worried about where your data will end up? By now I‘m feeding things to Codex that I‘d rather not have in a leak.

by manmal 4mo ago

Are you not worried about where your data will end up? By now I‘m feeding things to Codex that I‘d rather not have in a leak.

epolanski 4mo ago

Yes, that's exactly why I avoid OpenAI and Anthropic products. Besides the (quite true) joke, if sending data to DeepSeek is a concern the good thing is that the models are open weight, you can self host them or use third party providers.

SwellJoe 4mo ago
You can theoretically self-host. DeepSeek is big. DS4 (the 2-bit quantization of DeepSeek Flash) runs on my Strix Halo with 128GB, but it's slow as hell. Completely unusable for interactive work. But, I guess a company that cared about data privacy and wanted a Good Enough local model could spend $100,000 or more on hardware to run it properly.
- epolanski 4mo ago
  DS4 flash runs okay on MacBook Pro though: https://github.com/antirez/ds4#speed https://github.com/antirez/ds4#speed
- zozbot234 4mo ago
  The DS4 author has demoed upcoming work on Strix Halo that makes it roughly competitive with the Apple Silicon equivalent (i.e. Pro models with similar memory bandwidth figures, not Max or Ultra). Maybe even a bit faster for prefill, and with further potential for running small batches in parallel (since the GPU clearly has some amount of compute headroom during decode).
  - SwellJoe 4mo ago
    As far as I can tell you'll have a context limit of about 64k, which is also prohibitive for serious work. (My benchmark maxes out at 90k in context when running, so I'm giving the self-hosted models 128k to leave plenty of wiggle room.) But, still, it's cool that the work is happening. For some classes of problem it might be an option, and when the 192GB Strix Halo comes out, DS4 will probably become a real contender for self-hosting champ, as that leaves enough memory for a big context.
    - zozbot234 4mo ago
      > As far as I can tell you'll have a context limit of about 64k Source? The author has demoed a 100k ctx already, and I can't think of a reason why more wouldn't be supported. RAM is a bit tight but that only matters with really long contexts on DeepSeek V4, and proper support for SSD streaming would address this anyway. BTW, the official support is now merged too.
      - SwellJoe 4mo ago
        No source, just back of the envelope math. 100k seems optimistic, but I guess I'll try it and see. That would be usable for at least a few use cases, including the security scanning work I'm focused on at the moment (at least, so far, the peak token usage has been 90k, which would make 100k tight but probably fine).
      - SwellJoe 4mo ago
        OK, I just tried it with the new mainline ROCm and MTP support, and it is faster, but still uncomfortably slow for interactive coding agent use. It does about 14-15 t/s, which is faster than the 10-11 t/s I was seeing before, but still a crawl. I set it loose on a small 300-line Perl file, and it's still chewing several minutes later. So, it's super cool that such a solid model can run locally and it's probably useful for batched work overnight. But, I'm not going to sit around twiddling my thumbs while working. I think I can write code by hand faster than this. I'll gladly pay for a cloud model so I don't have to wait (especially since DeepSeek models are so cheap).
        zozbot234 4mo ago
        Well, that performance figure seems consistent with memory bandwidth on that machine (and its upcoming successor Gorgon Halo; Medusa Halo is projected to be faster) and even on DGX/RTX Spark. You'd get the same outcome on Apple Silicon Mn Pro (not Max or Ultra) if there was one with enough memory capacity. It's likely possible to raise aggregate tok/s on Strix Halo or DGX/RTX Spark (not realistically on Apple Silicon, at least not on a single machine) by batching multiple inference flows together, but that's admittedly a bit fiddly to implement and not what you're interested in anyway. It seems that you'll want either top-of-the-line Apple Silicon (Max/Ultra) or cloud inference, which will always be super competitive if your focus is on low latency.

axus 4mo ago

It might be a while before DeepSeek shows up on GovCloud

SwellJoe 4mo ago

These days I'm also worried about US companies having my data. I hate that we're at that point, but with Trump talking about taking an ownership stake in AI companies, and tech companies, including the leading AI companies, lining up to participate in the war crime of the day, I don't have a lot of faith my data is any safer with US companies than those in China. Though, I added Mistral's latest model to the mix in the hope that some European model could be a contender, but it failed completely. I don't know if it hit safety guardrails or is just not competent at security work, but it scored 0/9. No errors, it returned the empty JSON set it was supposed to return if it didn't find anything. But, there were plenty of real bugs to find, and some very small self-hosted models found at least some of them.

epolanski 4mo ago
I think it is a bit naive to assume that companies that have built their moats on violating copyright, scraping and ddosing all of the internet, and distilling each other's models will not leverage our data if they can have financial benefits out of it. I don't think that the country matters, whoever you send data to among these AI labs you are at security risk and data risk.
- SwellJoe 4mo ago
  I hope that someday there are AI companies for whom ethical behavior is a selling point. We're certainly not there for the current leaders, though vibes vary a little bit between them. Some seem scarier than others.

fc417fc802 4mo ago

What is there to worry about? OpenRouter currently lists 13 alternate providers for V4 Pro, many of them in the US. https://openrouter.ai/deepseek/deepseek-v4-pro/providers https://openrouter.ai/deepseek/deepseek-v4-pro/providers Unless you meant being concerned about hosted AI in general, not specifically DeepSeek. In which case yeah that's a huge concern to me but I can't reasonably afford a half million dollar appliance to self host a large model at reasonable performance and don't have anywhere to put one even if I could.