9 ms·

Why DeepSeek is cheap at scale but expensive to run locally

• comrade1234 1y ago
  I haven't looked for awhile but is deepseek online still about 1/100th the cost of its competitors?
  • ALLTaken 1y ago
    I don't know the exact cost-breakdown, but they've come up with a few really inspiring and qualitatively high value papers that demonstrate how they further increased efficiency at their scale. Along with it they also published quite a few repositories with fully open-source code. I stopped using ChatGPT as it was just reinforcing my prompts and not ever giving deeper insights, except something I call manipulative behaviour. DeepSeek was seriously cool, but it started behaving similar to Google Gemini Pro, which just tries to be lazy, if you give it a hard task to chew on. It basically gives you patch-files instead of printing out the whole code, which is more tedious doing manually, than c/p the code. It also started indexing our private repository and some corporate repositories that were on GitHub behind MFA and stringent lock. Definitely illegal.
    • diggan 1y ago
      > It also started indexing our private repository and some corporate repositories that were on GitHub behind MFA and stringent lock. Definitely illegal. What is "it" in this context, the DeepSeek weights? Sounds like you're talking about some application, but AFAIK, DeepSeek doesn't maintain any applications, only their API + released weights.
      • ALLTaken 1y ago
        ChatGPT indexed our private GitHub repositories and those our corporations I worked with.
    • simianwords 1y ago
      How did it have access to your private repo and how did you find out?
      • ALLTaken 1y ago
        I made a video of it with a friend. The repository is of a large corporate automative industry company. I also have my own private repositories which were always private and OpenAI printed my files in the first prompt. When I prompted again it acted as if it didn't know. But my friend tried on his account and could access the Corp and my private repository without ever being linked. The Corporate repository was of Volkswagen. It's quite serious of a breach. I only gave it the name of the repository and it printed the files, which shouldn't be possible. Maybe OpenAI exploits Microsoft to access GitHub fully to train their AI on all of humanity's code for free, violating privacy, security, IP and copyright.
        • Legend2440 1y ago
          >I only gave it the name of the repository and it printed the files, which shouldn't be possible. Are you sure these weren't just plausible guesses at file names? It's just a hallucination. I asked it for the list of files in some public repositories (which are definitely in the training data) and it gave me a plausible-but-wrong list of files. It can't remember that kind of detail.
          • ALLTaken 1y ago
            I am sure about that, I also thought about hallucination, but it was precise in the first prompt. 2nd and follow-ups had plausible denial and created similar, but clearly different code. It could even print the list of files and their exact names if triggered right. They may have surely patched that, so that nobody sues them with digital proof. But we recorded it. It was when their new model came out. Don't remember the date, but a few months ago. We have two videos and different repositories it should not have access to at all. Microsoft owns GitHub. OpenAI has a multi-billion dollar investment from Microsoft and access to their Infrastructure "for training" and seems likely, they got access to GitHub. Something that they shouldn't do, since that's illegal and very unethical.
    • singularity0808 1y ago
      ChatGPT is reinforcing your prompts, DeepSeek is cool but starts acting lazy like Gemini. So what are you working with now? Deepseek or something else?
    • ants_everywhere 1y ago
      > as it was just reinforcing my prompts and not ever giving deeper insights, except something I call manipulative behaviour. Try telling Deepseek you want to murder political dissidents. In my experiments Deepseek will start enthusiastically reinforcing your prompts.
      • johnisgood 1y ago
        It just simply does its job. We can add sorts of arbitrary safeguards, but then what is the point of using an LLM? Perhaps local modals are the future, because reverse engineers may not even be able to use the new Claude (just read its system prompt to not help with backdoors, and so forth).
        • ants_everywhere 1y ago
          Yes that's true. But in this case it's the (probably) unintended consequence of an intentional safeguard. Namely, Deepseek has an obligation to spread the Chinese version of socialism, which means it's deliberately trained on material advocating for or justifying political violence.
          • johnisgood 1y ago
            Well, I do not like that, for sure. Just put the politics and all that aside, I think it should lean towards neutrality, even if humans cannot... they should still make the LLM more neutral instead of pushing their own agenda, see Grok and white genocide in South Africa (Elon Musk's political opinion).
      • MangoToupe 1y ago
        Is this a reference to something? Political dissidents relative to which state? Does it change if you swap out the states? How did you discover this to begin with? Why did you initially suggest murdering political dissidents? this comment really raises so many questions I must have missed something Still, chatbots are just as vulnerable to state-driven propaganda as the rest of us. Probably even more so. I imagine if you just referred to dissidents as "terrorists" the rhetoric would fit right in in most opinion pages across the globe. The distinction between "terrorist" and "dissident" and "freedom fighter" seems quite subjective. I probably would avoid such heavily connoted floating signifiers if you want the chatbot to be useful. LLMs have nothing to contribute to political discourse aside from regurgitation of propaganda. Almost by definition.
    • VectorLock 1y ago
      >It basically gives you patch-files instead of printing out the whole code I've noticed on the Aider leaderboard that Google Gemini Pro has an "Edit Format" listed as "diff-fenced" and things like ChatGPT have "architect" edit format where Aider asks separate "architect" and "code" models. Seems like Gemini Pro prefers the diff format.
      • zxexz 1y ago
        The diff-fenced is iirc specific to Gemini models, they really don’t like the file path outside of the fence. The architect mode still uses one of the other edit format, the prompt just ends up a little different.
        • ALLTaken 1y ago
          Maybe they optimized for one of their TREEE SOLUTIONS: - https://Jules.google https://Jules.google - NotebookLm - GoogleCollab How can a company have 3 contenders to Windsurf and Cursor, which are VSCode forks with a little sugarcoating and not make any impact?? The CPO should be fired. I think also after seeing Google Gemini's Video that their entire department is now fully Indian, including the CEO. If that isn't racially biased, then idk. See yourself: https://www.youtube.com/watch?v=6GO7bPb5cTA&t=2270s https://www.youtube.com/watch?v=6GO7bPb5cTA&t=2270s
      • ALLTaken 1y ago
        I met a Googler when I was in Dubai for an event and he shared that he and others had access to LLMs internally for many years before it was made popular by OpenAI. I know Google has an internal AI everything policy, maybe they internally have awesome tools to rearchitect everything based on diffs and in the typical google way they adapted it to their own internal tools. You know, Google.. like they don't give a damn about the user, the product design or actually anything other than profit/roi. So many great discontinued products.. I think they killed RSS.
    • ashirviskas 1y ago
      > DeepSeek was seriously cool, but it started behaving similar to Google Gemini Pro You should be able to use the version of DeepSeek that you prefer indefinitely if you host it yourself or choose that specific version with your preferred provider.
    • zxexz 1y ago
      You should self host not trust a third party application if you run into either of those things. The weights are open. DeepSeek didn’t change, the application you’re accessing it through did. Or use an enterprise-ready service. Bedrock, firecracker, etc
      • ALLTaken 1y ago
        I like your thinking. Nobody can use ChatGPT offline or retrain it, but DeepSeek is fully opensource. It's technology, I don't care which country made it, if it's high quality engineering, it's just that. The data it was trained on doesn't matter if you can train a wholly new model using the exact same principles and stack they opensourced with your own data. Which is really awesome. I use openrouter.ai to have no timeouts and offtimes, since DeepSeek seems to get DDoS attacks somehow, or there are too many users, idk.
        • ElectricalUnion 1y ago
          > Nobody can use ChatGPT offline or retrain it, but DeepSeek is fully opensource. Well, you likely can't train DeepSeek yourself either. You most likely: * you philosophically don't have all the training data to train it yourself (so the claim it's opensource or open-whatever are dubious in the first place); or * you don't have the compute to "press the train button" and getting the weights back before the sun expires. While considered ridiculously ground-breaking cheap, those costs were still estimated to be around 6 million USD (DeepSeek claimed the model training took 2,788 thousand H800 GPU hours, which, at a cost of $2/GPU hour, comes out to a "mere $5.576 million"). I remember that when it was released, the mere thought that "people" cound "train AI cheaply with only 6 million USD" made one of the worst drops in the Nvidia valuation.
          • ALLTaken 1y ago
            This is really not true my friend. I would love to help you if I had some more time, but let me look for a tutorial. Because the FineWeb Dataset is already super good. You can train 7B or 32B Param models at home The >600B Param model isn't really using all the data effectively, but with a MacStudio Farm you can also train that one at home (if you have enough money to buy at least 100). Here's the easy way: https://github.com/FareedKhan-dev/train-deepseek-r1 https://github.com/FareedKhan-dev/train-deepseek-r1 More details: https://www.bentoml.com/blog/the-complete-guide-to-deepseek-models-from-v3-to-r1-and-beyond https://www.bentoml.com/blog/the-complete-guide-to-deepseek-... Here's how DeepSeek-R1-Zero was built, basically from 0 to Hero, including weights the FULL Training Data and everything you need to get it running locally or on servers.https://medium.com/@GenerationAI/how-deepseek-r1-zero-was-reproduced-in-30-4e394cd3dd58 https://medium.com/@GenerationAI/how-deepseek-r1-zero-was-re... For $30 USD you can also train a small DeepSeek at home! https://github.com/Jiayi-Pan/TinyZero https://github.com/Jiayi-Pan/TinyZero https://huggingface.co/deepseek-ai/DeepSeek-R1-Zero https://huggingface.co/deepseek-ai/DeepSeek-R1-Zero (the model)
    • davidmurdoch 1y ago
      Had Gemini 2.5 Pro preview running in agent mode in VSCode on a 3000+ line file. It patched it to about 200 lines with a comment in the middle: "// the rest of the code is unchanged".
      • ALLTaken 1y ago
        Exactly my experience too and it's soo annoying. It doesn't matter how you prompt it or what your system prompt is. It tries to end the session as early as possible, claiming to have fulfilled everything. Although it just causes more work for the user, less for itself. The tokens saved are easily multiplied by the amount you have to prompt it again. This I experienced partially in DeepSeek since their recent update too, not as aggresively as in Gemini 2.5 Pro, but similar lazyness or cleverness, if you may call that clever.
  • deleted 1y ago
    [deleted]
  • clippyplz 1y ago
    Depends on who you think its competitors are - deepseek-chat ($0.27/M in; $1.10/M out) is twice as expensive as Gemini 2.5 Flash ($0.15; $0.60) but far cheaper than Claude Sonnet 4 ($3; $15).
  • Hilift 1y ago
    That was a pretty good back to reality flex. There really isn't much of a market for expensive products. An inexpensive product that has a few tradeoffs will probably have the advantage. Given how proficient China is at accessing technology resources, it seems likely to me that any chip sanctions against them will probably not be effective.
  • dist-epoch 1y ago
    1/10-20th is a more realistic ratio.
• perching_aix 1y ago
  For those looking to save time, the answer is batched inference. Pretty much running multiple people's "prompts" through a model instance at the same time instead of just really tightly timesharing each model instance. This is also why you may experience a variance in replies when using these services, even when you set the temperature to 0 and the seed to a fixed value. It's cause you don't control the other prompts yours get batched with. Could this be a data exfiltration attack vector? Probably, I didn't "research" that far.
  • pcwelder 1y ago
    > other prompts yours get batched with Why would batching lead to variance?
    • Hendrikto 1y ago
      Because these models are context-sensitive. Every token can influence the output.
      • simianwords 1y ago
        I believe they are talking about latency variance. Batching can increase variance because you may have to wait for enough prompts to get to the batch size.
        • perching_aix 1y ago
          No, I meant that the responses will be different run-to-run. [0] [0] https://152334h.github.io/blog/non-determinism-in-gpt-4/ https://152334h.github.io/blog/non-determinism-in-gpt-4/
          • exe34 1y ago
            Variance based on actual randomness would be one thing, but to me variance based on what other people are running seems concerning, for reasons I can't quite articulate. I don't want the model to reply to a question in one domain based on what a large group of other people are thinking in a different domain (e.g. if they're discussing the news with chatgpt).
• jsnell 1y ago
  I'm not a ML research or engineer, so take this with a grain of salt, but I'm a bit confused by this post. Deepseek V3/R1 are expensive to run locally because they are so big compared to the models people usually run locally. The number of active parameters is obviously lower than the full model size, but that basically just helps with the compute requirements, not the memory requirements. Unless you have multiple H100s lying around, V3/R1 are only run locally as impractical stunts with some or all the model being stored on low bandwidth memory. We can't compare the size of Deepseek V3 to that of any proprietary frontier models because we don't know the size of those models at all (or even their architecture). The models being compared to that are "expensive at scale" you can't run locally at all, but surely we have no reason to believe that they'd somehow be cheap to run locally? But I thought you'd typically expect exactly the opposite effect than is claimed here? MoE should be the better tradeoff for the local/single-user scenario since the downside of batching being harder / less efficient doesn't matter. > Bigger batches raise latency because user tokens might be waiting up to 200ms before the batch is full enough to run, but they boost throughput by allowing larger (and thus more efficient) GEMMs in the feed-forward step Is it really that the matrixes being multiplied are larger? My mental model is that the purpose of batching isn't to get larger input matrices. It's to move the bottleneck from memory bandwidth to compute. The matrices are already sharded to a much smaller size than the size of the entire model or even layer. So you'll basically load some slice of the weights from the HBM to SRAM, do the multiplication for that slice, and then aggregate the results once all tiles have been processed. Batching lets you do multiple separate computations with the same weights, meaning you get more effective FLOPS per unit of memory bandwidth. > The fact that OpenAI and Anthropic’s models are quick to respond suggests that either: Is that actually a fact? The post has no numbers on the time to first token for any of the three providers.
  • gfysfm 1y ago
    Hi, I wrote the post! Also not a ML researcher, just an interested engineer, so I'm sure I got some things wrong. > MoE should be the better tradeoff for the local/single-user scenario since the downside of batching being harder / less efficient doesn't matter. What I meant was that the single-user scenario is going to get dramatically worse throughput-per-GPU, because they're not able to reap the benefits of multi-user batching (unless they're somehow doing massively parallel inference requests, I suppose). > Is it really that the matrixes being multiplied are larger? My mental model is that the purpose of batching isn't to get larger input matrices. It's to move the bottleneck from memory bandwidth to compute. As I understand it, you want larger input matrices in order to move the bottleneck from memory to compute: if you do no batching at all, your multiplications will be smaller (the weights will be the same, of course, but the next-token data you're multiplying with the weights will be 1xdim instead of batch-size x dim), so your GPUs will be under-utilized and your inference will spend more time doing memory operations and less time multiplying. > The post has no numbers on the time to first token for any of the three providers. I probably should have hunted down specific numbers, but I think people who've played with DeepSeek and other models will notice that DeepSeek is noticeably more sluggish.
• yekanchi 1y ago
  this statement holds true for all large parameter open weight models.
• freehorse 1y ago
  > mixture of experts requires higher batch sizes Or apple silicon for low batch size (=1 ideally). The unified memory allows for running larger models on the expense of them running slower, because of lower bandwidth/flops than a normal gpu. But MoEs require computing only few parameters every time, so the computational needs are low. I have seen people reporting decent speeds for deepseek for single batch inference on macs. It is still expensive though to many people's standards because it requires a lot of $$$ to get enough memory. In some ways, MoE models are perfect fit for macs (or any similar machines that may come out). In contrast, ordering a mac with upgraded ram size and running dense models that just fit in the vram can be very painful.
• DavidSJ 1y ago
  Here’s a concise explanation: - High sparsity means you need a very large batch size (number of requests being processed concurrently) so that each matrix multiplication is of sufficient arithmetic intensity to get good utilization. - At such a large batch size, you’ll need a decent number of GPUs — 8-16 or so depending on the type — just to fit the weights and MLA/KV cache in HBM. But with only 8-16 GPUs your aggregate throughput is going to be so low that each of the many individual user requests will be served unacceptably slowly for most applications. Thus you need more like 256 GPUs for a good user experience.
  • zxexz 1y ago
    I’m serving it on 16 H100s (2 nodes). I get 50-80 tok/s per request, and in aggregate I’ve seen several thousand. TTFT is pretty stable. Is faster than any cloud service we can use.
    • majke 1y ago
      Using vllm?
      • zxexz 1y ago
        Oh, SGLang. Had to make a couple modifications, I forget what they were, nothing crazy. Lots of extra firmware, driver and system config too.
    • latchkey 1y ago
      You could do it on one node of 8xMI300x and cut your costs down.
    • zackangelo 1y ago
      H200s are pretty easy to get now. If you switched I'm guessing you'd get a nice bump because the nccl allreduce on the big mlps wouldn't have to cross infiniband.
    • DavidSJ 1y ago
      You're presumably using a very small batch size compared to what I described, thus getting very low model FLOP utilization (MFU) and high dollar cost per token.
• dist-epoch 1y ago
  Do the individual requests in a batch influence each-other? Not in a floating point non-deterministic kind of way, where exact ordering might introduce some non-determinism (begin position 5th versus being position 10th in the batch lets say). I'm asking in a semantic way, can context from one request leak into another because they are in the same batch?
• ipieter 1y ago
  This is an interesting blogpost. While the general conclusion ("We need batching") is true, inference of mixture of experts (MoE) models is actually a bit more nuanced. The main reason we want big batches is because LLM inference is not limited by the compute, but my loading every single weight out of VRAM. Just compare the number of TFLOPS of an H100 with the memory bandwidth, there's basically room for 300 FLOP per byte loaded. So that's why we want big batches: we can perform a lot of operations per parameter/weight that we load from memory. This limit is often referred to as the "roofline model". As models become bigger, this does not scale anymore because the model weights will not fit into GPU memory anymore and you need to distribute them across GPUs or across nodes. Even with NVLink and Infiniband, these communications are slower than loading from VRAM. NVlink is still fine for tensor parallelism, but across nodes this is quite slow. So what MoE allows is expert parallelism, where different nodes keep different experts in memory and don't need to communicate as much between nodes. This only works if there are enough nodes to keep all experts in VRAM and have enough overhead for other stuff (KV cache, other weights, etc). So naturally the possible batch size becomes quite large. And of course you want to maximize this to make sure all GPUs are actually working.
  • zozbot234 1y ago
    You could load different "experts" in a round-robin way on a single node and only aggregate "batches" opportunistically, when you just have multiple requests in-flight that all happen to rely on the same "expert". The difference being that instead of "batches", you would only really have queues. Of course this would come with a sizeable increase in latency, but that's acceptable for many applications (such as for "deep research" workflows)
    • jchrisa 1y ago
      This is very much like Erlang's actor model. The same compute can be run in parallel, or managed via queues. With Erlang's strong support for FFI and process control, I wonder if it's being used as a dispatcher for these sorts of workloads.
  • iwontberude 1y ago
• dgfitz 1y ago
  I am so sincerely amused that “we” figured out how to monetize LLMs from the jump using tokens. It isn’t tech for techs sake, it’s a money grab. Reminds me of paying to send a text message or buying minutes for a phone plan. Purely rent-seeking.
  • kaashif 1y ago
    Can you explain how this is rent seeking? It seems to be straightforwardly not rent seeking. 1. Company develops model, invests in research, hardware, and software. 2. Company sells access to the model. (1) is the step that makes this not rent seeking. Rent seeking is when you profit from something you didn't earn - land rent, monopoly profits, protectionism.
    • dgfitz 1y ago
      That’s fair. My thought was, when there is an interesting new technology, it usually takes time to figure out how to monetize it. Figuring out how to monetize LLMs took no time at all.
      • davidmurdoch 1y ago
        "GPT 1.0" was released in 2018, I think that's a decent amount of time.
        • dgfitz 1y ago
          We must have different definitions of released.
          • davidmurdoch 1y ago
            Don't shoot the messenger. It's just what wikipedia says. Generative Pre-trained Transformer 1 (GPT-1) Original author(s) OpenAI Initial release June 2018; 7 years ago
  • vikramkr 1y ago
• imtringued 1y ago
  >It’s a peculiar feature of transformer-based LLMs that computing a batch of completions at the same time is almost as fast as computing a single completion. Why is that? Incorrect. Transformers usually contain a classical MLP layer. Only the MLP layer can be batched. Hence all classical neural networks including convolutional networks (via im2col) can be batched. If there's anything that the transformer architecture changes, it is that the attention layer cannot be batched.
  • krackers 1y ago
    Yeah this part was confusing, because it's only mentioned halfway through the article that the attention step can only be batched across matching context-window sizes.
• gok 1y ago
  MoE is in general kind of a stupid optimization. It seems to require around 5x more total parameters for the same modeling power as a dense model in exchange for around 2x less memory bandwidth needs. The primary win of MoE models seems to be that you can list an enormous parameter count in your marketing materials.
  • hansvm 1y ago
    Stupid? By paying 5x (normally 2-4x, but whatever) of a thing you don't care about at inference you can gain 2x in the primary thing you care about at inference. It's like handing out 4 extra bricks and getting back an extra lump of gold.
  • bick_nyers 1y ago
    The general rule of thumb when assessing MoE <-> Dense model intelligence is SQRT(Total_Params*Active_Params). For Deepseek, you end up with ~158B params. The economics of batch inferencing a ~158B model at scale are different when compared to something like Deepseek (it is ~4x more FLOPS per inference after all), particularly if users care about latency.
• philipodonnell 1y ago
  Isn’t this an arbitrage opportunity? Offer to pay a fraction of the cost per token but accept that your tokens will only be processed when the batch window isn’t big enough, then resell that for a markup to people who need non-time sensitive inference?
  • pama 1y ago
    You may have already noticed that many providers have separate, much lower, prices for offline inference.
• angry_octet 1y ago
  This is a great explainer from an LLM perspective, and it would be interesting to see a computational scheduling explanation in depth. I presume that hyperscale LLM companies extensively examine the computation trace to identify bottlenecks and idle bubbles, and develop load balancers, pipeline architectures and schedulers in order to optimise their workload. The batching requirement for efficiency makes high security applications quite difficult, because the normal technique of isolating unrelated queries would become very expensive. The nVidia vGPU GPU virtualisation time shares GPU memory, and every switch requires unload/reload context switches, doubtful they have deduplication. Multi-Instance GPU (MIG) splits GPU memory between users, but it is a fixed partitioning scheme (you have to reboot the GPU to change), and nobody wants to split their 96GB GPU into 4x24GB GPUs. Makes me wonder what the tradeoff is for putting second level memory on the GPU board (i.e. normal DRAM), so that different matrix data can be loaded in faster than over PCIe, i.e. the HBM becomes a cache. (I'm also really liking the honesty in the authors book on Software Engineering, not in the dry IEEE sense, but as a survival guide in a large enterprise. https://www.seangoedecke.com/book/ https://www.seangoedecke.com/book/ )
• slavboj 1y ago
  It is not "slow and expensive", although it could be "or". You can get 3 tokens / second running on DDR4 memory on a two generation old workstation system that costs ~1K, via llama.cpp .
  • KolmogorovComp 1y ago
    You’re most likely confusing the real deepseek with a distilled version. Unless you have more than 192Gb of RAM.
    • slavboj 1y ago
      Workstations routinely accommodate much more than that. The "under $1K" price referred to a 768gb build (12x 64gb sticks on a Skylake based system), you could also do a dual-socket version with twice that, at the cost of messing with NUMA (which could be a pro or a con for throughput depending on how you're spreading bandwidth between nodes).
• deleted 1y ago
  [deleted]
• deleted 1y ago
  [deleted]
• bick_nyers 1y ago
  There's still a lot of opportunity for software optimizations here. Trouble is that really only two classes of systems get optimizations for Deepseek, namely 1 small GPU + a lot of RAM (ktransformers) and the system that has all the VRAM in the world. A system with say 192GB VRAM and rest standard memory (DGX station, 2xRTX Pro 6000, 4xB60 Dual, etc.) could still in theory run Deepseek @4bit quite quickly because of the power law type usage of the experts. If you aren't prompting Deepseek in Chinese, a lot of the experts don't activate. This would be an easier job for pruning, but still I think enthusiast systems are going to trend in a way the next couple years that makes these types of software optimizations useful on a much larger scale. There's a user on Reddit with a 16x 3090 system (PCIE 3.0 x4 interconnect which doesn't seem to be using full bandwidth during tensor parallelism) that gets 7 token/s in llama.cpp. A single 3090 has enough VRAM bandwidth to scan over its 24GB of memory 39 times per second, so there's something else going on limiting performance.
  • latchkey 1y ago
    A single MI300x has 192GB of vram.
    • ElectricalUnion 1y ago
      Sad reality is that the MI300x isn't a monolithic die, so the chiplets have internal bandwidth limitations (ofc less severe that using PCIe/nvlink). In AMD own parlance, the "Modular Chiplet Platform" presents itself as either single-I-don't-care-about-speed-or-latency "Single Partition X-celerator" mode or in multiple-I-actually-totally-do-care-about-speed-and-latency-NUMA-like "Core Partitioned X-celerator" mode. So you kinda still need to care what-loads-where.
      • latchkey 1y ago
        I have never heard of a GPU where a deep understanding of how memory is managed was not critical towards getting the best performance.
  • MoonGhost 1y ago
• corey_moncure 1y ago
  If I understand it correctly, the effect of experts is a weighted sum of the individual calculation of each token meeting each expert, where experts to be met by a token are selected on an individual basis. Since a sum is commutative, though, it should be possible to send a large batch of tokens copied to multiple GPUs, where experts are streamed into VRAM, partitioned across GPUs. Then the bottleneck is your PCI-E bandwidth. With 2 GPUs at Gen 4 x16, you should have 60 GB/s of TX bandwidth, allowing you to upload a half precision quant of DeepSeek (about 360 GB) in about 6 seconds. 1 GPU - 30 GB/s TX - 12 seconds 2 GPUs - 60 GB/s TX - 6 seconds 4 GPUs - 120 GB/s TX - 3 seconds Then you just optimize your batch size to match the compute time to the upload time of each GPU. The expert calculation results can be retrieved from the GPUs and summed up.
• briian 1y ago
  This reminded me that the economies of scale in AI, especially inference, is huge. When people say LLMs will be commoditised, I am not sure that means that the market is going to be super competitive. As the economies of scale of AI get even bigger (larger training costs + batch inference etc.) it just seems likely only around 3 companies will dominate LLMs.
  • riku_iki 1y ago
    For inference cost, I don't see how this is different from cloud providers vs dedicated server providers, where AWS is 5-10x more expensive than hetzner. Somehow cloud providers manage to add lots of extra-cost on offering.
• ryan_glass 1y ago
  I run Deepseek V3 locally as my daily driver and I find it affordable, fast and effective. The article assumes GPU which in my opinion is not the best way to serve large models like this locally. I run a mid-range EPYC 9004 series based home server on a supermicro mobo which cost all-in around $4000. It's a single CPU machine with 384GB RAM (you could get 768GB using 64GB sticks but this costs more). No GPU means power draw is less than a gaming desktop. With the RAM limitation I run an Unsloth Dynamic GGUF which, quality wise in real-world use performs very close to the original. It is around 270GB which leaves plenty of room for context - I run 16k context normally as I use the machine for other things too but can up it to 24k if I need more. I get about 9-10 tokens per second, dropping to 7 tokens/second with a large context. There are plenty of people running similar setups with 2 CPUs who run the full version at similar tokens/second.
  • nardi 1y ago
    Whats your prompt processing speed? That’s more important in this situation than output TPS. If you have to wait minutes to start getting an answer, that makes it much worse than a cloud-hosted version.
    • pclmulqdq 1y ago
      I assume KV caching makes this a non issue, but I'm also curious.
      • idonotknowwhy 1y ago
        If you're just chatting with it starting with "Hi", that's correct. The conversation remains in the KV cache as it grows gradually. But if you're posting code, writing drafts, or even small snippets of articles, etc in there it becomes a huge problem.
        • pclmulqdq 1y ago
          Usually, when people think about the prompt tokens for a chat model, the initial system prompt is the vast majority of the tokens and it's the same regardless for many usage modes. You might have a slightly different system prompt for code than you have for English or for chatting, but that is 3 prompts which you can permanently put in some sort of persistent KV cache. After that, only your specific request in that mode is uncached.
• fdfofeoijfeoie 1y ago
  [flagged]
• cycomanic 1y ago
  I was talking with a colleague the other day and we came to the conclusion that in our experience if you're using llms as a programming help models are really being optimised for the wrong things. At work I often compare locallly run 4-30B models against various GPTs (we can only use non-local models for few things, because of confidentiality issues). While e.g. GPT-4o gives better results on average, the chances of it making parts of the response up is high enough that one has to invest significant amount to check and iterate over results. So the difference in effort is not much lower compared to the low parameter models. The problem is both are just too slow to really iterate quickly, which makes things painful. I'd rather have a lower quality model (but with large context) that gives me near instant responses instead of a higher quality model that is slow. I guess that's not giving you the same headlines as the improved score on some evaluation.
• mikewarot 1y ago
  Imagine an FPGA big enough to hold the whole model in LUTS (and not RAM) with latches in appropriate places to keep race conditions in check. Even a 100 Mhz clock cycle would beat almost anything else in the world running it. Even if there's 500 stages of pipeline involved, you could still get 200,000 tokens per second for a single stream and have 499 streams ready for other uses. With an FPGA like that, you could translate all of the matrix multiplies and weights directly into binary logic, optimizing out every multiply or add of a zero bit. This alone could cut the number of gates and computations, and power consumption in half. Because you wouldn't need to throw data to/from RAM, you'd save a huge percentage of the usual latency and eliminate memory bandwidth issues. The effective equivalent memory bandwidth would likely be measured in exabytes per second. This is the type of compute load that would perfectly match a bit level systolic array.
  • saagarjha 1y ago
    You'd need an insanely big FPGA for this.
    • mikewarot 1y ago
      Thanks to gigabit SERDES links, it should be reasonably easy to send the vectors between chips if you need to distribute the work to fit available FPGA hardware. Note this could also be done if you're just emulating a systolic array on cheap hardware, like Raspberry pi picos, using the built-in PIOs to handle the much lower signal rates.
      • saagarjha 1y ago
        Gigabit is incredibly slow for training
• deleted 1y ago
  [deleted]
• shing3232 1y ago
  It's not expensive to run locally at all if you know how big of GPT4.
• b0a04gl 1y ago
  [dead]