7 ms·

In the last year, I have bought an M3 Ultra Mac Studio with 512 GB, a Macbook Pro M5 MAX with 128 GB and an RTX 6000 Pro. I have spent around $25k so far, not i

In the last year, I have bought an M3 Ultra Mac Studio with 512 GB, a Macbook Pro M5 MAX with 128 GB and an RTX 6000 Pro. I have spent around $25k so far, not including electricity. I figured worst case scenario I can sell them in the next year and only take a haircut as opposed to losing my entire investment. In comparison to just spending for tokens, the tokens would have been much cheaper and much much faster. I've been running against Gemma4:31b, Qwen3.5 and 3.6, and getting local LLMs to solve AMC 8/10 math questions and it's about 10-100x slower than just doing it online. When I tried it with ChatGPT late last year, it took about one night and $25 to solve about 1000 questions. Using my RTX 6000 and M3 Ultra and Gemma4:31b on both, it answered about 40 questions in 7 hours and I haven't checked how good the answer is yet. At 800 watts (600 for RTX and 200 for M3 Ultra) and running for 7 hours, it solved around 40 questions. At the very least I'm going to try to sell my M3 Ultra if I can find a reliable place to sell it without getting ripped off by scammers.

ahmadyan 5mo ago
If you are in the bay area, i'm happy to buy that M3 Ultra from you, i've been unsuccessfully looking for one and can't find any.
jon-wood 5mo ago
I’m not usually one to ask this because learning to do a thing can be fun, but why exactly have you spent 25 thousand dollars on getting an LLM someone else made to answer maths exam questions?
- freediddy 5mo ago
  It's just a project I'm working on. I'm working on projects where AIs are processing and classifying large amounts of data that would be a lot of work for humans to do.
  - wutwutwat 5mo ago
    I think of LLMs as being well equipped for handling dynamic data or adapting to unforeseen circumstances well (random code requests, website's ever changing layouts, typos, non-standard formatting in docs, groking out important info, etc), but math problems are be definition a very specific set of instructions to run, so is the overhead and "thinking" aspect of a LLM/AI even needed here? I'm genuinely curious, btw, I'm not asking sarcastically. Can't these math problems just be yanked from some test file and rapid fired directly at a gpu/compute unit?
    - freediddy 5mo ago
      > Can't these math problems just be yanked from some test file and rapid fired directly at a gpu/compute unit? Yes this is exactly what I'm doing. I isolated the actual math question, and then sent it to my two servers to process and that's what's taking 10m+ to return. I'm asking them to solve the question and return the full answer along with their steps. I care about correctness so taking time is okay but I can't use 10m per solution.
      - jachee 5mo ago
        Nono, parent was asking “They’re bad and inefficient at that, so why have an LLM do math? Why not just use some code and the CPU/GPU that’s already good and efficient at basic math?”
- hnuser123456 5mo ago
  Privacy and offline operation are valuable or non-negotiable in some cases, but the difference is pretty categorical between what can run on a single card and what can run on a DGX GB200 NVL72 cabinet. Doesn't mean it's not worth seeing how far local models can be pushed. Not every problem needs a senior engineer.
  - dylan604 5mo ago
    I know it's one of those "if you have to ask" situations, but curiosity got the better part of me. Here's the search assist response: "The DGX GB200 NVL72 AI server costs approximately $3 million per unit. This system includes 72 Blackwell GPUs and 36 Grace CPUs, making it one of the most powerful AI servers available." The search assist actually credited a source used with: https://www.tweaktown.com/news/98292/nvidias-new-gb200-superchip-costs-up-to-70-000-full-b200-nvl72-ai-server-3-million/index.html https://www.tweaktown.com/news/98292/nvidias-new-gb200-super... That $25k spend by GGGP seems like nothing in comparison. That's ~1/3 of one chip in that cabinet. God gawd I'm old and out of touch with modern AI data centers.
    - TheOtherHobbes 5mo ago
      It's The Circle of Computing Life. The pendulum swings between centralised mainframe timesharing-for-hire and desktop individuality. We've been in a centralised phase for longer than usual - first cloud everything, then AI - but at some point in the next decade prices will crash and a market will appear for personal, local intelligence.
      - brianwawok 5mo ago
        Depends how local models keep up
    - nl 5mo ago
      By comparison, the Colossus 1 data center had 32,000 GB200s (as well as 150,000 H100 GPUs, 50,000 H200 GPUs), and they are bringing another 110,000 GB200s online (although this might be Colossus 2?) There are bigger data centers than Colossus 1 around too. There is a reason NVidia is the most valuable company on the planet. https://en.wikipedia.org/wiki/Colossus_(supercomputer)#Current_progress https://en.wikipedia.org/wiki/Colossus_(supercomputer)#Curre...
- Retric 5mo ago
  That hardware is costing him ~1$/hour over 3 years. Presumably having it answer math questions was a tiny fraction of what he was using it for.
- nickthegreek 5mo ago
  The cost is obviously not that big of factor for OP as it might be for others. It's actually refreshing to hear the candid viewpoint that he expresses here.
  - freediddy 5mo ago
    25k is definitely a lot but I did the risk analysis and I figured worst case I would lose a 1000-2000 after a year of playing around with it, so I look at it more like renting (I'm going to keep the Macbook Pro no matter what since I needed a new one).
    - cronin101 5mo ago
      Nitpicking, but the worst case of spending $25k is unforeseen circumstances that write off the entire asset. I don’t think -$2000 is a conservative enough figure for standard depreciation either (a lot can happen in a year)
      - throwawaytea 5mo ago
        Either I don't understand the used apple market.. or I agree this is crazy. Someone spends $25k on new hardware, waits a year, and expects to sell it for $23k? Unless the ram issues save him, and cost of new goes up, I don't see how that was going to work.
        hnav 5mo ago
        This is the case in lots of markets, e.g. look at used cars, luxury goods and more. Some of it is driven by inflation/the rapid devaluation of the dollar. General and AI-adjacent compute in particular hasn't come down in price in a long while.
        cortesoft 5mo ago
        Well, Apple is literally not offering the M3 512GB studios currently. You can’t even back order one. They are selling on EBay for over $20k, used.
- iwontberude 5mo ago
  I’ve spent twice that on hosting movies and tv for Plex, so… I think they are worthy of my praise. What a healthy outlet for money.
  - root_axis 5mo ago
    You spent 50k for plex hosting? Why so expensive?
    - iwontberude 5mo ago
      Half a petabyte of RAID6 is the biggest line item, then the redundant 40gb networking and compute follow closely. I have a lot… too much even?
      - iamacyborg 5mo ago
        That’s a lot of blurays…
        iwontberude 5mo ago
        I found them dumped out in the “street” in a place ordained by law as public domain. So I just grabbed up the media and use it in private.
        marai2 5mo ago
        How many Blurays are we talking about?
        carlob 5mo ago
        500 TB/25 GB = 20.000 if some have more than one layer it could fewer but that's the order of magnitude
        bronson 5mo ago
- _the_inflator 5mo ago
  One year ago finetuned local LLMs had a significant edge over ChatGPT or Claude. Look up in YouTube all the DIY videos testing LLMs on their own machines with different setups. Remember: one year showed up to be a gigantic leap in regards to quality of results and innovation in the AI space. Agents weren't really a thing and vibe coding wasn't even invented as a term because the top notch tools at the time were lousy, with lovable being the frontrunner with its - in my view - sorry Tailwind recombination tool shaming AI to do the work. Then fall hit 2025 hit us, new year's eve and suddenly there was such a massive surge of innovation and competition with ChatGPT Codex suddenly showing up. Remember: one year ago many now commonly used tools weren't yet available like Nano Banana or Codex. "The 25k are so vast" - Yes, and no. For example, if the machine is bought for business usage I can deduct the costs from taxes. This roughly amount for 50% of the financial burden. So I jokingly use to say, that I pay only half the price for my Apple business machines. And yes, I am strict in this regard. Business means business. No private emails etc. nothing on my company computers. Maybe there are other options as well to reduce the financial expenses the dude mentions, but it doesn't seem so. I would also go for leasing, this way already the monthly payments can be deduced and I don't need to buy and maybe resell the machine. Apple is a luxury good. Without business usage or at least partly using it for business as well as private (mixed usage in tax reports) I wouldn't buy the devices or think twice. Apple under Cook evolved into a Gucci like luxury brand, that is more and more a rip off than quality delivered, especially considering the latest OS updates for Mac, iOS and iPad. Apple is a mess, following Microsoft Windows' footsteps happily, because the CEO is as has been correctly assessed, no product guy. But I stop with my rant here. Always try to use tax deduction as leverage for your computer expenses. Every citizen should invest in basic knowledge about that. Even a 10-20% professional usage for work (mixed usage) gives you a noticeable advantage over normal pay.
- girvo 5mo ago
  I didn't spend that much, only $6500 AUD for a GB10 based Asus GX10 which is even slower than OPs, but I spent that because it makes for a great learning platform. Theres not much else that lets me fiddle with 128GB of RAM for my graphics processor, and it's quite lovely to be able to run things as long as I like without worrying about my cloud instance being shut down. It's not financially a good idea: renting really does beat owning, and cloud beats both if you're only running inference on these machines. But I'm not just doing inference, and as a thing I can do silly stuff on to learn, it's hard to beat!
  - a1o 5mo ago
    When you say you are not just doing inference, you mean you are also training your own llms? I am curious what other things can be done.
    - girvo 5mo ago
      Fine tuning, and yeah training my own, experimenting with architectures and learning how it all works. Been a lot of fun
  - nostrebored 5mo ago
    $6500 AUD can get you a good chunk of B200 time on any of the GPU neoclouds :)
    - girvo 5mo ago
      Less than I expected, though! And I get to run this all through the night I do still use Vast and Runpod for things too, but it’s much nicer to test a fine tuning run here to make sure I’m in the ballpark I also did literally say “It's not financially a good idea, renting is better than owning” so I’m confused why I have two people telling me that Also it’s just far more fun to play with something tangible to me :)
  - andoando 5mo ago
    You could just rent a bare metal server with those specs
- wotsdat 5mo ago
  [dead]
- ActorNightly 5mo ago
  Because buying Macs is not about performance, its about feeling like you are rich. That money could have been spent on way more bang/buck performance in the form of a set of 4 graphics cards. Also I would probably put the odds 70:30 that Apple marketing is astroturfing on HN from the amount of posts about running llms on Macbooks, because in reality, the inference speed of any decent llm is unusable on a Macbook despite the ability to fit it into RAM.
  - topaz0 5mo ago
    Or it could have had way more bang/buck by feeding a family of real brains for a year or two
    - yard2010 5mo ago
      Excuse me for this comment, really, but I can't comprehend the absurdity, some people are buying GPUs when other people have no money for insulin so they literally die. I don't mean anything towards op or gp, quite the opposite I'm truly happy they have this kind of freedom, it must feel really nice, I just hate this game so much.
      - topaz0 5mo ago
        You and me both
    - deleted 5mo ago
      [deleted]
  - gbgarbeb 5mo ago
    40-80 tok/s is unusable to you? Ok. If you like having a box with 8-12 fans blasting hot air and noise into your office all day, nobody's stopping you.
    - ActorNightly 5mo ago
      If you are not being paid by Apple, I feel sorry for you. Cause that means you are so bought into the cult that you are delusional. the 40-80 tok/sec is only for initial prompt processing, and with the "medium" models, like Qwen3.6:27b. The actual token generation is in the 10 token/second Thats very slow. And your Macbook pro will stop being a LAP-top, because it will get very warm. Meanwhile, my 2x3090s happily crank out ~100 tok/sec generation. Oh and I can run 100 tok/sec on my phone as well, because I can just access ollama on my home desktop over ssh from termux.
- pjc50 5mo ago
  This is making me feel a lot better about my plan to lease a $25k EV simply because it's available at a massive discount. I'll probably end up using less electricity, too.
CamperBob2 5mo ago
How do you use the RTX 6000 with the Macs? Exo? I would think that would be pretty snappy if configured properly.
- freediddy 5mo ago
  This is on a separate Windows PC, I don't have it integrated with the Macs.
  - CamperBob2 5mo ago
    If you don't need cash right away, I'd wait until the M5 Ultra comes out and see how things shape up. There have been some early efforts aimed at combining the prefill performance of a GPU with the high throughput achievable with the Mac's unified memory architecture (see various YouTube videos by Ziskind and others, as well as https://old.reddit.com/r/LocalLLM/comments/1r6drpi/exo_cluster_with_rtx_5090_and_mac_studio/ https://old.reddit.com/r/LocalLLM/comments/1r6drpi/exo_clust... ). Point being, once the M5 Ultra is available, I suspect a lot of people will get very serious about making Macs work with RTX GPUs because that will yield an inference platform with a good bang:buck ratio. If so, you may find that your existing hardware is more powerful than it seems today. And it may be a lot more expensive to replace later if you sell it now.
arjie 5mo ago
All of these have appreciated in value. How much are you looking for the Ultra?
- freediddy 5mo ago
  I've seen a lot of sales on eBay for over $20k, but I don't know if I believe it. Plus the lack of seller protection and the prevalence of scams on eBay make me too hesitant to actually want to risk it so I don't know what to do haha
  - arjie 5mo ago
    Haha, yeah, it's about $23k or so. Should be twice the price what you bought it for if you got it last year. Tbh I don't know why. The RAM is large but the bandwidth and the compute isn't nearly enough. You can fit DeepSeek V3 on it quantized but inference is like 10 tok/s. Honestly, you'll be able to sell it locally for that in cash, and I would in your place. I saw your heat comments about the RTX 6000 Pro as well. I bought a few of them recently and I'm running 2 of them in a 2U case in a colo. You need a lot of active airflow to keep them cool. Mine range from 23 C to 80 C.
bethekind 5mo ago
Which of these has been the most productive for you? Sounds like you've enjoyed the RTX6000 the most?
- freediddy 5mo ago
  RTX 6000 is some-what obviously my fastest card but my biggest problem with the RT 6000 is the immense heat. The GPU itself is almost 200F and the exhaust from the fans itself is over 150F. I'm worried that my hard drives are going to fail. I was told that the GDDR7 is even hotter than the GPU which is surprising to me. After my last run, I'm going to wait for the new case I ordered to come in and cannibalize my kid's PC that we built beginning of this year to form an entirely separate computer. And then figure out better ways to deal with the heat, especially with summer coming up. I'll have to play around with undervolting and running vents directly outside my house to see if that helps.
  - vladgur 5mo ago
    From my failed and expensive affair with GPU mining 5 years ago, You can get a great heat dissipation outcome by using an open case with a lot of directed fans at the expense of a bit of dust and lots of noise
  - ericd 5mo ago
    I take it this wasn't the half-wattage Max Q version with blower fan?
  - dylan604 5mo ago
    Since you are not running realtime 3d grafix, could you put the card in an external chassis so the heat is not in the same box as the SSDs?
  - theYipster 5mo ago
    That's about what my OC'd and watercooled 4090 runs at. The cards are designed for it. Only problem I have is when sitting next to the computer under load -- I either have to open windows or blast the AC. Too bad I don't live in a cold climate -- that 60c heat output would come in handy :)
plasticsoprano 5mo ago
You'll probably make a profit by selling them today. I bought a M1 Max Studio with 64 GB last year off FB Marketplace for $1000 and today I'm seeing numerous 32 GB M1 Maxes for $1200-1500.
- freediddy 5mo ago
  Yes the prices on eBay for the Mac Studio are all over the place, but I've seen sales for over $20k. I don't know if I believe it but there's enough to make me think if I can sell it for that price it would be worth it, but eBay has basically no seller protection so I'm not willing to take that chance.
jmyeet 5mo ago
I looked into the M3 Ultra 512GB Mac Studio before it was discontinued and the as best as I could determine it just wasn't worth it... yet. The GFLOPS and memory bandwidth just arne't there even though it can hold a much larger model in memory. But the trend here is interesting. I think by 2030 you'll be able to buy fairly cheap hardware that is currently $10k+. I don't know what this does to the trillions invested in AI data centers because the next NVidia architecture after Blackwell will essentially half the value of purchased cards overnight. I'm not convinced Apple has yet pivoted the Mac Studio line towards this market and the expected M5 Ultras in Q3 2026 will likely be an incremental improvement rather than big leap forward but I'd like to be proven wrong.
- freediddy 5mo ago
  I agree that all these datacenter companies like Coreweave are investing billions in technology that has a very fast depreciation curve and I don't know how they will sustain income. The same goes for datacenters in space, what happens when those chips are obsolete? Will they sent astronauts to replace them or will they let them burn up and send new ones into orbit every year? I feel that the open weight models pale in comparison to the frontier models, and I believe that if the gap closes quickly, that the open weight vendors will stop releasing it for free.
  - Atotalnoob 5mo ago
    Data centers in space aren’t realistic. Higher radiation, space insulations, etc. Underwater data centers provide a lot of the same benefits and can (much more) easily be hauled to the surface
  - deleted 5mo ago
    [deleted]
iooi 5mo ago
I'll buy your macbook if you're trying to get rid of it!
- freediddy 5mo ago
  I'm keeping that one for sure, I love it!
wslh 5mo ago
I'm not really asking this from the perspective of whether I should buy hardware. I'm trying to understand the economics. The AI space is moving so fast that it is hard to know which conclusions are stable. After all the discussion around local models, is the practical conclusion still that API/frontier providers have a huge structural advantage because of datacenter hardware, high utilization, batching, optimized inference stacks, and perhaps strategic pricing? In a comparison like this, a $25k local setup versus buying tokens, what multiple are we really talking about? 10x? 100x? Or is it too workload-dependent to reduce to a single number? Has someone written a good breakdown that separates true infrastructure efficiency from temporary underpricing/subsidy? The part I'm trying to understand is less ideological (local vs. cloud) and more basic economics.
- freediddy 5mo ago
  The speed of results for an API call to ChatGPT is 10-100x faster than my local LLM. I haven't exactly quantified the results but I was getting results in a few seconds vs 10+ minutes for my local LLM. I'm going to do a deep dive this weekend and try to get better results, but it was staggering. I'll also do a deep dive on how to optimize my setup and see if I can get things to perform much quicker.
LarsDu88 5mo ago
Well if it makes you feel better those frontier LLMs are all technically taking a big loss, and they may all be in your shoes after a few years.
de6u99er 5mo ago
You definitely want to get rid of your M3 Ultra before the M5 Ultra get officially announced.
- digitaltrees 5mo ago
  Give the global memory shortage, the m5 will be both delayed and restricted to lower ram tiers, I dont think we will see a 512gb ram model until 2030
mountainriver 5mo ago
Running LLMs on Macs is still terribly slow. They simply lack the optimizations other platforms have. An RTX 6000 pro Blackwell is a pretty good card
- speedgoose 5mo ago
  A M3 ultra mac Studio can run models that do not fit in similarly priced computers with multiple Nvidia GPUs. And it will use a lot less electricity while still having good enough performance. Except the pre-filing perfs that are quite poor on the M3.
- ttoinou 5mo ago
  M5 pro 48GB should be good and future proof
  - thefounder 5mo ago
    If you buy Mac get at least 256GB ram otherwise just buy a bunch of nvidia cards. It really does not make sense otherwise if you are looking for performance / $. The mac (studio) is unique as it has more ram than the alternatives(I.e consumer nvidia cards or spark stuff) so it can fit bigger models but otherwise its performance is worse.
    - ttoinou 5mo ago
      I’d rather use a 128GB mac which is super flexible than always configuring a desktop with a huge GPU. Unless you’re doing others things with that gpu such as gaming, video, video AI etc.
  - plutokras 5mo ago
    Last time I looked into it, I realized how severely it's limited by memory bandwidth. Only the M5 Ultra compares to a dedicated graphics card, but it still falls short.
    - ttoinou 5mo ago
tpurves 5mo ago
>> find a reliable place to sell it without getting ripped off by scammers. This is a real problem and why I've just about given up on ebay or fb marketplace, esp for computers. If you are in Canada though sellit9.com is a great solution to having to deal with sketchy buyers.
tencentshill 5mo ago
No harm in listing it for $20k, and if it sells, that's an easy $5-10k for you.
deleted 5mo ago
[deleted]
tracker1 5mo ago
If you're in a decent sized city, you should be able to find a local buyer on Craigslist or FB Marketplace... Beyond that, for higher value, smaller items like your M3 Ultra, I would talk to your local police department and/or library to see if you can do the exchange there. Larger libraries usually have a police officer on site or nearby, and the PD office near you may also provide a "safe" exchange location... I'd bring a monitor/keyboard/mouse so you can demonstrate the system working properly. YMMV but between your nearest PD office and Library, you should be able to use one or the other for your exchange of goods/money. The biggest thing I've sold is a mid-range video card during late covid (I managed to get a better one via newegg shuffle) so I sold the old one (RX 5700XT -> RTX 2080) to make up the difference a bit. I just did the exchange at the Starbucks near me for that.
- thefounder 5mo ago
  Something is very wrong in some countries if you have to get police protection to sell a f* computer. I get it’s on the expensive side but still….
  - tracker1 5mo ago
    You don't have to... but it's a matter of a safe location for both parties. If it was more expensive, I'd probably work through a broker (like a car or house). The buyer doesn't know who the seller is, and vice-versa... the level of trust you can bear depends on how much you're willing to lose. My advice is only in that there are safe venues you can use to make such an exchange.
  - michaelt 5mo ago
    See e.g. https://www.murphytx.org/843/Safe-Exchange https://www.murphytx.org/843/Safe-Exchange https://www.ottawapolice.ca/en/community-safety-and-crime-prevention/safe-trade-zones.aspx https://www.ottawapolice.ca/en/community-safety-and-crime-pr... Police "safe trade zones" are basically a parking space outside a police station, with a sign.
  - fc417fc802 5mo ago
    Not really. Every country has a nonzero number of criminals. It's entirely a matter of the risk/reward tradeoff. A small consumer item over $10k is well into dangerous territory.
skiing_crawling 5mo ago
I got an RTX 6000 pro too. I like running locally, I've learned a lot more than if I had used an API and there's less worry about overspending tokens. I accidentally spent $100 on claude api in like 2 days because I didn't know what I was doing. The problem is that while one these gpus is a huge improvement over a laptop or a single 3090, you very quickly wish you had more. I would buy a second one, but I did the math and realized that with the current crop of models, 2 Blackwells doesn't buy me any new capability that I didn't have with one. So I would need a 3rd one. And when I buy a 3rd one I will feel like I want to running a higher quant, so then I will want a 4th.
- CamperBob2 5mo ago
  A pair of RTX6000 cards will give you a good performance boost due to tensor parallelism, though. I haven't tried the newest predictive quants but I see about 35 tps when running the 8-bit Qwen 3.6 27B model on one board and about 50 tps on two. Probably could come close to 100 tps on an optimized setup with the latest GGUFs. Also, the 4-bit quants of MiniMax 2.7 will run at 100 tps or so with two cards, which is pretty decent. It doesn't go any faster at all with 4 GPUs from what I've seen, so if you don't actively need 384 GB of VRAM, 2x RTX6000 is a good place to be.
  - skiing_crawling 5mo ago
    You can get 70-80 tps on qwen3.6-27b f16 with MTP on a single card
- arjie 5mo ago
  You can fit Deepseek 4 Flash on two with TP 2 and 6 different streams at 65k context. 150 tok/s
- Melatonic 5mo ago
  What kind of machine did you build around it ?
  - skiing_crawling 5mo ago
onlyrealcuzzo 5mo ago
> I figured worst case scenario I can sell them in the next year and only take a haircut as opposed to losing my entire investment. It's going to be a non-trivial haircut. This stuff depreciates pretty fast.
- michaelt 5mo ago
  Bizarrely, I brought a GPU new in Jun 2024, and there are sold ebay listings saying the used GPU is worth 4% more today. Of course, this is an unusual state of affairs; I see my GPU purchase as consumption, not investment.
digitaltrees 5mo ago
I have three m3 512gb units and want a fourth to run an exo set up. Like you, I am worried about scammers. Let’s discuss if you still want to sell. https://calendly.com/ryanwmartin/open-office-hours https://calendly.com/ryanwmartin/open-office-hours
wotsdat 5mo ago
[dead]
miki123211 5mo ago
This is, sadly, obvious and inevitable in retrospect. The two major drivers of inference costs are GPUs and electricity. You can't get cheaper GPUs, but you can make existing GPUs not sit idle, and you do that by utilizing them 24/7, processing user B's request when user A is thinking, and handling many requests in parallel, neither of which you can do as an individual. You can get cheaper electricity... by moving, and it's much easier to move your AI workload than to move yourself. This is a completely different dynamic than renting houses or apartments, as you can't really rent out the same house to different people at different times of day.
- cootsnuck 5mo ago
  Yea. LLM inference requires batch processing to have a shred of hope at being cost efficient. Batch processing requires a not so insignificant amount of scale (but probably not as much as people think). I'm very pro local models, but not to have parity with SoTA frontier models. Just contextually trained small models doing smaller specific tasks. Trying to run bigger LLMs for an individual user to do big tasks is not going to be a good time.
  - MichaelZuo 5mo ago
    Wasnt this pretty evident to pretty much anyone who knew even a bit about inferencing? Idk what people were thinking. I’ve never seen anyone offer a plausible way to sidestep batch processing for example.
  - miki123211 5mo ago
    I think people overstate the importance of batch processing and understate the importance of consistent workloads. A GPU is much like a hotel or an airplane. To make hotel and airplane economics work out, you need to keep your seats / rooms filled at all times. Because the number of spots is limited, an empty seat, room or request slot today is an opportunity forever wasted. Unlike hotels and airplanes, though, GPU requests are short-lived. This means you need somebody to provide you with useful work to do 24/7. If you're an individual who needs to sleep or a single-country business that works 9 to 5, most of your request processing opportunities will go to waste. You're much better off renting your GPU from somebody who can provide it with useful work 24/7, and hence get much better per-request prices.
Craighead 5mo ago
I'd buy that Mac studio m3
jma24 5mo ago
Better sell it fast before the M5 ones come out.
daemin 5mo ago
Given that the tokens are being subsidised by a couple orders of magnitude, would it still be as cost effective long term?
throwaway2037 5mo ago
> if I can find a reliable place to sell it without getting ripped off by scammers. I don't follow this last part. What is the scam they try to run?
- gbgarbeb 5mo ago
  A buyer can claim they never received it or that the box was empty, thus receiving a full refund. For something listed at $25k I would not list on eBay at all. eBay corporate will pocket $3400 in fees and will also dock you local taxes on the $25k.
balls187 5mo ago
I’ve had the best luck selling in Craigslist. Every other platform has been sub par.
Melatonic 5mo ago
How are you using the 6000 with a Mac ?
jrmg 5mo ago
If you run it in the winter the electricity is “free” because it’s replacing a portion of whatever else heats your house.
- brianwawok 5mo ago
  Only feee if you are using 100% efficient resistance heating. A heat pump can be up to 400% efficient. And natural gas is a lot cheaper per dollar. So for most people… it saves a tiny bit but not free.
beernet 5mo ago
Yep, the great theoretical promise of local models remains theoretical, no matter how much die hard-engineers want to push it...Who would have thought, right?
vessenes 5mo ago
I don't think this changes the final conclusion - but have you considered calculating against depreciation -- i.e. figuring out how much your M3 ultra is worth today, and only charging yourself for the delta? In my mind you might even have made money on the hardware.
glaslong 5mo ago
Same as it ever was with "cloud" eh? The advantages of small-scale on-prem are never cost or quality, they are strictly privacy and sovereignty. No one can rug pull you with a week of Claude regressions. No one has access to your sensitive data.
kappuchino 5mo ago
I have two of the M3s due testing of models at work and with exo I can run decent quantization with 1 millon tokens for memory and derailment tests. Slow? Yes, but ... private. Unconditionally. And recently with https://github.com/antirez/ds4 https://github.com/antirez/ds4 one can use just one system to a very, very decent speed and ttft for chat inference. Again, private.

ahmadyan 5mo ago

If you are in the bay area, i'm happy to buy that M3 Ultra from you, i've been unsuccessfully looking for one and can't find any.

jon-wood 5mo ago

I’m not usually one to ask this because learning to do a thing can be fun, but why exactly have you spent 25 thousand dollars on getting an LLM someone else made to answer maths exam questions?

freediddy 5mo ago
It's just a project I'm working on. I'm working on projects where AIs are processing and classifying large amounts of data that would be a lot of work for humans to do.
- wutwutwat 5mo ago
  I think of LLMs as being well equipped for handling dynamic data or adapting to unforeseen circumstances well (random code requests, website's ever changing layouts, typos, non-standard formatting in docs, groking out important info, etc), but math problems are be definition a very specific set of instructions to run, so is the overhead and "thinking" aspect of a LLM/AI even needed here? I'm genuinely curious, btw, I'm not asking sarcastically. Can't these math problems just be yanked from some test file and rapid fired directly at a gpu/compute unit?
  - freediddy 5mo ago
    > Can't these math problems just be yanked from some test file and rapid fired directly at a gpu/compute unit? Yes this is exactly what I'm doing. I isolated the actual math question, and then sent it to my two servers to process and that's what's taking 10m+ to return. I'm asking them to solve the question and return the full answer along with their steps. I care about correctness so taking time is okay but I can't use 10m per solution.
    - jachee 5mo ago
      Nono, parent was asking “They’re bad and inefficient at that, so why have an LLM do math? Why not just use some code and the CPU/GPU that’s already good and efficient at basic math?”
hnuser123456 5mo ago
Privacy and offline operation are valuable or non-negotiable in some cases, but the difference is pretty categorical between what can run on a single card and what can run on a DGX GB200 NVL72 cabinet. Doesn't mean it's not worth seeing how far local models can be pushed. Not every problem needs a senior engineer.
- dylan604 5mo ago
  I know it's one of those "if you have to ask" situations, but curiosity got the better part of me. Here's the search assist response: "The DGX GB200 NVL72 AI server costs approximately $3 million per unit. This system includes 72 Blackwell GPUs and 36 Grace CPUs, making it one of the most powerful AI servers available." The search assist actually credited a source used with: https://www.tweaktown.com/news/98292/nvidias-new-gb200-superchip-costs-up-to-70-000-full-b200-nvl72-ai-server-3-million/index.html https://www.tweaktown.com/news/98292/nvidias-new-gb200-super... That $25k spend by GGGP seems like nothing in comparison. That's ~1/3 of one chip in that cabinet. God gawd I'm old and out of touch with modern AI data centers.
  - TheOtherHobbes 5mo ago
    It's The Circle of Computing Life. The pendulum swings between centralised mainframe timesharing-for-hire and desktop individuality. We've been in a centralised phase for longer than usual - first cloud everything, then AI - but at some point in the next decade prices will crash and a market will appear for personal, local intelligence.
    - brianwawok 5mo ago
      Depends how local models keep up
  - nl 5mo ago
    By comparison, the Colossus 1 data center had 32,000 GB200s (as well as 150,000 H100 GPUs, 50,000 H200 GPUs), and they are bringing another 110,000 GB200s online (although this might be Colossus 2?) There are bigger data centers than Colossus 1 around too. There is a reason NVidia is the most valuable company on the planet. https://en.wikipedia.org/wiki/Colossus_(supercomputer)#Current_progress https://en.wikipedia.org/wiki/Colossus_(supercomputer)#Curre...
Retric 5mo ago
That hardware is costing him ~1$/hour over 3 years. Presumably having it answer math questions was a tiny fraction of what he was using it for.
nickthegreek 5mo ago
The cost is obviously not that big of factor for OP as it might be for others. It's actually refreshing to hear the candid viewpoint that he expresses here.
- freediddy 5mo ago
  25k is definitely a lot but I did the risk analysis and I figured worst case I would lose a 1000-2000 after a year of playing around with it, so I look at it more like renting (I'm going to keep the Macbook Pro no matter what since I needed a new one).
  - cronin101 5mo ago
    Nitpicking, but the worst case of spending $25k is unforeseen circumstances that write off the entire asset. I don’t think -$2000 is a conservative enough figure for standard depreciation either (a lot can happen in a year)
    - throwawaytea 5mo ago
      Either I don't understand the used apple market.. or I agree this is crazy. Someone spends $25k on new hardware, waits a year, and expects to sell it for $23k? Unless the ram issues save him, and cost of new goes up, I don't see how that was going to work.
      - hnav 5mo ago
        This is the case in lots of markets, e.g. look at used cars, luxury goods and more. Some of it is driven by inflation/the rapid devaluation of the dollar. General and AI-adjacent compute in particular hasn't come down in price in a long while.
      - cortesoft 5mo ago
        Well, Apple is literally not offering the M3 512GB studios currently. You can’t even back order one. They are selling on EBay for over $20k, used.
iwontberude 5mo ago
I’ve spent twice that on hosting movies and tv for Plex, so… I think they are worthy of my praise. What a healthy outlet for money.
- root_axis 5mo ago
  You spent 50k for plex hosting? Why so expensive?
  - iwontberude 5mo ago
    Half a petabyte of RAID6 is the biggest line item, then the redundant 40gb networking and compute follow closely. I have a lot… too much even?
    - iamacyborg 5mo ago
      That’s a lot of blurays…
      - iwontberude 5mo ago
        I found them dumped out in the “street” in a place ordained by law as public domain. So I just grabbed up the media and use it in private.
        marai2 5mo ago
        How many Blurays are we talking about?
        carlob 5mo ago
        500 TB/25 GB = 20.000 if some have more than one layer it could fewer but that's the order of magnitude
        bronson 5mo ago
_the_inflator 5mo ago
One year ago finetuned local LLMs had a significant edge over ChatGPT or Claude. Look up in YouTube all the DIY videos testing LLMs on their own machines with different setups. Remember: one year showed up to be a gigantic leap in regards to quality of results and innovation in the AI space. Agents weren't really a thing and vibe coding wasn't even invented as a term because the top notch tools at the time were lousy, with lovable being the frontrunner with its - in my view - sorry Tailwind recombination tool shaming AI to do the work. Then fall hit 2025 hit us, new year's eve and suddenly there was such a massive surge of innovation and competition with ChatGPT Codex suddenly showing up. Remember: one year ago many now commonly used tools weren't yet available like Nano Banana or Codex. "The 25k are so vast" - Yes, and no. For example, if the machine is bought for business usage I can deduct the costs from taxes. This roughly amount for 50% of the financial burden. So I jokingly use to say, that I pay only half the price for my Apple business machines. And yes, I am strict in this regard. Business means business. No private emails etc. nothing on my company computers. Maybe there are other options as well to reduce the financial expenses the dude mentions, but it doesn't seem so. I would also go for leasing, this way already the monthly payments can be deduced and I don't need to buy and maybe resell the machine. Apple is a luxury good. Without business usage or at least partly using it for business as well as private (mixed usage in tax reports) I wouldn't buy the devices or think twice. Apple under Cook evolved into a Gucci like luxury brand, that is more and more a rip off than quality delivered, especially considering the latest OS updates for Mac, iOS and iPad. Apple is a mess, following Microsoft Windows' footsteps happily, because the CEO is as has been correctly assessed, no product guy. But I stop with my rant here. Always try to use tax deduction as leverage for your computer expenses. Every citizen should invest in basic knowledge about that. Even a 10-20% professional usage for work (mixed usage) gives you a noticeable advantage over normal pay.
girvo 5mo ago
I didn't spend that much, only $6500 AUD for a GB10 based Asus GX10 which is even slower than OPs, but I spent that because it makes for a great learning platform. Theres not much else that lets me fiddle with 128GB of RAM for my graphics processor, and it's quite lovely to be able to run things as long as I like without worrying about my cloud instance being shut down. It's not financially a good idea: renting really does beat owning, and cloud beats both if you're only running inference on these machines. But I'm not just doing inference, and as a thing I can do silly stuff on to learn, it's hard to beat!
- a1o 5mo ago
  When you say you are not just doing inference, you mean you are also training your own llms? I am curious what other things can be done.
  - girvo 5mo ago
    Fine tuning, and yeah training my own, experimenting with architectures and learning how it all works. Been a lot of fun
- nostrebored 5mo ago
  $6500 AUD can get you a good chunk of B200 time on any of the GPU neoclouds :)
  - girvo 5mo ago
    Less than I expected, though! And I get to run this all through the night I do still use Vast and Runpod for things too, but it’s much nicer to test a fine tuning run here to make sure I’m in the ballpark I also did literally say “It's not financially a good idea, renting is better than owning” so I’m confused why I have two people telling me that Also it’s just far more fun to play with something tangible to me :)
- andoando 5mo ago
  You could just rent a bare metal server with those specs
wotsdat 5mo ago
[dead]
ActorNightly 5mo ago
Because buying Macs is not about performance, its about feeling like you are rich. That money could have been spent on way more bang/buck performance in the form of a set of 4 graphics cards. Also I would probably put the odds 70:30 that Apple marketing is astroturfing on HN from the amount of posts about running llms on Macbooks, because in reality, the inference speed of any decent llm is unusable on a Macbook despite the ability to fit it into RAM.
- topaz0 5mo ago
  Or it could have had way more bang/buck by feeding a family of real brains for a year or two
  - yard2010 5mo ago
    Excuse me for this comment, really, but I can't comprehend the absurdity, some people are buying GPUs when other people have no money for insulin so they literally die. I don't mean anything towards op or gp, quite the opposite I'm truly happy they have this kind of freedom, it must feel really nice, I just hate this game so much.
    - topaz0 5mo ago
      You and me both
  - deleted 5mo ago
    [deleted]
- gbgarbeb 5mo ago
  40-80 tok/s is unusable to you? Ok. If you like having a box with 8-12 fans blasting hot air and noise into your office all day, nobody's stopping you.
  - ActorNightly 5mo ago
    If you are not being paid by Apple, I feel sorry for you. Cause that means you are so bought into the cult that you are delusional. the 40-80 tok/sec is only for initial prompt processing, and with the "medium" models, like Qwen3.6:27b. The actual token generation is in the 10 token/second Thats very slow. And your Macbook pro will stop being a LAP-top, because it will get very warm. Meanwhile, my 2x3090s happily crank out ~100 tok/sec generation. Oh and I can run 100 tok/sec on my phone as well, because I can just access ollama on my home desktop over ssh from termux.
pjc50 5mo ago
This is making me feel a lot better about my plan to lease a $25k EV simply because it's available at a massive discount. I'll probably end up using less electricity, too.

CamperBob2 5mo ago

How do you use the RTX 6000 with the Macs? Exo? I would think that would be pretty snappy if configured properly.

freediddy 5mo ago
This is on a separate Windows PC, I don't have it integrated with the Macs.
- CamperBob2 5mo ago
  If you don't need cash right away, I'd wait until the M5 Ultra comes out and see how things shape up. There have been some early efforts aimed at combining the prefill performance of a GPU with the high throughput achievable with the Mac's unified memory architecture (see various YouTube videos by Ziskind and others, as well as https://old.reddit.com/r/LocalLLM/comments/1r6drpi/exo_cluster_with_rtx_5090_and_mac_studio/ https://old.reddit.com/r/LocalLLM/comments/1r6drpi/exo_clust... ). Point being, once the M5 Ultra is available, I suspect a lot of people will get very serious about making Macs work with RTX GPUs because that will yield an inference platform with a good bang:buck ratio. If so, you may find that your existing hardware is more powerful than it seems today. And it may be a lot more expensive to replace later if you sell it now.

arjie 5mo ago

All of these have appreciated in value. How much are you looking for the Ultra?

freediddy 5mo ago
I've seen a lot of sales on eBay for over $20k, but I don't know if I believe it. Plus the lack of seller protection and the prevalence of scams on eBay make me too hesitant to actually want to risk it so I don't know what to do haha
- arjie 5mo ago
  Haha, yeah, it's about $23k or so. Should be twice the price what you bought it for if you got it last year. Tbh I don't know why. The RAM is large but the bandwidth and the compute isn't nearly enough. You can fit DeepSeek V3 on it quantized but inference is like 10 tok/s. Honestly, you'll be able to sell it locally for that in cash, and I would in your place. I saw your heat comments about the RTX 6000 Pro as well. I bought a few of them recently and I'm running 2 of them in a 2U case in a colo. You need a lot of active airflow to keep them cool. Mine range from 23 C to 80 C.

bethekind 5mo ago

Which of these has been the most productive for you? Sounds like you've enjoyed the RTX6000 the most?

freediddy 5mo ago
RTX 6000 is some-what obviously my fastest card but my biggest problem with the RT 6000 is the immense heat. The GPU itself is almost 200F and the exhaust from the fans itself is over 150F. I'm worried that my hard drives are going to fail. I was told that the GDDR7 is even hotter than the GPU which is surprising to me. After my last run, I'm going to wait for the new case I ordered to come in and cannibalize my kid's PC that we built beginning of this year to form an entirely separate computer. And then figure out better ways to deal with the heat, especially with summer coming up. I'll have to play around with undervolting and running vents directly outside my house to see if that helps.
- vladgur 5mo ago
  From my failed and expensive affair with GPU mining 5 years ago, You can get a great heat dissipation outcome by using an open case with a lot of directed fans at the expense of a bit of dust and lots of noise
- ericd 5mo ago
  I take it this wasn't the half-wattage Max Q version with blower fan?
- dylan604 5mo ago
  Since you are not running realtime 3d grafix, could you put the card in an external chassis so the heat is not in the same box as the SSDs?
- theYipster 5mo ago
  That's about what my OC'd and watercooled 4090 runs at. The cards are designed for it. Only problem I have is when sitting next to the computer under load -- I either have to open windows or blast the AC. Too bad I don't live in a cold climate -- that 60c heat output would come in handy :)

plasticsoprano 5mo ago

You'll probably make a profit by selling them today. I bought a M1 Max Studio with 64 GB last year off FB Marketplace for $1000 and today I'm seeing numerous 32 GB M1 Maxes for $1200-1500.

freediddy 5mo ago
Yes the prices on eBay for the Mac Studio are all over the place, but I've seen sales for over $20k. I don't know if I believe it but there's enough to make me think if I can sell it for that price it would be worth it, but eBay has basically no seller protection so I'm not willing to take that chance.

jmyeet 5mo ago

I looked into the M3 Ultra 512GB Mac Studio before it was discontinued and the as best as I could determine it just wasn't worth it... yet. The GFLOPS and memory bandwidth just arne't there even though it can hold a much larger model in memory. But the trend here is interesting. I think by 2030 you'll be able to buy fairly cheap hardware that is currently $10k+. I don't know what this does to the trillions invested in AI data centers because the next NVidia architecture after Blackwell will essentially half the value of purchased cards overnight. I'm not convinced Apple has yet pivoted the Mac Studio line towards this market and the expected M5 Ultras in Q3 2026 will likely be an incremental improvement rather than big leap forward but I'd like to be proven wrong.

freediddy 5mo ago
I agree that all these datacenter companies like Coreweave are investing billions in technology that has a very fast depreciation curve and I don't know how they will sustain income. The same goes for datacenters in space, what happens when those chips are obsolete? Will they sent astronauts to replace them or will they let them burn up and send new ones into orbit every year? I feel that the open weight models pale in comparison to the frontier models, and I believe that if the gap closes quickly, that the open weight vendors will stop releasing it for free.
- Atotalnoob 5mo ago
  Data centers in space aren’t realistic. Higher radiation, space insulations, etc. Underwater data centers provide a lot of the same benefits and can (much more) easily be hauled to the surface
- deleted 5mo ago
  [deleted]

iooi 5mo ago

I'll buy your macbook if you're trying to get rid of it!

freediddy 5mo ago
I'm keeping that one for sure, I love it!

wslh 5mo ago

I'm not really asking this from the perspective of whether I should buy hardware. I'm trying to understand the economics. The AI space is moving so fast that it is hard to know which conclusions are stable. After all the discussion around local models, is the practical conclusion still that API/frontier providers have a huge structural advantage because of datacenter hardware, high utilization, batching, optimized inference stacks, and perhaps strategic pricing? In a comparison like this, a $25k local setup versus buying tokens, what multiple are we really talking about? 10x? 100x? Or is it too workload-dependent to reduce to a single number? Has someone written a good breakdown that separates true infrastructure efficiency from temporary underpricing/subsidy? The part I'm trying to understand is less ideological (local vs. cloud) and more basic economics.

freediddy 5mo ago
The speed of results for an API call to ChatGPT is 10-100x faster than my local LLM. I haven't exactly quantified the results but I was getting results in a few seconds vs 10+ minutes for my local LLM. I'm going to do a deep dive this weekend and try to get better results, but it was staggering. I'll also do a deep dive on how to optimize my setup and see if I can get things to perform much quicker.

LarsDu88 5mo ago

Well if it makes you feel better those frontier LLMs are all technically taking a big loss, and they may all be in your shoes after a few years.

de6u99er 5mo ago

You definitely want to get rid of your M3 Ultra before the M5 Ultra get officially announced.

digitaltrees 5mo ago
Give the global memory shortage, the m5 will be both delayed and restricted to lower ram tiers, I dont think we will see a 512gb ram model until 2030

mountainriver 5mo ago

Running LLMs on Macs is still terribly slow. They simply lack the optimizations other platforms have. An RTX 6000 pro Blackwell is a pretty good card

speedgoose 5mo ago
A M3 ultra mac Studio can run models that do not fit in similarly priced computers with multiple Nvidia GPUs. And it will use a lot less electricity while still having good enough performance. Except the pre-filing perfs that are quite poor on the M3.
ttoinou 5mo ago
M5 pro 48GB should be good and future proof
- thefounder 5mo ago
  If you buy Mac get at least 256GB ram otherwise just buy a bunch of nvidia cards. It really does not make sense otherwise if you are looking for performance / $. The mac (studio) is unique as it has more ram than the alternatives(I.e consumer nvidia cards or spark stuff) so it can fit bigger models but otherwise its performance is worse.
  - ttoinou 5mo ago
    I’d rather use a 128GB mac which is super flexible than always configuring a desktop with a huge GPU. Unless you’re doing others things with that gpu such as gaming, video, video AI etc.
- plutokras 5mo ago
  Last time I looked into it, I realized how severely it's limited by memory bandwidth. Only the M5 Ultra compares to a dedicated graphics card, but it still falls short.
  - ttoinou 5mo ago

tpurves 5mo ago

>> find a reliable place to sell it without getting ripped off by scammers. This is a real problem and why I've just about given up on ebay or fb marketplace, esp for computers. If you are in Canada though sellit9.com is a great solution to having to deal with sketchy buyers.

tencentshill 5mo ago

No harm in listing it for $20k, and if it sells, that's an easy $5-10k for you.

deleted 5mo ago

[deleted]

tracker1 5mo ago

If you're in a decent sized city, you should be able to find a local buyer on Craigslist or FB Marketplace... Beyond that, for higher value, smaller items like your M3 Ultra, I would talk to your local police department and/or library to see if you can do the exchange there. Larger libraries usually have a police officer on site or nearby, and the PD office near you may also provide a "safe" exchange location... I'd bring a monitor/keyboard/mouse so you can demonstrate the system working properly. YMMV but between your nearest PD office and Library, you should be able to use one or the other for your exchange of goods/money. The biggest thing I've sold is a mid-range video card during late covid (I managed to get a better one via newegg shuffle) so I sold the old one (RX 5700XT -> RTX 2080) to make up the difference a bit. I just did the exchange at the Starbucks near me for that.

thefounder 5mo ago
Something is very wrong in some countries if you have to get police protection to sell a f* computer. I get it’s on the expensive side but still….
- tracker1 5mo ago
  You don't have to... but it's a matter of a safe location for both parties. If it was more expensive, I'd probably work through a broker (like a car or house). The buyer doesn't know who the seller is, and vice-versa... the level of trust you can bear depends on how much you're willing to lose. My advice is only in that there are safe venues you can use to make such an exchange.
- michaelt 5mo ago
  See e.g. https://www.murphytx.org/843/Safe-Exchange https://www.murphytx.org/843/Safe-Exchange https://www.ottawapolice.ca/en/community-safety-and-crime-prevention/safe-trade-zones.aspx https://www.ottawapolice.ca/en/community-safety-and-crime-pr... Police "safe trade zones" are basically a parking space outside a police station, with a sign.
- fc417fc802 5mo ago
  Not really. Every country has a nonzero number of criminals. It's entirely a matter of the risk/reward tradeoff. A small consumer item over $10k is well into dangerous territory.

skiing_crawling 5mo ago

I got an RTX 6000 pro too. I like running locally, I've learned a lot more than if I had used an API and there's less worry about overspending tokens. I accidentally spent $100 on claude api in like 2 days because I didn't know what I was doing. The problem is that while one these gpus is a huge improvement over a laptop or a single 3090, you very quickly wish you had more. I would buy a second one, but I did the math and realized that with the current crop of models, 2 Blackwells doesn't buy me any new capability that I didn't have with one. So I would need a 3rd one. And when I buy a 3rd one I will feel like I want to running a higher quant, so then I will want a 4th.

CamperBob2 5mo ago
A pair of RTX6000 cards will give you a good performance boost due to tensor parallelism, though. I haven't tried the newest predictive quants but I see about 35 tps when running the 8-bit Qwen 3.6 27B model on one board and about 50 tps on two. Probably could come close to 100 tps on an optimized setup with the latest GGUFs. Also, the 4-bit quants of MiniMax 2.7 will run at 100 tps or so with two cards, which is pretty decent. It doesn't go any faster at all with 4 GPUs from what I've seen, so if you don't actively need 384 GB of VRAM, 2x RTX6000 is a good place to be.
- skiing_crawling 5mo ago
  You can get 70-80 tps on qwen3.6-27b f16 with MTP on a single card
arjie 5mo ago
You can fit Deepseek 4 Flash on two with TP 2 and 6 different streams at 65k context. 150 tok/s
Melatonic 5mo ago
What kind of machine did you build around it ?
- skiing_crawling 5mo ago

onlyrealcuzzo 5mo ago

> I figured worst case scenario I can sell them in the next year and only take a haircut as opposed to losing my entire investment. It's going to be a non-trivial haircut. This stuff depreciates pretty fast.

michaelt 5mo ago
Bizarrely, I brought a GPU new in Jun 2024, and there are sold ebay listings saying the used GPU is worth 4% more today. Of course, this is an unusual state of affairs; I see my GPU purchase as consumption, not investment.

digitaltrees 5mo ago

I have three m3 512gb units and want a fourth to run an exo set up. Like you, I am worried about scammers. Let’s discuss if you still want to sell. https://calendly.com/ryanwmartin/open-office-hours https://calendly.com/ryanwmartin/open-office-hours

wotsdat 5mo ago

[dead]

miki123211 5mo ago

This is, sadly, obvious and inevitable in retrospect. The two major drivers of inference costs are GPUs and electricity. You can't get cheaper GPUs, but you can make existing GPUs not sit idle, and you do that by utilizing them 24/7, processing user B's request when user A is thinking, and handling many requests in parallel, neither of which you can do as an individual. You can get cheaper electricity... by moving, and it's much easier to move your AI workload than to move yourself. This is a completely different dynamic than renting houses or apartments, as you can't really rent out the same house to different people at different times of day.

cootsnuck 5mo ago
Yea. LLM inference requires batch processing to have a shred of hope at being cost efficient. Batch processing requires a not so insignificant amount of scale (but probably not as much as people think). I'm very pro local models, but not to have parity with SoTA frontier models. Just contextually trained small models doing smaller specific tasks. Trying to run bigger LLMs for an individual user to do big tasks is not going to be a good time.
- MichaelZuo 5mo ago
  Wasnt this pretty evident to pretty much anyone who knew even a bit about inferencing? Idk what people were thinking. I’ve never seen anyone offer a plausible way to sidestep batch processing for example.
- miki123211 5mo ago
  I think people overstate the importance of batch processing and understate the importance of consistent workloads. A GPU is much like a hotel or an airplane. To make hotel and airplane economics work out, you need to keep your seats / rooms filled at all times. Because the number of spots is limited, an empty seat, room or request slot today is an opportunity forever wasted. Unlike hotels and airplanes, though, GPU requests are short-lived. This means you need somebody to provide you with useful work to do 24/7. If you're an individual who needs to sleep or a single-country business that works 9 to 5, most of your request processing opportunities will go to waste. You're much better off renting your GPU from somebody who can provide it with useful work 24/7, and hence get much better per-request prices.

Craighead 5mo ago

I'd buy that Mac studio m3

jma24 5mo ago

Better sell it fast before the M5 ones come out.

daemin 5mo ago

Given that the tokens are being subsidised by a couple orders of magnitude, would it still be as cost effective long term?

throwaway2037 5mo ago

> if I can find a reliable place to sell it without getting ripped off by scammers. I don't follow this last part. What is the scam they try to run?

gbgarbeb 5mo ago
A buyer can claim they never received it or that the box was empty, thus receiving a full refund. For something listed at $25k I would not list on eBay at all. eBay corporate will pocket $3400 in fees and will also dock you local taxes on the $25k.

balls187 5mo ago

I’ve had the best luck selling in Craigslist. Every other platform has been sub par.

Melatonic 5mo ago

How are you using the 6000 with a Mac ?

jrmg 5mo ago

If you run it in the winter the electricity is “free” because it’s replacing a portion of whatever else heats your house.

brianwawok 5mo ago
Only feee if you are using 100% efficient resistance heating. A heat pump can be up to 400% efficient. And natural gas is a lot cheaper per dollar. So for most people… it saves a tiny bit but not free.

beernet 5mo ago

Yep, the great theoretical promise of local models remains theoretical, no matter how much die hard-engineers want to push it...Who would have thought, right?

vessenes 5mo ago

I don't think this changes the final conclusion - but have you considered calculating against depreciation -- i.e. figuring out how much your M3 ultra is worth today, and only charging yourself for the delta? In my mind you might even have made money on the hardware.

glaslong 5mo ago

Same as it ever was with "cloud" eh? The advantages of small-scale on-prem are never cost or quality, they are strictly privacy and sovereignty. No one can rug pull you with a week of Claude regressions. No one has access to your sensitive data.

kappuchino 5mo ago

I have two of the M3s due testing of models at work and with exo I can run decent quantization with 1 millon tokens for memory and derailment tests. Slow? Yes, but ... private. Unconditionally. And recently with https://github.com/antirez/ds4 https://github.com/antirez/ds4 one can use just one system to a very, very decent speed and ttft for chat inference. Again, private.