13 ms·
For applications that aren't latency sensitive my company has found that Apple hardware inference is far less expensive than the competition when calculating fo
by leetharris 3y ago
For applications that aren't latency sensitive my company has found that Apple hardware inference is far less expensive than the competition when calculating for electricity usage. I wish they would make a cloud offering.
- crooked-v 3y agoYou could always rack mount some Mac Minis.
- ninkendo 3y agoIt’s sad that racking and maintaining your own physical hardware is becoming such a lost art… I appreciate the up-front simplicity of cloud offerings as much as anyone, but there’s something to be said for owning your own hardware and avoiding the continual rent payments you’re sending the cloud providers. The wisdom is that cloud providers are better at infra than you, and that the economies of scale make it better to piggy back on what they’re doing, but… AWS is the most profitable part of Amazon for a reason. They’re overcharging you.
- cherryswitch 3y agoSometimes the flexibility and time savings is worth the added upfront cost. Similar to how companies like to hire consultants or lease office space. Being able to walk away is better for short term, because companies value profits in the short term.
- ilc 3y agoFor most orgs: AWS is not overcharging IMHO. When you look at the cost of the hardware + hosting. Yes, it certainly looks and feels that way. But if you've dealt with corporate IT, and had to deal with 3-6 month lead times on getting hardware, or politics to get your hands on hardware to get stuff done. AWS is cheap. It gives you velocity. If your company is large enough that it can offer the elasticity of resources that Amazon offers or even 1/4 of it... and you have an IT org that will let it happen. Yes, AWS is a waste. But with AWS... when a project dies, you can wipe its costs out, people won't hold onto hardware so they have hardware for the next project, etc... Trust me. I've been IT, I can spec and build rack systems. I am a software dev. And I've been a dev most all my career. For 90%+ of orgs... they don't have the maturity and skills to handle that type of infra without substantially distracting from their primary business.
- ttt3ts 3y agoI find at AWS I am always wasting engineer time optimizing dumb things. Do you know how much a TB of ram costs? Or 10TB of blazing fast NVME costs? Less than $5K. How much does that cost at AWS?! This is not even considering bandwidth which AWS overcharges soo much. Yet I waste time. Also, maintaining servers is not hard at a proper data center. It is often more hands off than the migrations cloud providers force on their customers.
- wredue 3y agoYeah. Also, to be honest, we still have process and approvals to get through to spin up AWS stuff. It’s not like process disappears just cause you’re not on your own hardware. Infra is still its own team with its own budgets poking and prodding at every damn turn for every little thing till rejecting your requests, you escalate and then have a 4 week battle over needing the space.
- JCharante 3y agoReally? At my old company each division had its own budget and account and then you’d be an iam member of an account and spin up services under that account, but there’s no central authority to send a request to. There were tools to analyze underused services across all accounts (like EC2 instances constantly under 2% cpu load meaning they were flagged for downsizing if possible).
- ilc 3y agoThen your company should have stayed on prem. If you are paying the price of being on prem, which is really lack of ability to provision and de-provision infra quickly. There's little point to the cloud, unless you just have no infra to begin with (small companies). I'm in a small firm now. I can't imagine having an approval process to spin up a few instances to run my tests and spin them down after. That'd be silly.
- sitkack 3y agoI work in cloud and you sound like a shill.
- solardev 3y agoWhere does this stop? Do you produce your own electricity? Farm your own food? Make your own silverware and shoes? Sometimes it's just easier to outsource the things you don't want to (or aren't good at) doing yourself. If I wanted to host a website, sure, I can build a server out of parts and negotiate with my ISP and get a business pipe and handle all caching and such. Or like I can pay a provider $5/mo and get better performance and reliability with no management overhead. Yeah, maybe over 5 years I'd save more money doing it myself... but it's not worth the time. If I wanted to generate a photo or a dozen, or a few paragraphs of text, that's like a few cents worth of cloud AI. Maybe low single-digit dollars. Or I could spend thousands on fat GPUs or a Macbook, spend forever training it, and still end up with a sub-par result. AWS is profitable not just because they're overcharging you but because they are providing a hugely useful service for millions of businesses that don't want to deal with that infrastructure themselves, any more than they'd want to manage their own plumbing or electrical grid or roads and bridges leading to their office. DIY makes sense if you're doing it as a hobby or if your scale is so big that you would incur significant savings to in-house it, but for millions of small and medium businesses, it's just not the most practical approach. Nothing wrong with that. I mean, it's like saying development is such a lost art... why hire a dev if you can learn to code yourself? Sure, but not everyone wants to, can, or has time.
- Spooky23 3y agoDepends on the financials. My employer has generated its own electricity and steam for decades. For a small business - different story.
- ClimaxGravely 3y agoI hate to say this but it has gotten to the point where I'm starting to farm some of my own food (I'm starting to get fed up with produce quality issues in my hometown). Still haven't started on the silverware or shoes yet. I do agree with you though. If you are a non-tech company or a company that lacks the human resources you might as well go with the cloud.
- solardev 3y ago
- sbarre 3y agoThe question I've often heard asked when deciding on build vs buy (which can apply to cloud vs. bare metal) is: Are we in the business of building, maintaining and operating <thing to build> or do we want to buy that as a service instead and focus on our actual core business? There's more to the cost of building and operating than just the hard costs. Retaining good modern IT talent is getting harder and harder - and I'm not even talking about salaries.. You need a whole department including strong leaders who can hire, train, and lead the right people, etc.. This is something most companies wouldn't even know where to start with.
- hamandcheese 3y agoI don't think you need a whole department to rack a few minis and run some buildkite agents on them.
- londons_explore 3y agoYou need one good motivated sysadmin. But if you can motivate that same sysadmin to spend his skills on something more directly benefiting your company, then you should still buy it in.
- c0pium 3y agoNow your business depends on that one person, congratulations. Hope they don’t realize that their skills are better suited to working at AWS/Azure etc. for 2x the money. Which they are.
- londons_explore 3y agoIt's not core stuff. If the one sysadmin leaves, you buy in a solution.
- c0pium 3y agoI mean, fair enough I guess. Why play the OpEx vs CapEx game when you can just pay both.
- whalesalad 3y agoYou’ve never had to drive to the colo at 3am huh?
- ninkendo 3y agoNo, I had proper OOB access and could do any kind of power cycling or remote console access from my desktop.
- hnfong 3y agoNever seen a hardware failure, I see.
- ninkendo 3y agoThe snark is getting a bit tiring. No, plenty of hardware failures but we had redundancy. Our availability wasn’t dependent on how fast I cold drive to the data center at 3am. Drive dies, who cares, there are plenty of hot spares, we’ll deal with it during business hours. Server dies, who cares, there are lots of them. We have remote hands too, so simple hardware replacement is something you can get cheap onsite labor to do. If your operations halt until some poor sysadmin has to drive to the colo, you are absolutely doing it wrong.
- ClimaxGravely 3y agoI'm a former build engineer and used to do everything on prem and I gotta say I miss it (not being a build engineer, the on prem experience). Since those days pretty much every company I've worked at moved their CI/CD to the cloud and I gotta say it feels so much slower, even when working from home. I remember twice switching from an in-house jenkins/teamcity/whatever type of CI to Azure devops and the thing I remember the most was how much longer it took a build to complete as well as the massively longer time downloading a build from Azure vs from within the office. Even when working from home the on-prem stuff was faster. The thing is, the build/devops teams seem to be about the same size in both cases. It's just kind of worse in pretty much every case when we do CI in the cloud. Notes - My experiences are largely for game development so the build times and artifact sizes can be quite large. - I've only ever had CI/CD experience with Azure, I've not tried other cloud providers - Since this is game development and we're using CI downtime is more acceptable than other cases. That said, I don't remember much downtime when I was working as a build engineer. I have seen periods of 1-2 hours of downtime once in a blue moon but then again I've seen that with Azure. In both cases it wasn't so much the setup but a build script deployment issue. Also being able to cool off in the rack room when it's a hot day is always a treat :)
- nre 3y agoInterestingly enough I was actually discussing this with a friend (who works in enterprise IT) the other day. Basically rack servers are purpose build for the task, with hot swappable components, redundant power/storage, multiple NICs, ECC, remote management, and so on. They come with enterprise support and can be easily maintained in the field. Meanwhile a Mini cluster is literally a bunch of mini pcs in a rack, and idk if Apple even supports this kind of industrial use. While it's a quality product the Mini isn't really designed for the datacenter.
- xienze 3y ago> and idk if Apple even supports this kind of industrial use. While it's a quality product the Mini isn't really designed for the datacenter. I think they know of it and tacitly approve of this use case, as evidenced by the Mac Mini having the same form factor for ages. They’re well aware that a lot of people use Minis (and Studios now) in data centers, and that the Mini footprint is sort of “standardized” at this point.
- nxobject 3y agoIIRC, Apple did indeed have a server SKU for the Intel Mac mini at some point.
- ytch 3y agoIt's called Xserve: https://en.wikipedia.org/wiki/Xserve#Intel_Xserve https://en.wikipedia.org/wiki/Xserve#Intel_Xserve But since Apple discontinued Xserve and macOS Server, they seems like don't care about this business anymore.
- astrodust 3y agoThere was also a "server" model Mini but it was very short lived and was basically a regular Mini with the "Server" software pre-installed, something that you could just throw in via the App Store with one click anyway.
- GravityLab 3y agoI have a base-model m1 Mac Mini and it's a beast. I'm using it as my build/deploy server and also as a back-end server (for running jobs) for the prototype I'm working on. I also do development on it when I want to use my big monitor rather than my laptop. And I listen to music and run Cookie Clicker at the same time while doing development. Got three databases up and running too. It's a beast. I'd definitely consider self-hosting with a few Mac Minis, that would be fun and they're really cute, sleek devices too. I paid $650 for it and consider it a great deal. Definitely should've gotten it with more than 8gb of ram but I got it to try it out and haven't yet really needed to upgrade to a unit with more memory.
- api 3y agoApple really should license the M chip IP to someone to make a server chip out of it, or do it themselves. It's money on the table for them and would not cannibalize their Mac business at all. It's a very nice core.
- mschuster91 3y agoFully agree. It's time for some serious competition not just against Intel/x86 but also in the ARM space.
- dzhiurgis 3y agoIDK about ML part, but equivalent performing Ryzen mini pc cost me 3x less over m1 macbook (yes im aware you get more with macbook)
- solardev 3y agoIt's easy to outperform Apple Silicon on pure power, but what about efficiency & heat? (like FLOPS/watt or whatever). Does anything else come close yet?
- blacksmith_tb 3y agoThat matters a lot in a laptop, but not so much in a 1U rack? Not that datacenters love heat, but the competition isn't extra hot, it's just hotter than Apple Silicon?
- solardev 3y agoI guess it depends on what you're doing with them? If you're running them 24/7 to train or model something, the energy costs might add up. Even if you're not, having more efficient chips might mean more data centers don't need as complex cooling equipment.
- ska 3y agoA 1U rack is a small machine, but a few hundred of them+ in a constrained space is a different story. At larger scales, moving electricity in and heat out are usually the defining factors.
- whalesalad 3y agoAmazon has Apple servers for rent. There is also https://macminicolo.net https://macminicolo.net and https://www.macminivault.com https://www.macminivault.com
- nathancahill 3y agoI'm so happy MacMiniColo is still around. I had projects hosted there when they were just getting started. They've stayed true to their mission. I love that their website still feels like Wordpress 3.0.
- hamandcheese 3y agoThey aren't accepting new Mac minis for colocation. Which means if you want to go from, say, 8gb to 16gb of ram then that's an extra $90 a month for the privilege.
- jzombie 3y agoI am using https://www.macminivault.com https://www.macminivault.com for colocation.
- cyberge99 3y agoThere’s also MacStadium.
- deleted 3y ago[deleted]
- smoldesu 3y agoFor applications that aren't latency sensitive, I run inference on a free 4 core Ampere server from Oracle. Once you ditch the "fast" prerequisite, a lot of hardware becomes viable.
- garciasn 3y agoWe went with a M2 Studio with maxed out RAM because we simply cannot get reliable GPU availability with cloud providers and for $6000 (with tax) we can have the equivalent VRAM of ~2 80GB GPUs instead of paying $5/hr for the pleasure.
- ttt3ts 3y agoWith a 70B param model how many tokens/second? Did the math and assuming 100% util and equal performance (which is certainly not the case) payback on your Mac is 9 months...
- garciasn 3y agoYou need to pay for dedicated because they’re generally unavailable in the moment. So it’s more like 45 days, if we’re only talking about a single GPU—but we’re talking about ~2x.
- ttt3ts 3y agoHow many tokens a second? Really trying to figure out viability. 4x NVIDIA A100 at lamda labs is $4.40 an hour and I really have not had an issue getting them.
- hnfong 3y agoNote this is a M1, not M2. https://www.reddit.com/r/LocalLLaMA/comments/16o4ka8/running_ggufs_on_an_m1_ultra_is_an_interesting/ https://www.reddit.com/r/LocalLLaMA/comments/16o4ka8/running...
- ttt3ts 3y agoThanks! Ya, I opted for dual 3090 for my workstation (keeping full LLM in VRAM is crit) was wondering what lift was for M2. OP implied that there were workloads where it out competes renting in terms of cost. Was hoping it was true for something than a single user interactive session (which can be done a lot cheaper)
- chmod775 3y agoNowadays GPUs factory defaults try to squeeze out the last bit of performance for a huge cost in power. You can run them at half the power usage and only lose a fraction of the performance - at least in gaming. Try for AI tasks.
- macwebcomputing 3y agomacweb.com is also a solution :-)