6 ms·
Ask HN: Running production server on M1 mini?
We have a performance intensive application running on a AMD Epyc dedicated instance with 32 cores (our application is highly parallelizable)
We just noticed locally on our dev environment that our M1 is actually performing better performance wise (don’t ask me how).
We are now considering switching our production servers to M1 minis which is also offered by our cloud provider. Do you have any experience on running M1s / Mac in an production environment regarding stability / uptime etc?
Edit: it’s a Rust application which uses the Rayon crate. The application gets on average one request a minute which crunches some numbers for an average of 2 seconds - so it’s mostly idle. No disk IO.
- al2o3cr 5y agodon’t ask me how _You_ should be asking you how - there are lots of reasons why this could be happening and knowing which one is important if you're changing stuff. Based on a "highly parallelizable" application performing better on 8 cores than 32, I'd guess you're running out of something else: memory or disk bandwidth.
- gjsman-1000 5y agoIf his application is running better on 8 instead of 32, that reeks to me of a dependency on single-core performance somewhere. An example of this would be Minecraft, which performs worse on heavily-multi core systems compared to a few fast cores (like M1).
- alpaca128 5y agoAlso Dwarf Fortress, which runs tons of simulations but is a single-threaded 32bit application, which makes multithreaded performance and RAM beyond ~2GB meaningless.
- groundthrower 5y agoIt does not consume much memory but do lots of allocations/deallocations. No disc operations whatsoever.
- userbinator 5y agoIt sounds like you might be running into some sort of contention.
- gjsman-1000 5y agoI’d suggest investigating single core performance. If you have the money, buy an i9-12900K (slightly faster single-core than M1 but much hotter) and do some testing on that. If my theory is correct, performance will be even better.
- groundthrower 5y agoWe have examined that as well, last week we tried a AMD 5950X which has half the amount of cores but much better single core performance - the result was still at 60% of the Epyc performance
- gjsman-1000 5y agoWhat was the M1 % relative to your Epyc?
- groundthrower 5y agoRoughly 10% faster
- gjsman-1000 5y agoHN makes us wait for replies… so if we need to continue this further I’m open at muse.theses-0z@icloud.com . My next question would be if you ran the 12900K in dual-channel memory.
- CobaltFire 5y agoHave you investigated memory constraints? Ryzen is 2 channels; Epyc is 4-8 (depending on CPU). M1 has that stupidly fast/wide setup. If your Epyc is one of the 4 channel optimized SKUs or is only running in 4 channel mode, you would get pretty close to the quoted ratios on a memory bandwidth test. Correlation, not causation, but worth looking into.
- gwbas1c 5y agoProbably the hardest thing to clean up is a codebase where very complicated "optimizations" were built because someone didn't understand some very basic bottlenecks. I recently inherited an app that makes heavy use of Redis caching because someone didn't first try to optimizing SQL. The complexity that Redis caching adds is insane to maintain compared to spending a few minutes optimizing SQL. The original poster really needs to hook up a profiler. Also: having written lots of parallel code: Parallelization isn't a magic way to make things faster. If the codebase is breaking up tasks into lots of tiny tasks that run in parallel, there might be more overhead in parallelization than needed. Sometimes the fastest (performance and implementation) way to parallelize is to keep most of the codebase serial, but only parallelize at the highest level and never share data among operations.
- tmnstr85 5y agoThe old... anything but reviewing the execution plan approach... throw more vCPU's at it! Thank god for query store.
- Matthias247 5y ago+1. They should start profiling their application. If its running on alpine linux e.g. the default memory allocator is extremely bad and would degrade performance - but it could also be tons of other things. Taking random actions without understanding what the current bottleneck is will never be great long term.
- DarthNebo 5y agoFeels like your provisioned disk or IOPS could be the missing factor instead of core counts.
- groundthrower 5y agoWe do not do any disk operations at all
- DarthNebo 5y agoOhkk, just see how the network stats on both setups are. How are you testing the remote env? Is the traffic from local env or same cloud env?
- joshdev 5y agoHave you considered looking at Amazon for their ARM offering (Graviton)? I'd be hesistant to use M1 minis for a production workflow as they are not really production grade (lacking ECC memory, not sure how long they are rated to run at high CPU, lack of user replaceable disks, no RAID, etc...).
- p_papageorgiou 5y agoAny more details about the workload type of your application? Single threaded / Multithreaded / AI etc
- Someone1234 5y agoNo currently offered M1 Mini has redundant fail-over power or storage. Also, without knowing how your cloud provider has cooling setup it is unclear how well it will operate under heavy load for extended periods of time (blade servers are designed for that specific workload and have cooling solutions to match). My point is: If your workload is time critical, and you cannot afford downtime/outages then it may not be for you. If your workload can afford the time it would take to adopt a new M1 Mini when the old one dies, then maybe?
- bpicolo 5y agoDoes that include the AWS launched M1 instances last week?
- Someone1234 5y agoI don't know, Amazon's press releases don't talk very much about how the offering works under the hood.
- groundthrower 5y agoWell, it waits for calculations which take about 2 seconds to complete on average - the vast majority of the time it’s idle
- gjsman-1000 5y agoCould this maybe, someday, be simplified or implemented to a AWS Lambda function?
- groundthrower 5y agoWe have already tried that and use Lambdas as last point of backup if the other servers aren’t available- however the performance is about 1/50 compared to our current production server
- jagger27 5y ago
- gjsman-1000 5y agoNo - but I can give a few suggestions. One would be to look, if you haven’t, at MacStadium and what they’ve got there. You can get an M1 Mini there and it will be run by experts who know all about using M1 minis for servers. Considering your application is highly parallelizable, this would also make it easy to upgrade to the M1 Pro with double the performance cores down the line. Secondly, if your application is running better on M1, that reeks of an application which is somehow greatly benefiting at single-threaded performance somewhere, which the M1 excels and the Epyc is poor at. That probably needs some investigation.
- krageon 5y agoIf you do not understand why your performance profile is as it is, how do you know next week's patch won't make it perform better on AMD machines suddenly? You should understand your problem before you solve it.
- DrBenCarson 5y agoI don’t think any amount of historical or present-state analysis will shed light on next week’s patch. That being said, it would prepare one to better analyze next week’s patch,
- krageon 5y agoIf you know how your application performs and why, you are well equipped for estimating potential impact of patches and hardware. Obviously you still need to profile. In any case complete ignorance should not be the accepted approach.
- crankyadmin 5y agoKnowing what the development language is as well would help a lot - but the first thing you want to do is get some instrumentation on both your 32 Core AMD box and your M1 and compare the two. The M1 is very fast at doing certain things and your application may just be making good use of the M1 instruction set... both without knowing a bit more its difficult to tell.
- svacko 5y agoHow do you actually compare performance/bechmark the app - are you testing/benchmarking both prod and dev directly on the box itself? I'm thinking, there might be other infrastructure shielding the production like load balancers, proxies and other involved (observability/security tooling running and slowing the prod server) compared to accessing the dev on M1 directly..
- hvgk 5y agoThey are perfectly stable machines for running batch jobs. I have had one running a bunch of build automation as a jenkins slave for about 9 months now. Never skipped a beat. It just works and the thing is damn fast. If it’s doing it offline it’s probably cheaper to buy one and chuck it in your office than borrow one from a cloud provider. The ass end ones are really really really cheap. Much cheaper than just the CPU in an equivalent server machine. If they blow up, just mill down to the apple store and buy another one. Disclaimers of course: (1) it doesn’t have ECC RAM (2) it doesn’t have redundant power. We ignore (1) and solve (2) by running a prometheus node exporter on it and seeing if it disappears.
- jagger27 5y agoI'd be curious to know if your application scales even further onto an M1 Pro/Max. If that's the case, then something about Apple silicon makes your application scream.
- marban 5y agoI have one sitting on my desk that generates videos 24/7 and hasn't been down in a year. https://imgur.com/a/VAxpGCL https://imgur.com/a/VAxpGCL
- nobbis 5y agoWe use MacStadium's M1 mini servers for Metascan's photogrammetry batch processing. They've only been running a few months, but no downtime yet and I'm impressed with MacStadium's customer support, responsiveness, and price.
- tyingq 5y agoHave you tried using taskset or similar to force the production application onto fewer cores? Perhaps something about thread/ipc/locking overhead?
- usefulcat 5y agoThis is a good thing to check. However, do be aware that a lot of apps check the number of physical processors in the system rather than the CPU affinity mask for the process, even though the latter is almost always what they ought to be using.
- maksimpiriyev 5y agoI was thinking the same, as switching to M1 server,and also next version of M1 Mac minis will probably be x2 faster than the current one, so next year you buy mac mini will be double benefit :)
- awinter-py 5y agocan you describe the program? just broad strokes about language, framework, what kind of traffic it's receiving?
- groundthrower 5y agoIt’s written in Rust and uses Rayon to a big extent. It’s receiving data to crunch maybe once every 5 minutes
- awinter-py 5y agothe msg from dragontamer to set up a profiler seems like one approach to diagnose this and also from joshdev to try aws graviton, which is also arm based but potentially more suited for cloud hosting than an m1 if you figure this out, definitely write it up -- very cool tech blog topic, most people never get to debug cpu architecture firsthand
- errcorrectcode 5y agoTerrible idea. I supported a dozen Xserves back in the day. They were crap because they weren't designed for production use. They used nonswappable, commodity retail IDE drives not meant for 100% duty cycle operation. Fixed power supplies. Real enterprise servers were cheaper. Mac minis don't have redundant power or ECC. You might as well run a bunch of RPis or PICs. Get yourself some real enterprise servers or rent some via a VPS. Disclaimer: I use a Mac mini as my living room HTPC. I wouldn't run anything real on it. That's what I have a 96 thread EPYC virtualized box for.
- throwaway4good 5y agoMac OS X will require updates from time to time. Otherwise they will run 24/7 with no problem. You can consider building a hybrid setup where you leave the stuff that requires no / little downtime at your cloud provider.
- caeril 5y agoNo personal experience other than a slightly different experience running production services (involving money!) on another box without ECC DRAM (to save money!) and experiencing random permission flags flips and actual balance/amount flips. Only a small handful over many years, but it does happen, and when it matters, it REALLY matters. My advice is to always use ECC DRAM in production unless you're serving cat photos, porn, social media posts, or other societally useless applications. For anything that actually matters, please use ECC.
- groundthrower 5y agoYes this is one concern. Are you sure it was a result of using non ECC mem and how did you find out it was because of that?
- caeril 5y agoWe could never be absolutely sure, due to the true Heisenbug nature of the behavior, but after tons of code audits and the observation after reverse proxy traffic analysis that it only occurred on processing by the non-ECC hosts, and never on the ECC hosts, that it was the most likely culprit. The fact that the errors were single bit errors also strongly pointed in that direction.
- poulsbohemian 5y ago>our M1 is actually performing better performance wise I did performance analysis work for a long span of my career. While I'm reading between the lines of what you wrote, my first question is - what do you mean by performing better? As in, is it somehow able to process more of these tasks over a given timeframe? If so, I'd want to understand more about the workloads you are running to make sure it's a proper comparison. There's a whole lot more questions we need to answer here to understand the results you are seeing before we can have any kind of discussion of whether M1s would be "better."
- BizarroLand 5y ago"Dedicated instance" makes me think it's a cloud-based system. Are you actually getting what you're paying for?
- Andys 5y agoThere are now 3 generations of AMD Epyc. The latest is noticeably quicker than the older ones, and competitive with the M1.
- toast0 5y agoUnless it's changed recently, OS X has essentially no protection from synfloods. The TCP stack predates FreeBSD's syncache, and it was never ported. It doesn't have syncookies either. The pf port's synproxy stuff doesn't seem to work either. You've got to put some sort of firewall or something in front, don't let it accept tcp connections directly. You might be OK, but not great if you just set the listen queue really short; at least that should prevent the machine from falling over when it's flooded, but without syncookies, chances are you won't be able to make new connections either.
- skw-hn 5y agoScaleway is also providing M1 mac minis. The price is around 0.10€/hr which is quite cheaper. As per the stability, my scaleway m1 has never had any issues. works just fine for some CI.