26 ms·
AWS Graviton vs. M1 vs. M1 Pro Node.js Benchmarks
- jeffbee 5y ago> Right now i have to build and deploy using a cloud server or an older Intel MacBook Is it really weird to me how little-known the art of cross-compilation is. You can target x86 from an ARM build box, or the other way around.
- BackBlast 5y agoIn this case, if there are binary dependencies besides node.js. They would be rebuilt on the box when 'npm install' is run. I've never seen a setup for cross compilation in this case. Though I don't think you really need it.
- mbreese 5y agoIf the application and test suite are built in Node.js, I'm really not sure why there are architecture issues to begin with. Especially if the test suite ran without alteration on the ARM machines, then I don't see why you'd get different results with running the tests on Intel vs ARM. > I installed node v16 on both my MacBook and the cloud instance, installed our app and then ran the full end to end test suite. Given this statement, it doesn't seem like there should be any cross-platform issues at all here (even in included npm packages). It doesn't sound like any of the tests are arch specific. But I don't use Node.js very often, so maybe I am missing something?
- tecleandor 5y agoThe app itself is not ARM specific, but the base Docker image and its dependencies (Debian + libraries + nodejs binaries...) are platform specific, so they can't build that image on a different platform than their target. See that, for the same tag (version) you have images with different hashes (hence content) for different architectures: https://hub.docker.com/_/node?tab=tags https://hub.docker.com/_/node?tab=tags
- mbreese 5y agoThat makes no sense to me. It doesn’t matter what the base image is — the application and test suite are platform agnostic. So, why not build two different Docker images that are the same except for arch? It seems that the problem isn’t the difference between dev and production environments, but the way Docker is being used. Sure, run an arch specific test before moving something to production, that makes sense. But for development, just use the Docker image that matches the dev arch. This seems like a much easier problem than the author is making it out to be. Before Docker, I don’t think this would have been an issue.
- tecleandor 5y agoIt makes sense if you take in account that you're packaging the whole stack in a container. It's the same than when you make 'golden images' for a VM with Packer or whatever tool is available: you end up with a platform specific image. But you aren't far fetched at all with that idea you have, and can be done using a Docker multi-stage build: You build all your application with whatever platform you have (either locally or for CI), and then you import your application layer from an ARM64 image in one side, and from and AMD64 in the other. We aren't still very used to multi-platform development environments, so the tooling isn't still perfect (at all).
- paxys 5y agoAnd how exactly do you test the cross-compiled bits before pushing them to a production server?
- jolux 5y agoYou should already be using continuous integration, so it should already be moot.
- paxys 5y agoContinuous integration also needs a server to run on. The point is, if you have a dev laptop running M1 ARM and a prod server running Intel x86, you need something in the middle to build/test on unless you want to cross-compile on your laptop and yeet it to production.
- throwawaylinux 5y agoQEMU with TCG can do wonders. It's not perfect but it's surprisingly fast and emulates a good amount of instructions, can do SMP, etc.
- pabs3 5y agoPerhaps on a staging server?
- catlifeonmars 5y agoEasy: push it to a preproduction server first.
- ungamedplayer 5y agoBoot in qemu, or have a test instance.
- hughrr 5y agoAlso you can run x86-64 and arm64 VMs side by side with UTM on the M1’s. That’s what I am doing. Although I don’t need to use the x86-64 one these days as Debian/arm64 is fine for my needs.
- itsthecourier 5y ago
- TheGoodBarn 5y agoHow much did it cost to run your Graviton research for this article? Just curious for how beefy it is
- paxys 5y agoThey mentioned it was a c6g.metal instance, which will cost you $1000-$1500 per month.
- koolba 5y agoAnd the test was for under a minute so even with additional start up and tear down time, it’s peanuts to run this type of thing using on-demand instances.
- deleted 5y ago[deleted]
- lIIIllllIIII 5y agoThey should have compared it with a cluster of M1s to the value of $1000-1500 a month.
- ketzo 5y agoMan, the M1 Macs are the first piece of tech in a while that I’ve felt myself actively pining over. They just seem… really goddamn fast. Everybody talks about it so glowingly. Hoping I can pick one up soon. I figure the Pro is probably the right move for a developer workload, although I do like the size of the Air.
- christkv 5y agoI got a MacBook Pro with the max cpu and it’s made such a difference in my day to day Kotlin work and the battery time is stunning considering how high the cpu usage is when working in IntelliJ
- andy_ppp 5y agoI wish they would make an Air with a 15 inch screen, could be incredible - the one that Ive wanted all the machines to become…
- deleted 5y ago[deleted]
- raihansaputra 5y agoYes yes yes. I think Apple is scared to cannibalize the 16" Pro, although it's already quite differentiated. Pro is quite focused on media workloads rather than pure performance/dev workloads. I can see some dev workloads needing the pro, but most would be great on the base chip. With the level of performance on the M1, I'll take the battery life and less weight on the Air over more cores. I hope the next gen base chip (M2?) will have more display output and more RAM, and the MBAir line up will have a 15/16" size (with the Pro speakers).
- tokamak-teapot 5y agoLots of corporate users here who love their Macs but don't need the media features. 13" Airs are lovely, but plenty of us would go for a bigger screen if it was an option.
- 5y ago
- JohnHaugeland 5y agoI love how every time someone wants to say the M1 is fast, they actively avoid comparing it to Intel or AMD, except in some nonsensical way like "single core performance"
- mwint 5y agoGo buy an M1 machine, use it for a week, and return it. Except you won't return it, because you'll recognize it's amazing beyond its benchmarks. And single core performance can't be discounted - turns out, it's often a huge component of user experience. Lots of software out there that hasn't been, or can't be, multithreaded.
- dboreham 5y agoIt only runs macos, so I'll definitely return mine.
- mattl 5y agoAnd Linux and OpenBSD
- smoldesu 5y agoThere are so many issues with the current implimentation that I think saying "it runs" is a bit of a stretch. I'll give it to you when you have a hardware-accelerated desktop, until then "it runs" Linux as much as a toaster SOC can "run" Linux.
- pabs3 5y agoThat is being worked on too: https://rosenzweig.io/blog/asahi-gpu-part-1.html https://rosenzweig.io/blog/asahi-gpu-part-1.html https://rosenzweig.io/blog/asahi-gpu-part-2.html https://rosenzweig.io/blog/asahi-gpu-part-2.html https://rosenzweig.io/blog/asahi-gpu-part-3.html https://rosenzweig.io/blog/asahi-gpu-part-3.html https://rosenzweig.io/blog/asahi-gpu-part-4.html https://rosenzweig.io/blog/asahi-gpu-part-4.html
- jtbayly 5y agoThe price comparisons are odd/wrong, because it appears he simply calculated the AWS price for a year of that machine. But on the one hand he didn’t need to run the AWS machine for a year. And on the other hand, he owns the Mac’s for life, as he pointed out. If you wanted to calculate the cost of having the AWS machine always available to you like the Mac, you’d probably at least want to use three years, which is a fairly normal timeframe for a business to write down the cost of a machine.
- RussianCow 5y agoGiven that the use cases for a Mac and an EC2 instance are very different, I don’t think there is a “right” way to compare them.
- ceeplusplus 5y agoThe benchmark itself has some issues; namely, the unit tests are run on a single thread but the end to end tests are merely parallelized one per thread. I would guarantee he's not getting 100% CPU utilization on the M1 machine if those tests are anything like the ones I see. BDD/E2E tests have a lot of waiting for buttons to load on pages, clicking/moving the cursor, and waiting for backend results. A proper test would simply run _all_ the BDD tests in parallel to properly max out the CPU and let the OS handle the thread switching.
- Seirdy 5y agoAre there any non-Apple ARM laptops that can reliably run "normal" Linux distros (e.g. Fedora, Alpine, Debian derivatives)? Currently the only options on my radar are the Pinebook (and Pinebook Pro) and MNT Reform.
- mbreese 5y agoAFAICT, The Pinebook Pro can run "normal" distros. Fedora and Manjaro are supported directly, and Armbian has Ubuntu images. It also looks like Arch is somewhat supported. Now, reliably on the other hand... that's another story. I still have hardware issues surrounding booting, but getting an OS installed isn't an issue anymore.
- dangus 5y agoIt doesn’t really matter the processor architecture, Apple has never been a good choice for Linux. Sure you can get it to work sometimes but it’s just not ideal.
- pjmlp 5y agoIt was when they shortly played with MkLinux project.
- fulafel 5y agoThere have been some Macbook models that have been very good Linux laptops for their time (if you don't mind the price and kb layout).
- goosedragons 5y agoAnd there's been some models that are pretty much nightmares (basically 2016-2020) even years later. I think their point stands. Some have been okay and others near useless.
- fulafel 5y agoTo clarify, I meant to make a point against the "has never been a good choice" claim. I agree the relative rarity of good Linux Mac models is not ideal, hopefully the current Arm model Linux activity will bring improvement.
- fulafel 5y ago(More specifically this is Graviton2) Interesting that the bang for the buck is not that far off for a Mac instance, the calculator says a M1 instance costs a bit under 1000 per month.
- azinman2 5y ago> Given i develop on a ARM Mac, i’d like to deploy to an ARM server. This line of reasoning must be scaring Intel.
- deleted 5y ago[deleted]
- neogodless 5y agohttps://www.tomshardware.com/news/intel-amd-4q-2021-2022-market-share-desktop-notebook-server-x86 https://www.tomshardware.com/news/intel-amd-4q-2021-2022-mar... While this is AMD-focused, you can scroll down to see "Server Shipments by CPU Type" and get a feel for how Intel is doing currently, and how quickly the market share changes. (Pretty good, and slowly.) Of course, not being on top in efficiency should light a fire under the market leader.
- bdcravens 5y ago> Given i develop on a ARM Mac, i’d like to deploy to an ARM server.... Amazon have been trying to solve this issue (and many others) by developing a line of ARM servers call Graviton." Graviton (2018) came before the M1 (2020).
- scns 5y agoThe posthumous reality distortion field.
- tomerbd 5y ago99% of reviews of M1 talk about video editing and music production. As a programmer I care more about RAM , storage, no reflections display , lightweight laptop long lasting replaceable battery great keyboard trackpad and self service repair. Give me mediocre cpu with all these call it Z1 and I'm happy programmer.
- grapeskin 5y agoVideo editors and music producers are using far more ram and storage than virtually any programmer out there, and loads of them are traveling with their work and draining batteries faster than anyone else. Asking them whether a laptop’s hardware is good is like asking a power lifter whether a gym has good equipment.
- timc3 5y agoNot a lot of music producers do use that much RAM (speaking as someone that has run multi-machine music making setup before). Its only the large sample libraries that do and you need multiple instances running to really eat it up. All our devs need than my current music making setup, and I am thinking of going to 128Gb because I do dev work around video.
- rfoo 5y agoThe point is M1 is extremely fast at video editing because it has DSAs designed for these workloads. So using them as a general purpose workload proxy is just bad.
- ungamedplayer 5y agoThe powerlifter only cares about the squat rack. They don't give a hoot about your leg press machine. Very good analogy.
- matwood 5y ago> only cares about the squat rack This is all anyone should care about, but I digress...
- chaboud 5y ago“ Lets start with the M1 Mac mini. It’s a nice baseline as it’s the Mac i was developing on up untill recently. The Mac mini completed the tests in 45.13 seconds across 8 cores. ………… The AWS Graviton instance (c6g.metal to be specific) delivered a score of 14.63 seconds across its 64 cores. Around 68% faster than the Mac mini.” “Faster” would be a matter of rate. So if we have times and we know D = R*T, we’d have: (D/T2)/(D/T1) as the speed up of R2, which is just (T1/T2), or 3.08. If we were to say how much “faster” c6g.metal is than a Mac Mini, wouldn’t we say 208% rather than 68% if the speed of the Mac Mini is our baseline? Am I missing something?
- lostmsu 5y agoJust never use percentages to show ratio. Graviton instance is approximately 3 times faster.
- chaboud 5y agoI completely agree. X times the rate of Y is unambiguous.
- samhw 5y agoI highly recommend this blog post, which makes the same point among many (great) others: https://sled.rs/perf.html https://sled.rs/perf.html > When we speak about comparative metrics, it is also important to avoid saying commonly misunderstood things like “workload A is 15% slower than workload B”. Instead of saying “faster” it is helpful to speak in terms of latency or throughput, because both may be used to describe “speed” but they are in direct opposition to each other. Speaking in terms of relative percentages is often misleading. What does A (90) is 10% lower than B (100) mean if we don’t know their actual values? Many people would think that B is 1.1 * A, but in this case, 1.1 * 90 = 99. It is generally better to describe comparative measurements in terms of ratios rather than relative percentages. > The phrase workload A is 20% slower than workload B can be more clearly stated as workload A was measured to have a throughput of 4:5 that of workload B. Even though many people will see that and immediately translate it to “80%” in their heads, the chances of improperly reasoning about the difference are lower.
- qbasic_forever 5y agoGraviton 2 instances are really amazing in my experience. I can't believe every other major cloud player (Google, Azure) still don't have any ARM instances available. Get with the times folks.
- hencoappel 5y agoGiven Google have made their TPUs for machine learning and Tensor processors for phones it may be possible they are investigating it. AMD is killing it though from the server side so it's hard to justify your own CPUs right now.
- reilly3000 5y agoNothing them is stopping GPC from buying Ampere servers and renting time on them. AWS made a great move in the space to get ahead of it, but there are totally viable commercial options: https://amperecomputing.com/ https://amperecomputing.com/
- datatrashfire 5y agoJust remember, containers don't just run everywhere. Your team will need to devote time to understanding multi-arch images and builds. If you're a python/data science team, I hope you have someone on your team that actually knows how to debug these problems. Otherwise you'll be using your $3k Mac as a thin client to an x86 resource to do your development work.
- daibo 5y agoNothing wrong with that though
- datatrashfire 5y agoYea, I personally think local development is fraught with problems that remote resources solve quite well.
- frant-hartm 5y agoI personally think remote development is fraught with problems that don't exist on local.
- kergonath 5y agoBoth can be true at the same time…
- ungamedplayer 5y agoWorse case scenario too.
- datatrashfire 5y agoYea it can be picking your own poison in some respects. My preferred workflow is something like spinning up a dedicated dev VM within your provider, install a standard image, then use a local text editor that's saving files over ssh or sftp. I have found virtually no one has problems with "well I can't get this to run on my machine and here are the n undocumented quirks". Everyone is using a homogeneous environment with local IDE help. This is in many respects similar to the idea of GitHub codespaces and other remote dev environment tooling that has started to pop up. While I haven't used any of them, I think the general principle makes a lot of sense. You can also develop on very light local hardware, while taking advantage of an elastic resource depending on what you're doing.
- outcoldman 5y agoThe company that I have built develop mostly for amd64 (Linux), because our customers run things on amd64. Mostly Kubernetes and OpenShift. Kubernetes does exist on arm64, but not OpenShift. So to test/develop I have to go sometimes to my MBP 16" i9 2019 (top specs). I use 2 5K LG monitors. And just development experience is not that great on Intel, especially after switching to M1 Max, when I don't hear fans, don't see glitches in window manager, don't see constant CPU throttling, because laptop of hot running GPU and CPU to run those 2 5K monitors.
- timthorn 5y agoOpenShift 4.10 has just been released and adds Arm support.
- deleted 5y ago[deleted]
- systemvoltage 5y agoI guess I’m confused about who this article is for. Graviton AFAIK is available for cloud compute on AWS. M1 MacBook Pro is a developer’s machine. There are different aspects of performance that’s important depending on who is inquiring. 1) AWS cares about performance/watt (operating expenses) and performance/$ (capital expense) 2) As a user, if I provision an EC2 instance, I have no interest in hardware costs or power consumption. I just care about performance/$ (EC2 instance pricing). So raw performance is basically moot point here. There are some non-linear concerns about scaling (Amdahl’s law), but generally, compute boils down to even more abstracted picture something like: requests/$, etc. that’s specific to your workload and company.
- dx034 5y agoYou can get Mac minis at many cloud providers (expensive at AWS, cheap at Hetzner). They weren't designed as servers, but are good value for many server use cases.
- systemvoltage 5y agoDoesn't have ECC RAM support.
- meragrin_ 5y ago> I guess I’m confused about who this article is for. "Given i develop on a ARM Mac, i’d like to deploy to an ARM server."
- pjmlp 5y agoEmbrace cloud native development, so enjoying CLI, vi, Emacs and not wanting to replicate the missing part of what used to be UNIX development experience, development servers with thin clients.
- qeternity 5y agoI wish the unit tests had been run separately. We’re comparing an 8 core 8gb M1 Mini to server with 8x as many cores and 16x as much ram. Of course the multithreaded tests run faster…
- bufferoverflow 5y agoBut not 8x faster.
- jwilliams 5y agoIt's worth highlighting that the M1 series has efficiency and performance cores. For our test suite, the efficiency cores struggle and can tank the whole test run. We set our node test suite (jest) to use the number of performance cores, not the total count (which is the default). That actually sped up the test process, particularly on the M1 which is 50/50 efficiency and performance cores. It's less pronounced on the M1 Pro, but still a benefit.
- nilsmagnus 5y agoI realize that this is not a very detailed benchmark, but it would be interesting to see a more head-to-head benchmark of the graviton2 vs m1 cpu. Running on 64 cores induces a lot of overhead vs running on 8/10 cores. How much work is needed to orchestrate the 64 cores vs 10 cores? A more accurate benchmark could be to run on 10 cores on the graviton and compare it to M1.
- tyingq 5y agoAnd even then a bit slanted, as the M1 is half performance cores and half efficiency cores.
- jfbaro 5y agoThanks! It would be good to check it again with Graviton3.
- captainbland 5y agoI'm quite impressed with the M1 to the point that I'm considering my own MacBook, not to mention familiarity with it from work etc... But the pricing is driving me crazy. Ok the MacBook Air base model can be had for a reasonable price, but quite modest upgrades to 16gb/512gb ram/SSD effectively increase the price by about £550 (from ~£850->£1400 for cheapest available comparable units from trustable retailers) or £400 on the Apple store. At which point it seems worth considering the £1800-900 14" pro model since basically everything is a lot better on it. But I guess that's the point. I suppose one way of looking at it is that the base air is discounted because the market sees it as something with a short shelf life and the 16/512 model is paying the full Apple premium for something that is actually likely to still be useful for anything more than being a glorified Chromebook in 3 years. Assuming, you know, the screen doesn't crack, if that's a real problem.
- joshstrange 5y agoI mean even base models will normally be fine for 4-6+ years depending on your work. Also Apple computers hold their value unlike any other computer in the industry. Go look on eBay or similar for what 4-5 year old MBPs are selling for, I think you'll be surprised.
- captainbland 5y agoWell this is what's tricky for me. I'm into software development, etc. but I also have a desktop which is suitable for such work. That's fine. But I'd like it to at least be suitable for light development work as a more portable device for the foreseeable, so 8GB feels like it's already being pushed on my existing Ubuntu laptop (which is ~4 years old). But it's hard to justify such a price increase when it would still ultimately be a secondary machine.
- wilg 5y agoJust chiming in to say I have a M1 Air with the base 8GB RAM and I regret not upgrading that.
- 5y ago
- bastawhiz 5y ago> each Apple CPU core is performing twice as quickly as each AWS Graviton core. It’s 1.5x slower than the AWS server, but it’s ~6x cheaper. Thats remarkable value and makes me very excited for higher core count ARM Macs in the future. This isn't necessarily true. I'd be surprised if the CPU was the single bottleneck. Lots of other factors, like memory and the disk, are going to make your tests run faster or slower.