10 ms·
Review of Hetzner ARM64 servers and experience of WebP cloud services on them
- mduggles 3y agoI’ve been migrating workloads away from x86 and towards ARM on AWS and GCP since they’ve been available. This review does a great job of kinda giving you an idea of what you are gonna get as a platform, but if you are interested I strongly recommend the experience on any cloud provider. While there was some work to benchmark and validate, the cost savings have been non-trivial. Plus this change happened as we were all switching to the M series Macs so ironically now our entire chain end to end is off x86.
- Arnavion 3y agoAlas all my stuff is in Azure, and I'm still waiting for them to offer smaller VM sizes comparable to their existing B line. I currently use a B1s (1 CPU 1GiB) that comes to ~$5/mo while the cheapest ARM VM would be ~$25/mo (2 CPU 4GiB).
- Hamuko 3y agoI just refer to ARM Lambda runners as a free 20% discount since it makes absolutely no difference in runtime but costs less. I'd also run ARM database instances but I think those are still not really that readily available.
- aidos 3y agoFor us it was driven in the other direction. With the introduction of the M1s we knew that we’d be on arm locally soon enough. There was a bit of work in the transition but things have improved since then. Definitely happy running on all arm now though.
- dijit 3y agoI was keen on migrating to ARM, but there seems to be no benefits from doing so on GCP; I'm open to be wrong here. From what I understand they're using Ampere Altra, which have single thread performance similar to Skylake; but the cost is equivelant or worse than the x86 e2 series. e2-standard-4: USD 97.84/mo t2a-standard-4: USD 112.42/mo (sustained use discounts apply to neither). EDIT: I see you're in Denmark and are operations focused. I am too operations focused and just across the bridge in Malmö, maybe we could hang out.
- re-thc 3y agoThe real hidden gems of GCP are the 90% off spot instances in a lot of regions for e.g. N2D. ARM makes 0 sense on GCP if you can use those.
- mduggles 3y agoYeah sorry I should have been more clear. Currently the ARM instances in GCP when you use them as spot basically never get interrupted. We’re big into GKE so use them as a preferred node group for interruptible pods. I assume due to the pricing you mentioned usage is very low. So basically any background jobs or big batch processing jobs that required a lot of CPU time. We have multi-arch container builds so if we can’t scale out the ARM node group not a problem, go back to x86. But it was worth the optimizing to get effectively always available spot instances. Yeah always open to meet up with folks. I’m on mastodon at matdevdug@c.im.
- johncolanduoni 3y agoT2A vCPUs are full cores though right? While E2 and most other instances are hyperthreads.
- bennythink 3y agoActually I’m in Sweden. Of course we could hang out in sometime. Just cross the bridge. Here’s my email emeries-atolls.0w@icloud.com
- kramerger 3y agoGreat no nonsense article! I'm surprised how bad xeon scales to 8 cores. But isn't the xeon instance the only one not running bare metal?? Maybe he is paying for 8 cores but gets only 2-4 physical cores?
- adrian_b 3y agoThat "Xeon" is a very old (10-year old) quadruple-core (8-thread, i.e. 8 "virtual CPUs") desktop Haswell CPU rebranded as "Xeon". A current Intel NUC Pro with a Core i3 CPU would be a much faster (67% faster ST, 43% faster MT) dedicated server than this one and it would cost to own less than $500 with DRAM and SSD, so about $8 per month for a 5-year lifetime (so the performance per $ would be at least 5 to 6 times higher than that of the compared Intel server). That "Xeon" is a good comparison point only because it was available for them in the same price range, not because it would be representative for the performance of any modern x86 CPUs. Also the "Epyc" is probably a very old model. Somebody who wants to spend their money for cloud services as efficiently as possible should better ensure that it is possible to migrate back and forth their applications between x86 and ARM instances, because which one is cheaper for a certain performance at a given time depends a lot on non-technical reasons, so it is unpredictable which will be cheaper a few months later.
- novakwok 3y ago@kramerger No, Xeon Server is a dedicated server(a.k.a Bare metal), I've looked at it's console and found it's Dell PowerEdge R220(Motherboard Dell Inc. 081N4V). I'm quite confused about it's performance as well. CPU Info Name Intel Xeon E3-1230 v3 Topology 1 Processor, 4 Cores, 8 Threads Geekbench Link is at: https://browser.geekbench.com/v6/cpu/1533259 https://browser.geekbench.com/v6/cpu/1533259
- mkl 3y agoThat CPU seems to be from 2011: https://ark.intel.com/content/www/us/en/ark/products/52271/intel-xeon-processor-e31230-8m-cache-3-20-ghz.html https://ark.intel.com/content/www/us/en/ark/products/52271/i...
- tmikaeld 3y agoGreat article, thanks for sharing! We're using Hetzners new ARM servers ourselves, to convert images to WebP (Yes, your company name is really confusing!) and they perform almost as good as the Hetzner AMD instances. But since they're so much cheaper, we can easily fire up many of them and use a load-balancer in front, saving a ton of money compared to dedicated servers.
- novakwok 3y ago[Yes, your company name is really confusing] LOL, we're not a company, we are just a small team of three individuals(Nova Kwok,Benny Think and Tuki Deng). [convert images to WebP] May I ask your use case on this? (As we've recently launched a product called WebP Cloud might fit this need. (And we're actively seeking seed users.)) WebP Cloud documentation here: https://docs.webp.se/webp-cloud/ https://docs.webp.se/webp-cloud/
- tmikaeld 3y agoAlright, they way it was mentioned in the article made it sound like a business, sorry about that. Your service looks great, but we long since concluded that using an API for image conversion would be many times more expensive than using our own setup. And we also have mixed in fetches from external sources, storage in S3, Cloudflare workers and generative AI mixed in the bag - no single service supports all that yet (hint).
- adventured 3y ago> they way it was mentioned in the article made it sound like a business, sorry about that It is a business. They're selling a service. I don't know why they're protesting at the notion of being a company, they're de facto a business (selling service behind a brand, which they're openly promoting to sell more services).
- novakwok 3y agoHmmm, maybe calling this a start-up/business might be more appropriate? (WebP Cloud Services starts by providing a free service of Gravatar/GitHub Avatar reverse proxy with WebP optimization at first, and now it's our first attempt to make a paid services of private proxy as more of our users want this to be a more generally available service. (And we are currently not a company indeed) ´･ᴗ･` No intentional protesting at the notion of being a company, just unsure if "company," "business," and "startup" have the same meaning in certain contexts.
- throwaway2990 3y ago> Hetzner CAX11, with a virtualized ARM64 processor, 42 cores, 4GB memory, priced at $4.91 USD per month, referred to as CAX11 for simplicity. Haha I wish it was a 42 core for $4.91 Small typo for them to fix.
- novakwok 3y agoThanks for pointing out, now fixed!
- daneel_w 3y agoAccording to Oracle's documentation their Arm servers are not virtualized cores but instead actual on-core tenancy, referred to as OCPU instead of conventional vCPU. https://blogs.oracle.com/cloud-infrastructure/post/vcpu-and-ocpu-pricing-information https://blogs.oracle.com/cloud-infrastructure/post/vcpu-and-...
- deleted 3y ago[deleted]
- spaniard89277 3y agoIn terms of software hiccups, for someone with little time to debug, is it worth the cost savings?
- adql 3y agoIf you're not using proprietary software but common programming languages and OSS tooling there should be no difference.
- archerx 3y agoI have been developing on ARM servers for a while. I use Raspberry Pis and Tinkerboards as dev and staging servers and push releases to an x86-64 server on digital ocean. With docker it has been pretty easy, docker-compose usually finds the right packages for the CPU and it works quite well. I am curious about maybe trying on of the ARM servers on Hertzner and see how it compares.
- PaoloBarbolini 3y agoI've been trying their Arm servers for a while and I've noticed some differences in the colors in htop for Debian 12, as if there were a slight difference between the x86_64 and the aarch64 image. Other than that everything's going fine and I'm planning to use Arm for every server in the Falkenstein datacenter (the only one with Arm dedicated and cloud servers for now)
- archerx 3y agoNice, how’s the performance? The cheapest digitalocean x86 single core cpu is a lot faster than a quad core pi or tinkerbord. I know it’s not the same as an arm server cpu but how much difference is there?
- philliphaydon 3y ago[dead]
- novakwok 3y agoI've searched for Geekbench result: https://browser.geekbench.com/v6/cpu/1584694 https://browser.geekbench.com/v6/cpu/1584694, it says DO-Premium-Intel 1 Processor, 1 Core, so I'm assuming it's the 7USD/mo plan from https://www.digitalocean.com/pricing/droplets#basic-droplets https://www.digitalocean.com/pricing/droplets#basic-droplets. The score on Geekbench is Single-Core: 838,Multi-Core: 842. While in our tests the cheapest ARM64 plan on Hetzner is CAX11 2Core, 4G RAM, about 5USD/mo, the Geekbench result is Single-Core: 1072,Multi-Core: 1921, so assuming it about 20% faster than DigitalOcean. We've done the same test on Rpi4B too: Processor : Cortex-A72 CPU cores : 4 @ 1500.0000 MHz Score is Single-Core: 247,Multi-Core: 387 For your reference.
- mpweiher 3y agoSince moving to Apple Silicon, I've been wanting more ARM options in the cloud. Although it is possible to host x86_64 VMs, having fewer differences is obviously better. I've been using Oracle's free tier for a while, and it's been OK. Performance-wise, my Objective-S and libµhttpd based web-server appears to be doing around 1800 requests per second, and held up fine to a HN hug of death. Hetzner was far, far easier to set up, both from their console and via the API. Performance was comparable.
- shepherdjerred 3y agoAWS has great support for arm64 instances
- nezirus 3y agoInteresting read. I'd like to know more about alpine problems (even just to confirm my bias against it, unless space savings are the most important thing). For me, Hetzner is mostly baremetal provider. They have dedicated RX line, and if you have base load, a couple of those could run it all (use hetzner cloud instances for scalling and failover)
- novakwok 3y ago[I'd like to know more about alpine problems] Sure, and we're planning to share another post later on the whole procedure of our migration from AMD64 to ARM64, and in that post we'll include more details about Clickhouse's problem if we can definitively establish that the problem was caused by alpine. (After this incident I personally have bias against alpine images too Comparing alpine and non-alpine images on DockerHub: https://hub.docker.com/layers/clickhouse/clickhouse-server/23.4-alpine/images/sha256-8160aa2ad15b7c922c08fc2ff826c5e0702a54150337791e57c9f738d90d5d43?context=explore https://hub.docker.com/layers/clickhouse/clickhouse-server/2... https://hub.docker.com/layers/clickhouse/clickhouse-server/23.4/images/sha256-343b921e14c91baef6592e266e2c47941f2e3d6f23ee62f5997ae39f93688bb9?context=explore https://hub.docker.com/layers/clickhouse/clickhouse-server/2... There is just ~66MB(255MB vs 321MB) of size difference, my personal advice after this to to avoid alpine images in production as much as possible :P
- e145bc455f1 3y agoHow do you get your account verified at hetzner without sending a government ID to them?
- novakwok 3y agoI've sent my passport image to them to get my account verified.(When the second time I register Hetzner) (My first attempt on registration got my account closed even I've provided by passport.(maybe it's because I've used VPN for registration as it's website is too slow to open in China(might caused by china GFW)
- pulpfictional 3y agoPay 20€ up-front through PayPal. This becomes available as credit, a top-up in a sense.
- e145bc455f1 3y agoThey have disabled my account, i can't even login anymore.
- pulpfictional 3y agoAh sucks. From what I hear their support will probably not help you register but it's worth a shot.
- tehlike 3y agoGet a new account.
- brian_cunnie 3y agoI like the article, but I wish there had been an "Abstract" or "Executive Summary" at the top so that I'd be spared having to read the entire article to find out the results. I'd like to have seen something along the lines of the following: "We found Hetzner's ARM64 offering, specifically the CAX21 with 4 cores, 8GB at $8.40/month, to be a performant and cost-effective alternative to x86_64-based solutions."
- langsoul-com 3y agoAlso add, based on tests arm performed 8% worse than amd64, but this is offset by the 14% savings.
- petercooper 3y agoAlso, notably, the team who did the benchmarking were impressed enough to have actually switched entirely to said CAX instances for their app.
- novakwok 3y agoGood idea! I've added some TL;DRs at the beginning of the article.
- CodesInChaos 3y agoAt least two links don't work because they contain a closing parentheses.
- novakwok 3y agoThanks for pointing out! Now fixed.
- FlyingSnake 3y agoThis is a great article and it’s nice to see we’ve lots of alternatives to run ARM servers. I ran the now defunct Scaleway ARM server mentioned in the article for several years. For €2,99 it was a surprisingly useful machine. I ran several projects (.net core) on it and it was quite good for those simple workloads. I looked for alternatives for a while but nothing turned up until Apple restarted the ARM revolution with M1.
- xrd 3y agoI'm excited about arm in more places but my experience with arm and docker isn't as easy as i expected Is it just me? When I've started using arm more, I've noticed that docker images are often incomplete or behind the x86 release cycle. I love the ease of wiring docker images together for all my services (corollary: never having to understand the myriad packaging issues with whatever language the service is written in, python, nodejs, etc). But when I'm using an arm image, often it is not the same version as the latest on x86, or even worse, is packaged by someone random on the internet. If I were to install the JavaScript service myself, I could audit it (not that I ever do!) by looking into the package.json file, or reviewing the source code, or whatever. There is a clear path to reviewing it. But with a docker image from the Internet, I'm not sure how I would assert it is a well behaved service. Docker itself gives me some guarantees, but it still feels less straightforward. I've packaged things for an arm container myself and it isn't always exactly the same as for x86. Is this just me? Am I doing it wrong on arm?
- shepherdjerred 3y agoI felt this a year or two back, but today I've had as good of an experience on Docker w/ arm64 as I do w/ x86_64. I use arm64 Docker a lot since I work on a M1 MacBook. I usually stick to the common base images, e.g. ubuntu, alpine, nodejs, golang, etc. and install based off of that. Also, I rarely write Dockerfiles these days and instead use Earthly [0], which is a tool that really shines as a CI/make alternative, but it incidentally also has a nicer syntax which makes it easier to write multi-platform Docker images. What images or other problems have you ran into on arm64? [0]: https://earthly.dev/ https://earthly.dev/
- xrd 3y agoFor example gitlab. The latest arm image, as far as I could tell, isn't the same as the most recent x86. And, iirc, it was from some other person, not gitlab. It's often hard to tell what you are getting when you run an image, because docker pull can pull an image that isn't a multi platform build. I've had issues where the SSL certificates don't work, and I'm assuming it is because the stack could listen on 443, but the full ssl when running on arm didn't work. I'm not sure if that was because it is emulating using Rosetta or whether the software inside the container built correctly but isn't actually running on the arm platform correctly or what. It just feels like the wild west with arm images right now. I'm sure it will get better but it is still a minority platform and that comes with those issues. And, this might just be exposing my ignorance. Until recently I hadn't needed to use arm but now with macos it's gotten more interesting and more complicated.
- EVa5I7bHFq9mnYK 3y agoI've been using a cax41 (16 cores) instance for numerical computations recently. Geekbench scores are 774/10221, costs $0.04 hourly ($27 monthly). Perfectly stable. No throttling (probably not that popular yet hehe). For my specific program it's 10% slower than my laptop's 11980HK processor (8 threads, 16 hyperthreads).
- nikita2206 3y agoI’m always so taken aback when I compare VM prices from Hetzner/OVH and AWS/GCP. Similarly sized machine in AWS seems to be around $300 monthly, that’s 10x cost.
- jeroenhd 3y agoAmazon/Google has fallbacks across regions, several layers of data storage redundancy, high-speed. highly configurable software based networking and so much more. Hetzner/OVH has machines with almost no failover, with no extra availability zones, with no backup guarantees, very little in the way of custom networking, and doesn't integrate with dev tools quite as much. They're different products. For most people, going Amazon/Google makes no sense. However, if you absolutely MUST keep your data available after or during a fire [0] and keep your systems running during datacenter downtime, you're better off with AWS/GCP/Azure. SLAs with many nines can't afford cheap servers, and that's where the big cloud providers make a lot of money. Up until recently I saw a lot of people and companies move back from the cloud to self-managed dedicated hardware in data centers. All most companies need is half a rack in two places and a competent sysadmin team, but externalizing the risks is often attractive because disasters and bad failovers do happen sometimes. [0]: https://www.datacenterdynamics.com/en/opinions/ovhclouds-data-center-fire-one-year-on-what-do-we-know/ https://www.datacenterdynamics.com/en/opinions/ovhclouds-dat...
- nikita2206 3y agoAbsolutely no arguing that AWS adds more value. Another thing about AWS/GCP, they are also good at locking you in. For example you want to shift some workloads to Hetzner while leaving others in AWS, you will get a bill for egress out of AWS.
- qingcharles 3y agoWow, this is timely. I just bought their cheapest one last night (about $4/mo) to play with and performance test it for ASP.Net Core, vs. their x86 boxes. I tried to be ultra cheap and not buy a v4 IP but it appears Microsoft doesn't have v6 IPs on all their download servers which is causing me pain.
- foobarbazetc 3y agoTheir RX-220 servers are also amazing. Ampere 80 core machines for $220/m. We use these for anything requiring a lot of threads.
- NorwegianDude 3y agoWeird using a E3-1230 v3 in 2023, it's over 10 years old. A similar modern low end CPU would be many times faster.
- smarx007 3y agoI guess they used whatever CPU gives a relative price parity with the VMs: https://www.hetzner.com/sb?country=de https://www.hetzner.com/sb?country=de
- justsomehnguy 3y agoDepends on the task/load. Modern low end wouldn't be many times faster, at least not with a lower core/thread count https://ark.intel.com/content/www/us/en/ark/products/75054/intel-xeon-processor-e31230-v3-8m-cache-3-30-ghz.html https://ark.intel.com/content/www/us/en/ark/products/75054/i...
- toast0 3y agoI think a modern low end Xeon (desktop socket) is something like this. https://ark.intel.com/content/www/us/en/ark/products/212263/intel-xeon-w1350-processor-12m-cache-up-to-5-00-ghz.html https://ark.intel.com/content/www/us/en/ark/products/212263/... That's 6 cores, so not quite fair, but w1350 also has a much higher boost frequency, bigger caches, more bandwidth to ram, and it's built on smaller lithography and several generations of core designs later. It's hard to find a comparison between rocket lake and haswell, but between all the differences in lowend xeon parts, you're probably seeing a significant increase in throughput if your load isn't bottlenecked on something else, but even then, 20 pci-e 4.0 lanes vs 16 pci-e 3.0 lanes is more than double the i/o capacity.
- justsomehnguy 3y ago> you're probably seeing a significant increase As someone who had a chance to see the difference by it's own eyes: yes, it's faster, especially when the memory is the bottleneck. But it's not times faster in everyday tasks. Aside from the synthetic tests you are usually see more performance improvement from the overall system being faster (ie SATA to NVMe, more faster RAM) than just by CPU alone. There are some apps that are CPU bound (GHz first, RAM BW second) which gladly run way faster on these E3-16xx CPUs, than contemporary E5 multisocket monsters with tons of RAM... but waaay less GHz. These apps would be better on W1350, no questions.
- CathalMullan 3y agoCAX11 looks like a great deal, especially with IPv4 disabled.
- throwaway81523 3y agoTFA describes the E3-1230 as an 8 core server when it is actually a 4 core server with 8 threads. That means the ARM vs x86 per-core performance comparisons are off by a factor of 2. I stopped reading when I noticed that. For cheap sustained compute, it's hard to beat a Hetzner auction dedi.
- mike503 3y agoThat's interesting. I feel like I see benchmarks almost always showing ARM outperforming for all kinds of specific workloads. This is the first one I can recall showing it's not as good performance-wise, however when you add the power efficiency, cost savings, it winds up being better overall.
- rankun203 3y agoI’ve been using Hetzner’s EX line for some years, it’s super cost effective, until now I still can’t find any other provider with cheaper offerings.