5 ms·
Cheapest Source of x86 Cores?
I'm doing embarrassingly parallel simulations (think Monte Carlo runs of a legacy scientific binary) and am trying to find the cheapest possible host source of x86 compute, at scale. These are jobs that are single-threaded, use maybe 2-4GB of ram, last an hour, and can be checkpointed if necessary. A c5.18xlarge on AWS has 36 physical (real) cores and on the spot market is $0.74/hr which works out to $0.02/core-hour. Does anyone know of cheaper options?
- asah 3y agoContact joinmassive.com - much cheaper than all of this.
- justinclift 3y agoBased on what?
- vbrandl 3y agoSounds like a botnet with extra steps
- justinclift 3y agoYeah. I'm inclined to agree with you now too. :)
- asah 3y agoNo. Founded by ex-Google engineer and whitelisted with all the major anti-virus companies. Disclosure: I'm an investor and advisor.
- justinclift 3y agoUnfortunately, the other commenters descriptions of it make it sound super shady. Perhaps you made a bad call with this one?
- asah 3y agoNo seriously, I know the team, work with them every day, read the code and have repo access, etc etc etc. The other commenters are social media randos, who actually don't know WTF they're talking about. Lookup my profile and you can decide if I do. Sometimes HN are just a herd of lazy jerks.
- justinclift 3y agok, well using Tor I just loaded the website. "The power of 100,000 computers in a few clicks" No strong impression either way from that. So, lets look a bit more... k, there's an "About" button up the top. Clicked that. Nothing. It just drops down a list of "Blog", "Team", or "Contact". I don't give a shit about any of those, nor have any interest in them. Why isn't "About" actually taking me to a page with info telling me WTF it's About? That's not super shady anyway, just really dumb design. Moving on, Lets look at the "Developers" options. So I click "Developers" then pick the 1st option "Desktop apps". That open's a new submenu, so I pick the first page there... "Dashboard". Instead of anything useful, that takes me to a website where I need to login. Well. That's the end of my interest. Closed website, never to return. At least it doesn't seem shady, as it never took me to anything other than the front page even though I tried (briefly). You might want to advise them a bit harder or differently or something, as it's clearly not great currently. :/
- asah 3y agoThanks for the feedback, but haha it sounds like it worked actually: you're not the target customer and avoided wasting yours or the company's time :-) At this stage of the company, the goal of the website is to provide a validating presence for people who already heard about them, because they're selling to carefully vetted partners. The typical flow is (strong reason to engage) => homepage => "deploy on Massive" => book a time to talk. Massive never seems short of customer interest, and the challenge is more on engineering to safely and efficiency grow. I guess now is the time to plug the jobs page: https://www.joinmassive.com/jobs https://www.joinmassive.com/jobs (I'm personally leading the key searches and feedback very welcome on those listings)
- delfinom 3y agohttps://www.joinmassive.com/faq#users https://www.joinmassive.com/faq#users >Massive is like Airbnb or Turo for your computer. Rather than letting you share your home or vehicle when you’re not using them, Massive lets you share any unused computing resources you have. In exchange, app developers give you access to their premium features. This doesn't have security concerns, not at all, nope >Massive combines the small amounts of resources contributed by users like you into a supercomputer that funds app features by mining cryptocurrency, decentralizing blockchain infrastructure, running scientific simulations, and performing general distributed tasks. Yea so it's totally just going to be abused to crpytomine 24/7 on your hardware >Massive doesn’t develop consumer apps but has partnered with many developers who offer to let you upgrade with Massive in their apps. Depending on your geolocation, you’ll get such offers in apps like Boom 3D for Windows and Digg Desktop. ....Yea this is basically malware and botnetting. There is 10000% perverse incentive for those "apps" to trick users into the botnet
- asah 3y agoIndeed, there are terrible actors in this space, but they're not one of them. See above response.
- hkgjjgjfjfjfjf 3y ago[dead]
- hkgjjgjfjfjfjf 3y ago[dead]
- re-thc 3y agoBoth Azure and GCP have really good spot pricing in many regions (~90% off). E.g. GCP C3 pricing is $0.003/vCPU hour and $0.004 for 1GB RAM -- where C3 = Intel Sapphire Rapids (latest gen), which should be quite a bit faster than a C5 (older Intel). c3-highcpu-88 (44 real cores) is $0.344/hr hence N2D (AMD Milan) has equivalents that's also in a similar price bracket See: https://cloud.google.com/compute/vm-instance-pricing#general-purpose_machine_type_family https://cloud.google.com/compute/vm-instance-pricing#general... Azure has similar prices for spot with AMD CPU. On a "smaller" cloud, Oracle cloud E4 (AMD Milan) prices are also a lot cheaper than AWS. For an even smaller cloud there's also Hetzner cloud with dedicated cores or dedicated servers even.
- mlthrowaway1953 3y agoThis is fantastic, thank you!
- rhplus 3y agoThe prices are great, but you may need to search across different regions for spot VM availability.
- rcarmo 3y agoI have a pre-built query for that on all my templates using spot instances: https://github.com/rcarmo/azure-stable-diffusion#cheapest-spots https://github.com/rcarmo/azure-stable-diffusion#cheapest-sp...
- sz4kerto 3y agoAn AX161 at Hetzner with 32 physical cores (EPYC 7502P, 64 threads) costs ~$130 per month, that works out to $0.17/hr ($0.0053 per core per hour).
- ysleepy 3y agoThat comes with a setup fee of 40$ though.
- hampereddustbin 3y agoAnnual purchases don't have setup fees as far as a I know, but it's probably just a way to charge more if you're only using it for a short while
- justinclift 3y agoYeah. That being said, Hetzner sometimes has weeks where there's no setup fee on some of their servers. Doesn't seem to be the case for these ones at the moment though.
- actionfromafar 3y agoHetzner is good value, but I had a long conversation with their support, and we were separated by a common language.
- kobalsky 3y agomy experience with hetzner's support is excellent since I've never spoken to them 8 dedicated 64t/128gb servers for 2 years, 0 cumulative seconds of downtime, perfect record.
- kjs3 3y agoYou aren't experiencing support (you're talking about ops if anything) and you don't rate support based on when you don't need them.
- tyingq 3y agoI'm guessing ~$0.02 is probably the floor. Even a dedicated server from OVH with similar specs is $0.02-0.04/real-not-smt-core-hour.
- amelius 3y agoWhy does one have to ask HN for this? It looks to me like the market could be more transparent here.
- mewmew07 3y agothe price confusion/obscurity facilitates higher prices, none of the players on the sell side would benefit from lower margins, so they all do the same
- prhrb 3y agoI also want cheapest compute for software video encoding, ARM or x86 is fine. can someone give suggestions? I am currently using oracle cloud
- re-thc 3y agoWhere are your videos? Encoding is a more complicated problem in that the source and destination would impact things. Bandwidth transfer can cost more than the compute on some clouds (and impact speeds). If it's on S3 it might still be best to use AWS EC2 in the same region. Oracle is definitely a good choice with 10TB free tier. Hetzner cloud likely has cheaper compute than Oracle. Upcloud has the fastest CPU.
- justinclift 3y agoDepending upon the type of encoding being used, GPU instances might be worth having a look at as well. :)
- prhrb 3y agoI have videos on Google drive. For hetzner can you tell which one to buy? They have many offerings- dedicated, cloud etc.
- namibj 3y agoDedicated, get some AMD Zen and use benchmarks for your encoder to compare performance across instance types.
- hampereddustbin 3y agoSpecifically for Video encoding I'd look into benchmarks about what certain ARM architectures can do. Not all encoders can correctly utilize them and they may end up very slow.
- justinclift 3y agoWhen you say "at scale", how many cores are you likely wanting to use at once, and for how long? If it's for several hundred hours worth of compute, then dedicated servers can start to look pretty good. eg: https://www.hetzner.com/dedicated-rootserver/ax161 https://www.hetzner.com/dedicated-rootserver/ax161 €119.80/mo, for a 32 (real) core Zen2 EPYC with 128GB ram They generally tend to have better local disk performance than most cloud servers, but that might be irrelevant for your use case. :) If you're just doing short bursts of use though, then it's probably not the right option.
- tyingq 3y agoYou'll would have to bump up from 128GB ram for a fair comparison. I didn't look exhaustively, but $.02/core-hour is hard to beat, especially since he's quoting that much memory and real (not smt/hyperthreads) core hours.
- justinclift 3y agoOh, good point. Yeah, that'd need to double the ram if they're going to use 4GB ram per SMT core. :) Doing that moves the cost up to €165.80 for 64 SMT cores with 256GB ram for a full month. (Note - I've updated those prices as I previously forgot to set the region to "US" for the Hetzner pricing thing) -- Lets see if my math is working today. ;) €165.80 / 64 cores = 2.5906 per core, per month Using 720 hours a month for our purposes, that's €0.0036 per core hour. -- Use xe.com to do pricing conversion, that €165.80 is US$179.79. So, $179.79 / 64 cores = $2.809 per core, per month Using 720 hours a month for our purposes, that's US$0.0039 per core hour. So, about a fifth of the price of $.02/core-hour.
- tyingq 3y agoThat is comparing an SMT core to an actual core. The c5.18xlarge is 36/72 cores/vcpus. Still cheaper, yes. Though I don't know how much you lose in scheduling trying to keep it fully utilized 24/7 versus the spot pricing.
- 3y ago
- ysleepy 3y agoThe hetzner server auction has pretty cheap dedicated boxes, 30 bucks for a month of 4core/8thread 64G ram. Which is roughly 0.006/thread hour, including cpu/ram/ssd.
- redat00 3y agoYou should check out Qarnot Computing (qarnot.com).
- Triangle9349 3y agoA little off topic. I was looking for a cheap cpu render farm for the project and ended up buying some chinese amd mini pcs. After a few months, I sold them at a retail price in just 3 days, there was a huge demand for them. I was very surprised by the power and almost silent operation compared to the insanely expensive and noisy servers that were used recently.
- ericpauley 3y agoIf you’re using aws spot pay careful attention to the region. In many aws regions spot prices have skyrocketed and offer less discount over on demand. Other regions have very cheap spot prices still. Also consider looking at equivalent instances like the c5d family.
- zombielinux 3y agoHave you looked at old Xeon PHI devices? They're not fast, and they may not meet your ram requirements, but DAMN are they cheap.
- aclindsa 3y agoWhy x86 only? Is that a hard requirement?
- b20000 3y agobuy a threadripper
- turtlebits 3y agoI use lowendbox.com when I want a cheap VPS. Not necessarily the most performant, but you can usually find pretty good deals on lower specced servers.
- johnklos 3y agoOne thing to consider would be to use something like the BOINC model, if the amount of data that needs to be exchanged isn't too great. You could also just start a small project to pay people. If 2¢ per core hour is a reasonable price, that means a Ryzen 5950X user could make $7.68 a day not using SMT threads when the machine is mostly idle.
- numpad0 3y agoSounds like a Xeon Phi workload, I always thought they would look cool as a collectible on Device Manager
- genmud 3y agoPeople don't realize how cheap hardware has become. Most vendors also have the ability to lease stuff for 0% (though that might have started to change), so you end up paying for a 3 or 5 year lease, which allows you to deduct vs. depreciate the asset on taxes. Sure you have to colocate it, but you could get a 48u colo rack for ~1k/month + ~1k/month for 1gbps link. I used to buy dell outlet servers (which is their refurbed gear) all the time for folks who insist on dell, they end up being about 20-50% of the cost of cloud pricing if you are utilizing it with a full workload. Their stuff also includes a 3 year warranty with next day support, so it isn't very risky buying a refurbished server. There have been times where I found deals where the components could be parted out on ebay for more than the cost of the server. If you don't need support, there are tons of used servers on ebay / craigslist where you can get deals on 1-3 year old hardware that are good enough to purchase multiple spares. For example, you can purchase a 2x 32 core server [1] for ~11k from the outlet store with next day support. If you are running 24/7, with 64 cores over a 3 year period... Server Cost / Years / Days in Year / Hours in Day / Cores = Core per hour cost 11000 / 3 / 365 / 24 / 64 = $0.0065/hour If you wanted to go supermicro [2] with somewhat less reliable support, its a similar story. It would be a 1u dual 96 core (192 cores) box w/ 1.5tb memory for around ~30k. 30000 / 3 / 365 / 24 / 192 = $0.0059/hour You can recoup your hardware costs within the first year vs. running in AWS if your workloads are high enough to justify the hardware expense. [1] - https://outlet.us.dell.com/ARBOnlineSales/Online/SecondaryInventorySearch.aspx?c=us&cs=28&l=en&s=dfb&sign=PXhcOSHtr1T4IOw%2fPR7UdaavHoUbtm0%2bKb7bKA8KhOvHPR8wIo1pmB%2fxdC8iGNzWIGfCL0rgrtD11xH%2bveqk8lrOBUiU8LbggJrNAxmLHjTyKbhovUt9P3%2bW7tGL22gPc%2fhQgXhAYq7oDb6isoqb5XN8JriqOdOqoCYnXRXCIFHQPjTWFP1paLqadxAtg%2bFGW25CQbCAFEI%3d https://outlet.us.dell.com/ARBOnlineSales/Online/SecondaryIn... [2] - https://www.itcreations.com/configurator/model/supermicro-as1125hstnr-hyper-a-server/833/5152#tab=1 https://www.itcreations.com/configurator/model/supermicro-as...