6 ms·

These sorts of core-density increases are how I win cloud debates in an org. * Identify the workloads that haven't scaled in a year. Your ERPs, your HRIS, you

These sorts of core-density increases are how I win cloud debates in an org. * Identify the workloads that haven't scaled in a year. Your ERPs, your HRIS, your dev/stage/test environments, DBs, Microsoft estate, core infrastructure, etc. (EDIT, from zbentley: also identify any cross-system processing where data will transfer from the cloud back to your private estate to be excluded, so you don't get murdered with egress charges) * Run the cost analysis of reserved instances in AWS/Azure/GCP for those workloads over three years * Do the same for one of these high-core "pizza boxes", but amortized over seven years * Realize the savings to be had moving "fixed infra" back on-premises or into a colo versus sticking with a public cloud provider Seriously, what took a full rack or two of 2U dual-socket servers just a decade ago can be replaced with three 2U boxes with full HA/clustering. It's insane. Back in the late '10s, I made a case to my org at the time that a global hypervisor hardware refresh and accompanying VMware licenses would have an ROI of 2.5yrs versus comparable AWS infrastructure, even assuming a 50% YoY rate of license inflation (this was pre-Broadcom; nowadays, I'd be eyeballing Nutanix, Virtuozzo, Apache Cloudstack, or yes, even Proxmox, assuming we weren't already a Microsoft shop w/ Hyper-V) - and give us an additional 20% headroom to boot. The only thing giving me pause on that argument today is the current RAM/NAND shortage, but even that's (hopefully) temporary - and doesn't hurt the orgs who built around a longer timeline with the option for an additional support runway (like the three-year extended support contracts available through VARs). If we can't bill a customer for it, and it's not scaling regularly, then it shouldn't be in the public cloud. That's my take, anyway. It sucks the wind from the sails of folks gung-ho on the "fringe benefits" of public cloud spend (box seats, junkets, conference tickets, etc...), but the finance teams tend to love such clear numbers.

zbentley 8mo ago
That’s definitely the right call in some cases. But as soon as there’s any high-interconnect-rate system that has to be in cloud (appliances with locked in cloud billing contracts, compute that does need to elastically scale and talks to your DB’s pizza box, edge/CDN/cache services with lots of fallthrough to sources of truth on-prem), the cloud bandwidth costs start to kill you. I’ve had success with this approach by keeping it to only the business process management stacks (CRMs, AD, and so on—examples just like the ones you listed). But as soon as there’s any need for bridging cloud/onprem for any data rate beyond “cronned sync” or “metadata only”, it starts to hurt a lot sooner than you’d expect, I’ve found.
- stego-tech 8mo ago
  Yep, 100%, but that's why identifying compatible workloads first is key. A lot of orgs skip right to the savings pitch, ignorant of how their applications communicate with one another - and you hit the nail on the head that applications doing even some processing in a cloud provider will murder you on egress fees by trying to hybrid your app across them. Folks wanting one or the other miss savings had by effectively leveraging both.
  - hedora 8mo ago
    Any experience with the mid-to-small cloud providers that provide un-metered network ports and/or free interconnect with partner providers? (For various reasons, I just care about VPS/bare metal, and S3-compatiblity.) I'm looking at those because I'm having difficulty forecasting bandwidth usage, and the pessimistic scenarios seem to have me inside the acceptable use policies of the small providers while still predicting AWS would cost 5-10x more for the same workload.
    - stackskipton 8mo ago
      Vultr and Digital Ocean both offer Direct Connects. I've had good experience with their VPSes.
    - Imustaskforhelp 8mo ago
      Netcup and OVH provide free un-metered ports. There are actually lots of options available on the market. BuyVM is another good one.
mschuster91 8mo ago
> If we can't bill a customer for it, and it's not scaling regularly, then it shouldn't be in the public cloud. That's my take, anyway. It sucks the wind from the sails of folks gung-ho on the "fringe benefits" of public cloud spend (box seats, junkets, conference tickets, etc...), but the finance teams tend to love such clear numbers. I agree, but. For one, it's not just the machines themselves. You also need to budget in power, cooling, space, the cost of providing redundant connectivity and side gear (e.g. routers, firewalls, UPS). Then, you need a second site, no matter what. At least for backups, ideally as a full failover. Either your second site is some sort of cloud, which can be a PITA to set up without introducing security risks, or a second physical site, which means double the expenses. If you're a publicly listed company, or live in jurisdictions like Europe, or you want to have cybersecurity insurance, you have data retention, GDPR, SOX and a whole bunch of other compliance to worry about as well. Sure, you can do that on-prem, but you'll have a much harder time explaining to auditors how your system works when it's a bunch of on-prem stuff vs. "here's our AWS Backup plans covering all servers and other data sources, here is the immutability stuff, here are plans how we prevent backup expiry aka legal hold". Then, all of that needs to be maintained, which means additional staff on payroll, if you own the stuff outright your finance team will whine about depreciation and capex, and you need to have vendors on support contracts just to get firmware updates and timely exchanges for hardware under warranty. Long story short, as much as I prefer on-prem hardware vs the cloud, particularly given current political tensions - unless you are a 200+ employee shop, the overhead associated with on-prem infrastructure isn't worth it.
- Imustaskforhelp 8mo ago
  > Then, you need a second site, no matter what. At least for backups, ideally as a full failover. Either your second site is some sort of cloud, which can be a PITA to set up without introducing security risks, or a second physical site, which means double the expenses. You can technically have backblaze's unlimited backup option which costs around 7$ for a given machine although its more intended for windows, there have been people who make it work and Daily backups and it should work with gdpr (https://www.backblaze.com/company/policy/gdpr https://www.backblaze.com/company/policy/gdpr) with something like hetzner perhaps if you are worried about gdpr too much and OVH storage boxes (36 TB iirc for ~55$ is a good backup box) and you should try to follow 3-2-1 strategy. > Then, all of that needs to be maintained, which means additional staff on payroll, if you own the stuff outright your finance team will whine about depreciation and capex, and you need to have vendors on support contracts just to get firmware updates and timely exchanges for hardware under warranty. I can't speak for certain but its absolutely possible to have something but iirc for companies like dell, its possible to have products be available on a monthly basis available too and you can simply colocate into a decent datacenter. Plus points in that now you can get 10-50 GB ports as well if you are too bandwidth hungry and are available for a lot lot more customizable and the hardware is already pretty nice as GP observed. (Yes Ram prices are high, lets hope that is temporary as GP noted too) I can't speak about firmware updates or timely exchanges for hardware under security. That being said, I am not saying this is for everyone as well. It does essentially boils down to if they have expertise in this field/can get expertise in this field or not for cheaper than their aws bills or not. With many large AWS bills being in 10's of thousands of dollars if not hundreds of thousands of dollars, I think that far more companies might be better off with the above strategy than AWS actually.
jmward01 8mo ago
Cloud = the right choice when just starting. It isn't about infra cost, it is about mental cost. Setting up infra is just another thing that hurts velocity. By the time you are serving a real load for the first time though you need to have the discussion about a longer term strategy and these points are valid as part of that discussion.
- CyberDildonics 8mo ago
  Is infra really that hard to set up? It seems like infra is something a infra expert could establish to get the infra going and then your infra would be set up and you would always have infra.
  - readthenotes1 8mo ago
    Based on the evidence, not only is infrastructure really hard to set up in the first place, it is incredibly error-prone to adjust to new demand.
  - estimator7292 8mo ago
    You have to pay that infra person and shield them from "infra works, why are we paying so much for IT staff" layoffs. Then you have ongoing maintenance costs like UPS battery replacement and redundant internet connections, on top of the usual hardware attrition. It's unfortunately not so cut and dry
  - UltraSane 8mo ago
    Secure and reliable infrastructure is hard to set and keep secure and reliable over time.
  - ocdtrekkie 8mo ago
    As a big on-prem guy, I think cloud makes sense for early startups. Lead time on servers and networking setup can be significant, and if you don't know how much you need yet you will either be resource starved or burn all your cash on unneeded capacity. On-prem wins for a stable organization every time though.
user5994461 8mo ago
> These sorts of core-density increases are how I win cloud debates in an org. The core density is bullshit when each core is so slow that it can't do any meaningful work. The reality is that Intel is 3 times behind AMD/TSMC on performance vs power consumption ratio. People would be better off having a look at the high frequency models (9xx5F models like the 9575F), that was the first generation of CPU server to reach ~5 GHz and sustain it on 32+ cores.
- matja 8mo ago
  Intel seem to be deliberately hiding the clock frequency of this thing, the xeon-6-plus-product-deck.pdf has no mention of clock frequency or how LLC is shared.
  - deleted 8mo ago
    [deleted]
deleted 8mo ago
[deleted]
madduci 8mo ago
Is your calculation also taking cost of energy and personnel that keeps your own infra running?
- matsemann 8mo ago
  Is that personnel cost more than running on someone else's infra? Just counting the amount of people a company now need just to maintain their cloud/kubernetes/whatever setup, paired with "devops" meaning all devs now have to spend time on this stuff, I could almost wager we would spend less on personnel if we just chucked a few laptops in a closet and sshed in.
carefree-bob 8mo ago
The main cost with on-prem is not the price of the gear but the price of acquiring talent to manage the gear. Most companies simply don't have the skillset internally to properly manage these servers, or even the internal talent to know whether they are hiring a good infrastructure engineer or not during the interview process. For those that do, your scaling example works against you. If today you can merge three services into one, then why do you need full time infrastructure staff to manage so few servers? And remember, you want 24/7 monitoring, replication for disaster recovery, etc. Most businesses do not have IT infrastructure as a core skill or differentiator, and so they want to farm it out.
- boltzmann-brain 8mo ago
  As opposed to talent to manage the AWS? Sorry, AWS loses here as well.
  - carefree-bob 8mo ago
    I know of AWS's reputation as a business and what the devs say who work there, so I have no argument against your point, except to say that they do manage to make it work. Somewhere in there must be some unsung heroes keeping the whole thing online.
    - rcxdude 8mo ago
      The point being that AWS runs AWS, they don't run your business on AWS. You still need someone to actually set up AWS to do what you want, much like you would need someone to run your on-premises servers. And in my experience, the difference is not much.
      - boltzmann-brain 8mo ago
        The biggest issue is that with colo you're building a skill pool that can be used forever, with AWS you're building a skill pool centered around a corporate entity's business strategies and an inscrutable, closed-source system, which is not sustainable.
jfindley 8mo ago
Do note though that AIUI these are all E-cores, have poor single-threaded performance and won't support things like AVX512. That is going to skew your performance testing a lot. Some workloads will be fine, but for many users that are actually USING the hardware they buy this is likely to be a problem. If that's you then the GraniteRapids AP platform that launched previously to this can hit similar numbers of threads (256 for the 6980P). There are a couple of caveats to this though - firstly that there are "only" 128 physical cores and if you're using VMs you probably don't want to share a physical core across VMs, secondly that it has a 500W TDP and retails north of $17000, if you can even find one for sale. Overall once you're really comparing like to like, especially when you start trying to have 100+GbE networking and so on, it gets a lot harder to beat cloud providers - yes they have a nice fat markup but they're also paying a lot less for the hardware than you will be. Most of the time when I see takes like this it's because the org has all these fast, modern CPUs for applications that get barely any real load, and the machines are mostly sitting idle on networks that can never handle 1/100th of the traffic the machine is capable of delivering. Solving that is largely a non-technical problem not a "cloud is bad" problem.
- formerly_proven 8mo ago
  E-cores aren't that slow, yesteryear ones were already around Skylake levels of performance (clock for clock). Now one might say that's a 10+ year old uarch, true, but those ten years were the slowest ten years in computing since the beginning of computing, at least as far as sequential programs are concerned.
- adrian_b 8mo ago
  These Intel Darkmont cores are in a different performance class than the (Crestmont) E-cores used in the previous generation of Sierra Forest Xeon CPUs. For certain workloads they may have even a close to double performance per core. Darkmont is a slightly improved variant of the Skymont cores used in Arrow Lake/Lunar Lake and it has a performance very similar to the Arm Neoverse V3 cores used in Graviton5, the latest generation of custom AWS CPUs. However, a Clearwater Forest Xeon CPU has much more cores per socket than Graviton5 and it also supports dual-socket motherboards. Darkmont also has a greater performance than the older big Intel cores, like all Skylake derivatives, inclusive for AVX-using programs, so it is no longer comparable with the Atom series of cores from which it has evolved. Darkmont is not competitive in absolute performance with AMD Zen 5, but for the programs that do not use AVX-512 it has better performance per watt. However, since AMD has started to offer AVX-512 for the masses, the number of programs that have been updated to be able to benefit from AVX-512 is increasing steadily, and among them are also applications where it was not obvious that using array operations may enhance performance. Because of this pressure from AMD, it seems that this Clearwater Forest Xeon is the final product from Intel that does not support AVX-512. Both next 2 Intel CPUs support AVX-512, i.e. the Diamond Rapids Xeon, which might be launched before the end of the year, and the desktop and laptop CPU Nova Lake, whose launch has been delayed to next year (together with the desktop Zen 6, presumably due to the shortage of memories and production allocations at TSMC).
zer00eyz 8mo ago
> These sorts of core-density increases are how I win cloud debates in an org. AMD has had these sorts of densities available for a minute. > Identify the workloads that haven't scaled in a year. I have done this math recently, and you need to stop cherry picking and move everything. And build a redundant data center to boot. Compute is NOT the major issue for this sort of move: Switching and bandwidth will be major costs. 400gb is a minimum for interconnects and for most orgs you are going to need at least that much bandwidth top of rack. Storage remains problematic. You might be able to amortize compute over this time scale, but not storage. 5 years would be pushing it (depending on use). And data center storage at scale was expensive before the recent price spike. Spinning rust is viable for some tasks (backup) but will not cut it for others. Human capital: Figuring out how to support the hardware you own is going to be far more expensive than you think. You need to expect failures and staff accordingly, that means resources who are going to be, for the most part, idle.
bearjaws 8mo ago
I just don't know if the human capital is there. At my job we use HyperV, and finding someone who actually knows HyperV is difficult and expensive. Throw in Cisco networking, storage appliances, etc to make it 99.99% uptime... Also that means you have just one person, you need at least two if you don't want gaps in staffing, more likely three. Then you still need all the cloud folks to run that. We have a hybrid setup like this, and you do get a bit of best of both worlds, but ultimately managing onprem or colo infra is a huge pain in the ass. We only do it due to our business environment.
- sounds 8mo ago
  > I just don't know if the human capital is there. > At my job we use HyperV, and finding someone who actually knows HyperV is difficult and expensive... Try offering significantly higher pay.
  - ponector 8mo ago
    Or even try to educate people. It was common to have learning programs but nowadays managers only complain you cannot find cheap experts.
    - eastabrooka 8mo ago
      "We educated the people and they left because they could get better elsewhere" - Some Manager
- to11mtm 8mo ago
  I think you're hitting on a general problem statement a lot of orgs run into, even ignoring the uptime figure... All of the complexity of onprem, especially when you need to worry about failover/etc can get tricky, especially if you are in a wintel env like a lot of shops are. i.e. lots of companies are doing sloppy 'just move the box to an EC2 instance' migrations because of how VMWare jacked their pricing up, and now suddenly EC2/EBS/etc costing is so cheap it's a no brain choice. I think the knowledge base to set up a minimal cost solution is too tricky to find a benefit vs all the layers (as you almost touched on, all the licensing at every layer vs a cloud provider managing...) That said, rug pulls are still a risk; I try to push for 'agnostic' workloads in architecture, if nothing else because I've seen too many cases where SaaS/PaaS/etc decide to jack up the price of a service that was cheap, and sure you could have done your own thing agnostically, but now you're there, and migrating away has a new cost. IOW, I agree; I don't think the human capital is there as far as infra folks who know how to properly set up such environments, especially hitting the 'secure+productive' side of the triangle.
justsomehnguy 8mo ago
> The only thing giving me pause on that argument today is the current RAM/NAND shortage Not a shortage - price gouging. And it would mean an increase in the 'cloud' prices because they need to refresh the HW too. So by the summer the equation would be back to it.
varispeed 8mo ago
That only works if purchasers in the organisation are immune to kickbacks.
PaulKeeble 8mo ago
What has surprised me about the cloud is that the price has been towards ever increasing prices for cores. Yet the market direction is the opposite, what used to be a 1/2 or a 1/4 of a box is now 1/256 and its faster and yet the price on the cloud has gone ever up for that core. I think their business plan is to wipe out all the people who used to maintain the on premise machines and then they can continue to charge similar prices for something that is only getting cheaper. Its hard drive and SSD space prices that stagger me on the cloud. Where one of the server CPUs might only be about 2x the price of buy a CPU for a few years if you buy less in a small system (all be it with less clock speed usually on the cloud) the drive space is at least 10-100x the price of doing it locally. Its got a bit more potential redudency but for that overhead you can repeat that data a lot of times. As time has gone on the deal of cloud has got worse as the hardware got more cores.
- ragall 8mo ago
  > What has surprised me about the cloud is that the price has been towards ever increasing prices for cores. That makes a lot of sense. Cloud providers are selling compute, and as cores get faster, the single core gets more expensive.
umvi 8mo ago
Is using virtualization the only good way of taking a 288-core box and splitting it up into multiple parallel workloads? One time I rented a 384-core AMD EPYC baremetal VM in GCP and I could not for the life of me get parallelized workloads to scale just using baremetal linux. I wanted to run a bunch of CPU inference jobs in parallel (with each one getting 16 cores), but the scaling was atrocious - the more parallel jobs you tried to add, the slower all of them ran. When I checked htop the CPU was very underutilized, so my theory was that there was a memory bottleneck somewhere happening with ONNX/torch (something to do with NUMA nodes?) Anyway, I wasn't able to test using proxmox or vmware on there to split up cpu/memory resources; we decided instead to just buy a bunch of smaller-core-count AMD Ryzen 1Us instead, which scaled way better with my naive approach.
- Dylan16807 8mo ago
  How did the speed of one or two jobs on the EPYC compare to the Ryzen? And 384 actual cores or 384 hyperthreading cores? Inference is so memory bandwidth heavy that my expectations are low. An EPYC getting 12 memory channels instead of 2 only goes so far when it has 24x as many cores.
- PunchyHamster 8mo ago
  They are used for VMs because the load is pretty spiky and usually not that memory heavy. For just running single app smaller core count but higher clocked ones are usually more optimal >Anyway, I wasn't able to test using proxmox or vmware on there to split up cpu/memory resources; we decided instead to just buy a bunch of smaller-core-count AMD Ryzen 1Us instead, which scaled way better with my naive approac If that was single 384 (192 times 2 for hyperthreading) CPU you are getting "only" 12 DDR5 channels, so one RAM channel is shared by 16c/32y So just plain 16 core desktop Ryzen will have double memory bandwidth per core
klooney 8mo ago
Man, how do you get box seats out of AWS, I'm missing out
fnord77 8mo ago
on prem = capex cloud = opex The accounting dept will always win this debate.
MrBuddyCasino 8mo ago
It seems a lot of people have forgotten how BigCorp IT used to work. - request some HW to run $service - the "IT dept" (really, self-interested gatekeeper) might give you something now, or in two weeks, or god help you if they need to order new hardware then its in two months, best case - there will be various weird rules on how the on-prem HW is run, who has access etc, hindering developer productivity even further - the hardware might get insanely oversubscribed so your service gets half a cpu core with 1GB RAM, because perverse incentives mean the "IT dept" gets rewarded for minimizing cost, while the price is paid by someone else - and so on... The cloud is a way around this political minefield.
- misswaterfairy 8mo ago
  > The cloud is a way around this political minefield. Until the bills _really_ start skyrocketing...
fennecfoxy 8mo ago
>Realize the savings to be had moving "fixed infra" back on-premises or into a colo versus sticking with a public cloud provider As other people have pointed out: what happens when the PSU or mobo shits itself, what happens when the new version of vmware or docker (or whatever) shits itself, etc
penglish1 7mo ago
I don't post much on HN, but this topic is near and dear to my heart, so here we go. Context: been helpdesk, sysadmin & network admin, DevOps, Site Reliability Engineer, in that progression, starting in the 90's. Max on-prem was 40 racks, scaled up and down over years. Many comments talking about how staffing is a key element of this equation that can't be overlooked, but I decided to reply to the root comment, which doesn't say whether/how it considers staffing. This is a complex equation - and it is relatively easy to present an incomplete or misleading picture management to push the move into the cloud.. or out of the cloud. Some factors, in no particular order: 1) Scaling: it is self-evident that pulling a single physical server worth out of the cloud is not worth it.. even for 288 cores. Or perhaps 1152 for 4xXeon in a single server. Still likely not worth it. Why? Because a single server is never just that. Someone has to swap components when it goes down. When it goes down.. ALL 1152 cores are down, along with everything they are doing. Is that acceptable for all applications running on all those cores? It is also appropriate supporting infrastructure - power, cooling, physical space. The "fairly obvious" minimum scaling is "enough servers that one can be entirely down for maintenance while keeping everything else running." But now you're paying for some overhead. At 2 servers, you're buying 2x what you need, half that capacity is idle all the time. And so on. On this point - I think the other comments talking about "each SRE managing 4 (or 5, or 7)" racks missed the point entirely. SRE's should be doing scalable work, whether in the cloud, or on-prem. And they should NOT be swapping failed hard drives and power supplies. Designing a larger-than-one rack install is probably worth hiring consultants for if you don't have that expertise in-house, though the SREs that would be supporting it would need to supply lots of input. To some extent, server & network equipment vendors can also help. It is not trivial as the scale goes up. But then it should run for some years, with relatively unskilled people handling hardware failures and you can re-engage consultants if necessary to do upgrades as hardware and needs evolve. But your SREs should be on-staff, and probably on-call to handle the software running on that hardware.. and to some extent to call the remote hands to deal with hardware failures. 2) Business needs: does the business need the tech skills that self-hosting requires for the core business? For example - if the business itself is cloud SAAS, maybe DIY-ing at least some of your infrastructure is right in your wheelhouse. If so - a modest increase in staff could mean a huge cost savings. But if not, all the cost of skilled staff to run it is simply part of the cost of in-housing this stuff. 3) Staffing: the people that swap broken hardware are not the same people that respond to pages because the business critical application crashed due to a bug. You can pay a colo facility for all this, typically by the hour - but it isn't cheap and you've got to supply all the spares etc. Is that part of your budget for on-prem? 4) On-call: maybe your self-hosted ERP system can be down every night and weekend without issues.. and even business hours can tolerate 98% uptime. But that doesn't mean you can get away without having someone on call - presumably you're hosting more than just this lowish-requirement ERP system. I'll disagree with other comments - the "no burnout" number of on-call staff you need is 6-7, not 4! Remember people take vacations too. This is well studied, and established, I'll reference Tom Limoncelli's books. This could be relatively cheap and require fewer staff with geo-distributed staff, and it would tend to overlap with staff you already use to provide on-call for anything you host on the cloud - so maybe for your situation it is close to a wash. But you can't forget to budget for it even if the line item is $0. 5) Vendor support: maybe you already have your own data center or colo and are hosting a ton of stuff. Why not move all your Atlassian stuff in house and save the hosting cost. Oh.. wups, Atlassian simply doesn't support that any more. Host it with Atlassian or GTFO. A minor point as most vendors would give you enough notice you can simply run out the lifetime of the hardware it is on and not replace it. 6) Market pricing: At one point Amazon was starting "by the minute, by the core cloud" (as opposed the older "cloudish" model of leasing only an entire physical server by the entire year) and priced a bit under market to get going. Then once they established dominance, they cranked the price WAY up for profit extraction. But now they do have some competition and they're a bit more selective about how they extract profits. In my perception they've shifted a lot of the profit taking to the value-added services rather than raw instance time, but I could be wrong. And they have HUGE costs - it is beyond naive to look at the per-hour cost of an instance and compare it to purchasing an identical physical server solely on the purchase price of that server. Corey Quinn / Duckbill group has spent a huge part of his career in this space - if you're already in the cloud 100% it is well worth optimizing those costs before you start comparing it to what on-prem might cost.

zbentley 8mo ago

That’s definitely the right call in some cases. But as soon as there’s any high-interconnect-rate system that has to be in cloud (appliances with locked in cloud billing contracts, compute that does need to elastically scale and talks to your DB’s pizza box, edge/CDN/cache services with lots of fallthrough to sources of truth on-prem), the cloud bandwidth costs start to kill you. I’ve had success with this approach by keeping it to only the business process management stacks (CRMs, AD, and so on—examples just like the ones you listed). But as soon as there’s any need for bridging cloud/onprem for any data rate beyond “cronned sync” or “metadata only”, it starts to hurt a lot sooner than you’d expect, I’ve found.

stego-tech 8mo ago
Yep, 100%, but that's why identifying compatible workloads first is key. A lot of orgs skip right to the savings pitch, ignorant of how their applications communicate with one another - and you hit the nail on the head that applications doing even some processing in a cloud provider will murder you on egress fees by trying to hybrid your app across them. Folks wanting one or the other miss savings had by effectively leveraging both.
- hedora 8mo ago
  Any experience with the mid-to-small cloud providers that provide un-metered network ports and/or free interconnect with partner providers? (For various reasons, I just care about VPS/bare metal, and S3-compatiblity.) I'm looking at those because I'm having difficulty forecasting bandwidth usage, and the pessimistic scenarios seem to have me inside the acceptable use policies of the small providers while still predicting AWS would cost 5-10x more for the same workload.
  - stackskipton 8mo ago
    Vultr and Digital Ocean both offer Direct Connects. I've had good experience with their VPSes.
  - Imustaskforhelp 8mo ago
    Netcup and OVH provide free un-metered ports. There are actually lots of options available on the market. BuyVM is another good one.

mschuster91 8mo ago

> If we can't bill a customer for it, and it's not scaling regularly, then it shouldn't be in the public cloud. That's my take, anyway. It sucks the wind from the sails of folks gung-ho on the "fringe benefits" of public cloud spend (box seats, junkets, conference tickets, etc...), but the finance teams tend to love such clear numbers. I agree, but. For one, it's not just the machines themselves. You also need to budget in power, cooling, space, the cost of providing redundant connectivity and side gear (e.g. routers, firewalls, UPS). Then, you need a second site, no matter what. At least for backups, ideally as a full failover. Either your second site is some sort of cloud, which can be a PITA to set up without introducing security risks, or a second physical site, which means double the expenses. If you're a publicly listed company, or live in jurisdictions like Europe, or you want to have cybersecurity insurance, you have data retention, GDPR, SOX and a whole bunch of other compliance to worry about as well. Sure, you can do that on-prem, but you'll have a much harder time explaining to auditors how your system works when it's a bunch of on-prem stuff vs. "here's our AWS Backup plans covering all servers and other data sources, here is the immutability stuff, here are plans how we prevent backup expiry aka legal hold". Then, all of that needs to be maintained, which means additional staff on payroll, if you own the stuff outright your finance team will whine about depreciation and capex, and you need to have vendors on support contracts just to get firmware updates and timely exchanges for hardware under warranty. Long story short, as much as I prefer on-prem hardware vs the cloud, particularly given current political tensions - unless you are a 200+ employee shop, the overhead associated with on-prem infrastructure isn't worth it.

Imustaskforhelp 8mo ago
> Then, you need a second site, no matter what. At least for backups, ideally as a full failover. Either your second site is some sort of cloud, which can be a PITA to set up without introducing security risks, or a second physical site, which means double the expenses. You can technically have backblaze's unlimited backup option which costs around 7$ for a given machine although its more intended for windows, there have been people who make it work and Daily backups and it should work with gdpr (https://www.backblaze.com/company/policy/gdpr https://www.backblaze.com/company/policy/gdpr) with something like hetzner perhaps if you are worried about gdpr too much and OVH storage boxes (36 TB iirc for ~55$ is a good backup box) and you should try to follow 3-2-1 strategy. > Then, all of that needs to be maintained, which means additional staff on payroll, if you own the stuff outright your finance team will whine about depreciation and capex, and you need to have vendors on support contracts just to get firmware updates and timely exchanges for hardware under warranty. I can't speak for certain but its absolutely possible to have something but iirc for companies like dell, its possible to have products be available on a monthly basis available too and you can simply colocate into a decent datacenter. Plus points in that now you can get 10-50 GB ports as well if you are too bandwidth hungry and are available for a lot lot more customizable and the hardware is already pretty nice as GP observed. (Yes Ram prices are high, lets hope that is temporary as GP noted too) I can't speak about firmware updates or timely exchanges for hardware under security. That being said, I am not saying this is for everyone as well. It does essentially boils down to if they have expertise in this field/can get expertise in this field or not for cheaper than their aws bills or not. With many large AWS bills being in 10's of thousands of dollars if not hundreds of thousands of dollars, I think that far more companies might be better off with the above strategy than AWS actually.

jmward01 8mo ago

Cloud = the right choice when just starting. It isn't about infra cost, it is about mental cost. Setting up infra is just another thing that hurts velocity. By the time you are serving a real load for the first time though you need to have the discussion about a longer term strategy and these points are valid as part of that discussion.

CyberDildonics 8mo ago
Is infra really that hard to set up? It seems like infra is something a infra expert could establish to get the infra going and then your infra would be set up and you would always have infra.
- readthenotes1 8mo ago
  Based on the evidence, not only is infrastructure really hard to set up in the first place, it is incredibly error-prone to adjust to new demand.
- estimator7292 8mo ago
  You have to pay that infra person and shield them from "infra works, why are we paying so much for IT staff" layoffs. Then you have ongoing maintenance costs like UPS battery replacement and redundant internet connections, on top of the usual hardware attrition. It's unfortunately not so cut and dry
- UltraSane 8mo ago
  Secure and reliable infrastructure is hard to set and keep secure and reliable over time.
- ocdtrekkie 8mo ago
  As a big on-prem guy, I think cloud makes sense for early startups. Lead time on servers and networking setup can be significant, and if you don't know how much you need yet you will either be resource starved or burn all your cash on unneeded capacity. On-prem wins for a stable organization every time though.

user5994461 8mo ago

> These sorts of core-density increases are how I win cloud debates in an org. The core density is bullshit when each core is so slow that it can't do any meaningful work. The reality is that Intel is 3 times behind AMD/TSMC on performance vs power consumption ratio. People would be better off having a look at the high frequency models (9xx5F models like the 9575F), that was the first generation of CPU server to reach ~5 GHz and sustain it on 32+ cores.

matja 8mo ago
Intel seem to be deliberately hiding the clock frequency of this thing, the xeon-6-plus-product-deck.pdf has no mention of clock frequency or how LLC is shared.
- deleted 8mo ago
  [deleted]

deleted 8mo ago

[deleted]

madduci 8mo ago

Is your calculation also taking cost of energy and personnel that keeps your own infra running?

matsemann 8mo ago
Is that personnel cost more than running on someone else's infra? Just counting the amount of people a company now need just to maintain their cloud/kubernetes/whatever setup, paired with "devops" meaning all devs now have to spend time on this stuff, I could almost wager we would spend less on personnel if we just chucked a few laptops in a closet and sshed in.

carefree-bob 8mo ago

The main cost with on-prem is not the price of the gear but the price of acquiring talent to manage the gear. Most companies simply don't have the skillset internally to properly manage these servers, or even the internal talent to know whether they are hiring a good infrastructure engineer or not during the interview process. For those that do, your scaling example works against you. If today you can merge three services into one, then why do you need full time infrastructure staff to manage so few servers? And remember, you want 24/7 monitoring, replication for disaster recovery, etc. Most businesses do not have IT infrastructure as a core skill or differentiator, and so they want to farm it out.

boltzmann-brain 8mo ago
As opposed to talent to manage the AWS? Sorry, AWS loses here as well.
- carefree-bob 8mo ago
  I know of AWS's reputation as a business and what the devs say who work there, so I have no argument against your point, except to say that they do manage to make it work. Somewhere in there must be some unsung heroes keeping the whole thing online.
  - rcxdude 8mo ago
    The point being that AWS runs AWS, they don't run your business on AWS. You still need someone to actually set up AWS to do what you want, much like you would need someone to run your on-premises servers. And in my experience, the difference is not much.
    - boltzmann-brain 8mo ago
      The biggest issue is that with colo you're building a skill pool that can be used forever, with AWS you're building a skill pool centered around a corporate entity's business strategies and an inscrutable, closed-source system, which is not sustainable.

jfindley 8mo ago

Do note though that AIUI these are all E-cores, have poor single-threaded performance and won't support things like AVX512. That is going to skew your performance testing a lot. Some workloads will be fine, but for many users that are actually USING the hardware they buy this is likely to be a problem. If that's you then the GraniteRapids AP platform that launched previously to this can hit similar numbers of threads (256 for the 6980P). There are a couple of caveats to this though - firstly that there are "only" 128 physical cores and if you're using VMs you probably don't want to share a physical core across VMs, secondly that it has a 500W TDP and retails north of $17000, if you can even find one for sale. Overall once you're really comparing like to like, especially when you start trying to have 100+GbE networking and so on, it gets a lot harder to beat cloud providers - yes they have a nice fat markup but they're also paying a lot less for the hardware than you will be. Most of the time when I see takes like this it's because the org has all these fast, modern CPUs for applications that get barely any real load, and the machines are mostly sitting idle on networks that can never handle 1/100th of the traffic the machine is capable of delivering. Solving that is largely a non-technical problem not a "cloud is bad" problem.

formerly_proven 8mo ago
E-cores aren't that slow, yesteryear ones were already around Skylake levels of performance (clock for clock). Now one might say that's a 10+ year old uarch, true, but those ten years were the slowest ten years in computing since the beginning of computing, at least as far as sequential programs are concerned.
adrian_b 8mo ago
These Intel Darkmont cores are in a different performance class than the (Crestmont) E-cores used in the previous generation of Sierra Forest Xeon CPUs. For certain workloads they may have even a close to double performance per core. Darkmont is a slightly improved variant of the Skymont cores used in Arrow Lake/Lunar Lake and it has a performance very similar to the Arm Neoverse V3 cores used in Graviton5, the latest generation of custom AWS CPUs. However, a Clearwater Forest Xeon CPU has much more cores per socket than Graviton5 and it also supports dual-socket motherboards. Darkmont also has a greater performance than the older big Intel cores, like all Skylake derivatives, inclusive for AVX-using programs, so it is no longer comparable with the Atom series of cores from which it has evolved. Darkmont is not competitive in absolute performance with AMD Zen 5, but for the programs that do not use AVX-512 it has better performance per watt. However, since AMD has started to offer AVX-512 for the masses, the number of programs that have been updated to be able to benefit from AVX-512 is increasing steadily, and among them are also applications where it was not obvious that using array operations may enhance performance. Because of this pressure from AMD, it seems that this Clearwater Forest Xeon is the final product from Intel that does not support AVX-512. Both next 2 Intel CPUs support AVX-512, i.e. the Diamond Rapids Xeon, which might be launched before the end of the year, and the desktop and laptop CPU Nova Lake, whose launch has been delayed to next year (together with the desktop Zen 6, presumably due to the shortage of memories and production allocations at TSMC).

zer00eyz 8mo ago

> These sorts of core-density increases are how I win cloud debates in an org. AMD has had these sorts of densities available for a minute. > Identify the workloads that haven't scaled in a year. I have done this math recently, and you need to stop cherry picking and move everything. And build a redundant data center to boot. Compute is NOT the major issue for this sort of move: Switching and bandwidth will be major costs. 400gb is a minimum for interconnects and for most orgs you are going to need at least that much bandwidth top of rack. Storage remains problematic. You might be able to amortize compute over this time scale, but not storage. 5 years would be pushing it (depending on use). And data center storage at scale was expensive before the recent price spike. Spinning rust is viable for some tasks (backup) but will not cut it for others. Human capital: Figuring out how to support the hardware you own is going to be far more expensive than you think. You need to expect failures and staff accordingly, that means resources who are going to be, for the most part, idle.

bearjaws 8mo ago

I just don't know if the human capital is there. At my job we use HyperV, and finding someone who actually knows HyperV is difficult and expensive. Throw in Cisco networking, storage appliances, etc to make it 99.99% uptime... Also that means you have just one person, you need at least two if you don't want gaps in staffing, more likely three. Then you still need all the cloud folks to run that. We have a hybrid setup like this, and you do get a bit of best of both worlds, but ultimately managing onprem or colo infra is a huge pain in the ass. We only do it due to our business environment.

sounds 8mo ago
> I just don't know if the human capital is there. > At my job we use HyperV, and finding someone who actually knows HyperV is difficult and expensive... Try offering significantly higher pay.
- ponector 8mo ago
  Or even try to educate people. It was common to have learning programs but nowadays managers only complain you cannot find cheap experts.
  - eastabrooka 8mo ago
    "We educated the people and they left because they could get better elsewhere" - Some Manager
to11mtm 8mo ago
I think you're hitting on a general problem statement a lot of orgs run into, even ignoring the uptime figure... All of the complexity of onprem, especially when you need to worry about failover/etc can get tricky, especially if you are in a wintel env like a lot of shops are. i.e. lots of companies are doing sloppy 'just move the box to an EC2 instance' migrations because of how VMWare jacked their pricing up, and now suddenly EC2/EBS/etc costing is so cheap it's a no brain choice. I think the knowledge base to set up a minimal cost solution is too tricky to find a benefit vs all the layers (as you almost touched on, all the licensing at every layer vs a cloud provider managing...) That said, rug pulls are still a risk; I try to push for 'agnostic' workloads in architecture, if nothing else because I've seen too many cases where SaaS/PaaS/etc decide to jack up the price of a service that was cheap, and sure you could have done your own thing agnostically, but now you're there, and migrating away has a new cost. IOW, I agree; I don't think the human capital is there as far as infra folks who know how to properly set up such environments, especially hitting the 'secure+productive' side of the triangle.

justsomehnguy 8mo ago

> The only thing giving me pause on that argument today is the current RAM/NAND shortage Not a shortage - price gouging. And it would mean an increase in the 'cloud' prices because they need to refresh the HW too. So by the summer the equation would be back to it.

varispeed 8mo ago

That only works if purchasers in the organisation are immune to kickbacks.

PaulKeeble 8mo ago

What has surprised me about the cloud is that the price has been towards ever increasing prices for cores. Yet the market direction is the opposite, what used to be a 1/2 or a 1/4 of a box is now 1/256 and its faster and yet the price on the cloud has gone ever up for that core. I think their business plan is to wipe out all the people who used to maintain the on premise machines and then they can continue to charge similar prices for something that is only getting cheaper. Its hard drive and SSD space prices that stagger me on the cloud. Where one of the server CPUs might only be about 2x the price of buy a CPU for a few years if you buy less in a small system (all be it with less clock speed usually on the cloud) the drive space is at least 10-100x the price of doing it locally. Its got a bit more potential redudency but for that overhead you can repeat that data a lot of times. As time has gone on the deal of cloud has got worse as the hardware got more cores.

ragall 8mo ago
> What has surprised me about the cloud is that the price has been towards ever increasing prices for cores. That makes a lot of sense. Cloud providers are selling compute, and as cores get faster, the single core gets more expensive.

umvi 8mo ago

Is using virtualization the only good way of taking a 288-core box and splitting it up into multiple parallel workloads? One time I rented a 384-core AMD EPYC baremetal VM in GCP and I could not for the life of me get parallelized workloads to scale just using baremetal linux. I wanted to run a bunch of CPU inference jobs in parallel (with each one getting 16 cores), but the scaling was atrocious - the more parallel jobs you tried to add, the slower all of them ran. When I checked htop the CPU was very underutilized, so my theory was that there was a memory bottleneck somewhere happening with ONNX/torch (something to do with NUMA nodes?) Anyway, I wasn't able to test using proxmox or vmware on there to split up cpu/memory resources; we decided instead to just buy a bunch of smaller-core-count AMD Ryzen 1Us instead, which scaled way better with my naive approach.

Dylan16807 8mo ago
How did the speed of one or two jobs on the EPYC compare to the Ryzen? And 384 actual cores or 384 hyperthreading cores? Inference is so memory bandwidth heavy that my expectations are low. An EPYC getting 12 memory channels instead of 2 only goes so far when it has 24x as many cores.
PunchyHamster 8mo ago
They are used for VMs because the load is pretty spiky and usually not that memory heavy. For just running single app smaller core count but higher clocked ones are usually more optimal >Anyway, I wasn't able to test using proxmox or vmware on there to split up cpu/memory resources; we decided instead to just buy a bunch of smaller-core-count AMD Ryzen 1Us instead, which scaled way better with my naive approac If that was single 384 (192 times 2 for hyperthreading) CPU you are getting "only" 12 DDR5 channels, so one RAM channel is shared by 16c/32y So just plain 16 core desktop Ryzen will have double memory bandwidth per core

klooney 8mo ago

Man, how do you get box seats out of AWS, I'm missing out

fnord77 8mo ago

on prem = capex cloud = opex The accounting dept will always win this debate.

MrBuddyCasino 8mo ago

It seems a lot of people have forgotten how BigCorp IT used to work. - request some HW to run $service - the "IT dept" (really, self-interested gatekeeper) might give you something now, or in two weeks, or god help you if they need to order new hardware then its in two months, best case - there will be various weird rules on how the on-prem HW is run, who has access etc, hindering developer productivity even further - the hardware might get insanely oversubscribed so your service gets half a cpu core with 1GB RAM, because perverse incentives mean the "IT dept" gets rewarded for minimizing cost, while the price is paid by someone else - and so on... The cloud is a way around this political minefield.

misswaterfairy 8mo ago
> The cloud is a way around this political minefield. Until the bills _really_ start skyrocketing...

fennecfoxy 8mo ago

>Realize the savings to be had moving "fixed infra" back on-premises or into a colo versus sticking with a public cloud provider As other people have pointed out: what happens when the PSU or mobo shits itself, what happens when the new version of vmware or docker (or whatever) shits itself, etc

penglish1 7mo ago

I don't post much on HN, but this topic is near and dear to my heart, so here we go. Context: been helpdesk, sysadmin & network admin, DevOps, Site Reliability Engineer, in that progression, starting in the 90's. Max on-prem was 40 racks, scaled up and down over years. Many comments talking about how staffing is a key element of this equation that can't be overlooked, but I decided to reply to the root comment, which doesn't say whether/how it considers staffing. This is a complex equation - and it is relatively easy to present an incomplete or misleading picture management to push the move into the cloud.. or out of the cloud. Some factors, in no particular order: 1) Scaling: it is self-evident that pulling a single physical server worth out of the cloud is not worth it.. even for 288 cores. Or perhaps 1152 for 4xXeon in a single server. Still likely not worth it. Why? Because a single server is never just that. Someone has to swap components when it goes down. When it goes down.. ALL 1152 cores are down, along with everything they are doing. Is that acceptable for all applications running on all those cores? It is also appropriate supporting infrastructure - power, cooling, physical space. The "fairly obvious" minimum scaling is "enough servers that one can be entirely down for maintenance while keeping everything else running." But now you're paying for some overhead. At 2 servers, you're buying 2x what you need, half that capacity is idle all the time. And so on. On this point - I think the other comments talking about "each SRE managing 4 (or 5, or 7)" racks missed the point entirely. SRE's should be doing scalable work, whether in the cloud, or on-prem. And they should NOT be swapping failed hard drives and power supplies. Designing a larger-than-one rack install is probably worth hiring consultants for if you don't have that expertise in-house, though the SREs that would be supporting it would need to supply lots of input. To some extent, server & network equipment vendors can also help. It is not trivial as the scale goes up. But then it should run for some years, with relatively unskilled people handling hardware failures and you can re-engage consultants if necessary to do upgrades as hardware and needs evolve. But your SREs should be on-staff, and probably on-call to handle the software running on that hardware.. and to some extent to call the remote hands to deal with hardware failures. 2) Business needs: does the business need the tech skills that self-hosting requires for the core business? For example - if the business itself is cloud SAAS, maybe DIY-ing at least some of your infrastructure is right in your wheelhouse. If so - a modest increase in staff could mean a huge cost savings. But if not, all the cost of skilled staff to run it is simply part of the cost of in-housing this stuff. 3) Staffing: the people that swap broken hardware are not the same people that respond to pages because the business critical application crashed due to a bug. You can pay a colo facility for all this, typically by the hour - but it isn't cheap and you've got to supply all the spares etc. Is that part of your budget for on-prem? 4) On-call: maybe your self-hosted ERP system can be down every night and weekend without issues.. and even business hours can tolerate 98% uptime. But that doesn't mean you can get away without having someone on call - presumably you're hosting more than just this lowish-requirement ERP system. I'll disagree with other comments - the "no burnout" number of on-call staff you need is 6-7, not 4! Remember people take vacations too. This is well studied, and established, I'll reference Tom Limoncelli's books. This could be relatively cheap and require fewer staff with geo-distributed staff, and it would tend to overlap with staff you already use to provide on-call for anything you host on the cloud - so maybe for your situation it is close to a wash. But you can't forget to budget for it even if the line item is $0. 5) Vendor support: maybe you already have your own data center or colo and are hosting a ton of stuff. Why not move all your Atlassian stuff in house and save the hosting cost. Oh.. wups, Atlassian simply doesn't support that any more. Host it with Atlassian or GTFO. A minor point as most vendors would give you enough notice you can simply run out the lifetime of the hardware it is on and not replace it. 6) Market pricing: At one point Amazon was starting "by the minute, by the core cloud" (as opposed the older "cloudish" model of leasing only an entire physical server by the entire year) and priced a bit under market to get going. Then once they established dominance, they cranked the price WAY up for profit extraction. But now they do have some competition and they're a bit more selective about how they extract profits. In my perception they've shifted a lot of the profit taking to the value-added services rather than raw instance time, but I could be wrong. And they have HUGE costs - it is beyond naive to look at the per-hour cost of an instance and compare it to purchasing an identical physical server solely on the purchase price of that server. Corey Quinn / Duckbill group has spent a huge part of his career in this space - if you're already in the cloud 100% it is well worth optimizing those costs before you start comparing it to what on-prem might cost.