19 ms·

We decided to move 90% of our workload from the cloud to on-prem infrastructure

227 points by Protostome 4y ago

StreamBright 4y ago
"First, it’s important to note that Enzymit’s use of cloud computing mainly entailed computationally intensive calculations for protein design" So they got a very specific workload that is easy to run on any infra including on-prem.
mattymf 4y ago
> After doing the math, We decided to purchase three workstations at a total cost of $17k. Why not share the math? Isn't this article supposed to answer why it's more economical to use on-prem?
bob1029 4y ago
> We do not have (yet) any public-facing applications that need to scale across multiple geographical zones and handle millions of requests per minute. Most don't. 1mm requests per minute is very pedestrian for a single vm in virtually all cases. 1mm per second is totally reasonable too if you are careful with a few things... I genuinely believe you could put the literal public Netflix biz experience on a single VM. Account management, billing, preferences, view history, etc. The only pieces that need cloud scale are ddos mitigation, 4k video streams and media-dense static web content. Most businesses do not have strong need for these things.
- recuter 4y ago
  Correctish in a sense: https://openconnect.netflix.com/en/appliances/#the-hardware https://openconnect.netflix.com/en/appliances/#the-hardware They throw one of these boxes at an ISP and interconnect. 4-6 year no touch reliability, couple hundred TB storage. Modern hardware is quite something. They will saturate 2x100GE. That's in the thousands of concurrent streams per box.
  - Nextgrid 4y ago
    > Modern hardware is quite something. The one weird trick that cloud providers hate.
- charcircuit 4y ago
  What about Netflix's recommendation system and analytics?
  - bob1029 4y ago
    Certainly not going to fit on the real-time production instance, but two computers isn't a whole lot more than one considering how many extra features we just added. There is a lot of elegance with this type of setup too. You can have your analytics system receive a synchronous replicated log from the production system (what else is it gonna analyze?), so it can also double as a manual failover site.
- karmakaze 4y ago
  Here's more of that context that paints a better picture. > First, it’s important to note that Enzymit’s use of cloud computing mainly entailed computationally intensive calculations for protein design. We do not have (yet) any public-facing applications that need to scale across multiple geographical zones and handle millions of requests per minute. Our primary use case is running CPU and GPU heavy analyses, and for that use case, we have found the IaaS/public cloud solution to be far from cost-effective in the long term.
- rhplus 4y ago
  1 million/sec is basically line speed on a 10Gbps link if each request is coming in at MTU of 1500 bytes. Sure, you might be able to push that much data through a VM on a test bench with well-behaved local clients, but you ain’t gonna be doing that rate once you add TLS, authz, logging, throttling, non-trivial serialization, non-trivial database access, A/B tests, metrics, fraud detection, recommendations, and everything else that makes an API like Netflix tick. Whether you do that all on one VM or split across service roles, you’re gonna be much more realistically in the range of 1000 rps per CPU core.
  - bob1029 4y ago
    1500 bytes is a pretty big average payload size when you consider information theory and what actually must be communicated for this kind of business (on average). A user clicking "Watch later" on a video could theoretically be communicated in something as small as 64-bit integer for the user/session id, one for the command type, and another for the identity of the actual video. With serialization, padding, etc., you are still probably well under 50 bytes for this one event. Being sloppy with data throughout is certainly a good reason to need more pipes and servers. With enough discipline, you can process events at rates far exceeding 1 million per second with a single box and non-exotic network stack.
    - TheCoelacanth 4y ago
      1500 bytes isn't even enough to send a list of video titles and thumbnail images for a single page of videos. If you imagine that the client already has a full database of all the available videos and metadata about them, then you could get by with tiny amounts of data, but that's not even close to the actual circumstances that Netflix operates under.
recuter 4y ago
After spending $100k in a year.. After doing the math, We decided to purchase three workstations at a total cost of $17k. One is a GPU-based workstation with two RTX 3090s and an Intel i9–12900 CPU, and another two workstations with 16 cores AMD Ryzen 5950X CPUs. It took us a few FTE days to set those up to our satisfaction with slurm, NFS, backups, and several other services. We noticed that our RTXs, although considered gaming cards, are comparable (if not better) in performance to Tesla V100, which some cloud providers rent at the staggering price of $3.06 an hour. tl;dr - They did the math.
- justsomehnguy 4y ago
  > They did the math. And bought 3 workstations instead of 3 servers. IMHO the headline of "Why Enzymit Decided to Build its Own On-Prem HPC Infrastructure" is a bit... stretched.
  - raelmebrand 4y ago
    This is how universities (or people from academic background) interpret...
    - HelloNurse 4y ago
      They have a workload that is entirely dictated by their protein design etc. projects, not by external access to a web app. So "high performance" means doing a single definite task quickly (so they can *proceed to the next run), not scaling to many users and requests. If they need to scale, they first hire protein designers or the like and then they set up some new workstation, probably something different from existing ones for diversification and obsolescence reasons. No cattle.
  - uniqueuid 4y ago
    Well, they do use slurm, so it's technically a HPC stack.
- ldoughty 4y ago
  They also had a static workload, with no changing requirements, or really any need for scalability. They also had FTEs with experience to configure the systems (which for a lot of us technical people is a no-brainer, but if you had data scientists with no hardware experience, that might be different) This worked for them, and I'm happy for them.. the cloud does not solve all problems... It's simply one hardware strategy you can pick.. it's important to review the options!
lvl102 4y ago
I hate the fact that hiring now basically requires cloud experience specific to vendors. This is basically going to force people into one of the three major cloud platforms.
- pzduniak 4y ago
  Eh, it depends on the features that you're using. As long as you stick to the basic stack of Terraform + Kubernetes / IaaS with cloudinit + networking + S3-compatible storage API, you can quite easily jump between clouds. Sure, the logic that sets them up is different, but the concepts are generally roughly the same. Even if I end up choosing a managed service, I always implement a second OSS backend that gets regularly tested. Every day I deal with AWS, Google Cloud and Oracle Cloud. Previously I used DigitalOcean and OVH. I have no issues onboarding people to work with the less popular options - as long as they get how Kubernetes / Linux / containers work, it's pretty good.
  - lillecarl 4y ago
    This is what I've come to realise too, as long as you can and do stick your workloads in Kubernetes it doesn't really matter what he logo says. EKS, AKE, GKE, LKE, DOKS, OKD, Rancher... Whatever they're all compatible with what you want to do. There are definitely upsides to the cloud, but Kubernetes is the common denominator everywhere. Wanna run GPU workloads on-prem? Buy some servers and do so. The only hairy thing is managing your own storage, quite the responsibility. (Look at Atlassian right now).
    - lillecarl 4y ago
      GCE https://kubernetes.io/docs/concepts/cluster-administration/cluster-management/ https://kubernetes.io/docs/concepts/cluster-administration/c... GKE https://cloud.google.com/container-engine/docs/cluster-autoscaler https://cloud.google.com/container-engine/docs/cluster-autos... AWS https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... Azure https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/azure/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... Alibaba Cloud https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/alicloud/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... Brightbox https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/brightbox/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... OpenStack Magnum https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/magnum/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... DigitalOcean https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/digitalocean/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... CloudStack https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/cloudstack/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... Exoscale https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/exoscale/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... Equinix Metal https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/packet/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... OVHcloud https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/ovhcloud/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... Linode https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/linode/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... OCI https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/oci/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... Hetzner https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/hetzner/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... Cluster API https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/clusterapi/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... Vultr https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/vultr/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... TencentCloud https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/tencentcloud/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... These are all cloud providers that invested into their own managed Kubernetes, I'm certain all of them aren't as sleek as the big three, but it shows that there's definitely momentum behind sticking your workloads into Kubernetes.
ldoughty 4y ago
This is a good explanation of cloud issues for a company with resources and consistent workload... IaaS is not really competitive in this space, I don't think... If you have access to system admins, and have a consistent work load, you could avoid the cloud, trading the cloud premium for more employees and skills in your team. This is fine, if that is what you're team needs. The cloud is not a silver bullet that solves all companies infrastructure... But they have a very profitable space, especially in small businesses, or businesses that benefit from multiple data centers. AWS simply made it easy to scale up and down, as well as scale around the world... if you don't need to dynamically scale or easy access to multiple data centers, the cloud begins to lose it's best (cost-effective) competitive edge to self hosting... Though at the micro scale, the cloud can do dead simple basics for free -- or near enough (e.g. static websites), which is fun for personal projects
- Nextgrid 4y ago
  > If you have access to system admins Clouds also require sysadmins, they're just called "DevOps engineers" now. Those YAML & Terraform files aren't going to write themselves.
- dahfizz 4y ago
  It takes just as much, if not more, IT work to maintain cloud infrastructure as local infra. People who have never managed servers imagine them blowing up once a week or something.
bmj 4y ago
Starting a web-based or SaaS (Software as a Service) business was virtually unheard of before the age of IaaS (Infrastructure as a Service) companies. There were simply too many hurdles, and the expenses were too high — you’d have to purchase dedicated servers and a high bandwidth connection to handle a load of incoming visitors to your website, hire engineers to build a scalable system, and if you planned to go international, you would have to purchase servers in other geographical locations. The first internet boom was exactly what the author describes. I worked for an internet e-commerce start-up in 1999, and guess what? We had dedicated hosting bandwidth coming into our office, and the hardware and software in place to serve our application. Of course, one of the founders was a sysadmin, but I had friends working for similar start-ups, and they leveraged one of the many co-location providers in the city to manage all the details. Yet their employer still owned actual hardware that they could touch when necessary.
- vidarh 4y ago
  I cofounded an ISP in '95. There were plenty of alternatives offered by ISPs like mine. People often chose to host themselves when they had the skills, sure.
manigandham 4y ago
> "Starting a web-based or SaaS (Software as a Service) business was virtually unheard of before the age of IaaS" Nonsense. There were plenty of SaaS startups. There was even a little event called the dotcom boom all about internet companies. This lack of history and experience is why new companies get into this cloud-first mess in the first place. Cloud is primarily for flexibility in iteration, dynamic scaling, or complex configurations that would be otherwise hard to do. If you have steady-state load like this then a few servers in your colo is pretty simple and far cheaper. Companies also vastly overestimate their scale when their entire business could probably fit on a single commodity server.
- xuki 4y ago
  I'd say don't even consider colo unless you have a specific use case. Rented dedicated servers are cheap, let someone else take care of the hardware.
  - manigandham 4y ago
    True, there's a wide spectrum in the middle from rented colo to managed servers.
  - bsenftner 4y ago
    Yes, they are cheap. Running one's own server is also easy peasy; far too many think it is difficult, it is not. The most expensive part is the electricity.
    - maccard 4y ago
      > The most expensive part is the electricity. No, the most expensive part is the persons time for managing it. I can rent a monstrous Dedicated Server for $400/month from OVH, but even with a UK salary, if I have to spend more than 1 day a month on it in any shape or form (and that includes the initial setup), it's cheaper to use "the cloud" or some form of a managed service.
      - 4y ago
baybal2 4y ago
lioeters 4y ago
Just want to highlight how futuristic the author's title is: "Computational Biologist, Head of Protein Design @ Enzymit". Why they moved to on-prem: lower and more predictable cost. At a public cloud provider, they lost thousands of dollars (of free credit they had) through "architectural blunders". And the running cost of GPU, CPU, storage, and data transfer summed up to $10K a month - at which point they figured they might as well purchase their own compute servers.
- tonyedgecombe 4y ago
  >"architectural blunders" I wonder how much of this is by design. It seems it is in Amazon's interest to keep their systems and pricing as opaque as possible.
  - mathattack 4y ago
    At MSFT the Azure solutions architects are part of the sales organization. Their commissions are tied to usage (revenue) which tells you everything about their skill sets. And their lack of standard clear pricing means you have to estimate everything yourself.
  - maccard 4y ago
    I've spent a lot of time with Amazon's Architects who have given us a huge amount of advice on how we could reduce our costs. Maybe we've gotten lucky with our contact, but their approach for keeping us on AWS seems to be "provide enough value on top of AWS that the extra pricing doesn't really matter" mostly in the form of handing us architecture templates for common workflows that we've never set up.
- sokoloff 4y ago
  Now, when they make blunders, they don’t see an overt bill for them and are thereby happier.
  - emteycz 4y ago
    Yeah, but when they do a blunder, they see they're not getting the results they expected and fix it, without any fear of 10x costs.
fxtentacle 4y ago
Wholeheartedly agree. After my cloud storage costs exploded (mostly S3 egress over to Hetzner/OVH), I noticed that renting a 1GBit/s fiber connection to my office is actually quite affordable at $80 per month (in northern Germany). Of course, it's not globally distributed and there is no fail-safe, but for the work we're doing, that is no issue. If we employees are offline, it doesn't matter if our tools are offline, too. And now that all AI storage is local anyway, building a GPU compute node is easy. I'm still waiting for 3090 prices to drop further, though, in contrast to the article. But I also went with Ryzon 5950 and Linux. I was positively surprised that 10G fiber networking is now down to $70 for a PCIe card + 20m cable kit. My workstation now has 1010MB/s 4k random write on the network filesystem. (We used SAMBA 3.1 and CIFS mounts) I also grabbed the python/ubuntu package lists off Google Colab and created my own Docker to imitate it, and now data processing and AI training is fast (I always get the good GPU, no luck involved) and dirt cheap. Originally the idea was to run it on OVH, but I'm now also running it locally. https://github.com/fxtentacle/ovh-colab-sagemaker-compatibility-mode/blob/master/Dockerfile https://github.com/fxtentacle/ovh-colab-sagemaker-compatibil...
- GordonS 4y ago
  > I was positively surprised that 10G fiber networking is now down to $70 for a PCIe card + 20m cable kit. Wow, that is surprising, maybe it's time I started upgrading my LAN...
  - fxtentacle 4y ago
    I use Intel 82599ES SFP+ and a TL-SX3008F router. But let me warn you: Things are affordable, but NOT consumer-friendly. I needed to study the 500 page PDF manual and do basic link configuration through telnet via USB before I could connect to the router via Ethernet and use its web-GUI to finish the setup.
    - sofixa 4y ago
      How's the routing performance on that switch? I'm really struggling finding a device to fit my needs ( routing, 6-8 ports, Ethernet, at least two 2.5/5/10G).
zmmmmm 4y ago
Hope he's prepared for a letter from nVidia's lawyers for breaking their license agreement for the RTX3090's.
- piker 4y ago
  Is all commercial use of the consumer grade nVidia products prohibited under that license? I thought (and this seems to agree: https://www.nvidia.com/en-gb/drivers/geforce-license/ https://www.nvidia.com/en-gb/drivers/geforce-license/) that it was just use in a datacenter that was prohibited.
  - zmmmmm 4y ago
    I reckon they will consider what he has constructed to be a data center. A lot of places do it and just fly under the radar, but if you're going to publish a blog post bragging about it and how much money you are saving ...
    - penultimatename 4y ago
      They run two GPUs. I highly doubt NVIDIA, their lawyers, or a judge would consider that a datacenter. The license is clearly directed at a different crowd.
    - TheGuyWhoCodes 4y ago
      I don't think so. These are just workstations. It's not like they have a a rack full of 4u servers with RTX3900s and even then it's for their own usage. The RTX datacenter restriction as far as read, but not a lawyer, is for data center providers like aws, ovh, hetzner etc to provide servers with these gpu and rent them.
    - athorax 4y ago
      It doesn't matter if they consider these 3 servers to be a "datacenter." There are legal definitions[0] that this usage doesn't fit in at all. Unless nvidia provides a different definition in their license (which they don't) [0]https://www.law.cornell.edu/uscode/text/42/17112#a_1 https://www.law.cornell.edu/uscode/text/42/17112#a_1
jollybean 4y ago
Yeah, for on-prem scientific work, heavy CPU usage, probably bursty, no need for all the fancy things, security etc. - this is a pretty straight forward case for on-prem use. Or not even on-prem, just renting some physical boxes from some place where they're only going to have a basic markup. You have to wait a week to get few more boxes but that's not a big deal. This is the kind of thing I'd imagine a corp would start doing pretty early, because the 'flexibility' of IaaS just isn't worth the cost.
_0w8t 4y ago
5 years ago we evaluated cold storage for 100 TB of data. Even at that relatively small amount magnetic tapes were much more cheaper even after accounting for an extra copy to ship to a remote location.
reacharavindh 4y ago
The flexibility and scalability of the cloud comes at a cost. Scientific, static and non web but IO heavy workloads are almost always better off run on-prem or in a co-located data centre with servers paid up front. As is most things in businesses, it’s a trade-off one needs to think about and make. The smart ones will come out ahead, and those that blindly follow FAANG and drink their kool-aid may or may not come ahead, albeit with a heavy hit to their(or their VC’s) pocket. If you’re a startup that simply have a bunch of web apps and APIs where uptime and network are your major costs, on-prem is only going to become a worthless headache. A good Systems Engineer should help to figure out such choices. Anybody need one?
nikanj 4y ago
1) You save a metric boatload of money this way 2) Onprem experience looks bad on your resume, compared to cloud experience 3) Incentives work for people, and resume-driven development is key, as our industry is very stingy in passing the savings from 1) to the developers
- dgb23 4y ago
  Many care more about cs and engineering fundamentals rather than bespoke products, even more people only care about having their problem solved and how much it costs.
Maxburn 4y ago
We came to the same conclusion. We do hosing for commercial HVAC systems and due to software requirements and the human factor of training the VM's we had in azure cost us significantly more than on premises servers. Add to that we generally keep those servers a little longer than some others would just increases the savings. This is still true even though we pay a colocation to host those servers. The cloud flexibility is completely irrelevant for us in light of multi year service contracts from each customer we work with.
SpaceMartini 4y ago
I work in HPC for a cloud provider, and fully endorse this move. Anonymously, of course. You can make an economic argument for or against cloud in practically every IT domain, but in HPC the case for on-prem is really compelling; none of the cloud networking/resiliency value-add is relevant to batch workflows, and costs per core-hour are only remotely comparable if you use spot - which is itself a major compromise. The only real advantage cloud has for science is object storage, which is genuinely a much better idea than trying to manage your own long-term archival storage. If I were independent I would recommend people buy and build on-prem clusters and shuffle data out of fast scratch into Glacier, but other than that just don't worry about cloud until price pressure kicks in and we are down to 1-2 cents per core-hour on-demand. I'd love a role where I can say these things non-anonymously, but the salary for such a position would be at least 50% lower than working for a cloud provider. Keep that in mind when talking to your supplier - we may not believe the pitch ourselves, but making it is just part of the job.
- SpaceMartini 4y ago
  As an addendum to this: if you absolutely must use cloud, stick with AWS. Using Azure is (IMO) a fucking miserable experience and their only advantage (InfiniBand) is better served by buying your own hardware. GCP and OCI might be fine if you are getting a lot of credits, but the skills will not be useful down the line - while AWS is expensive, you will at least learn a bunch of in-demand operational skills.
  - briffle 4y ago
    So nobody got fired for buying IBM?
    - smm11 4y ago
      I can hang 32 terminals off just one PC. You're still blowing your budget on standalones.
      - earleybird 4y ago
_pdp_ 4y ago
There are a lot of hidden costs for not using a cloud too. One of them is security. Frankly, you won't come up with anything better than AWS IAM in the short term. If you don't care much about that, then it is a different story, but I am happy to pay the premium knowing that all data is adequately encrypted and all lambda functions have the minimum permissions required to run.
dbrowne 4y ago
It is a question of the firm's model. A one time cost + electricity - depreciation or a recurring, variable monthly cost that may exceed the the cost of the hardware? I have two used workstations that cost less than 6 months of GCE time. Does that work for me? yes might not work for others. (especially if the workstation dies)
kuon 4y ago
On premise can be a lot cheaper for bandwidth intensive app. We have a 10gbit/s dedicated fiber we use at like 80%, it cost us 1500$ per month (power+fiber). This kind of bandwidth is 10 to 100 times more expensive on the cloud depending on the service. Of course, no CDN, but as our customers are mostly local, we don't need global presence. Also, we serve this from a single epyc based server, using elixir/phoenix. It's at about 6gbit/s outbound traffic. I realize this is not uber redundant, but it works and keep the costs low.
yubiox 4y ago
It is on-premises not on-premise. Why is this so hard?
- ternaryoperator 4y ago
  > Why is this so hard? Well, for one, because it's an exceptionally rare English word that is singular in meaning but plural in its usage. In addition, there is an unrelated English word that is singular in both meaning and construction. That confusion would naturally arise seems almost pre-ordained.
stephen_g 4y ago
As a random aside, I’m glad to see ‘on-prem’ emerge as a common shorthand for this, only because it always grates me to see people make the extremely common mistake of saying ‘on-premise’. Premise, of course, only ever means “an idea or theory on which a statement or action is based”, whereas the actual term is premises (“the land and buildings owned by someone, especially by a company or organisation”, as in “The security guards escorted the protesters off the premises”).
AtlasBarfed 4y ago
Why did openstack fail? Or did it, was it just not adopted? I think there is still a lot of potential for open source management of core EC2/S3/networking capabilities (aka "core AWS IAAS service"). We have a fair number of cloud abstraction layers now, and obviously kubernetes, you'd think we could as an industry produce core apis for doing resource listing, availability, etc. Maybe some of the problem is that devs have a LOT of experience with the "ask" side of IaaS: gimme storage, gimme vms, set. But they have no experience with the "provide" side, and the sort of one-off manual nature of installing networking and machines doesn't have good standardization for "reporting available resources". At this point, aws apis are somewhat stable. (I would bitch about the error codes and documentation... but anyway). It's obviously "good enough" after 10-15 years of them. Are there projects that try to marry an aws-ish api, which really is a reporting and request api, with a "available resources" reporting api? Are some of these things out there? AWS ten years ago was liberating. It was progress. It was a good thing. But Amazon is not a "do no evil" corporation, much the opposite. And you see this in AWS with its treatment of startups, open source projects, and other manipulations. They are a monopoly now, or at a minimum a dangerous cartel. A real open source alternative would be a good thing. It would be good for the rest of FAANG, it would encourage competition by allowing lesser clouds to offer core competencies that are drop-in.
gnarcoregrizz 4y ago
<tinfoil hat> I always got the impression that there has been a lot of cloud propaganda/astroturfing, even on HN. I'm seeing more of these "on-prem infrastructure" posts, citing costs, efficiency, and cloud complexity. We run most of our infra on-prem, and have looked to moving a few bits and pieces to the cloud, but the math almost always works out to buying hardware. Meanwhile, I talk to some <cloud stack> friends, and their opex costs are astronomical for the traffic and size of their products. The cloud is extremely convenient, and I would choose it if I were launching a new product, but past certain sizes and expenses I would start to do some math. It's not terribly difficult to run these cloud "shrinkwrapped" products (such as load balancers) on-prem. Things like object storage seem more difficult to me. I'm also hesitant to admin a database, they intimidate me :)
- thefunnyman 4y ago
  I think the reality is that most companies don’t have the skill set needed to maintain on-prem infra. A lot of us here take this kind of knowledge for granted, but most businesses don’t have the time or resources to build some of this stuff themselves so they reach for the cloud. You also see a lot of VC backed companies splurge on cloud as a trade off of money for rapid growth.
slac 4y ago
The issue I see is that the public clouds often offer carbon free or carbon offset power. When you deploy onprem... This often gets ignored.
paxys 4y ago
If you are paying the cloud premium make sure you actually utilize the advantages it offers – global distribution, elasticity, rapid scaling, high availability, disaster recovery. Otherwise what's the point? If your needs can be met by shoving a couple thousand dollars worth of hardware into a closet somewhere that is obviously going to be a lot cheaper.
nijave 4y ago
>Adding storage, CPU time, and many other costs makes understanding and verifying the cost structure a task suitable for certified experts Is that easier on-prem? I was under the impression it was even more difficult--especially with shared tenancy (how much incremental cost does App A auth add to our Active Directory deployment?) You're going to need to know server power utilization under load to calculate power/cooling costs and probably some additional data on network utilization to figure out incremental costs for that Maybe if it's a colo or managed data center that gets rolled up for you, but if you're managing yourself, you still have to figure it out Blog post also doesn't mention cost of downtime (maybe not an issue for them) or a metrics solution (you usually get basic machine and service metrics for free on the big cloud providers)
Delitio 4y ago
The calculation is very thin in my opinion. 10k of infra costs are not a lot of money in a business context. A person to operate a private cloud with OnCall, backup hardware, capacity planing etc. costs what? I would always try to have GPU on prem as those prices are quite high bit others I would use managed. Cloud providers are just much better in operating infrastructure ad normal ops.
ksec 4y ago
>> "Starting a web-based or SaaS (Software as a Service) business was virtually unheard of before the age of IaaS" > "...off-load many of its processes to an on-premise private cloud?" On-Premise Private Cloud I dont know why reading this my brain just cant compute. You mean you have three consumer grade computer with Dedicated GPU running 24x7?
jtsymonds 4y ago
Many companies that do this look at MinIO for object storage. Given they run in AWS, GCP and Azure, they will minimize or eliminate your application rewrites. They are cloud-native by design and very fast.

227 points by Protostome 4y ago

"First, it’s important to note that Enzymit’s use of cloud computing mainly entailed computationally intensive calculations for protein design" So they got a very specific workload that is easy to run on any infra including on-prem.

mattymf 4y ago

> After doing the math, We decided to purchase three workstations at a total cost of $17k. Why not share the math? Isn't this article supposed to answer why it's more economical to use on-prem?

bob1029 4y ago

> We do not have (yet) any public-facing applications that need to scale across multiple geographical zones and handle millions of requests per minute. Most don't. 1mm requests per minute is very pedestrian for a single vm in virtually all cases. 1mm per second is totally reasonable too if you are careful with a few things... I genuinely believe you could put the literal public Netflix biz experience on a single VM. Account management, billing, preferences, view history, etc. The only pieces that need cloud scale are ddos mitigation, 4k video streams and media-dense static web content. Most businesses do not have strong need for these things.

recuter 4y ago
Correctish in a sense: https://openconnect.netflix.com/en/appliances/#the-hardware https://openconnect.netflix.com/en/appliances/#the-hardware They throw one of these boxes at an ISP and interconnect. 4-6 year no touch reliability, couple hundred TB storage. Modern hardware is quite something. They will saturate 2x100GE. That's in the thousands of concurrent streams per box.
- Nextgrid 4y ago
  > Modern hardware is quite something. The one weird trick that cloud providers hate.
charcircuit 4y ago
What about Netflix's recommendation system and analytics?
- bob1029 4y ago
  Certainly not going to fit on the real-time production instance, but two computers isn't a whole lot more than one considering how many extra features we just added. There is a lot of elegance with this type of setup too. You can have your analytics system receive a synchronous replicated log from the production system (what else is it gonna analyze?), so it can also double as a manual failover site.
karmakaze 4y ago
Here's more of that context that paints a better picture. > First, it’s important to note that Enzymit’s use of cloud computing mainly entailed computationally intensive calculations for protein design. We do not have (yet) any public-facing applications that need to scale across multiple geographical zones and handle millions of requests per minute. Our primary use case is running CPU and GPU heavy analyses, and for that use case, we have found the IaaS/public cloud solution to be far from cost-effective in the long term.
rhplus 4y ago
1 million/sec is basically line speed on a 10Gbps link if each request is coming in at MTU of 1500 bytes. Sure, you might be able to push that much data through a VM on a test bench with well-behaved local clients, but you ain’t gonna be doing that rate once you add TLS, authz, logging, throttling, non-trivial serialization, non-trivial database access, A/B tests, metrics, fraud detection, recommendations, and everything else that makes an API like Netflix tick. Whether you do that all on one VM or split across service roles, you’re gonna be much more realistically in the range of 1000 rps per CPU core.
- bob1029 4y ago
  1500 bytes is a pretty big average payload size when you consider information theory and what actually must be communicated for this kind of business (on average). A user clicking "Watch later" on a video could theoretically be communicated in something as small as 64-bit integer for the user/session id, one for the command type, and another for the identity of the actual video. With serialization, padding, etc., you are still probably well under 50 bytes for this one event. Being sloppy with data throughout is certainly a good reason to need more pipes and servers. With enough discipline, you can process events at rates far exceeding 1 million per second with a single box and non-exotic network stack.
  - TheCoelacanth 4y ago
    1500 bytes isn't even enough to send a list of video titles and thumbnail images for a single page of videos. If you imagine that the client already has a full database of all the available videos and metadata about them, then you could get by with tiny amounts of data, but that's not even close to the actual circumstances that Netflix operates under.

recuter 4y ago

After spending $100k in a year.. After doing the math, We decided to purchase three workstations at a total cost of $17k. One is a GPU-based workstation with two RTX 3090s and an Intel i9–12900 CPU, and another two workstations with 16 cores AMD Ryzen 5950X CPUs. It took us a few FTE days to set those up to our satisfaction with slurm, NFS, backups, and several other services. We noticed that our RTXs, although considered gaming cards, are comparable (if not better) in performance to Tesla V100, which some cloud providers rent at the staggering price of $3.06 an hour. tl;dr - They did the math.

justsomehnguy 4y ago
> They did the math. And bought 3 workstations instead of 3 servers. IMHO the headline of "Why Enzymit Decided to Build its Own On-Prem HPC Infrastructure" is a bit... stretched.
- raelmebrand 4y ago
  This is how universities (or people from academic background) interpret...
  - HelloNurse 4y ago
    They have a workload that is entirely dictated by their protein design etc. projects, not by external access to a web app. So "high performance" means doing a single definite task quickly (so they can *proceed to the next run), not scaling to many users and requests. If they need to scale, they first hire protein designers or the like and then they set up some new workstation, probably something different from existing ones for diversification and obsolescence reasons. No cattle.
- uniqueuid 4y ago
  Well, they do use slurm, so it's technically a HPC stack.
ldoughty 4y ago
They also had a static workload, with no changing requirements, or really any need for scalability. They also had FTEs with experience to configure the systems (which for a lot of us technical people is a no-brainer, but if you had data scientists with no hardware experience, that might be different) This worked for them, and I'm happy for them.. the cloud does not solve all problems... It's simply one hardware strategy you can pick.. it's important to review the options!

lvl102 4y ago

I hate the fact that hiring now basically requires cloud experience specific to vendors. This is basically going to force people into one of the three major cloud platforms.

pzduniak 4y ago
Eh, it depends on the features that you're using. As long as you stick to the basic stack of Terraform + Kubernetes / IaaS with cloudinit + networking + S3-compatible storage API, you can quite easily jump between clouds. Sure, the logic that sets them up is different, but the concepts are generally roughly the same. Even if I end up choosing a managed service, I always implement a second OSS backend that gets regularly tested. Every day I deal with AWS, Google Cloud and Oracle Cloud. Previously I used DigitalOcean and OVH. I have no issues onboarding people to work with the less popular options - as long as they get how Kubernetes / Linux / containers work, it's pretty good.
- lillecarl 4y ago
  This is what I've come to realise too, as long as you can and do stick your workloads in Kubernetes it doesn't really matter what he logo says. EKS, AKE, GKE, LKE, DOKS, OKD, Rancher... Whatever they're all compatible with what you want to do. There are definitely upsides to the cloud, but Kubernetes is the common denominator everywhere. Wanna run GPU workloads on-prem? Buy some servers and do so. The only hairy thing is managing your own storage, quite the responsibility. (Look at Atlassian right now).
  - lillecarl 4y ago
    GCE https://kubernetes.io/docs/concepts/cluster-administration/cluster-management/ https://kubernetes.io/docs/concepts/cluster-administration/c... GKE https://cloud.google.com/container-engine/docs/cluster-autoscaler https://cloud.google.com/container-engine/docs/cluster-autos... AWS https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... Azure https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/azure/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... Alibaba Cloud https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/alicloud/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... Brightbox https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/brightbox/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... OpenStack Magnum https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/magnum/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... DigitalOcean https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/digitalocean/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... CloudStack https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/cloudstack/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... Exoscale https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/exoscale/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... Equinix Metal https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/packet/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... OVHcloud https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/ovhcloud/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... Linode https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/linode/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... OCI https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/oci/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... Hetzner https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/hetzner/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... Cluster API https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/clusterapi/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... Vultr https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/vultr/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... TencentCloud https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/tencentcloud/README.md https://github.com/kubernetes/autoscaler/blob/master/cluster... These are all cloud providers that invested into their own managed Kubernetes, I'm certain all of them aren't as sleek as the big three, but it shows that there's definitely momentum behind sticking your workloads into Kubernetes.

ldoughty 4y ago

This is a good explanation of cloud issues for a company with resources and consistent workload... IaaS is not really competitive in this space, I don't think... If you have access to system admins, and have a consistent work load, you could avoid the cloud, trading the cloud premium for more employees and skills in your team. This is fine, if that is what you're team needs. The cloud is not a silver bullet that solves all companies infrastructure... But they have a very profitable space, especially in small businesses, or businesses that benefit from multiple data centers. AWS simply made it easy to scale up and down, as well as scale around the world... if you don't need to dynamically scale or easy access to multiple data centers, the cloud begins to lose it's best (cost-effective) competitive edge to self hosting... Though at the micro scale, the cloud can do dead simple basics for free -- or near enough (e.g. static websites), which is fun for personal projects

Nextgrid 4y ago
> If you have access to system admins Clouds also require sysadmins, they're just called "DevOps engineers" now. Those YAML & Terraform files aren't going to write themselves.
dahfizz 4y ago
It takes just as much, if not more, IT work to maintain cloud infrastructure as local infra. People who have never managed servers imagine them blowing up once a week or something.

bmj 4y ago

Starting a web-based or SaaS (Software as a Service) business was virtually unheard of before the age of IaaS (Infrastructure as a Service) companies. There were simply too many hurdles, and the expenses were too high — you’d have to purchase dedicated servers and a high bandwidth connection to handle a load of incoming visitors to your website, hire engineers to build a scalable system, and if you planned to go international, you would have to purchase servers in other geographical locations. The first internet boom was exactly what the author describes. I worked for an internet e-commerce start-up in 1999, and guess what? We had dedicated hosting bandwidth coming into our office, and the hardware and software in place to serve our application. Of course, one of the founders was a sysadmin, but I had friends working for similar start-ups, and they leveraged one of the many co-location providers in the city to manage all the details. Yet their employer still owned actual hardware that they could touch when necessary.

vidarh 4y ago
I cofounded an ISP in '95. There were plenty of alternatives offered by ISPs like mine. People often chose to host themselves when they had the skills, sure.

manigandham 4y ago

> "Starting a web-based or SaaS (Software as a Service) business was virtually unheard of before the age of IaaS" Nonsense. There were plenty of SaaS startups. There was even a little event called the dotcom boom all about internet companies. This lack of history and experience is why new companies get into this cloud-first mess in the first place. Cloud is primarily for flexibility in iteration, dynamic scaling, or complex configurations that would be otherwise hard to do. If you have steady-state load like this then a few servers in your colo is pretty simple and far cheaper. Companies also vastly overestimate their scale when their entire business could probably fit on a single commodity server.

xuki 4y ago
I'd say don't even consider colo unless you have a specific use case. Rented dedicated servers are cheap, let someone else take care of the hardware.
- manigandham 4y ago
  True, there's a wide spectrum in the middle from rented colo to managed servers.
- bsenftner 4y ago
  Yes, they are cheap. Running one's own server is also easy peasy; far too many think it is difficult, it is not. The most expensive part is the electricity.
  - maccard 4y ago
    > The most expensive part is the electricity. No, the most expensive part is the persons time for managing it. I can rent a monstrous Dedicated Server for $400/month from OVH, but even with a UK salary, if I have to spend more than 1 day a month on it in any shape or form (and that includes the initial setup), it's cheaper to use "the cloud" or some form of a managed service.
    - 4y ago

baybal2 4y ago

lioeters 4y ago

Just want to highlight how futuristic the author's title is: "Computational Biologist, Head of Protein Design @ Enzymit". Why they moved to on-prem: lower and more predictable cost. At a public cloud provider, they lost thousands of dollars (of free credit they had) through "architectural blunders". And the running cost of GPU, CPU, storage, and data transfer summed up to $10K a month - at which point they figured they might as well purchase their own compute servers.

tonyedgecombe 4y ago
>"architectural blunders" I wonder how much of this is by design. It seems it is in Amazon's interest to keep their systems and pricing as opaque as possible.
- mathattack 4y ago
  At MSFT the Azure solutions architects are part of the sales organization. Their commissions are tied to usage (revenue) which tells you everything about their skill sets. And their lack of standard clear pricing means you have to estimate everything yourself.
- maccard 4y ago
  I've spent a lot of time with Amazon's Architects who have given us a huge amount of advice on how we could reduce our costs. Maybe we've gotten lucky with our contact, but their approach for keeping us on AWS seems to be "provide enough value on top of AWS that the extra pricing doesn't really matter" mostly in the form of handing us architecture templates for common workflows that we've never set up.
sokoloff 4y ago
Now, when they make blunders, they don’t see an overt bill for them and are thereby happier.
- emteycz 4y ago
  Yeah, but when they do a blunder, they see they're not getting the results they expected and fix it, without any fear of 10x costs.

fxtentacle 4y ago

Wholeheartedly agree. After my cloud storage costs exploded (mostly S3 egress over to Hetzner/OVH), I noticed that renting a 1GBit/s fiber connection to my office is actually quite affordable at $80 per month (in northern Germany). Of course, it's not globally distributed and there is no fail-safe, but for the work we're doing, that is no issue. If we employees are offline, it doesn't matter if our tools are offline, too. And now that all AI storage is local anyway, building a GPU compute node is easy. I'm still waiting for 3090 prices to drop further, though, in contrast to the article. But I also went with Ryzon 5950 and Linux. I was positively surprised that 10G fiber networking is now down to $70 for a PCIe card + 20m cable kit. My workstation now has 1010MB/s 4k random write on the network filesystem. (We used SAMBA 3.1 and CIFS mounts) I also grabbed the python/ubuntu package lists off Google Colab and created my own Docker to imitate it, and now data processing and AI training is fast (I always get the good GPU, no luck involved) and dirt cheap. Originally the idea was to run it on OVH, but I'm now also running it locally. https://github.com/fxtentacle/ovh-colab-sagemaker-compatibility-mode/blob/master/Dockerfile https://github.com/fxtentacle/ovh-colab-sagemaker-compatibil...

GordonS 4y ago
> I was positively surprised that 10G fiber networking is now down to $70 for a PCIe card + 20m cable kit. Wow, that is surprising, maybe it's time I started upgrading my LAN...
- fxtentacle 4y ago
  I use Intel 82599ES SFP+ and a TL-SX3008F router. But let me warn you: Things are affordable, but NOT consumer-friendly. I needed to study the 500 page PDF manual and do basic link configuration through telnet via USB before I could connect to the router via Ethernet and use its web-GUI to finish the setup.
  - sofixa 4y ago
    How's the routing performance on that switch? I'm really struggling finding a device to fit my needs ( routing, 6-8 ports, Ethernet, at least two 2.5/5/10G).

zmmmmm 4y ago

Hope he's prepared for a letter from nVidia's lawyers for breaking their license agreement for the RTX3090's.

piker 4y ago
Is all commercial use of the consumer grade nVidia products prohibited under that license? I thought (and this seems to agree: https://www.nvidia.com/en-gb/drivers/geforce-license/ https://www.nvidia.com/en-gb/drivers/geforce-license/) that it was just use in a datacenter that was prohibited.
- zmmmmm 4y ago
  I reckon they will consider what he has constructed to be a data center. A lot of places do it and just fly under the radar, but if you're going to publish a blog post bragging about it and how much money you are saving ...
  - penultimatename 4y ago
    They run two GPUs. I highly doubt NVIDIA, their lawyers, or a judge would consider that a datacenter. The license is clearly directed at a different crowd.
  - TheGuyWhoCodes 4y ago
    I don't think so. These are just workstations. It's not like they have a a rack full of 4u servers with RTX3900s and even then it's for their own usage. The RTX datacenter restriction as far as read, but not a lawyer, is for data center providers like aws, ovh, hetzner etc to provide servers with these gpu and rent them.
  - athorax 4y ago
    It doesn't matter if they consider these 3 servers to be a "datacenter." There are legal definitions[0] that this usage doesn't fit in at all. Unless nvidia provides a different definition in their license (which they don't) [0]https://www.law.cornell.edu/uscode/text/42/17112#a_1 https://www.law.cornell.edu/uscode/text/42/17112#a_1

jollybean 4y ago

Yeah, for on-prem scientific work, heavy CPU usage, probably bursty, no need for all the fancy things, security etc. - this is a pretty straight forward case for on-prem use. Or not even on-prem, just renting some physical boxes from some place where they're only going to have a basic markup. You have to wait a week to get few more boxes but that's not a big deal. This is the kind of thing I'd imagine a corp would start doing pretty early, because the 'flexibility' of IaaS just isn't worth the cost.

_0w8t 4y ago

5 years ago we evaluated cold storage for 100 TB of data. Even at that relatively small amount magnetic tapes were much more cheaper even after accounting for an extra copy to ship to a remote location.

reacharavindh 4y ago

The flexibility and scalability of the cloud comes at a cost. Scientific, static and non web but IO heavy workloads are almost always better off run on-prem or in a co-located data centre with servers paid up front. As is most things in businesses, it’s a trade-off one needs to think about and make. The smart ones will come out ahead, and those that blindly follow FAANG and drink their kool-aid may or may not come ahead, albeit with a heavy hit to their(or their VC’s) pocket. If you’re a startup that simply have a bunch of web apps and APIs where uptime and network are your major costs, on-prem is only going to become a worthless headache. A good Systems Engineer should help to figure out such choices. Anybody need one?

nikanj 4y ago

1) You save a metric boatload of money this way 2) Onprem experience looks bad on your resume, compared to cloud experience 3) Incentives work for people, and resume-driven development is key, as our industry is very stingy in passing the savings from 1) to the developers

dgb23 4y ago
Many care more about cs and engineering fundamentals rather than bespoke products, even more people only care about having their problem solved and how much it costs.

Maxburn 4y ago

We came to the same conclusion. We do hosing for commercial HVAC systems and due to software requirements and the human factor of training the VM's we had in azure cost us significantly more than on premises servers. Add to that we generally keep those servers a little longer than some others would just increases the savings. This is still true even though we pay a colocation to host those servers. The cloud flexibility is completely irrelevant for us in light of multi year service contracts from each customer we work with.

SpaceMartini 4y ago

I work in HPC for a cloud provider, and fully endorse this move. Anonymously, of course. You can make an economic argument for or against cloud in practically every IT domain, but in HPC the case for on-prem is really compelling; none of the cloud networking/resiliency value-add is relevant to batch workflows, and costs per core-hour are only remotely comparable if you use spot - which is itself a major compromise. The only real advantage cloud has for science is object storage, which is genuinely a much better idea than trying to manage your own long-term archival storage. If I were independent I would recommend people buy and build on-prem clusters and shuffle data out of fast scratch into Glacier, but other than that just don't worry about cloud until price pressure kicks in and we are down to 1-2 cents per core-hour on-demand. I'd love a role where I can say these things non-anonymously, but the salary for such a position would be at least 50% lower than working for a cloud provider. Keep that in mind when talking to your supplier - we may not believe the pitch ourselves, but making it is just part of the job.

SpaceMartini 4y ago
As an addendum to this: if you absolutely must use cloud, stick with AWS. Using Azure is (IMO) a fucking miserable experience and their only advantage (InfiniBand) is better served by buying your own hardware. GCP and OCI might be fine if you are getting a lot of credits, but the skills will not be useful down the line - while AWS is expensive, you will at least learn a bunch of in-demand operational skills.
- briffle 4y ago
  So nobody got fired for buying IBM?
  - smm11 4y ago
    I can hang 32 terminals off just one PC. You're still blowing your budget on standalones.
    - earleybird 4y ago

_pdp_ 4y ago

There are a lot of hidden costs for not using a cloud too. One of them is security. Frankly, you won't come up with anything better than AWS IAM in the short term. If you don't care much about that, then it is a different story, but I am happy to pay the premium knowing that all data is adequately encrypted and all lambda functions have the minimum permissions required to run.

dbrowne 4y ago

It is a question of the firm's model. A one time cost + electricity - depreciation or a recurring, variable monthly cost that may exceed the the cost of the hardware? I have two used workstations that cost less than 6 months of GCE time. Does that work for me? yes might not work for others. (especially if the workstation dies)

kuon 4y ago

On premise can be a lot cheaper for bandwidth intensive app. We have a 10gbit/s dedicated fiber we use at like 80%, it cost us 1500$ per month (power+fiber). This kind of bandwidth is 10 to 100 times more expensive on the cloud depending on the service. Of course, no CDN, but as our customers are mostly local, we don't need global presence. Also, we serve this from a single epyc based server, using elixir/phoenix. It's at about 6gbit/s outbound traffic. I realize this is not uber redundant, but it works and keep the costs low.

yubiox 4y ago

It is on-premises not on-premise. Why is this so hard?

ternaryoperator 4y ago
> Why is this so hard? Well, for one, because it's an exceptionally rare English word that is singular in meaning but plural in its usage. In addition, there is an unrelated English word that is singular in both meaning and construction. That confusion would naturally arise seems almost pre-ordained.

stephen_g 4y ago

As a random aside, I’m glad to see ‘on-prem’ emerge as a common shorthand for this, only because it always grates me to see people make the extremely common mistake of saying ‘on-premise’. Premise, of course, only ever means “an idea or theory on which a statement or action is based”, whereas the actual term is premises (“the land and buildings owned by someone, especially by a company or organisation”, as in “The security guards escorted the protesters off the premises”).

AtlasBarfed 4y ago

Why did openstack fail? Or did it, was it just not adopted? I think there is still a lot of potential for open source management of core EC2/S3/networking capabilities (aka "core AWS IAAS service"). We have a fair number of cloud abstraction layers now, and obviously kubernetes, you'd think we could as an industry produce core apis for doing resource listing, availability, etc. Maybe some of the problem is that devs have a LOT of experience with the "ask" side of IaaS: gimme storage, gimme vms, set. But they have no experience with the "provide" side, and the sort of one-off manual nature of installing networking and machines doesn't have good standardization for "reporting available resources". At this point, aws apis are somewhat stable. (I would bitch about the error codes and documentation... but anyway). It's obviously "good enough" after 10-15 years of them. Are there projects that try to marry an aws-ish api, which really is a reporting and request api, with a "available resources" reporting api? Are some of these things out there? AWS ten years ago was liberating. It was progress. It was a good thing. But Amazon is not a "do no evil" corporation, much the opposite. And you see this in AWS with its treatment of startups, open source projects, and other manipulations. They are a monopoly now, or at a minimum a dangerous cartel. A real open source alternative would be a good thing. It would be good for the rest of FAANG, it would encourage competition by allowing lesser clouds to offer core competencies that are drop-in.

gnarcoregrizz 4y ago

<tinfoil hat> I always got the impression that there has been a lot of cloud propaganda/astroturfing, even on HN. I'm seeing more of these "on-prem infrastructure" posts, citing costs, efficiency, and cloud complexity. We run most of our infra on-prem, and have looked to moving a few bits and pieces to the cloud, but the math almost always works out to buying hardware. Meanwhile, I talk to some <cloud stack> friends, and their opex costs are astronomical for the traffic and size of their products. The cloud is extremely convenient, and I would choose it if I were launching a new product, but past certain sizes and expenses I would start to do some math. It's not terribly difficult to run these cloud "shrinkwrapped" products (such as load balancers) on-prem. Things like object storage seem more difficult to me. I'm also hesitant to admin a database, they intimidate me :)

thefunnyman 4y ago
I think the reality is that most companies don’t have the skill set needed to maintain on-prem infra. A lot of us here take this kind of knowledge for granted, but most businesses don’t have the time or resources to build some of this stuff themselves so they reach for the cloud. You also see a lot of VC backed companies splurge on cloud as a trade off of money for rapid growth.

slac 4y ago

The issue I see is that the public clouds often offer carbon free or carbon offset power. When you deploy onprem... This often gets ignored.

paxys 4y ago

If you are paying the cloud premium make sure you actually utilize the advantages it offers – global distribution, elasticity, rapid scaling, high availability, disaster recovery. Otherwise what's the point? If your needs can be met by shoving a couple thousand dollars worth of hardware into a closet somewhere that is obviously going to be a lot cheaper.

nijave 4y ago

>Adding storage, CPU time, and many other costs makes understanding and verifying the cost structure a task suitable for certified experts Is that easier on-prem? I was under the impression it was even more difficult--especially with shared tenancy (how much incremental cost does App A auth add to our Active Directory deployment?) You're going to need to know server power utilization under load to calculate power/cooling costs and probably some additional data on network utilization to figure out incremental costs for that Maybe if it's a colo or managed data center that gets rolled up for you, but if you're managing yourself, you still have to figure it out Blog post also doesn't mention cost of downtime (maybe not an issue for them) or a metrics solution (you usually get basic machine and service metrics for free on the big cloud providers)

Delitio 4y ago

The calculation is very thin in my opinion. 10k of infra costs are not a lot of money in a business context. A person to operate a private cloud with OnCall, backup hardware, capacity planing etc. costs what? I would always try to have GPU on prem as those prices are quite high bit others I would use managed. Cloud providers are just much better in operating infrastructure ad normal ops.

ksec 4y ago

>> "Starting a web-based or SaaS (Software as a Service) business was virtually unheard of before the age of IaaS" > "...off-load many of its processes to an on-premise private cloud?" On-Premise Private Cloud I dont know why reading this my brain just cant compute. You mean you have three consumer grade computer with Dedicated GPU running 24x7?

jtsymonds 4y ago

Many companies that do this look at MinIO for object storage. Given they run in AWS, GCP and Azure, they will minimize or eliminate your application rewrites. They are cloud-native by design and very fast.