35 ms·

Ask HN: Is your company sticking to on-premise servers? Why?

• moonbug 6y ago
  TCO.
  • aspyct 6y ago
    Which, all things considered, seems lower on the cloud. Could you give more details on this answer?
    • throwaway888abc 6y ago
      https://techcrunch.com/2017/09/15/why-dropbox-decided-to-drop-aws-and-build-its-own-infrastructure-and-network/ https://techcrunch.com/2017/09/15/why-dropbox-decided-to-dro...
      • HatchedLake721 6y ago
        Dropbox has outgrown AWS. I’m not sure it’s a good example.
    • lazylizard 6y ago
      A 2u asus with 2 xeon silver, 64gb ram and 4 x 2080ti is maybe us$15k? We'll use it for as long as its producing useful output. Lets say 5 years? Probably a little longer? A 60bay 4u western digital with 14tb drives is under us$50k? definitely got a dell md1280+1u server with 70x10tb 2yrs ago for under us$50k. Fully populated the following year.. A 2u dell with maybe 20 cores n 128gb ram each should cost less than us$10k. And we just got 4 or 5 dell switches with 48x10g ports for 50-70k? I'm not sure. What's the equivalent in the cloud?
      • aspyct 6y ago
        Are you keeping tabs on the cost of the building and associated security? Video cameras, locks... Also the manpower needed to install and maintain that equipment. Similarly, your 70*10tb tells nothing. What's the redundancy on that, how much of that space did you lose to it, where do you store your backups and at what cost? As for networking, having those switches is nice, but you still need your internet connection if you're serving anything online from this.
        • lazylizard 6y ago
          The manpower to run that stuff is already doing desktop support. Storage is usually a stripe of 10 disks in raidz2 + 2 spares. Then we do a nightly zfs send.. The internet connection for the hosting is 100mbps. The users share 1gbps of internet. We pay only for hardware. Rent and utilities. And the internet connections. Everything else we can do it ourselves. I mean. Seriously. Whats the cost of running 100 x 4 x 2080ti in the cloud for a year? Or storing 500tb(1 instance only,no redundancy) ?
      • bluedino 6y ago
        So service contracts on any of those?
        • lazylizard 6y ago
          What service contracts? Only hardware warranties n extended hardware warranties..
    • deleted 6y ago
      [deleted]
    • deleted 6y ago
      [deleted]
    • ThrowawayR2 6y ago
      AWS EC2 m5d.2xlarge (8 vCPU / 32 GB / 300GB nVME SSD) for a 3 year term is $4,466. That's enough for 3 HP ML10s equivalently configured plus the power to run them for their entire lifetime, which is well over 3 years. You are paying out the nose for the fancy-pants deployment and administrative features that the cloud gives you, and for a lot of use cases, that isn't needed.
      • aspyct 6y ago
        You're forgetting the human part in this cost. Installing, securing and maintaining the machines costs as well.
        • ThrowawayR2 6y ago
          I'm not; for non-production compute workloads, the cost is negligible. Even for production workloads, the 4x premium over base hardware cost is absurd.
          • aspyct 6y ago
            Totally disagree with that. Humans are expensive. Cabling, configuring, training, ordering and replacing hardware, that all takes time. Over a period of 3 years, there's a high chance that you'll end up having, depending on your luck, at least a few dozen days used up by those tasks. Assuming you're not paying peanuts for your contractors, this will easily get to the $10k or over.
            • ThrowawayR2 6y ago
              If you disagree, give us your example TCO estimates for all of that vs cloud. I'm sure HN readers would be happy to dissect it.
              • aspyct 6y ago
                Unfortunately I don't have exact numbers, having never been responsible for the budget of those. This is only an estimation considering the pricing I see from the cloud providers, and the rates I know from my fellow contractors. Besides, I'm not exactly trying to win a war here. Simply trying to know under which circumstances it is better to go for on-prem vs public cloud.
    • nojito 6y ago
      Do you have a source on this? By any reasonable metric, onprem is much much cheaper than the cloud.
• reacharavindh 6y ago
  University research group here. Simply, _cost_ Our compute servers crunch numbers and data at > 80% util. Our servers are optimized for the work we have. They run 24/7 picking jobs from queue. Cloud burst is often irrelevant here. They deal with Terabytes or even Petabytes of moving data. I’d cry paying for bandwidth costs if charged €/GB. Sysadmin(yours truly) would be needed even if it were to be run in the cloud. We run our machines beyond 4 years if they are still good at purpose. We control the infra and data. So, a little more peace and self-reliance. No surprise bills because some bot pounded on a S3 dataset. Our heavy users are connected to the machines at a single hop :-) No need to go across WAN for work.
  • aspyct 6y ago
    Interesting. How many machines do you have, roughly? Are you the only sysadmin for this? Also, do you have any public facing services hosted from that infrastructure?
    • reacharavindh 6y ago
      I am the only one administering this cluster. We have approximately 1000 cores across 45 servers. A few public facing services - web servers, small APIs, a web front end tool for a big genetics database. Nothing big. Public services are cordoned off from the compute cluster.
      • pbnjay 6y ago
        We had a similar setup at our University as well. Execs decided we needed to be a part of school-wide clusters though, and are now trying to retire even that computing hardware and push us to the cloud without covering any of the costs.
  • Jugurtha 6y ago
    Can you share the architecture and stack? What form do these jobs have? How do you manage workloads? How do you manage resources? Do users have quota for compute and storage? Do you use GPUs? If so, how do you deal with malfunction? Is this a distributed processing? Are the machines heterogeneous? What do you use for that cluster? What if a job requires dependencies? Do you create a compute environment on the fly or do all jobs have the same dependencies and these don't change much? How do you do data governance? Is the data read only? Do you have an API to fetch the data from the job code?
• 2rsf 6y ago
  Security (we're a bank)
  • aspyct 6y ago
    How is the cloud less secure than your on-prem servers? I would argue that it's easier to keep track of all the threats with the tools available from big cloud providers.
    • badpun 6y ago
      The big question is - how much do you trust these providers.
      • deleted 6y ago
        [deleted]
      • quicklime 6y ago
        I would guess that approximately zero banks own data centers that are operated by their own employees. There might be a few exceptions to this, but the reality is that most banks don't view technology as part of their core business. So this largely gets outsourced to IT consulting firms like Infosys, IBM, Wipro, etc. The big question is - how much do you trust these providers, and do you think they are more competent at security than Amazon/Google/Microsoft?
        • dathinab 6y ago
          Or formulate it differently: 1. Trust a provider which whole existence relies on trust and which you can audit or at last cross-check the audit and security processes (as a Bank you are normally not a small customer). 2. Trust a provider which might 1st be a potential competitions in some business fields. 2nd is so big that it can easily afford losing you. 3nd for the same reason doesn't allow you any insights into there internal processes. etc. Plus many of the banks having their own hardware also have their own IT team. So it's often about trusting your own people. I mean either you keep your it or you outsource _and_ go into the cloud. Keeping local servers but outsourcing IT at the same time seems kind not very clever tbh.
• Jugurtha 6y ago
  Many organizations do have private clouds. If by cloud you mean a public cloud like Google, Amazon, or Microsoft, then forget about it; not with these companies piping data directly to U.S intelligence.
  • aspyct 6y ago
    I meant public cloud indeed.
  • p1necone 6y ago
    What's the difference between private cloud and on prem? Does it just mean you let everyone spin up VMs etc rather than requiring them to go through IT?
    • toomuchtodo 6y ago
      Mostly. Cloud is just bin packing compute and data storage. There are many use cases where it’s cheaper for you to host the hardware you cloud on instead of a public cloud provider.
    • dathinab 6y ago
      A private cloud is potentially on prem. But besides that it's a about how the infrastructure is manages, in the classical way. Or with somethings which allows easy setup of new "containers" and similar like Kubernets. This doesn't necessarily means VMs. It can be more lightweight containers, e.g. based on Linux namespaces. Nor does it means you can sidestep the IT ;=) But normally it mans you can much easier spin up new services or change scaling between them.
    • Jugurtha 6y ago
      > What's the difference between private cloud and on prem? Cloud and on-premises are not mutually exclusive, as many "bits I push to the cloud" get on-premises of a ponytailed guy named John. We focus on what people mean[^1], not what they say. Some even make a living saying "chemical-free"[^2] or "digital marketing" publicly. > Does it just mean you let everyone spin up VMs etc rather than requiring them to go through IT? This is the on-demand, self-service aspect. What the users get as a result of their demand depends on the need you're trying to fulfill and defines (SaaS, PaaS, IaaS, etc.). Do you want them to be able to interact with an application, deploy/run applications, or get a VM? Here's a resource that summarizes it[^3] and the Wikipedia page gives an overview[^4]. [^1]: https://en.wikipedia.org/wiki/DWIM https://en.wikipedia.org/wiki/DWIM [^2]: https://en.wikipedia.org/wiki/Chemical_free https://en.wikipedia.org/wiki/Chemical_free [^3]: https://www.redhat.com/en/topics/cloud-computing/iaas-vs-paas-vs-saas https://www.redhat.com/en/topics/cloud-computing/iaas-vs-paa... [^4]: https://en.wikipedia.org/wiki/Cloud_computing https://en.wikipedia.org/wiki/Cloud_computing
• grantlmiller 6y ago
  I always find it important to separate "cloud" into 2 categories: 1. IaaS - Which I mainly define as the raw programmable resources provided by "hypercloud" providers (AWS, GCP, Azure). Yes, it seems that using an IaaS provider with a VPC can provide many benefits over traditional on-prem data centers (racking & stacking, dual power supply, physical security, elasticity, programmability, locations etc). 2. SaaS - I lump all of the other applications by the hundreds of thousands of vendors into this category. I find it hard to trust these vendors the same way that I trust IaaS providers and am much more cautious of using these applications (vs OSS or "on-prem software" versions of these apps). They just don't have the same level of security controls in place as the largest IaaS providers can & do (plus the data is structured in a way that is more easily analyzed, consumed by prying eyes).
  • aspyct 6y ago
    I was referring to the IaaS in this question. As for the SaaS, I guess your mileage may vary. I trust some of them really make a point of securing your data :)
  • opportune 6y ago
    What about first-party SaaS? Those can also be big features that bring people to some cloud providers. Not all SaaS requires you to trust your data/availability to some random vendor. Of course those first-party SaaS aren't typically suitable for lift-and-shift by their very nature, and they can still have some rough edges, but IMO you can expect them to be almost as reliable as IaaS
    • grantlmiller 6y ago
      First-party SaaS meaning things like RDS, DBaaS, queues, LBs etc? Most of that I would sort of put into a IaaS controlled PaaS, rather than true IaaS SaaS. Yes, these are generally higher on the trust spectrum as they don't involve additional vendors accessing/managing/storing data.
      • opportune 6y ago
• shortlived 6y ago
  Small company here (30 total people, 8 in IT/software). - unwillingness to seed control of the critical parts of our software infrastructure to a third party. - given our small team size and our technical debt load we are not currently able to re-architect to make our software cloud-ready/resilient. - true cost estimates feel daunting to calculate, whereas on-prem costs are fairly easy to calculate.
  • aspyct 6y ago
    I agree with the last 2 points. Estimates are hard to get right, and rearchitecting an existing app is probably not worth it. What about your first point though? Do you not trust a 3rd party to maintain infrastructure properly? In what way?
  • AnonC 6y ago
    > unwillingness to seed control Typo alert: you mean “cede control”, not “seed control”.
• TedLePoireau 6y ago
  Not exactly on-premise but we rent 2 big dedicated server (ovh) + install VMWare ESXi on them. Going to the cloud would cost more, the price would be unpredictable, only to solve a scale problem we won't have. And customers love to know their data are hosted in France by a French company, not by Google or Amazon :)
• XCSme 6y ago
  I use a $5/mo DigitalOcean VPS droplet instead of AWS or other "cloud" service. I only have to host an analytics dashboard ( https://usertrack.net/ https://usertrack.net/ ), I don't need scaling and this way I know exactly how much I will pay. The resources are more than enough for my needs and I don't think it could be much cheaper even on the most optimized pay-per-minute of use cloud platforms. I also have some other APIs hosted in the same way (eg. website thumbnail generation API), for the very low traffic I have and no chance of getting burst traffic I think the use case of a VPS or dedicated server is perfect.
  • chickenpotpie 6y ago
    Whenever I need to host something small and I’m trying to decide between DO and AWS I always ask myself. Would I rather be surprised by the bill or my website crashing from too much traffic? I almost always pick DO because I don’t want to mess something up and lose a few hundred dollars.
    • jackson1442 6y ago
      Wholeheartedly agree. I think AWS is moving in the right direction with Lightsail[0], which is a service very similar to DO droplets and includes transfer. Nice if you want to use AWS for like one or two other services, but I tend to still go with DO for small things. [0]: https://aws.amazon.com/lightsail/ https://aws.amazon.com/lightsail/
      • XCSme 6y ago
        That sounds interesting. By "moving in the right direction" do you mean that it's still in beta or not released yet? Or that it's just the first step of many to come?
        • adventured 6y ago
          Lightsail works well now. It's a little over three years old. In the first year or so after it was released, they were notorious for being slow in most regards compared to their peers (it launched using rebranded instances from AWS, and used spinning disks, going up against SSDs their competitors were all using). They've largely caught up on performance with DigitalOcean, Linode, Vultr and similar. That said, I've stuck with DigitalOcean even though Lightsail tests fine. I've had a great experience over the years with DO and see no reason to leave.
• starpilot 6y ago
  PII and draconian security policies. We are not a tech company, so we can't fine-tune or have nuanced policies, we just have to build a wall around everything . In our web password recovery process, we can't tell people if their login was correct or not, because that might help a brute force attacker infer they got that right. Even though we have that rate limited anyway. I don't know why we can't just tell people the login was found or not found like banks etc.
  • vikramkr 6y ago
    What industry? It's good practice to not share information like that in any context, since attackers that have a bank if email addresses that are trying to figure out which if a few reused passwords might be used for a given website would have a harder time if they dont even know if the email has an account with a given website, or if an alternate email is used instead etc.
• centimeter 6y ago
  We are a 1000-2000 person company and we have probably on the order of $100M of servers and data centers and whatnot, and I think we spend about 2/3rds of that every year on power/maintenance/rent/upgrades/etc. We don't generally trust cloud providers to meet our requirements for: * uptime (network and machine - both because we are good at reliability [and we're willing to spend extra on it] and because we have lots of fancy redundant infrastructure that we can't rely on from cloud companies) * latency (this is a big one) * security, to some degree * if something crazy is happening, that's when we need hardware, and that's when hardware is hard to get. Consider how Azure was running out of space during the last few months. It would have cost us an insane amount of money if we couldn't grow our data centers during Corona! We probably have at least 20-30% free hot capacity in our datacenters, so we can grow quickly. We also have a number of machines with specs that would be hard to get e.g. on AWS. We have some machines on external cloud services, but probably less than 1% of our deployed boxes. We move a lot of bandwidth internally (tens of terabytes a day at least, hundreds some days), and I'm not sure we could do that cheaply on AWS (maybe you could). We do use <insert big cloud provider> for backup, but that's the only thing we've thought it was economical to really use them for.
  • H8crilA 6y ago
    Hundreds of terabytes a day is really not that much, depends on what latency can you accept. I often run computations over datasets that are petabytes in size, just for my own needs. A big data move would be at least tens of petabytes or more like hundreds, or thousands. Also surprised about latency, latency from what to what? Big cloud providers have excellent globally spanning networks. Long distance networking is crazy expensive, though, compared to the peanuts it costs to transfer data within a data center. Reliability - again, not sure I buy it. Reliability is "solved" at low levels (such as data storage), most failures occur directly at service levels, regardless of whether you have the service in house or in the cloud. The rest of your points make sense.
    • deleted 6y ago
      [deleted]
    • centimeter 6y ago
• throwaway6845 6y ago
  Mostly, headspace. If I run my own server, I just need to apply my existing Ubuntu sysadmin knowledge. If I use AWS, I have to learn a whole load of AWS-specific domain knowledge, starting with their utterly baffling product names. My time is more valuable than that. Also, sheer cost. Literally everyone I know in my particular part of the industry uses Hetzner boxes. For what I do, it’s orders of magnitude cheaper than AWS.
  • HatchedLake721 6y ago
    That’s how you get old, when your time is more valuable than a massive shift in technology.
    • henriquez 6y ago
      Nah, we already did mainframes in the 1970s. Renting CPU time only makes sense if you don’t need CPU time or you like wasting money.
    • Silhouette 6y ago
      Whether some new technology is worth learning and using depends on your circumstances: would you gain more than it would cost? Understanding this isn't about being old, it's about being wise.
    • throwaway6845 6y ago
      It's a common HN fallacy that everyone here is a developer whose main life aim is to remain employable as a developer. Lots of us are here to ship stuff; to provide viable products that customers will like. Moving to AWS from Hetzner would not help me ship stuff. Moving to AWS from Hetzner would not result in a better product.
  • sixothree 6y ago
    We're a small business and hosting costs are killing us. Spending $300+ on a small VM that I can only describe as measly really hurts.
    • 6y ago
• sdan 6y ago
  Founder of a growing startup: Started with a cluster of Raspberry Pis and expanded onto an old desktop. Primarily did this for cost (raspberry pis alone were more powerful than a GCP $35/mo instance). Everything was fine until I needed GPUs/handling more traffic than those Raspberrys could handle. So I expanded by including cloud instances in my Docker Swarm cluster (tidbit: Using Traefik and WireGuard) So half on-prem half in the cloud. Honestly just scared GCP might one day cancel my account and I'll lose all my data unless I meet their demands (has happened in the past) so that half on-prem stores most of the data.
  • chickenpotpie 6y ago
    At $35/month though GCP would only have to save you a half an hour of maintenance for it to be worth it though.
    • sdan 6y ago
      Well, given that I am using Docker it doesn't really matter much... but the bigger issue is: GCP in the past has completely blocked access from accounts when they detect random things. Unless I meet their demands, my entire infra is gone/down for days, which I can't deal with.
    • Silhouette 6y ago
      You write this as if it's unrealistic. Who spends half an hour or more every month managing each tiny server? The closest server to me right now has probably taken about that long in the past year, swapping out a failed drive that unfortunately wasn't in a hot-swap bay. And it's much more powerful than a $35/month cloud instance.
    • MattGaiser 6y ago
      A half-hour a month seems like a lot of maintenance for a server.
  • PhantomGremlin 6y ago
    Honestly just scared GCP might one day cancel my account and I'll lose all my data unless I meet their demands This hasn't been emphasized enough in these discussions. Once you're in the cloud, you're dealing with computers as a "utility". Or are you? When you buy electricity from your local utility, you don't just lose it in the middle of some afternoon for some unknown reason. There's a clear and established procedure, supervised by a Public Utilites Commission. You aren't forced to resort to sending emails and receiving some idiotic canned response from a bot. "Sorry we have reviewed your case and our decision is final. By the way, we don't tell you what you did wrong. Fuck you. Ha. Ha. Ha." I don't know if Amazon or Azure are better than Google. But I wouldn't trust Google with my worst enemy's data.
• gramakri 6y ago
  We use servers on the cloud (IaaS) but still self-host all our apps.
• kkielhofner 6y ago
  Not a company but a personal project: https://github.com/krisk84/retinanet_for_redaction_with_deepstream/wiki https://github.com/krisk84/retinanet_for_redaction_with_deep... I haven't analyzed the TCO yet but the bandwidth costs alone from hosting my curated model of 100GB in the cloud (Azure blob) have greatly exceeded my total operation spend from downloading the entire dataset and running training. By an order of magnitude.
• edoceo 6y ago
  Cost. We're a small company, five person team and we need a development environment. All our stuff is built around VM and Docker. So scores of little test nodes that get run, or beta environment in the cloud was costly (>$300/mo base, sometimes 3x). For $1000 we put a box in the office we all VPN to that runs all the necessary VM and Docker. The variable, sometimes expensive cost for Test/Beta in cloud was replaced with low fixed cost.
• altmind 6y ago
  to control the costs. after the original purchase, the MRC is a fixed cost for hosting and for the network access. also, for the total control - the network, io, cpu performance does not change randomly. with better predictability our IT team can give more precise SLA. we're not 100% on-prem, but aws, gcloud and azure are the worst examples of 3rd party hosting - unpredictable and with complicated billing. we're considering the alternatives to big 3 for the "cloud hosting"
• deleted 6y ago
  [deleted]
• ROFISH 6y ago
  Network storage. MacBooks don't have a lot of space and artists like to make HUGE psd files. Bonus points for small stuff like RADIUS for wifi and stuff. Groups charging $5/user for junk like that is absolutely awful with a high number of staff. With a staff of 100, a single box with a bunch of hard drives is two months worth of cloud and SaaS. TCO needs to come down by like at least 100x before I consider going server-less.
• hprotagonist 6y ago
  AI compute is so much cheaper on-prem that it's not even in question. And there are clients that demand it. And researchers, in general, like to do totally wacky things, and it's often easier/cheaper to let us if you have physical access.
  • sdan 6y ago
    +1 on this. Get a nice server with some GPUs and you'd save a lot more than paying the super expensive costs on cloud.
  • icelancer 6y ago
    Yep. This is where we are at.
  • adev_ 6y ago
    +1 that's one of the reason for us too. GPU machines are clearly too expensive in most cloud providers compared to on-prem.
• michaelt 6y ago
  On-prem makes your cost control proactive, rather than reactive. Nobody gets a new server without first having a purchase order approved - and the burden to get that approval falls on the person who wants the server. In the cloud, at least the way it's generally used, cost control is reactive: You get a bill from AWS every month, and if you're lucky you'll be able to attribute the costs to different projects. This is both a strength and a weakness: on-premise assets will end up at much higher utilisation, because people will be keen to share servers and dodge the bureaucracy and costs of adding more. But if you consider isolation a virtue, you might prefer having 100 CPUs spread across 100 SQL DBs instead of 50 CPUs across two mega-databases.
• Nextgrid 6y ago
  Bandwidth costs. Most dedicated servers come with unmetered bandwidth so not only is it cheap to serve large files but your bandwidth costs won't suddenly explode because of heavy usage or a DDoS attack.
• thecolorblue 6y ago
  Just talking about my side project here: a local server + ngrok is easier to use and cheaper than anything in the cloud. In general, I would say any noncritical system I would host on-prem.
• jarym 6y ago
  my 2cents: the 'big' clouds (AWS, GCP, Azure) and the big-brand clouds (Oracle, IBM, etc.) are attractive for BigCustomerCo because: 1. Replace capital expenditure of in-house infrastructure + staff with OpEx that can be dialled down 2. Get to benefit from the economies of scale that the cloud vendors get (those Intel CPUs get a lot cheaper when purchased in the 1000s) 3. Get to leverage big shiny new tech like Big Data and AI that's 'advertised' as 'out-of-the-box' My only concern really is that the big cloud players are all fighting for dominance and market share. What happens in the next 5-10 years time when they start raising prices? Different kind of lock-in - customers won't have the expertise in-house to migrate stuff back.
• the_svd_doctor 6y ago
  PhD student/researcher. Most of the compute I do are HPC/scientific computing style and run on University or US National Lab machines. We thought about cloud, but the interconnect (MPI-style code) is very important for us and it's not very clear to me what's available there in the cloud.
• jolmg 6y ago
  Unreliable internet. A retail company may decide that the best place to put up a new branch is coincidentally (though there might be a correlation) at the edge of what the available ISPs currently cover. They might have to make a deal to get an ISP to extend their area to where the store is going to be. However, because of lack of competing ISP options on the part of the retailer, and the lack of clients in the retailer's area on the part of the ISP, that service is probably not going to be all that reliable. Also, that retail company may experience a big rise in sales after a natural disaster occurs, when communications (phone/cell/internet) are mostly down for the area. One tends not to think about stuff like that until it happens at least once. It's very important for the ERP/POS systems to be as operational as possible even when the internet is down.
• jabroni_salad 6y ago
  I'm in rural iowa and you really can't bank on a solid internet connection. One of my clients decided to iaas-ify all their servers and it works great except when it is windy out. They're on fixed wireless and the remote mast has some sway to it. 3-4 times a year they get struck by lightning and have a total work stoppage until all their outside gear can get replaced. Even their VDI is remote so all the thin clients just disconnect and they are done for the day. Also, my clients aren't software development firms. They are banks and factories. They buy a software based on features and we figure out how to make it work, and most of the vendors in this space are doing on-prem non-saas products. A few do all their stuff in IAAS or colo but a lot of these places are single-rack operations and they really don't care as long as it all works. A lot of people in small/midsize banks feel like they are being left out. They go to conferences and hear about all the cool stuff in the industry but the established players are not bringing that to them. If you can stomach the regulatory overhead, someone with drive could replace finastra/fiserv/jackhenry. Or get purchased by them and get turned into yet another forever-maintenancemode graveyard app.
• trelliscoded 6y ago
  Zseries are a huge deal to move to the cloud; it’s just not worth the risk for most organizations.
• nitwit005 6y ago
  We're looking at moving from AWS to having some machine space rented in two data centers. The reason is purely cost. There are still some computers on site due to equipment being tied to it, telephony stuff, etc. My last company was looking at "moving to the cloud", with the idea that its data centers were too expensive, but found out that the cloud solutions would be even more expensive, despite possible discounts due to the size. They still invested in it due to some Australia customers wanting data to be located there.
• chime 6y ago
  1. 500TB of storage for 3-6mo of CCTV footage. 2. Bought a hanful of $700 24 core Xeons on eBay 2 years ago for 24/7 data crunching. Equivalent cloud cost was over $3000/mo. On-Prem paid off within a month! 3. Nutanix is nice. Awesome performance for the price and almost no maintenance. Got 300+ VDI desktops and 50+ VMs with 1ms latency.
  • gge 6y ago
    How high are your power costs?
    • chime 6y ago
      Manufacturing company with insane deal with local utility. No idea but server room cost is probably under 2% of machinery cost so hard to quantify.
  • junar 6y ago
    > $700 24 core Xeons on eBay 2 years ago Can you clarify? I don't think such a product exists as a single chip at that price point. The Threadripper 3960X costs $1400, and that released less than a year ago. Edit: Looking up Intel chips on Wikipedia, I think you might be using 12-core/24-thread chips... https://en.wikipedia.org/wiki/Skylake_(microarchitecture)#Xeon_Bronze_and_Silver_(dual_processor) https://en.wikipedia.org/wiki/Skylake_(microarchitecture)#Xe...
    • fiveguys94 6y ago
      I picked up three Dell R900's for an average of $200 each, with 4x Xeon E7450 and 128GB of ECC RAM. No hyperthreading (2011!), it's 24 real cores. They're noisy and use lots of power, but you can't argue with the value for money.
      • icedchai 6y ago
        I have one similar to this. I don't think it is an R900, but it's an older 1U rack mount. I forget if it's a 12 or 24 core xeon, but it was dirt cheap, came with 72 gigs of RAM, and sounds like a jet engine turning it on. I recently built a Ryzen box with 128 gigs of RAM and it's much quieter...
• snarfy 6y ago
  The idea of the cloud is to only pay for what you use. Your on-premise server is idle 99% of the time so why are you paying for a full server? If that's not true, it turns out it's quite expensive to run things in the cloud. If your workload is crunching numbers 24/7 at 100% cpu, it's better to buy the cpu than to rent it.
  • Polylactic_acid 6y ago
    Cloud servers tend to be more reliable as well if you don't run your own datacenters. We have lost our internet connection or power 3 times in the last year in the office. Its not the end of the world since we can go to 4g for our own usage but if our servers were hosted locally this would be a huge issue.
    • dathinab 6y ago
      Don't forget that between cloud and servers in the company there are still VPS and rented dedicated hardware in a data center. So you: 1. Don't manage hardware. 2. But manage a server (OS+software stack). 3. Have reliable internet, power and physical security from the data center you are renting your hardware from (if you trust them fully!). 4. Have fixed cost but also fixed resources. Tends to be cheaper for many tasks. Especially CPU/GPU heavy ones.
      • Polylactic_acid 6y ago
        I consider VPSs to be cloud servers. Is this not common?
        • XCSme 6y ago
          I mentioned in another comment that I use VPS and not cloud services. I think of cloud as the auto-scaling infrastructure with dynamic pricing. I think of VPS as just sharing a dedicated machine with others, so each one gets a few cores and shares other resources. The implementation of VPSs nowdays is probably more similar to cloud services, where your own space might be moved around to another physical machine without any downtime.
• janstice 6y ago
  I'm application architect at enterprise-type org. We have a few SaaS applications, but all the big dogs, including custom dev, run in-house on a dual data centre vmware environment. It's cheaper for us to spin up more VMs in-house, so there's no real cloud driver for things that just live on VMs. On the other hand, our ops team are still working on network segmentation and self-service, but I regularly get a standard VM provisioned in less than 20 mins. If we had to buy new data centres it might be different. But the real reason we're not deeper in the cloud is that our business-types insist on turn of the century, server-based software from the usual big vendors, and all the things that integrate with them need to use 20th century integration patterns, so for us migrating to the cloud (in stages at least) would have drawbacks from all options without the benefits. It's only where we have cloud-native stuff that we can sneak in under the radar for stand-alone greenfields projects, or convince the business types that they can replace the Oracles and Peoplesofts with cloud-first alternatives will things really change.
  • jasonv 6y ago
    Last company I was at was more or less as you describe. Now.. at a company in a different industry, there's a 5+ year plan to move 100% to cloud. Nascent efforts are about 18 mos old already, no apps are live yet. Fortunately, they've been using a container approach for their on-prem stuff for a while, so some stuff can move over pretty easily, a lot of things will get a touch-up or more interesting upgrade along the path to the cloud environment. Not even talking about de-commissioning the DCs yet, but those will get defunded as things go on.
• sokoloff 6y ago
  We have mostly switched to, or are in the midst of switching to, the cloud. Services that we will continue to run on-premises (as an exception to that rule) are some machine learning training clusters (where we need a constant, high-level amount of GPU and cloud provider pricing for GPU machines is very far off the mark of what you can build/run yourself) and some file shares for our production facilities where very large raster files are created, manipulated, sent to production equipment, and then shortly afterwards deleted. Most everything else is going to the cloud (including most of our ML deployed model use cases).
• throwaway7281 6y ago
  We own infra because we need to own and control it and also because it's just vastly cheaper at the scale we use. Besides, we do have things like our own S3, k8s and other cloud-ish utilities running so we do not miss out that much, I guess.
• dogecoinbase 6y ago
  Yes. Three major reasons: - Cost. It's vastly cheaper to run your own infra (like, 10-100x -- really!). The reason to run in cloud is not to save money, it's to shift from capex to opex and artificially couple client acquisition to expenditure in a way that juices your sheets for VCs. - Principle. You can't do business in the cloud without paying people who also work to assemble lists of citizens to hand over to fascist governments. - Control. Cloud providers will happily turn your systems off if asked by the government, a higher-up VP, or a sufficiently large partner. EDIT: I should add. Cloud is great for something -- moving very fast with minimal staffing. That said, unless you get large enough to renegotiate you will get wedged into a cost deadend where your costs would be vastly reduced by going in-house, but you cannot afford to do so in the short term. Particularly for the HN audience, take care to notice who your accelerator is directing you to use for cloud services -- they are typically co-invested.
  • PaulWaldman 6y ago
    Regarding the shift in CapEx to OpEx, on-prem servers can also be leased, keeping their costs in OpEx.
• bcrosby95 6y ago
  We have around 20 servers in a colo center down the street. At this number of servers we can still host websites that have millions of users (but not tens of millions). They are not exotic servers either. In fact by now they are, on average, around 11 years old. And costed anywhere from 2k to 8k at the time of purchase. Some are as old as 19 years. Hell, when we bought some of them - with 32GB of memory each - AWS had no concept of "high memory" instances and you had to completely pay out your ass for a 32GB server, despite ram being fairly cheap at the time. We have no dedicated hardware person. Between myself and the CTO, we average maybe a day per month thinking about or managing the hardware. If we need something special setup that we have no experience in, we have a person we know that we contract, and he walks us through how and why he set it up as he did. We've used him twice in the last 13 years. The last time one of us had to visit the colocation center was months ago. The last time one of us had to go there in an emergency was years ago. It's a 5 minute drive from each of our homes. So, why exactly should we use the cloud? We have servers we already paid for. We rent 3 cabinets - I don't recall the exact cost, but I think its around $1k per month. We spend practically no time managing them. In our time being hosted in a colo center - the past 19 years - we've had a total of 3 outages that were the fault of our colo center. They all lasted on the order of minutes.
  • dahfizz 6y ago
    I think people who have no experience managing servers dramatically overestimate how much time it takes to manage servers. Depending on your team, it can definitely be easier to manage your own hardware than to manage your cloud infrastructure.
    • lykr0n 6y ago
      Yep. A lot of issues come from legacy products and configurations. You can do a greenfield infrastructure with OpenStack, VMware, or oVirt/RHEV and it's a pleasure to work with.
    • wooly_bully 6y ago
      In my experience, it's not the time required but that a lot of development teams don't have a sysadmin or ops skillset.
• tgamblin 6y ago
  I work in Livermore Computing at LLNL. We manage upwards of 30 different compute clusters (many listed here: https://hpc.llnl.gov/hardware/platforms https://hpc.llnl.gov/hardware/platforms). You can read about the machine slated to hit the floor in 2022/2023 here: https://www.llnl.gov/news/llnl-and-hpe-partner-amd-el-capitan-projected-worlds-fastest-supercomputer https://www.llnl.gov/news/llnl-and-hpe-partner-amd-el-capita.... All the machines are highly utilized, and they have fast Infiniband/OmniPath networks that you simply cannot get in the cloud. For our workloads on "commodity" x86_64/no-GPU clusters, we pay 1/3 or less the cost of what you'd pay for equivalent cloud nodes, and for the really high end systems like Sierra, with NVIDIA GPUs and Power9's, we pay far less than that over the life of the machine. The way machines are procured here is different from what smaller shops might be used to. For example, the El Capitan machine mentioned above was procured via the CORAL-2 collaboration with 2 other national labs (ANL and ORNL). We write a 100+ page statement of work describing what the machine must do, and we release a set of benchmarks characterizing our workload. Vendors submit proposals for how they could meet our requirements, along with performance numbers and test results for the benchmarks. Then we pick the best proposal. We do something similar with LANL and SNL for the so-called commodity clusters (see https://hpc.llnl.gov/cts-2-rfi https://hpc.llnl.gov/cts-2-rfi for the latest one). As part of these processes, we learn a lot about what vendors are planning to offer 5 years out, so we're not picking off the shelf stuff -- we're getting large volumes of the latest hardware. In addition to the cost savings from running on-prem, it's our job to stay on the bleeding edge, and I'm not sure how we would do that without working with vendors through these procurements and running our own systems.
  • theyboosting 6y ago
    didn't they EOL omnipath last year?
    • tgamblin 6y ago
      Yes but we still have many OP systems on the floor (see the link above).
  • glacials 6y ago
    I interned at LLNL on three separate occasions during school. Everything about what y'all do in high-performance computing is levels beyond anything I've gone on to see in the "real world". I cannot drop enough praise.
• arghwhat 6y ago
  Quite frankly, "cloud" is a convenience and elasticity service at a steep premium, with downsides. Contrary to popular belief, it does not in the slightest save you a sysadmin (most just end up unknowingly giving the task to their developers). And contrary to popular belief, the perf/price ratio is atrocious compared to just buying servers. For some of the loads I had been doing the math for, I could rent a colo and buy a new beefy server every year with money to spare for the yearly cost of something approximating the performance in AWS...
• shockinglytrue 6y ago
  Try running any service with an average egress exceeding 10 Mbit/s then tell me cloud still makes sense. By the time you reach 1 Gbit/s the very idea of it is enough to elicit a primitive biological defensive response. We don't do on-prem but we do make heavy use of colo. The thought of cloud growth and DC space consolidation some day pushing out traditional flat rate providers absolutely terrifies me. At some point those cloud premiums will trickle down through the supply chain, and eventually it could become hard to find reasonably priced colo space because the big guys with huge cash-flush pockets are buying up any available space with a significant premium attached. I don't know if this is ever likely, but growth of cloud could conceivably put pressure on available physical colo space. Similar deal with Internet peering. There may be a critical point after which cloud vendors, through their sheer size will be able to change how these agreements are structured for everyone.
  • jedberg 6y ago
    Netflix runs on the cloud and does 30% of all internet traffic. That being said, 99% of that traffic is served from servers in colos now, but 10 years ago it was all served from CDN providers like Akamai, which is just a specialized cloud.
    • toomuchtodo 6y ago
      This is kind of a big caveat (“Netflix is in the cloud but almost none of the work is done there”), and something I have to mention to non tech decision makers when they say “but Netflix!”. I even have a slide for presentations just for this (“You Are Not Netflix”).
      • jedberg 6y ago
        99% of the work is done on the cloud. What comes off of those colo servers is literally just bits streaming from disk to network. There is no transformation or anything. No authentication, no user accounts, no database. Nothing. Just static files served efficiently.
• dathinab 6y ago
  A company which: 1. Does "security" critical stuff (like affecting the security of people not data if breached) 2. Which besides certain kinds of breaches has lowish requirements to performance and reliability (short outages of a view minutes are not a big problems, even outages of half a day or so can be coped with) 3. Slightly paranoid founders, with a good amount of mistrust into any cloud company. 4. Founders and tech-lead which have experience in some areas but toughly underestimate the (time)cost of managing servers them self, and that _kinda_ hard to do secure by yourself (wrt. long term DDOS and similar). So was it a good reason? Probably not. But we still went with it. As a site note while we did not use the could _we didn't physically manage server either_. Instead we had some dedicated hardware in some compute center in Germany which they did trust. So no "physical" managements securing etc. needed and some DDOS and network protection by default. Still we probably could have it easier without losing anything. On the other side if you have a dedicated server hardware in some trusted "localish" compute center it's not _that_ bad either to manage.
• PaulWaldman 6y ago
  Manufacturing. The cost to a factory if the internet is down is too great. Each facility has its own highly redundant virtualization infrastructure hosting 50 to 100 VMs.
  • thisisnico 6y ago
    Manufacturing Here. This is also the Case with us. Also the cost to run our on-prem in the cloud would be almost 10x what we pay right now. Such an insane amount of CAD Data, there is 0% chance we can get an internet connection that comes close to what on-prem can handle for our massive cad data.
  • eitally 6y ago
    I was in manufacturing IT before moving to big tech. Our big campuses in the US & Europe had 40-80mbps internet circuits. The remote facilities in developing countries often only had 10mbps MPLS connections to a regional hub. To be 100% honest, we had 10x the outages caused by crappy local infrastructure than anything having to do with a SaaS service or IaaS/PaaS provider. Seriously, things like bad storms, a snake (cobra!) sneaking into the server room and frying itself and a machine it was snuggling against, utility workers accidentally severing cables, generators failing during power outages, labor strikes, and so much more. Moving to the cloud -- or even just hosting everything centrally -- was much more stable than maintaining a fleet of distributed machines.
  • generatorguy 6y ago
    Not sure what you are making but that’s way more advanced than any factory I’ve been to!
  • whatsmyusername 6y ago
    Manufacturing is the one place I still use on-prem, mostly because I work in a field that has compliance reasons for doing so and the, 'internets out, we can't print' problem.
• burnte 6y ago
  Yes. Why? Cost, availability, flexibility, bandwidth. For a lot of companies, on-prem servers are the best solution for efficiency and cost. One great example. We were paying $45k/yr for a hosted MS Dynamics GP solution. For $26k we brought it in house with only a $4k/yr maintenance fee. We bought a rackmount Dell, put on VMWare, have an app VM and a DB VM. My team can handle basic maintenance. In the past 11 months we haven't had to touch that server once. We have an automated backup, pulls VMs out daily and sends them off to Backblaze. Even if we need to call our GP partner for some specialized problem, it's not $45k/yr in consulting costs. We had a bunch of Azure servers for Active Directory and a few other things. When I came in 2 years ago I set up new on-prem DC VMs, and killed out absurd Azure monthly bill, we were saving money by month three. A meteor could take out Atlanta and the DCs are our satellite offices would handle the load just fine until we restored from backups and we'd STILL save money. We've had MORE uptime and reliability since then too. If I have a server go down, we have staff to get on it immediately, no toll free number to dial, no web chat to a level 1 person in India, etc. Our EMR is hosted, because that's big enough that I want to pay someone to be in control over it, and someone to blame. However, there have been many times where I'm frustrated with how they handle problems, and jumping from one EMR to another is not easy. And in the end they're all bad anyway. Sometimes I DO wish we were self hosted. The Cloud is just someone else's computer. If they're running those machines more cheaply than you are, they're cutting out some cost. The question is, do you need what they're cutting?
  • jedberg 6y ago
    > The Cloud is just someone else's computer. If they're running those machines more cheaply than you are, they're cutting out some cost. The question is, do you need what they're cutting? They're cutting overhead and getting better deals on hardware than you could ever get. Their efficiency is their profit margin.
    • ClumsyPilot 6y ago
      This person is not payong overhead of developing and running 500'th AWS service with confusing name that they will never need. Just saying 'economies of scale' is a banality, everyone past schoolage knows about them.
• loeg 6y ago
  Cost-wise is still a pretty compelling argument.
• dahfizz 6y ago
  There are some things you just can't do in a VM. The company I work for actually develops and hosts an AWS clone for the Linux Foundation, but with very specific requirements. They have special needs that requires baremetal machines and "real" networking between them across 6+ NICs per server.
• iseletsk 6y ago
  We are software development company. Most of our compute/storage needs are for build/test cycles. We recently bought ~100K worth of additional hardware to migrate some of that work off AWS. The storage / virtualization is done using Ceph & OpenNebula. Including colocation/electricity/networking costs, the investment will pay for itself in ~9 months. If I would include deployment costs & work to migrate the jobs off the AWS -- it will pay for itself in 11 months.
• whorleater 6y ago
  Yes, but we work in financial information so very weird requirements.
• ggm 6y ago
  We operate an X509 PKI with a Trust anchor and CA. Its not impossible to run the Hardware Security Module (HSM) in the cloud but its off the main path. Its more plausible to run it in a D.C. but it invites security threats you don't have, if you run it inside your own perimiter. Of course you also then invite other risks, but its a balancing act and it has to be formally declared in your Certification Practice Statement (CPS) We also run some registry data which we consider mission critical as a repository. We could run the live state off-prem, but we'd always have to be multi-site to ensure data integrity. We're not a bank, but like a bank or a land and titles administration office, registry implies stewardship in trust. That imposes constraints on "where" and "why". Take central registry and the HSM/related out of the equation, if I was building from scratch I'd build to pub/sub, event-sourcing, async and in-the-cloud for everything I could. private cloud. If you don't control your own data and logic, why are you in the loop?
• 32gbsd 6y ago
  Cloud servers tend to hide their limitations behind payment tiers which makes it hard to really know how far you can push things. Also there are various turns, conditions, cache rules, change management strategies that are hidden when dealing with someone else's constantly changing box of magic.
• FpUser 6y ago
  I normally host on prem and also rent dedicated servers elsewhere as standby. Way cheaper than the cloud and full control the way I want.
• caseyf 6y ago
  We colocate and as a small tech business, I can't imagine doing anything else. We don't spend more on payroll due to colocating and AWS/etc would easily double our annual non-payroll expenses.
• _wldu 6y ago
  Local caching recursive DNS servers work best close to the clients.
• mattbeckman 6y ago
  We spend ~$50k/mo on serverless infrastructure on AWS. It hurts sometimes, given we were fully colocated about 4 years back, and I know how much hardware that could buy us every month. However, with serverless infra we can pivot quickly. Since we're still in the beta stage, with a few large, early access partnerships, and an unfinished roadmap, we don't know where the bottlenecks will be. For example, we depended heavily on CloudSearch, until it sucked for our use case, so we shifted to Elasticsearch, and ran both clusters simultaneously until we were fully off of CS. If we were to do that on-prem, we'd have to order a lot more hardware (or squeeze in new ES cluster VMs across heavy utilization nodes). With AWS, a few minutes to launch a new ES cluster, dev time to migrate the data, followed by a few clicks to kill the CloudSearch cluster. Cloud = lower upfront, higher long term, but no ceiling. On-prem = higher upfront, lower long term, but ceiling.
  • brickbrd 6y ago
    If "Cloud = lower upfront, higher long term, but no ceiling. On-prem = higher upfront, lower long term, but ceiling" is true, then how come the revenue of cloud companies keeps going up? That would mean the incoming rate of users who are just starting off and find Cloud worthwhile is more exit rate of mature users who are finding on-prem more worthwhile than cloud
    • wvenable 6y ago
      If you're spending more than 50k/month on AWS where is the money to move to on-prem? When they got you, they got you.
  • nojito 6y ago
    Your costs are going to skyrocket in a nonlinear fashion as you exit the beta stage. That’s the biggest counterpoint to using the cloud as a initial setup.
• desc 6y ago
  1. Our customers run our software on their own machines for security and data-control reasons. As soon as something's running on someone else's hardware, the data is out of your control. Unless you're going to accept the (often massive) cost of homomorphic encryption, AND have a workload amenable to that, it's a simple fact. 2. Everything we do in house is small enough that the costs of running it on our own machines is far less than the costs of working out how to manage it on a cloud service AND deal with the possibility of that cloud service being unavailable. Simply running a program on a hosted or local server is far far simpler than anything I've seen in the cloud domain, and can easily achieve three nines with next to no effort. Most things which 'really need' cloud hosting seem to be irrelevant bullshit like Facebook (who run their own infrastructure) or vendor-run workflows layered over distributed systems which don't really need a vendor to function (like GitHub/Git or GMail/email). I'm trying to think of a counterexample which I'd actually miss if it were to collapse, but failing.
• kasey_junk 6y ago
  Bandwidth is the big reason to stay on-prem. Good peering contracts can more than make up for any cloud advantages for bandwidth intensive uses. Now the hard part is turning those cost advantages into operational improvements instead of deficiencies.
• skiril 6y ago
  One of the reason is a legacy systems. Some companies are too tied up to old custom made systems built on old software and hardware virtually not convertible to cloud. You will be surprised but there are big corporation still using AS400 and not planning to switch anytime soon. If you heard in recent news US unemployment system was still built on COBOL... In 2020... Another reason is a cost. I love AWS! Its fantastic to be able to create and launch servers or the whole farm of servers in the matter of minutes! And ability to convert physical servers to virtual and upload them to the cloud is breathtaking! But my monthly bill started at $300 and grew to $18K per month in less than 3 years. And that was for just a few virtual servers with Windows OS and SQL. My company realized that we can have a state of the art datacenter with WmWare and SAN on premises for the fraction of that price. Put second one on the other coast (one of our other offices) and you have your own cloud with better ping and six digits figure saving a year. For the last I would name vendor lock. With vSphere its very easy to move your virtual servers between AWS, Azure and Google (assuming you can afford all 3 and licensing cost of WmWare) but have you ever tried to "download" your server back to premise? It's virtually impossible or made so hard by cloud players trying to keep you up they're in the clouds. With all said I read that Netflix (I believe its Netflix) saving hundreds of millions dollars per year by using Amazon instead of its own servers. I also read somewhere that Dropbox moved away from AWS...
• Spooky23 6y ago
  It all depends. Outside of SaaS, If you have a mature data center operating model and truly understand your costs, there won’t be a strong cost savings story for many types of workflows. If you suck and don’t understand costs, or don’t automate, or spend a lot of time eating steak with your Cisco team, you’ll save money... at first.
• sarasasa28 6y ago
  we are not zoomers
• som33 6y ago
  >I honestly fail to see any good reason not to use the cloud anymore, at least for business. Cost-wise, security-wise, whatever-wise. Problem is point of failure, many businesses need to be independent and having data stored in the cloud is a bad idea overall. Because it produces point of failure issues. Consider if we ever got a real nasty solar wind and the electric grid goes down, the more we rely on the internet and centralize infrastructure into electric devices, the more it becomes a costly point of failure. While many see redundancy as "waste" in terms of dollars, notice that our bodies have many billions of redundant cells and that's what makes us resilient as a species, we can take a licking and keep on ticking. Trusting your data to out-side sources generally is a bad idea any day of the week. You always want to have backups and data available in case of diaster, mishap, etc. Like no one has learned from this epidemic yet. Notice that our economic philosophy didn't plan for viral infections and has forced our capitalist society to make serious adjustments. Helping people is an anathema to liberals and conservatives / republicans / democrats, so for void to come along and actually force co-operation was a bit tragically humorous. As a general rule you need redundancy if you want to survive, behaving as if the cloud is almighty is a bad idea, I'm not sold on "software as a service" or any of that nonsense. It's just there to lull you into a false sense of security. You always need to plan for the worst case scenario for surviveability reasons.
• Stierlitz 6y ago
  Annual costs, backups, security and latency.
• lettergram 6y ago
  I actually am moving my startups servers from AWS to a home server. Reasoning: * We know how much compute is need. * We know how much the new servers can compute. * We have the ability to load balance to AWS or Digital Ocean or another service as needed. * This move provides a 10x speed improvement to our services AND reduces costs by 70%. For reference, had to call the ISP (AT&T) and they agreed to let me host my current service. It’s relatively low bandwidth, but has high compute requirements.
• MorganGallant 6y ago
  I picked up a few refurbished dell servers off Ebay for super cheap a while back - and usually use these with Cloudflare Argo tunnels to host various servers. However, since these are just sitting on the floor next to my desk, usually rely on cloud for any applications with high uptime requirements. Recently though, I've been working on some distributed systems type projects which would allow these servers to be put in different physical locations (and power grids), and still continue to operate as a cohesive whole. This type of technology definitely increases my confidence in them being able to reliably host servers. I wouldn't want to be reliant on the cloud for large scale services though, from my understanding you can get some crazy cost savings by colocating some physical servers (especially for large data storage requirements).
• kjgkjhfkjf 6y ago
  On-prem can make sense where your computing needs are constant and predictable. You can arrange to have exactly what you need, and the total cost of buying and operating it may be less than it would be to get comparable resources in the cloud. If your computing needs vary over time, then provisioning on-prem for peak load will mean that some of your resources will be idle at non-peak times. It may be cheaper to use cloud resources in cases like these, since you only need to pay for the extra capacity when you need it.
• SoylentOrange 6y ago
  We are a research group at a company with 2000 or so employees. We have a few GPU machines to train models, which are utilized nearly around the clock. AWS and co’s GPU-enabled servers are exceedingly expensive. Most of the GPU models on those machines are also very old. We pay maybe 1/3 or less to maintain these machines and train models in-house vs paying AWS. Mind you, we use AWS for plenty of stuff...
• frellus 6y ago
  Self-driving company here. We're doing on-prem because we have a clear projection on the amount of data we'll need to injest, store and process using GPUs. The advantages of running things in a cloud are clear -- and as an infrastructure team we have challenges around managing physical assets at scale, however it's clear with the cost of cloud providers that eventually we would have to pull data into a datacenter to survive at some point. Co-location costs are fixed, and it's actually easy to make a phenomenal deal now-a-days given the pressure these companies are under. The real trick of it all is that regardless of running on-prem or in the cloud, we need to run as if everything is cloud native. We run Kubernetes, Docker, and as much as possible automate things to the point that running one of something is the same as running a million of it.
• walterbell 6y ago
  The (startup) Oxide podcast has good history/stories about on-prem servers, from veterans of pioneering companies. They are fans of open-source firmware and Rust, and are working to make OCP-based servers usable for on-prem. In one podcast, they observed that cloud is initially cheaper, but can quickly become expensive with growth. There is a time window where you can still switch from cloud to on-prem, but if that window is missed, you're left with high switching costs and high cloud fees. https://oxide.computer/podcast/ https://oxide.computer/podcast/
  • mapgrep 6y ago
    Their co founder Bryan Cantrill gave a talk at Stanford on what they are trying to do, essentially offer on prem servers comparable to what “hyperscalers” like Google and Facebook put in their data centers — highly efficient and customizable (in low level software) iirc. https://youtu.be/vvZA9n3e5pc https://youtu.be/vvZA9n3e5pc
• catlas3r 6y ago
  Why stay on premise? Cost. On-prem is roughly on-par in an average case, in my experience, but we've got many cases where we've optimized against hardware configurations that are significantly cheaper to create on-prem. And sunk costs are real. It's much easier to get approval for instances that don't add to the bottom line. But for that matter, we try to get our on-prem at close to 100% utilization, which keeps costs well below cloud. If I've got bursty loads, those can go to the cloud. Lock-in. I don't trust any of the big cloud providers not to jack my rates up. I don't trust my engineers not to make use of proprietary APIs that get me stuck there. Related to cost, but also its own issue, data transfer. Both latency and throughput. Yeah, it's buzzwordy, but the edge is a thing. I have many clients where getting processing in the same location where the data is being generated saves ungodly amounts of money in bandwidth, or where it wouldn't even been feasible to transfer the data off-site. Financial sector clients also tend to appreciate shaving off milliseconds. Also, regulatory compliance. And, let's be honest, corporate and actual politics. Inertia. Trust. Risk. Interoperability with existing systems. Few decisions about where to stick your compute and storage are trivial; few times is one answer always right. But there are many, many factors to consider, and they may not be the obvious ones that make the decision for you.
• jerzyt 6y ago
  Client data confidentiality. I know it's a weak argument, but if the contract requires that we store data in-house, there's no choice.
• MattGaiser 6y ago
  I work on two projects, neither of which use cloud. For my day job, it is privacy and legal constraints. I work for the government and all manner of things need to be signed off on to move to cloud. We could probably make it work, but in government, the hassle of doing so is so large that it is not going to happen for a long time. In my contract project, it is a massive competitive advantage. I won't go into too many details, but customers in this particular area are very pleased that we do not use a cloud provider and instead host it somewhat on-premise. I don't see a large privacy advantage over using the cloud, but the people buying the service do simply because they are paranoid about the data and every single one of them could personally get in a lot of trouble for losing the data. Not my project, but intensive computing requirements can be much more cheaply filled by on-premise equipment (especially if you don't pay for electricity), so my university does most of its AI and crypto research on-premise.
• Blackadderz 6y ago
  I work in R&D for a telecomms/optics company. All servers are on premises. Not allowed to have a laptop. No access to emails/data outside of the office. No USB drives, printing documents, etc. Reason? Protect IP. From who? Mostly Huawei. Good and bad: When I walk out the door... I switch off. The bad is that working from home isn't realy an option. Although they have accommodated somewhat for this pandemic.
• allenrb 6y ago
  Finance space, under 100 people. Most servers are either latency-sensitive or 24/7 fully-loaded HPC. Neither case fits the cloud model. We do use cloud for build/test, though.
• kgc 6y ago
  We moved all AWS servers to a colo. Saving 80% of the cost.
• sergiotapia 6y ago
  Just want to say I love threads like these. I hope one day where I work we can have two or three obscenely beefy servers and be done with it. I'm planning for something similar probably Q2 2021 as our expenses grow too large on a manged hosting platform like Heroku/Render/Aptible.
• withinrafael 6y ago
  Not many services are available in gov cloud regions, so we're stuck with on-prem nearly everything.
• AgentK20 6y ago
  Like many others have pointed out: Cost. I'm the CTO of a moderately sized gaming community, Hypixel Minecraft, who operates about 700 rented dedicated machines to service 70k-100k concurrent players. We push about 4PB/mo in egress bandwidth, something along the lines of 32gbps 95th-percentile. The big cloud providers have repeatedly quoted us an order of magnitude more than our entire fleet's cost....JUST in bandwidth costs. Even if we bring our own ISPs and cross-connect to just use cloud's compute capacity, they still charge stupid high costs to egress to our carriers. Even if bandwidth were completely free, at any timescale above 1-2 years purchasing your own hardware, LTO-ing, or even just renting will be cheaper. Cloud is great if your workload is variable and erratic and you're unable to reasonably commit to year+ terms, or if your team is so small that you don't have the resources to manage infrastructure yourself, but at a team size of >10 your sysadmins running on bare metal will pay their own salaries in cloud savings.
  • MCthrowaway12 6y ago
    [offtopic] Awesome seeing you here! I used to be super active on Hypixel when I was in high school (I was actually #1 on the leaderboards for one of the games for over a year). One of my first large scale programming projects ever was creating a Hypixel mod for my guild. Years later I am now a software engineer working at Google on the YouTube algorithm!
    • AgentK20 6y ago
      Great to hear! Lots of the people who learned how to code because of Minecraft have ended up all over the industry. It's where I got my start, as well as many of my coworkers. That's part of our passion for making Hytale, to empower the next generation of youth who want a game they can tinker with and use to learn. If you haven't seen it yet definitely go take a look, got lots of cool plans for moddability and customization :)
      • kldavis4 6y ago
        Also offtopic, but my 12 year old is obsessed with Hytale.. Loves to read the blog updates and watch the videos and can't wait to eventually play it :)
• kevlar1818 6y ago
  We use on-prem GPU nodes for training deep learning models. Our group estimated the cost vs. cloud and it was significantly cheaper to go on-prem. I can't speak to security-wise and whatever-wise though :)
• hdmoore 6y ago
  Anything with massive storage and massive compute that doesn't need low latency is a great fit. I still host ~300TiB and ~250 cores at home because the cloud cost would be astronomical. Edit: This is for personal stuff related to internet-wide scan data and domain tracking. See https://github.com/hdm/inetdata https://github.com/hdm/inetdata
• olivierduval 6y ago
  Security... not against hacker but against provider's government. I worked for some french or european companies, with IPs and sensitive informations, and US business competitors. By the US law, US companies may have to let the US gov spy on their customers (even non US, even on non US location), so this may be a problem for strategic sectors, like defense for example. In that case, sensitive informations is required to be hosted in country by a company of the country, under the country law. Of course, it's not against "cloud" in general... only against US cloud providers (and chinese, and...)
• deleted 6y ago
  [deleted]
• Groxx 6y ago
  Meta-comment: cost: https://news.ycombinator.com/item?id=23098576 https://news.ycombinator.com/item?id=23098576 cost: https://news.ycombinator.com/item?id=23097812 https://news.ycombinator.com/item?id=23097812 cost: https://news.ycombinator.com/item?id=23098658 https://news.ycombinator.com/item?id=23098658 abilities / guarantees: https://news.ycombinator.com/item?id=23097213 https://news.ycombinator.com/item?id=23097213 cost: https://news.ycombinator.com/item?id=23090325 https://news.ycombinator.com/item?id=23090325 cost: https://news.ycombinator.com/item?id=23097737 https://news.ycombinator.com/item?id=23097737 threat model: https://news.ycombinator.com/item?id=23098612 https://news.ycombinator.com/item?id=23098612 cost: https://news.ycombinator.com/item?id=23097896 https://news.ycombinator.com/item?id=23097896 cost: https://news.ycombinator.com/item?id=23098297 https://news.ycombinator.com/item?id=23098297 cost: https://news.ycombinator.com/item?id=23097215 https://news.ycombinator.com/item?id=23097215 That's just the in-order top comments I'm seeing right now. (please do read and upvote them / others too, they're widely varying in their details and are interesting) The answer's the same as it has always been. Cloud is more expensive, unless you're small enough to not pay for a sysadmin, or need to swing between truly extreme scale differences. And a few exceptions for other reasons.
  • cortesoft 6y ago
    There is also another answer... I work for a CDN, so we can't really use the cloud when in many ways we ARE the cloud. Although we do often make jokes about "what if we just move the CDN to AWS?"
    • mcny 6y ago
      It is a pity East Dakota won’t make jokes like these. Can you imagine cloudflare running on aws? What happens when someone tries to denial of service them while on aws? On a different note, Netflix still runs “on the cloud”, right? I mean what does it really mean? Dropbox can still have most of its stuff on aws and do the expensive part on premises if cost is a concern? The truly bizarre stuff happens at hybrid cloud.
      • ddorian43 6y ago
• horsawlarway 6y ago
  I'm slowly coming to the complete opposite opinion you seem to have. I've worked almost entirely for companies that run services in various cloud infrastructures - Azure/Heroku/Aws/GCP/Other. I recently started a tiny 1 man dev shop in my spare time. Given my experience with cloud services it seemed like a no brainer to throw something up in the cloud and run with it. Except after a few months I realized I'm in an industry that's not going to see drastic and unplanned demand (I'm not selling ads, and I don't need to drive eyeballs to my site to generate revenue). So while in theory the scaling aspect of the cloud sounds nice, the reality was simple - I was overpaying for EVERYTHING. I reduced costs by nearly 90% by throwing several of my old personal machines at the problem and hosting things myself. So long story short - Cost. I'm happy to exchange some scaling and some uptime in favor of cutting costs. Backups are still offsite, so if my place burns I'm just out on uptime. The product supports offline, so while no one is thrilled if I lose power, my customers can still use the product. Basically - cost, Cost, COST. I have sunk costs in old hardware, it's dumb to rent an asset I already own. There might well be a point when I scale into a point where the cloud makes sense. That day is not today.
  • tjbiddle 6y ago
    What's the time trade-off? I've been drawing out my plans lately for a hobby project, all 100% on AWS. Being able to spin up my entire infrastructure with Terraform, build out images with Packer, setup rules for off-site backups, ensure everything is secure to the level I want it, etc. - It takes me next to no time at all. I can't imagine buying hardware, ensuring my home is setup with proper Internet, configuring everything here, and then still needing off-site backups anyway. Now, keep in mind - I'm definitely coming in from a Millennial point of view. My entire career was built on cloud. I've never touched hardware apart from building a computer back when I was 15 or something. I understand virtual. But being able to build up and tear down an entire setup, having it completely self-restore in minutes. Can't beat that. Napkin math has me at ~$50/mo: Full VPC, private/public isolated subnets, secure NACLs and security groups, infinitely extendable block storage and flat-file storage, near-instant backups with syncing to a different continent, 5 servers, DNS configurations, etc. All depends what you're doing too - of course. But for me, just the trade-off of working with what I know and not needing to leave my cafe of choice, still not breaking the bank - and if I do, having instant tear down and restore. Bam.
• blackflame7000 6y ago
  If you have a lot of data but not a lot of users, it's prohibitively expensive to pay monthly hosting and network egress fees when you can buy cheap hard drives, use ZFS, and whatever server protocol you desire.
• MaulingMonkey 6y ago
  I work in gamedev. Build servers, version control, etc. are almost always on-premise, even if a lot of other stuff has been transitioned to the cloud. There's a few reasons: 1) Bandwidth. I routinely saturate my plebian developer gigabit NIC links for half an hour, an hour, longer - and the servers slurp down even worse. In an AAA studio I am but one of hundreds of such workers. Getting a general purpouse internet connection that handles that kind of bandwidth to your heavily customized office is often just not really possible. If you're lucky your office is at least in the same metro area as a relevant datacenter. If you're really lucky you can maybe build a custom fiber or microwave link without prohibative cost. But with those kinds of geographical limitations, you're not so much relying on the general internet, so much as you're expanding your LAN to include a specific datacenter / zone of "the cloud" at that point. 2) Security. These servers are often completely disconnected from the internet, on a completely separate network, to help isolate them and reducing data exfiltration when some idiot installs malware-laden warez, despite clear corporate policy threatening to fire you if you so much as even think about installing bootleg software. Exceptions - where the servers do have internet access - are often recent, regrettable, and being reconsidered - because of, or perhaps despite, draconian whitelisting policies and other attempts at implementing defense in depth. 3) Customizability. Gamedev means devkits with strict NDAs and physical security requirements, and a motley assortment of phone hardware, that you want accessible to your build servers for automatic unit/integration testing. Oddball OS/driver/hardware may also be useful for such testing. Sure, if you can track down the right parties, you might be able to have your lawyers convince their lawyers to let you move said hardware into a datacenter, expand the IP whitelists, etc... but at that point all you've really done is made it harder to borrow a specific popular-but-discontinued phone model from the build farm for local debugging when it's the only one reproducing a specific crash when you want to debug and lack proper remote debug tooling. ...there are some inroads on the phone farms (AWS Device Farm, Xamarin Test Cloud) but I'm unaware of farms varied desktop hardware or devkits. Maybe they exist and just need better marketing? I have some surplus "old" server hardware from one such gamedev job. Multiple 8gbit links on all of them. The "new" replacement hardware often still noticably bottlenecked for many operations.
• dijit 6y ago
  I work for a large video games publisher, as you might expect we use a lot of windows. Windows server licenses on AWS and GCP are hundreds of times more expensive at our scale. Incidentally we actually do have some cloud infra and we like it, but the licensing cost is half the total price of the instance itself. In fact, you might not know this but games are relatively low margin, and we have accidentally risked the companies financial safety by moving into the cloud.
  • whatsmyusername 6y ago
    TBF windows licensing in general is a shit show, to the point where just handling that is a specialized ability potentially warranting a full time position.
• strags 6y ago
  Cost and Latency. My team and I run the servers for a number of very big videogames. For a high-cpu workload, if you look around at static on-prem hosting and actually do some real performance bencharking, you will find that cloud machines - though convenient - generally cost at least 2x as much per unit performance. Not only that, but cloud will absolutely gouge you on egress bandwidth - leading to a cost multiplier that's closer to 4x, depending on the balance between compute and outbound bandwidth. That's not to say we don't use the cloud - in fact we use it extensively. Since you have to pay for static capacity 24/7 - even when your regional players are asleep and the machines are idle, there are some gains to be had by using the right blend of static/elastic - don't plan to cover peaks with 100% static - and spin up the elastic machines when your static capacity is fully consumed. This holds true for anything that results in more usage - a busy weekend, an in-game event, a new piece of downloadable content, etc... It's also a great way to deal with not knowing exactly how many players are going to show up on day 1. Regarding latency, we have machines in many smaller datacenters around the world. We can generally get players far closer to one of our machines than to AWS/GCP/Azure, resulting in better in-game ping, which is super important to us. This will change over time as more and more cloud DCs spring up, but for now we're pretty happy with the blend.
• erulabs 6y ago
  We hear from our customers mostly what has been said here: cost and mental overhead. There is a bit of a paradox - companies that plan to grow aggressively are wary of AWS bills chopping their runway in half - they're very aware of _why_ cloud providers give out a year for free to most startups - they recoup that loss very fast once the cash faucet opens up. What really gets me is that most cloud providers promise scalability, but offer no guard-rails - for example diagnosing performance issues in RDS - the goal for most cloud providers is to ride the line between your time cost and their service charges. Sure you can reduce RDS spend, but you'll have to spend a week to do it - so bust out the calculator or just sign the checks. No one will stop you from creating a single point of failure - but they'd happily charge for consulting fees to fix it. There is a conflict on interest - they profit from poor design. In my opinion, the internet is missing a platform that encourages developers to build things in a reproducible way. Develop and host at home until you get your first customers, then move to a hosting provider down the line. Today, this most appeals to AI/ML startups - they're painfully aware of their idle GPUs in their gaming desktops and their insane bill from Major Cloud Provider. It also appeals to engineers who just want to host a blog or a wedding website, etc. This is a tooling problem that I'm convinced can be solved. We need a ubiquitous, open-source, cloud-like platform that developers can use to get started on day 1, hosting from home if desired. That software platform should not have to change when the company needs increased reliability or better air conditioning for their servers. If its a Wordpress blog or a minecraft server or a petabyte SQL database - the Vendor should be a secondary choice to making things.
  • sbrother 6y ago
    I've found that Kubernetes mostly solves this problem. I say mostly because for AI/ML workloads that require GPUs, we still rely on running things on bare metal locally, and deploying with GKE's magic annotations and Deep Learning images. But for anything else, I haven't had an issue going all in on k8s at the beginning, even with very small teams.
    • erulabs 6y ago
      Yep! My startup is https://kubesail.com https://kubesail.com, so I agree :) As for ML on Kube, I agree, there have been and still are some rough edges. The kernel drivers alone make a lot of out-of-the-box Kubernetes solutions unusable. That said, we've had a lot of success helping people move entirely onto kube - the mental gain alone from ditching the bash scripts or ansible playbooks (etC) alone is pretty freeing.
• apetersonBFI 6y ago
  I'm one of two IT persons at a food processor, in a small town. Despite living in an area where a majority of IT & Programmers work at the hospital or an insurance company, my boss has run our own networks and servers since the days of Novell, and we continue to run Windows servers on-premise, instead of the cloud. It does lead to interesting situations, like finding out a Dell server shrieking next to you is running max fan speed because the idrac is not connected. Neither of us have any experience with the cloud, whereas we have a lot of Microsoft experience. We still rely on OEM licenses of Office, because Office 365 would be 3x or more expensive. We have a range of Office 2019, 2016, 2013 OEM, and we get audited by them nearly every year. We use LastPass, Dropbox and Github, but only the basic features, and LastPass was an addition last year after someone got into our network through a weak username/password. In our main location, we have three ESX boxes, running several virtual servers, and then we have a physical server for our domain controller, file sharing and DHCP, DNS in other locations. We also switched to a physical server for our new ERP application server, which hasn't yet been rolled out. Projects like upgrading our ERP version can take months, but we have a local consulting team, with a specialist in our particular ERP solution, as well as a Server and Network specialist, and we also have a very close relationship with our ISP, who provides WAN troubleshooting. Our IT budget is small relative to our company revenue, so most cloud proposals would raise our costs manyfold. We continue to use more services like Github and Lastpass, and we both have multiple hats. I'm a developer, in-house app support, Email support, HR systems support, ERP support, PC setup, and I run our Data Synchronization operation and my boss runs EDI. I do a lot of PowerShell and Task Scheduler, but I've got familiar with bash through git bash.
• throwaway028374 6y ago
  I worked for a famous large tech company that makes both hardware and software. On stuff that runs on customer datacenters. There are plenty of companies that run their infrastructure to keep their data secure and accessible. It's not the type of companies that blog about their infra or are popular on HN. Banks and financial institutions, telcos, airlines, energy production, civil infrastructure. Critical infrastructure need to survive events larger than a datacenter outage. FAANGs don't protect customers from legal threats, large political changes, terrorist attacks, war.
• cpascal 6y ago
  My company cannot run our infrastructure in the cloud because we do performance/availability monitoring from ~700 PoP’s around the world. Not running our infrastructure in the cloud is part of our value proposition. Our customers depend on us to detect and alert them when their services go down. We _have_ to be operational when the cloud providers are not, otherwise we aren’t providing our customer with a valuable service. Another reason we don’t run in the cloud is because we store a substantial amount of data that is ever increasing. It’s cheaper to run our own SAN in a data center than to store and query it in the cloud. The final reason is our workloads aren’t elastic. Our CPU’s are never idle. In that type of use case, it’s cheaper to own the hardware.
• jcrawfordor 6y ago
  For hobby projects, I own a moderately outdated 1U "pizzabox" installed in a downmarket colocation facility in a nearby major city. Considering the monthly colocation rate and the hardware cost amortized over two years (I will probably not replace it that soon but it's what I've used for planning), this works out to appreciably less than it would cost to run a similar workload on a cloud provider. It costs about the same or possibly a bit less than running the same workload on a downmarket dedi or VPS provider, but it feels more reliable (at least downtime is usually my fault and so under my control) and the specs on the machine are higher than it's easy to get from downmarket VPS operations. Because my application involves live video transcoding I'm fairly demanding on CPU time, which is something that's hard to get (reliably) from a downmarket VPS operation (even DO or what have you) and costly from a cloud provider. On the other hand, dual 8 core Xeons don't cost very much when they're almost a decade old and they more than handle the job. There are a few fairly reputable vendors for used servers out there, e.g. Unix Surplus, and they're probably cheaper than you think. I wouldn't trust used equipment with a business-critical workload but honestly it's more reliable than an EC2 instance in terms of lifetime-before-unscheduled-termination, and since I spend my workday doing "cloud-scale" or whatever I have minimal interest in doing it in my off-time, where I prefer to stick to an "old fashioned" approach of keeping my pets fed and groomed. And, honestly, new equipment is probably cheaper than you think. Dealing with a Dell account rep is a monumental pain but the prices actually aren't that crazy. Last time I purchased over $100k in equipment (in a professional context, my hobbies haven't gotten that far yet) I was able to get a lot for it - and that's well less than burdened cost for one engineer.
• tcbyrd 6y ago
  I haven't personally done detailed cost analysis lately, but if you have systems that regularly operate at 80+% of capacity, I can't see how the operating costs of any cloud operator can be cheaper than operating it yourself. Their whole pricing model is based on being able to over-provision compute and move workloads around within a datacenter. As much as people talk about failing hard drives and other components at scale, failure rates are still low enough that you could operate dozens of systems at full capacity for 3+ years with maybe a handful of legit hardware failures. To rent that same compute from any cloud provider would cost significantly more. The cheapest "Dedicated Host" on AWS will cost you almost $12k over 3 years if you pay for it on-demand, and it's equivalent in specs to something you can buy for ~$2k. > am I missing something? I'd want more background behind what you mean by "at least for business". What kind of business? Obviously IaaS providers like Digital Ocean and Linode are are type of business that would not use other clouds. Dropbox and Backblaze as well would probably never use something like S3. And there are legitimate use cases outside of tech that have needs in specific teams for low latency compute, or its otherwise cost and time prohibitive to shuttle terabytes of data to the cloud and back (3D rendering, TV news rooms, etc). If you're talking about general business systems that can be represented by a website or app with a CRUD API, then most of that probably doesn't require on-prem. But that's not the only reason businesses buy servers.
• timbre1234 6y ago
  Amazon/Azure/GCP - they're _businesses_. They charge you 50-70 points of margin in order to run your computers. If you're R&D-limited, that's not important to you, but if you're a more mature company that's cost-limited then it matters a lot. If it's a core part of your business, it'll never be cheaper to pay another company to profit from you. Period.
• ex3ndr 6y ago
  We are small team startup and i am personally annoyed about pricing of a clouds. I have a 3 smallish VM for build server + managed SQL. It cost 500$/mo. It doesn't make sense. Having my own VMs on ESXi makes everything very different - most of the time this VMs do nothing, but you want to make them performant from time to time, so there are a plenty of resources because all other vms are too mostly IDLE. In cloud they are billed as if they are 100% loaded all the time. I am not really satisfied with latencies and insane price for egress traffic. I just can't do backups daily since it could cost whooping 500$/mo just for the traffic. This is just insane, i can't see how it could scale anywhere for B2C market. For B2B it might work really well though since revenue per customer is much higher. We are not moving to our own DC, but just keep realtime stuff in the cloud, but anything that is not essential is being moved somewhere else. Bonus is that you need off-site backups anyway in the case if cloud vendor will just ban you and delete all your data. Startups might move fast and iterate, but if you don't have your own servers you always reduce your usage because it could grow fast effectively reducing your delivery capacity.
• hourislate 6y ago
  If you interested, here is an article about how Bank of America chose to build out its own cloud to save about 2 billion a year. https://www.businessinsider.com/bank-of-americas-350-million-internal-cloud-bet-striking-payoff-2019-10?op=1 https://www.businessinsider.com/bank-of-americas-350-million...
• axaxs 6y ago
  My company moved from all on prem, to all in AWS. Having used both, I'd much rather go back to on prem. I did architecture, capacity planning, system tuning, deployments, etc. I knew everything about all of them, and treated them as sacred. The next generation came in, deciding not to learn anything about systems and brought in the 'systems as cattle' attitude and everything that comes with it. I try to remain objective, there are some pros to AWS, but I still much prefer my on prem setup. It was way cheaper, and deployments were way faster.
• busterarm 6y ago
  I'm going to buck the trend and say cloud is great. We do cloud, on-prem and colo (16 racks in two different DCs). Procurement is a nightmare especially when your vendor is having problems with yields (thanks Intel!) and the ability to scale up and scale down without going through hardware procurement process saves us millions of dollars a year. We avoid the lock-in by running on basic services on multiple cloud providers and building on top of those agnostically. Spend is in the millions per month between the cloud providers, but the discounts are steep. We're essentially had to build our own global CDN and the costs are better than paying the CDN services and better than running our own hardware & staffing all those locales. It's a no brainer. We'll continue to operate mixed infrastructure for quite some time as certain things make sense in certain places.
• mathattack 6y ago
  We are going to the cloud, but you have to be careful. With on-prem the limit of the cost is the server. Someone writes inefficient code and it just doesn’t work. In the cloud there are 1000 ways to overspend and the vendors purposefully don’t make it easy to track or keep things under control. It’s kind of like outsourcing. If you don’t know what you are doing, cost goes up and quality goes down.
• _bxg1 6y ago
  One of my favorite things (at least for personal projects) about using the cloud is so-called "platform as a service" systems like Heroku, where I don't have to get down in the weeds, I just push code and the process starts (or restarts). Is there something like that I could use on my own hardware? I just want to do a fresh Linux install, install this one package, and start pushing code from elsewhere, no other configuration or setup necessary. If it can accept multiple repos, one server process each, all the better. I know things like Docker and Kubernetes exist but what I want is absolute minimal setup and maintenance. Does such a thing exist?
  • mappu 6y ago
    You're looking for Dokku Same git push deploys, heroku-compatible buildpacks or Dockerfiles, all on your own hardware, MIT license.
    • _bxg1 6y ago
      This looks perfect, thank you! I knew there was no way I could've been the first person to think of this
• nikisweeting 6y ago
  Quebec power and internet pricing is really competitive. For residential services I pay $0.06/kw + $80/mo for 1Gbps fiber with 1ms ping to 8.8.8.8 (USD). As a result, I run a power-hungry Dell r610 with 24 cores and 48GB of ram with 20+ services on it for many different aspects of my company. All the critical stuff runs on DigitalOcean / Vultr, but the 20+ non-critical services like demo apps, CI/CD, cron workers, archiving, etc. run for <$200/yr in my closet.
  • blaser-waffle 6y ago
    This is also why there are a lot of data centers in the Greater Montreal area, FWIW.
• deleted 6y ago
  [deleted]
• tpae 6y ago
  Because Elon doesn't believe in the cloud. We were one of the few teams that got AWS access, and we were told not to rely on it too much because it's temporary...
• GaryNumanVevo 6y ago
  We run one of the largest Hadoop clusters on the planet. On-prem is very cost efficient if you're running jobs flat out 24/7
• jaboutboul 6y ago
  There is not one clear cut answer on this. It depends on what your company values, i.e. cost vs. agility. If you are using the cloud for what it was meant for availability, scalability, elasticity--and those are the things that your org values--its the right fit for you. If on the other hand you value cost then it clearly isn't the right fit. One other point I'll make, the true value of cloud isn't in IaaS, renting VMs from anyone is relatively expensive compared to the costs of buying a server and maintaining it yourself for a number of years. The true value of the cloud is when can architect your solution to utilize the various services the cloud providers offer, RDS/DynamoDB, CDN, Lambda, API Gateway, etc. so that you can scale quickly when you need to.
• reilly3000 6y ago
  I’ve been studying like a fiend to get AWS certs and thoroughly understand the cloud value proposition, especially for ephemeral workloads and compliance needs. I’m all for cloud solutions that make sense and love when serverless/usage-only systems can’t be deployed. That said, I recently started work on a friend’s system that he has had running in colo for a long time. It’s absolutely insane how long his systems have been up. There are processes that have been alive since 2015, with some hosts having uptime linger than that. He’s got a nice HA configuration but hasn’t had any incidents that have triggered failover. He recently built a rack for his home with 384gb ram and gobs of cpu across 3 nodes, with rack, nice switch and UPS for just shy of $2500 ( he is quite the bargain hunter... ). I did some quick math and found a similarly equipped cluster (of just VMs, not dedicated hosts) has a 1.1 month break-even with on-demand costs, no bandwidth considered. Sure, maybe a 1 year reservation could make it a 2-3 month break even instead, but why? Those machines can easily give him 3-5 years of performance without paying another dime. If you can feasibly run workloads onpremise or colo and have a warm failover to AWS you could probably have the best of all worlds.
  • whatsmyusername 6y ago
    If he has processes that have been up since 2015 how is he patching? That's one of my biggest gripes with on-prem, it's easy to leave something that works alone... until it gets popped by a 5 year old vuln. In cloud I'm constantly looking at what we have because I have good billing tools in place to see what we're paying for.
• annoyingnoob 6y ago
  International Traffic in Arms Regulations (ITAR) Compliance - much easier to keep it on site, off-site compliance is costly. Also, cost over time. Better control of performance requirements for certain applications.
• gen220 6y ago
  My SO’s brother works in the studio video recording industry, and is a very IT-savvy guy. We had a long discussion last holiday season about the state of cloud adoption in that industry. He told me (this is obviously secondhand) that most of the movie industry is not only off the cloud, but exclusively working in the realm of colocating humans and data (footage). This is for many reasons. The one that comes back to me now is that the file sizes are HUGE, because resolution is very high, so bandwidth is a major concern. Editors and colorists need rapid feedback on their work, which demands beefy workstations connected directly with high bandwidth connections to the source files. Doing something like this over a long distance network (even if the storage was free) would be prohibitively expensive, and sometimes literally impossible. So the write loads are basically the antithesis of what cloud optimized for: “random sequential reads of typical short length, big append only writes”. The big production houses (lucasarts famously) are also incredibly secretive about their source material, and like to use physical access as a proxy for digital access. It leads to some seemingly strange (to me as a cloud SWE guy) decisions. He pretty much exclusively purchases top of the line equipment (hard drives/ssds), and keeps minimal if any backups for most projects because there simply isn’t any room. It’s a recipe for disastrous data loss, and apparently it’s something that happens quite often to this day. It’s just extremely prohibitively expensive to do version control for movie development. I don’t know to what extent cloud technologies can solve for this domain. I asked him if Netflix was innovating in this area, since they’re so famously invested in AWS, but he said that they mostly contracted out the production stuff, and only managed the distribution, which makes sense. The contractors don’t touch the cloud at all, for the most part. Again most of this is secondhand, I’d be curious to hear more details or reports from other people in the movie industry.
• frogbox12 6y ago
  After a big cloud-first initiative, several managers left; leaving implantation to Linux sysadmins now in charge of cloud. Treated cloud as some colo facility, dump all apps in one big project/account, cloud costs spun quickly out-of-control and lots of problems with apps not being segregated from each other. Cloud declared 'too expensive' and 'too insecure', things migrated back on-prem, team now actively seeks to build and staff colo facilities with less than 10ms latency somewhere outside coastal California (Reno, Vegas, PHX) which just isn't gonna happen because physics.
• ocdtrekkie 6y ago
  There are very few niches where the cloud makes sense: Namely, where you are either too small to benefit from a single server and a single entry-level IT guy (think three or four person companies with low need for technical competency), or where you are expecting rapid growth and can't really rationally size out your own hardware for the job (in this case, the cloud is useful initially, leave it later once your scale is more stable). In every other case, you are paying for the same hardware you could buy yourself, plus the cloud provider's IT staff, plus your own IT staff which you likely need anyways to figure out how to deal with the cloud provider, and then the cloud provider's profit margin, which is sizeable.
• acwan93 6y ago
  I agree with you OP. Our company provides on-premise ERP systems to small (we’re talking at most 20 person companies) wholesale distributors. Pre-COVID, I was pushing for a cloud solution to our product and pivoting our company towards that model. We’re at a hybrid approach when COVID hit. What ends up happening with an on-premise/hybrid cloud model is we end up doing a lot of the sysadmin/IT support work for our customers just to get our ERP working. This includes getting ahold of static IP addresses (and absolving responsibility), configuring servers/OSes, and several other things along the same vein that’s wholly irrelevant to the actual ERP like inventory management and accounting. Long story short, these customers of ours end up expecting us to maintain their on-premise server without actually paying for help or being knowledgable about how it all works. We keep pitching them the cloud but they’re not willing to pay us a recurring fee even though it actually saves the headaches of answering the question “who’s responsibility is it to make sure this server keeps running?" I think a lot of these answers here are dealing with large-scale products and services where the amount of data and capital costs is so massive it makes sense to start hiring your own admins solely to maintain servers. For these small mom-and-pop shops who are looking for automation, the cloud is still the way to go.
  • jolmg 6y ago
    Deja vu. I think you totally hit the nail on the head with that last paragraph. On-premise ERP systems probably only make real sense for (non-small) companies that wish to avoid relying on the internet (because e.g. their business strategy requires that freedom) and can hire long-term sysadmins/programmers that can provide support to those systems.
    • acwan93 6y ago
      Have you had experience selling on-premise systems? I’m really curious how other companies handle the sysadmin and IT support issue.
• INTPenis 6y ago
  Becuse we don't trust a foreign cloud provider with our client's data. Why is that so hard to understand? All the best cloud providers are from the US and as a european company with clients in european government and healthcare we are often not morally or legally allowed to use a foreign provider. The sad thing is that this is an ongoing battle between people on a municipal level who believe they can save money in clouds, and morally wiser people who are constantly having to put the brakes on those migration projects.
  • v4dok 6y ago
    What about the existing/upcoming technologies of let you use the cloud without trusting it?
    • INTPenis 6y ago
      Doesn't matter because all US companies are subject to US laws and agencies. Even if the data is in Ireland, they are obliged to cooperate and before you know it all our patient records are leaked in the states. And speaking of Ireland, I have a memory of a Microsoft EULA saying that even if the data is stored in Ireland they can't guarantee that it won't be transferred to the US.
• exabrial 6y ago
  Hybrid cloud is the moneymaker solution, but there are no out of box solutions for it.
• ps 6y ago
  Cost. We recently moved one rack to the different DC in the same city and used Digitalocean droplets to avoid downtime. Services running on Linux were migrated without high-availability (e.g. no pgsql replication, no redis cluster, single elasticsearch node...) and we just turned off Windows VMs completely due to licensing issues and no need to have them running at night. The price of this setup was almost 4x higher than what we pay for colo. Our servers are usualy <5 years old Supermicro, we run on Openstack and Ceph (S3, rbd) and provide VPNaaS to our clients. AWS/GCP/Azure was out of question due to cost. We considered moving Windows servers to Azure with the same result - the cost of running Windows Server (AD, Terminal, Tomcat) + MS SQL was many times higher than the price of colo per month. It is bizarre that you can buy the server running those VMs approximately every 3 months for the Azure expenses (Xeon Platinum, 512GB RAM).
• pavelevst 6y ago
  AWS is one of most expensive options, and far not perfect, I can’t understand why people consider it as default choice... To compare - dedicated server on hetzner (core i7, 32gb ram, 20tb network) is cheaper than medium VM on AWS. If the product is growing, cloud cost can quickly become the biggest expense for company. Than it will make sense to spend some time make things run in more cost effective way I think if you choose cloud hosting that costs about same as renting dedicated server plus settings virtualization by yourself - than it’s a fair choice (can check on https://www.vpsbenchmarks.com/ https://www.vpsbenchmarks.com/ or similar) Another sweet configuration is dedicated servers with kubernetes: good user experience for developers, easy to setup and maintain, easy to scale up/down
• quanto 6y ago
  > The more I learn, the more I believe cloud is the only competitive solution today, even for sensitive industries like banking or medical. I honestly fail to see any good reason not to use the cloud anymore, at least for business. Cost-wise, security-wise, whatever-wise. [emphasis mine] Most people here seem to point out cost and utilization. I would like to offer another perspective: security. I worked in both of these industries: finance ("banking", not crypto or PG) and medical (within a major hospital network). The security requirement, both from practical and legal perspectives, cannot be understated. In many situations, the data cannot leave an on-prem air-gapped server network, let alone use a cloud service. It costed us more to have on-prem servers as we need a dedicated real estate and an engineering team to maintain. Moreover, the initial capital expenditure is high -- designing and implementing a proper server room/data center with climate control, power wiring, and compliant fire extinguishing system are not trivial.
• icelancer 6y ago
  Yes. Cost of GPUs if you want them on regularly and reliably, and you don't need 2000 of them. We run small/mid-sized operations on-demand and the latency of spinning up instances is not competitive and the cost is outrageous to have them on standby.
• markc 6y ago
  on-premises not on-premise
• joshuaellinger 6y ago
  On the smaller end of the scale, I have a $12K/mo spend with Azure. I decided to go back to Coloc. For under $50K, I have 4 machines with an aggregate 1TB RAM, 48 cores, 1 pricy GPU, 16TB of fast SSD, 40 TB HHD, and infiniband @ 56GB/sec. Rent on the cabinet is less than $1K/mo. It's going to cost me about $20K in labor to migrate. So the nominal break-even point is six months but the real kicker is that this is effectively x10-30 the raw power of what I was getting on the cloud. I can offer a quantitatively different set of calculations. It also simplifies a bunch of stuff: 1. No APIs to read blob data -- just good old files on an ZFS share. 2. No need to optimize memory consumption. 3. No need for docker/k8s/etc to spin up under load. Just have a cluster sitting there. There are downsides but Coloc beats the cloud for certain problems.
• Mave83 6y ago
  Cloud is only good if you don't care about costs and plan to scale without looking back. For example, building a Ceph based Software Defined Storage with croit.io for S3 comes for 1/10 to 1/5 of the AWS price in TCO. Same goes for any other product in the cloud. If you only need it a short time up to 6 months, go to the cloud. If you plan to have the resources longer than 6 months go to Colocation.
• darrelld 6y ago
  I have a friend who is an IT manager for a large chain hotel in the Caribbean. I keep asking him about why they still use on premises equipment and it boils down to: * Cost for training / transitioning + sunk cost fallacy * Perceived security risk (right or wrong) * IT is mostly invisible and currently "works" with the current arrangement, why change?
• sudhirj 6y ago
  The way I think about it this: not using the cloud is like building your own code editor or IDE after assembling your own laptops and desktops. It may make you happier and it’s a great hobby, but if you’re trying to run a business you need to do a cost benefit analysis. We currently have double digit petabytes of data stored in our own data centres, but we’re moving it to S3 because we have far better things to do with our engineers that replace drives all day, and engineering plus hardware is more expensive than S3 Deep Archive - but it wasn’t until Deep Archive came out. We put out hundreds of petabytes of bandwidth, and AWS is horribly expensive at first glance, but if you’re at that scale you negotiate private pricing that brings it to spitting distance of using a COLO or Linode/Hetzner/OVH - the distance is small enough that the advantages of AWS outweigh it, and it allows us to run our business at known and predictable margins. Besides variability (most of our servers are shut nights, weekends and when not required), op ex vs cap ex, spikes in load (100X to 1000X baseline when tickets open), there’s also the advantage of not needing ops engineers and being to handle infrastructure with code. If you have a lot of ops people and don’t need any of the advantages, and you have lots of money lying around that you can use on cap ex, and you have a predictable load pattern, and you’ve done a clear cost benefit analysis to determine that building your own is cheaper, you should totally do that. Doesn’t matter what others are doing.
  • nihil75 6y ago
    You're not building everything from scratch on-prem. There are excellent tools for deploying and managing infrastructure like Terraform, Ansible, Puppet that funny enough are used to deploy to cloud as well. Add a self-hosted Kubernetes cluster to that and your on-prem is not that different than a cloud. As for ops people - you might not need an engineer to replace failed hard-drives, but you'll need a DevOps person to manage Cloud Formation templates and such, and they cost more.
  • fock 6y ago
    maybe it doesn't need a SV-salary engineer to swap harddrives. Just pick (hehe) three good ones of the AMZ-warehouse people, train him/her and you'll have a very happy employee which is around fulltime (or three of them) and paid well (1/3 of your engineers salary). They'll enjoy working predictable shifts in a nice office and in time you can even train them to do 2nd level support.
• otabdeveloper4 6y ago
  We're in the process of moving a greenfield project from AWS to a more traditionally hosted solution. AWS turned out to be 5-10 times more expensive; what's worse, our developers are spending more then half their time working around braindead AWS design decisions and bugs. A disaster any way you look at it. There are good reasons to chose AWS, but they're never technical. (Maybe you don't want to deal with cross-departmental communications, or you can't hire people into a sysadmin role for some reason, maybe you want to hide hosting in operational expenses instead of capital, etc.)
• lrpublic 6y ago
  - cost, as well evidenced in other comments here. The hyperscalers are orders of magnitude more expensive than dedicated hosting or using collocation providers. - lock in, all the hyper scalers want to sell you value add services that make it hard or impossible to move away. - concentration risk, hyper scale providers are a well understood target for malign actors. It’s true they are better protected than most. - complexity, if you think about how little time the hyperscalers have been operating in comparison with corporate IT they have created huge technical debt in the race to match features.
• totorovirus 6y ago
  I work in a data science company with 30+ engineers. We've spent 80k dollars on GKE last month only..
• mattmireles 6y ago
  CEO of an AI startup & former AWS employee here. The cloud sucks for training AI models. It's just insanely overpriced in a way that no "Total Cost of Ownership" analysis is going make look good. Every decent AI startup––including OpenAI––has made significant investments in on-premise GPU clusters for training models. You can buy consumer-grade NVIDIA hardware for a fraction of the price that AWS pays for data center-grade GPUs. For us in particular, the payback on a $36k on-prem GPU cluster is about 3-4 months. Everything after that point saves us ~$10k / month. It's not even close. When I was AWS, I tried to point this fact out to the leadership––to no avail. It simply seemed like a problem they didn't care about. My only question is why isn't there a p2p virtualization layer that lets people with this on-prem GPU hardware rent out their spare capacity?
  • blueblisters 6y ago
    Are TPUs too application specific to replace GPUs? It seems cloud TPUs could be competitive with GPUs in terms of $ per number of target epochs, provided you can do data parallelism for your workloads. Also, IBM offers bare metal pricing which is somewhat cheaper than virtualized instances attached to GPUs (and faster too). I think GPU virtualization is not quite there yet because Nvidia does not give access to core GPU functionality needed for efficient virtualization - you're stuck with using their closed-source libraries and drivers.
• nurettin 6y ago
  Sure, for tiny intel servers it makes sense to rent vms. It won't start to hurt until 1.5 years later at which point the project needs to become profitable anyways. I run a couple of on-premise xeon gold machines with 96 gb ram and 40+ cores on each. Their total purchase cost was the monthly cost of renting them on the cloud. Also, you will never get the full benefit of the servers you use unless they are dedicated instances with no virtualization layer.
• julienfr112 6y ago
  I think it depends on what is the alternative and what hardware you are buying. If you go for top of the line HPE or dell or Cisco hyperconverged stuff that allow you to be, sort of, your own cloud, you will end up with the same large bill.
• mlang23 6y ago
  Cost and data security, while cost actually weights more. It is simply not true that the cloud is cheaper.
• sqldba 6y ago
  - Uptime The number and frequency of outages in Azure are crazy. They happen non-stop all year around. You get meaningless RCAs but it never seems to get better, and if it did, you'd have no way of knowing. Compare this with doing stuff internally - you can hire staff, or train staff, and get better. In the long run outsourcing and trusting other companies to invest in "getting better" doesn't end very well. Just because they moved their overall metrics overall from 99.9 to 99.91 may not help your use case. - Reliability Their UIs change every day, there's no end to end documentation on how things work. There's no way to keep up. - Support Azure's own support staff are atrocious. You have to repeatedly bang your head against the wall for days to get anyone who even knows the basic stuff from their own documentation. But it's also difficult to find your own people to do the setup too. Sure, lots of people can do it, but because it's new they have little experience and end up not knowing much, unable to answer questions, and building garbage on the cloud platform. Because there's no cloud seniority, it hasn't been around for long enough. - Security Cloud providers have or can get access and sometimes use it. - Management I've seen too many last minute "we made a change and now everything will be broken unless you immediately do something" notifications to be happy about. - Cost It's ridiculously expensive above a certain scale, and that scale is not very big. I don't know if it's because people overbuild, or because you're being nickel-and-dimed, or if you're just paying so many times above normal for enterprise hardware and redundancy. It's still expensive. Yes, owning (and licensing) your own is expensive too. For smaller projects and tiny companies, totally fine! It's even great! - Maturity People can't manage cloud tools properly. This doesn't help with costs above. PS: I don't think any other cloud service is better.
• itworker7 6y ago
  In physics, a gallon of water weights 8.34 lbs. (For this analogy, a gallon of water is a unit of work.) And the gallon of water weighs 8.34 lbs irregardless if it is sitting on my desk in a physical building, or on your desk, in the cloud. Same weight, same unit of work. same effort. For a brand new, greenfield application, the cloud is a no brainer. I agree 100%. But for legacy applications, and there are so, soooo many, the cloud is just some one else's computer. Yes, the cloud is more scaleable, yes, the cloud is more manageable, and yes, you can control the cpu/storage/memory/network in much finer amounts. But legacy applications are very complicated. They have long tails, interconnections to other applications that cannot immediately be migrated to the cloud. I have migrated clients off of the cloud, back to on premise or to (co-lo) local hosting, because without rewriting the legacy application, the cloud costs are simply too great. The essence of IT is to apply technology to solve a business problem. Otherwise, why would the business spend the money? The IT solution might be crazy/stupid/complex but if it works, many business simply adopt it and move on. Now, move that crazy/stupid/complex process to the cloud and surprise, it is very, very expensive. So, yes, the cloud is better, but only for some things. And until legacy applications are rewritten on-premise will exist. One final insight. The cloud costs more. It has been engineered to be so, both from a profitability standpoint(Amazon is a for profit company) but also because the cloud has decomposed the infrastructure of IT into functional subcomponents, each of which cost money. When I was younger, the challenge for IT was explaining to management, the ROI of new servers, expanded networking, additional technology. We never quite got it right and often had it completely wrong. That was because we lacked the ability to account for/track and manage the actual costs of an on-premise operation. Accounting had one view, operations had another view and management had no idea really, why they were spending millions a year and could not get their business goals accomplished. The cloud changed all of that. You can do almost anything in the cloud, for a price. And I will humbly submit, that the cost of the cloud - minus the aforementioned profitability, is what on-premise organizations should have been spending all along. Anyone reading this and who has spent time in a legacy environment, knows that it is basically a futile exercise of keeping the plates spinning. On-premise failed because it could not get management to understand the value on in-house IT. As I said, the costs are the same. A gallon of water weighs what it weighs regardless of location. It will be interesting to see, I predict the pendulum will swing back.
• jordanbeiber 6y ago
  My last three employments have had me and my team build three different platforms with three different “providers”. Many lessons learned! Chronological order: 1. E-commerce, low volume (1000-5000RPM), very high value conversions, highly localized trade. We built an on-prem stack using hasicorp here. This place had on-prem stuff already in place, the usual vendor driven crap - expensive hypervisor, expensive spoffy SAN, unreliable network. Anyway, my platform team (4-5 guys) built a silo on commodity hardware to run the new version of this site. This is a few years back, but the power you get from cheap hardware these days is astounding. With 6 basic servers, in two DCs, stuffed with off the shelf SSDs we could run the site and dev teams no problem. Much less downtime compared to the expensive hyperconverged blade crap we started on at basically no cost. There’s a simplicity that wins out using actual network cables and 1u boxes... LXC is awesome btw! Using “legacy” vmware, emc, hp etc for non-essential on-prem? Cloud is tempting! 2. Very high volume (Billions of requests per day), global network. AWS. Team tasked with improving on-demand scalability. We implemented kubernetes on AWS and it really showed what it’s about! After 6-7 months of struggle with k8s < 1.12 things turned around when it hit 1.12-1.13-ish we got it to act how we wanted. Sort of, at least. Cloud just a no brainer for this type of round-the-clock, elastic workload. You’d need many millions up-front to even begin building something matching “the cloud” here. Lot of work spent tweaking cost though. At this scale cloud cost management is what you do. 3. Upstart dev-shop. No rpm outside dev (yet). Azure. About 30 devs building cool stuff. Azure sucks as IaaS, they want you to PaaS that’s for sure. Cloud decision had been made already when I joined. Do you need cloud for this? No. Are there benefits? Some. Do they outweigh the cost? Hardly. In the end it will depend on how and where your product drives revenue. We pay for a small local dev datacenter quarterly, which i find annoying. Just some quick thoughts off the top of my head (on the phone so excuse everything). Happy to discuss further!
• gorgoiler 6y ago
  For business logic at a fairly large school the free and open tools that make the cloud so productive get used here a lot. We get to leverage commodity hardware and network infra very effectively for on-premises[1]. You have to have a good recovery plan for when equipment X’s power supply fails but when deploy is all automated, it’s very easy to overcome swapping bare metal, and easy to drill (practice) during off hours. This makes it much easier to meet regulatory compliance: either statutory or regulations your org has created internally (e.g. financial controls in-org, working with vulnerable people or children, working with controlled substances, working with sensitive intellectual property.) Simply being able to say you can pull the plug on something and do forensic analysis of the storage on a device is an important thing to say to stakeholders (carers, carers families, pupil parents.) I’m so grateful to be living in the modern age when “cloud” software exists[2], but I don’t have to be in the cloud to use it. The downside: you need trained staff and it’s completely inappropriate if you need any kind of bandwidth, power consumption, or to support round the clock business (which we do not because, out here on the long tail, we work in single city so still have things like evenings weekends and holidays for maintenance!) — [1] Premise vs premises is one of those “oh isn’t the English language awful” distinctions. Premise is the logical foundation for some other theory (“the premises for my laziness is that because the sky is grey it will probably rain so I’m not going to paint the house”) where as the premise_s_ means physical real estate property (“this is my freshly painted house: I welcome you onto the premises”.) [2] Ansible, Ubiquiti, arm SBCs like raspberry pi, Docker, LXC, IPv6 (making global routing for more tractable, IPv4 for the public and as an endpoint to get on the VPN.)
• pmlnr 6y ago
  > The more I learn, the more I believe cloud is the only competitive solution today, even for sensitive industries like banking or medical. Then learn A LOT more and start with mainframes and their reliability.
• amq 6y ago
  Surprised no one mentioned a third option: cheap vps providers like digitalocean or vultr. They've become real contenders to big clouds recently, providing managed databases, storage and k8s. And their bandwidth costs are close to what you'd get in colo.
• doctor_eval 6y ago
  Lots of great insights here, which fully accord with my experience, even in the small end of town. About a year ago, I was in a meeting with my new CEO (who had acquired my company). My side of the business had kept hardware in-house, his was in AWS. We had broadly similar businesses in the same industry and with the same kind of customers. My side of the business needed to upgrade our 5+ year old hardware. The quote came to $100K; the CEO freaked out. I asked him how much he spent on AWS? The answer was that they spent $30K per month on AWS. The kicker is that we managed 10x as many customers as they did, our devops team was half the size, and we were rolling out continuous deployment while they were still struggling to automated upgrades. Our deployment environment is also far less complicated than theirs because there isn't a complex infrastructure stack sitting in front of our deployment stack. There was literally no dimension on which AWS was better than our on-prem deployment, and as far as I was able to tell before I quit, the only reason they used AWS was because everyone else was doing it.
  • pickle-wizard 6y ago
    With all the job hopping that goes on in tech, there is a lot of Resume Driven Development. People want to use AWS because it will help them get their next job. I'm finally in a job that I'm happy with and can see myself staying here until retirement. I have noticed that has changed my technology recommendations. For example we recently started looking at configuration management tools. Ansible is the obvious choice from a resume perspective as it is very popular. I ended recommending Powershell DSC. Why, because our environment is mostly windows, the team is familiar with Powershell, and for our use case is much faster. Powershell DSC is not as popular so it won't help me get another job. When it comes time to expand the team, I can hire someone who understands configuration management tools or powershell, and get them up to speed in a day or two.
  • propogandist 6y ago
    >There was literally no dimension on which AWS was better than our on-prem deployment The big pitch is opex vs capex. This is one of the leading "value propositions" being made to the C-suite. If you're a business that's worried about capex then opex model will look very appealling... until someone looks at how much you'll (generally) be screwed in the Opex model in the mid to long term. Most 'decision makers' in large corporations are short term oriented and the opex model ends up winning.
• benbro 6y ago
  How do you manage installation and upgrades of dedicated servers? What do you use for block storage and object storage? Kubernetes and Ceph seems to be hard to setup and maintain.
• adev_ 6y ago
  I worked in Switzerland and a reason to use on premise here is Security Many detail banks, asset management company or high security company refuse to use any public Cloud. They want to have a strict and traceable list of people who have physical access to their hardware. This in order to control any risk of dataleak [1]. In practice they use generally on-premise installation. They ren space in a computer center and own there a private cage monitored with multiple cameras. Meaning they know exactly anyone touching their hardware and enforce security clearance for them. [1]: https://en.m.wikipedia.org/wiki/Swiss_Leaks https://en.m.wikipedia.org/wiki/Swiss_Leaks
• mconstantinescu 6y ago
  Hi @agentK20 , I agree with you that cloud providers should not take the skin out of you. I work for Sardina Systems, an ISV, and it is very frustrating for me to hear your high costs feedback more and more often everywhere. Please drop me an email at mconstantinescu@sardinasystems.com about your cloud details and I will make sure to provide you the best cost for it, very competitive in the market, as we are very flexible to adjust to any workload size and personalise the offer accordingly. Thank you, Mihaela
• irrational 6y ago
  It is so so so much cheaper. We moved to AWS and tried setting up our servers with specs that were comparable to what our physical servers had been. We just about died after the first month. Our bill was higher for one month than for multiple years running our physical servers. We had to way dial them back to get the run rate down to a reasonable number. Now we have serious buyers remorse. Performance is terrible. The cost is still more per month than we ever had with our physical servers by a large amount.
• lvturner 6y ago
  We install in to remote locations - you can't access the cloud if your connectivity is down so local resources is a hard must. Though we have adopted something close to an "Edge Computing" solution... I guess it comes down to "Why not both?" :) I think it also depends on your definition of "server"
• Cthulhu_ 6y ago
  Our software runs in core (mobile) network systems, but that's at our customers. We ourselves have a rack in our office that runs things like Git repos, project management, virtual machines for development / testing, build servers, and instances for trainings. We're concerned about corporate espionage and infiltration, so we can't trust our servers being out of our sight. Most people don't have the code on their physical machines either; I'm a cocky breath of fresh air in that regard in that I prefer my stuff to run locally instead of the (slow, underpowered) VMs, I trust Apple's encryption a lot.
• alpenbazi 6y ago
  Security. And Independency.
• avifreedman 6y ago
  We're (Kentik) a SaaS company and one key to having a great public margin is buying and hosting servers. In our case, we use Equinix in the US and EU to host, with Juniper gear for routing/switching, and the customary transit, peering at the edge. One secondary factor is that we've only monotonically increased, and it's way cheaper to keep 10%-15% overprovisioned than to be on burst price with 50%+ constant load. But the simplest math is - we have > 100 storage servers that are 2u 26x2tb flash, 256gb RAM, 36 cores. They cost $18k once, which we finance at pretty low interest over 36 months (and really last longer than that). Factor in $200-400/mo to host each depending (I think it's more like $200, but it doesn't matter for the cloud math). That same server would be many $thousands/month on any cloud we've seen. Probably $4-6k/mo, depending on the type of EBS-ish attached. Or with the dedicated server 'alternate' they are moving to offer (and Oracle sorta launched with). It'd be cheaper but still > 2x as expensive on Packet, IBM dedicated, OVH, Hetzner, Leaseweb (OVH and Hetzner the cheapest probably). Three other factors for us: 1) Bandwidth would be outrageous on cloud but probably not as outrageously high as just the servers, given that our outbound is just our SaaS portal/API usage 2) We'd still need a cabinet with router/switch infra to peer with a couple dozen customers that run networks (other SaaS/digital natives and SPs that want to send infrastructure telemetry via direct network interconnect). 3) We've had 5-6 ops folks for 3 of the 6 years, 3-4 for the couple years before that. As we go forward, as we double we'll probably +1. It is my belief that we'd need more people in ops, or at least eng+ops mix, if we used public cloud. But in any case, the amount of time we spend adding and debugging our infra is really really really small, and the benefit of knowing how servers and switching stuff fails is huge to debugging (or, not having to debug). All that said - we do run private SaaS clusters, and 100% of them are on bare metal, even though we could run on cloud. Once we do the TCO, no one yet has wanted to go cloud for an always-on data-intensive footprint like ours. Good luck with your journey, whichever way you go! And happy to discuss more, email in profile
• adreamingsoul 6y ago
  Personally, I'd rather have capex than opex. My observations from working with, and in the "cloud": The "cloud" does benefit from it's scale in many ways. It has more engineers to improve, fix, watch, and page. It has more resources to handle spikes, whales, and demand. Almost everything is scale tested and the actual physical limits are known. It is damn right impressive to see what kind of traffic the cloud can handle. Everything in the "cloud" is abstracted which increases complexity. Knowledgeable engineers are few and far between. As an engineer you assume something will break, and with every deployment you hope that you have the right metrics in place and alarms on the right metrics. The "cloud" is best suited for whales. From special pricing to resource provisioning, they get the best. The rest is trickled down. Most services are cost-centers. Very few can actually pay for the team and the cost of its dependencies. It's insane how much VC money is spent building whatever the latest trend of application architecture is. Very few actually hit their utilization projections.
• drej 6y ago
  Many of the stories here are from large companies, where the costs are quite a different beast. I want to offer an opposite view - from a small company (20-30 people), which is usually the kind of company best suited for the cloud. We ran a number of modelling jobs, basically CPU intensive tasks that would run for minutes to hours. Investing in on-prem computers (mostly workstations, some servers), we got very solid performance, very predictable costs and no ops issues. Renting beefy machines in the cloud is very expensive and unless you get crafty (spot and/or intelligent deployment), it will be prohibitive for many. Looking at AMD's offering these days, you can get sustained on-prem perf for a few dollars. Three details of note: 1) We didn't need bursty perf (very infrequently) - had this been a need, the cloud would make a lot more sense, at least in a hybrid deployment. 2) we didn't do much networking (I'm in a different company now and we work with a lot of storage on S3 and on-prem wouldn't be feasible for us), 3) we didn't need to work remotely much, it was all at the office. Obviously, it was a very specific scenario, but given how small the company was, we couldn't afford people to manage the whole cloud deployment/security/scaling etc. and beefy workstations was a much simpler and more affordable endeavour.
• fulafel 6y ago
  Cloud is even more muddled as a term in this space than usual (Hybrid/private added to the mix).
• mrweasel 6y ago
  We sell hosting to large number of different customers who for whatever reason, mostly legal, are required to keep data within the borders of Denmark. There is no Google, Amazon, Azure or DigitalOcean data centers in Denmark, so cloud isn't an option for them. Regarding cost, well it depends. We try to help customers to move to cloud hosting if it's cheaper for them. It almost always will be if they take advantage of the features provided by the cloud providers. If you just view for instance AWS as a VMware in the cloud, then we can normally host the virtual machines for you cheaper and provide better service. You have to realize that many companies aren't developing software that's ready for cloud deployment. You can move it to EC2 instance, but that's not taking advantage of the feature set Amazon provides, and it will be expensive and support may not be what you expect. You can't just call up Amazon and demand that they fix your specific issue.
• natmaka 6y ago
  Confidentiality. Protecting sensible data (avoiding letting some hostile obtain or even modify them) seems impossible on the cloud.
• Areading314 6y ago
  In some industries, cloud is not an option. For example, certain privacy laws like HIPAA preclude uploading data to third parties, in which case, you need things to be on-prem. There are also a lot of places in the world where internet access is limited. Sometimes you need to solve problems beyond the simple "web saas in cloud" use case
• joaodlf 6y ago
  Like many said, cost. But also - legalities. Most cloud providers have very unclear rules and what exactly happen should you be in breach. For this reason, our business prefers to have most of the control.
• Xelbair 6y ago
  Cloud costs way way more compared to on-prem solution for my company. We need random access to about 50TB of files, and quite a decent number of VMs. For storage on-perm vs cloud: buy was cheaper to have after 3(!) months. For VMs(some of them could be containerized though): 1 year It was cheaper to buy second-hand decent server, slap SSDs and just install a decent hypervisor. Those costs also include: server room, power usage, admins etc. We do use cloud backups for the most important stuff. Cloud is cheaper if your business is a something that is user-based - as in you might need to scale it, hard. If you aren't doing anything like that it is absurdly expensive.
• mcv 6y ago
  I notice all the banks I'm working for are moving to the cloud. A few years ago they all had their own data centers, sometimes really big, well-designed custom data centers. But they're all moving to the cloud now. I've personally been wondering whether that's wise, because financial data and the handling of many banking processes are a bank's core business. It makes sense that a bank should be in control of that. And it needs to obey tons of strict banking data regulations. But apparently modern cloud services are able to provide all of that.
• Darkstryder 6y ago
  On top of what others have said, when outside the US the Cloud Act has been a big one for most previous companies I worked for. Using AWS / Azure / Google Cloud (even using datacenters from your own country) implies that the US government can access your data at will. As soon as you treat sensitive information, especially related to non-US governments, this becomes a blocking factor.
• samcrawford 6y ago
  Cost is the sole reason for us as well. We have ~600-700 dedicated servers around the globe, and generate a lot of egress traffic (~20PB/mo). We last ran the figures a year or so ago, and it'd cost us around 13-15x in network costs alone. A common thread of a lot of the replies to this post is network traffic costs. If one of the cloud providers can figure out a way to dramatically (and I mean at least 10x) reduce their network transfer pricing, then I think we'll see a second wave of companies adopting their services.
• hans_castorp 6y ago
  We are staying on-premise for security and privacy reasons. We can't store our data outside of the company (or even worse: outside of the EU)
• hendry 6y ago
  Bandwidth is the problem. You can't run a Youtube on any cloud provider.
• kortilla 6y ago
  It’s not secure if you have air gap requirements or issues with employees from another company technically having access to all of your data.
• DyslexicAtheist 6y ago
  this assumption doesn't consider threat models where the vendor could be part of your problem. E.g. if you're based in country A and work on super secret new Tech for an emerging industry, then hosting in country B may not be an option. Imagine a company in Europe that decides to host it's files on Alibaba Cloud in the US. Imagine the US Department of State hosting it's files with Google. Imagine an energy company working on new reactor Tech, ... Imagine a Certificate Authority which has an offline store of root certificates which need to come online to sync twice a day. Imagine cases where you need a hardware HSM. Then there is also Cost as others have pointed out. AWS cost structure is so complex that whole business models[1] have sprung up to help you optimize the price tags or reduce the risks of huge bills. that's right: you need to have a commercial agreement with another partner that has nothing to do with your cloud just to work around aggressive pricing. The guy who started this ~2 years ago has grown to 40+ people (organically), is based in Switzerland and is still hiring even in this recession. It should give you an idea of how broken the cloud is. Lastly there is also the lock-in. All the hours that you have to sit down and learn how the AWS IAM works is wasted once you decide to move to another cloud. The cost for learning how to use the 3rd party API is incurred by you not the cloud vendor. For people who think lock-in isn't much of a problem remember your whole hiring strategy will be aligned to whatever cloud vendor you're using (look at job description that already filter out based on AWS or GCP experience). Lock-in is so bad that for a business it is close to the rule of real-estate (location, location, location), only that it's to the advantage of the cloud vendor not you as the customer. [1] optimyze.cloud [2] "I have just tried to pull the official EC2 price list via JSON API, and the JSON file is 1.3gb" https://twitter.com/halvarflake/status/1258161778770542594 https://twitter.com/halvarflake/status/1258161778770542594
• sgt 6y ago
  We do a hybrid approach which I think makes sense for a lot of companies. Our mission critical stuff runs in the cloud, but anything that has to do with staging environments and development we do on-premises. It's pretty easy to host yourself, if you have a couple of decent engineers looking after it (depending on scope, of course!). Redundant power, redundant internet connections, and a few racks of Dell servers and gigabit switches. Why did I mention Dell? They just don't seem to die. We used HP for a few years but had a few bad experiences.
• sradman 6y ago
  Legacy seems to be missing from the comments. Before the advent of Cloud IaaS (Infrastructure as a Service) a very large ecosystem of On-Premise hardware and software flourished. The question needs to be considered in the context of greenfield vs brownfield systems as the trade-offs involved differ drastically.
• reverseengineer 6y ago
  Own servers cheaper than cloud. We calculated two years ago, it was up to 4x for HPC. Best cost-wise option is used servers.
• manishsharan 6y ago
  I work in a very large financial institution in small tech team. I would give an arm and a leg to not deal with soul sucking bureaucracy that is our IT department.
• skunkiferous 6y ago
  I work for a German company. They run their own DC (unfortunately, I'm not privy to precise numbers I could share, but we must be in the 1000s of hardware servers). Why? Because our (German) clients don't thrust US cloud providers.
• benbro 6y ago
  Can you recommend dedicated hosting provider that: 1. Has US, EU and Asia regions. 2. Let you rent servers per month. 3. Has decent pricing. Not premium, doesn't have to be low cost. I expect excellent egress pricing and 1/2-1/4 cost for CPU compared to the cloud. 4. Reliable network. GCP premium network is very good. How does dedicated providers and VPS providers (Linode and DO) compare? 5. Easy to use management and dashboard. Experienced really bad dashboards and hard to use Java tools to install and manage dedicated servers.
• alkonaut 6y ago
  What do you mean by "servers"? Anything that isn't a client machine, or just customer-facing infra? We have a pool of 15 build servers for our CI. They run basically 100% cpu during office hours and tranffer terabytes of data every day. They have no real requirements for backup, reliability etc, but they need to be fast. If I run a pricing calculator for hosting those in the cloud it's ridiculous. We are moving source and CI to cloud, but we'll probably keep all the build machines on-prem for the foreseeable future. For customer facing servers the calculation is completely different. More traffic means more business. Reliability, Scalability and backup is important and so on.
• PudgePacket 6y ago
  This thread has been illuminating, a lot more non-cloud people than I thought! Drinking the cloud kool-aid had me thinking cloud was the only realistic way to go.
• znpy 6y ago
  Many of the customers of my previous employer had hardware on premises, including the customer that I was handling. It had both compute and storage (netapp). It had two twin sites in two different datacenters. The infra in each site consisted basically in six compute servers (24c/48t, 128gb ram) and netapp storage (two netapp heads per site + disk shelves). Such hardware has basically paid itself across its seven or eight years of life, and having one of the sites basically in the building meant basically negligible latency. The workload was mostly fixed, and the user base was relatively small (~1000 concurrent users, using the services almost 24/7). It really checks all of the boxes, does all it is supposed to do and in a cheap manner.
• kokey 6y ago
  I think when the economy is hit hard a lot of companies are going to have to look at what they can do to make sure they remain profitable since investor appetite has changed. This means some companies will have to look at what is being spent on cloud providers. Renting RAM by the hour makes sense if you are optimistic about future revenue growth, but if the market changes and you have to worry about how you can sustain profits while just keeping your current customer base then this makes a lot less sense. The cloud vs on prem argument also really includes all the things in between, e.g. colo, managed servers, VPSes and also better tooling to manage your own VM and container clusters on these, which I think will now get increased attention and competition when people are considering alternatives to the big cloud providers in order to bring down costs.
• mikorym 6y ago
  Here is a stupid example: Excel vlookups work on a network drive, but not on a cloud service like Dropbox or OneDrive. The absolute path can't resolve if it's used across multiple Excel users. If the users store the file locally, each will have a different path on their computers. Excel stores actual paths. [1] There is one way around it: Mounting the cloud server as a network drive (some providers do this by default, but OneDrive is not one of them, neither is Dropbox). I don't know of a way of mounting OneDrive as a virtual drive; I would be interested to know. It sounds stupid, but the above was a real life scenario. [1] Only if the files are closed. Excel can change the path if you have the file open, but it can't change it to multiple option across different PCs. But as I have mentioned before, Excel doesn't seem to document all of their more subtle features.
• krageon 6y ago
  You've been managing servers for quite some time but you've never considered the security implications of hosting all of your sensitive data on someone else's computer? You say you're not trolling but I genuinely don't see how those two facts are compatible, except if you work in an industry where the data just doesn't matter. If that were the case though, you shouldn't feel compelled to judge what banking or medical institutions are doing.
  • deleted 6y ago
    [deleted]
  • aspyct 6y ago
    We use a public cloud, and believe me: we do consider the security implications. We veeery much do.
• tr33house 6y ago
  I'm a solo founder who's bootstrapped a saas that's in one state. I'd started out with the cloud then moved to a private cloud in a colocated data center. Saved more than 80% in monthly costs. Got faster speeds, better reliability and a ton of extra compute and network capacity. I just bought used servers from eBay that are retired from big corps. Nothing significant has really changed in the last five years on compute so I'll happily take their depreciated assets :) __Modern__ servers are really awesome and I totally recommend them. You can do a ton remotely.
• collyw 6y ago
  There is extra complexity with managing cloud based solutions. Logging in, setting up ssh keys. Ok it's all automatable but if I want a basic server set up for doing a simple task quickly it's often a lot easier to run it on an internal server.
• z3t4 6y ago
  There are several levels, not just on-prem vs cloud. You can for example co-locate, rent dedicated servers, rent a single VPS, or put your website up on a web hosting provider...
• deleted 6y ago
  [deleted]
• urza 6y ago
  I would like to add, that even for small teams/projects the cost is the reason. We have small business project, with only one server, few hundred customers, with varying traffic (database sync between thick clients, web portal,..). We were considering cloud, but with the features we needed (few different databases, few APIs,..) it would be cca $1000/month (with reasonable response times - could be cheaper but terribly slow). Having our own on premise server, the price is back just after few months and then just the minimal cost of connectivity, energy and occasional maintenance.. it just didn't make any sense for us to choose cloud.
• Yizahi 6y ago
  "Server" is such a broad term. In our case, aside from cost as others already mentioned, the distance and latency is very important. The servers must be located as close to client devices as possible and reasonable, and they are synced with clients using PTP to microseconds (depends on the actual distance and topology). Cloud is a no go, and we a using bare metal K8S for graceful handling of failures and updates.
• bluedino 6y ago
  A job or two ago: Everything on-site for a couple reasons (50 servers). Mainly because as a manufacturing company, machines on the shop floor need to talk to the servers. This brings up issues of security (do you really wan to put a 15 year old CNC machine 'on the internet'?). Also, if our internet connection has issues, we still need to build parts. The other big part of it is the mindset of management and the existing system, which was built to run locally, does Amazon offer cloud hosted Access and Visual Basic workers?
• leothecool 6y ago
  If your customers still want the system to run air-gapped from the internet, cloud is basically off the table.
• mrmrcoleman 6y ago
  There's a renewed interest in on-prem bare metal recently with a lot of different offerings helping to make various parts of the stack easier to manage. Awesome bare metal is a new repo created by Alex Ellis that tracks a lot of the projects: https://github.com/alexellis/awesome-baremetal https://github.com/alexellis/awesome-baremetal Also we (Packet) just open sourced Tinkerbell, our bare metal provisioning engine: https://www.packet.com/blog/open-sourcing-tinkerbell/ https://www.packet.com/blog/open-sourcing-tinkerbell/
• majkinetor 6y ago
  There is no cloud solution that lets you ship and forget a system. Come in 15 years ? It still works. Is that possible with cloud even in short periods, like 2 years ? No. Will it ever be possible? No. Thats primary reason for me. I can use cloud only for stuff that are nice but not mandatory to have for service to work, like status page. Plus, work is more enjoyable then using somebodies else stuff.
• yread 6y ago
  Yes just moved from 4 servers in AWS costing 700 usd/month to a single dedicated one that costs 40.
  • gizmodo59 6y ago
    Just out of curiosity, what downtime is in acceptable range for that single server?
    • yread 6y ago
      At night noone cares
• iso1631 6y ago
  Most of my equipment has physical interfaces, video and audio in and out Some equipment is very latency sensitive -- I'm talking microseconds, not milliseconds. More generic tasks need easy access to that specialist equipment (much of which doesn't quite grasp the concept of security) Given that we therefore have to run a secure network across hundreds of sites on multiple continents, adding a couple of machines running xen adds very little to the overhead.
• marvinblum 6y ago
  As most have mentioned already: cost We started out with Emvi [1] on Kubernetes at Google Cloud as it was the "fancy thing to use". I like Kubernetes, but we paid about 250€/month just to run some web servers and two REST APIs. Which is way too much considering that we're still working on the product and pivoting right now, so we don't have a lot of traffic. We then moved on to use a different cloud provider (Hetzner) and host Kubernetes on VMs. Our costs went down to about 50€ just because of that. And after I got tired managing Kubernetes and all the complexity that comes along with it, we now just use a docker-compose on a single (more powerful) VM, which reduced our cost even futher to about 20€/month and _increased_ performance, as we have less networking overhead. My recommendation is to start out as simple as possible. Probably just a single server, but keep scaling in mind while developing the system. We can still easily scale Emvi on different hardware and move it around as we like. We still use Google Cloud for backups (together with Hetzners own backup system). [1] https://emvi.com/ https://emvi.com/
• pachico 6y ago
  It depends on what you have to do. If your stack includes a series of microservices/monoliths connected to the typical OLTP DB then you might very well sit entirely on cloud. Things change when you need heavy lifting like having big ElasticSearch or ClickHouse clusters, or any other operation that requires heavy CPU and high RAM capacity. In that case using providers like Hetzner can cost you 1/10 of the bill compared to AWS.
• Scaevus 6y ago
  For our case, the need is very specific as we are working with mobile apps. Building iOS apps require macOS and even though there are some well-known "Mac hosting" services, none of them are actual cloud services similar DigitalOcean, Azure, AWS, etc. So it is much less expensive and actually easier to scale and configure to host the Macs onprem. (Off the record: If it is for internal use only, you can even stick in a few hackintoshes for high performance.)
• StreamBright 6y ago
  Yes, some of the leadership thinks that they can build a better cloud than MS or AWS. It is pretty hilarious to watch how spectacularly they fail. https://forrestbrazeal.com/2020/01/05/code-wise-cloud-foolish-avoiding-bad-technology-choices/ https://forrestbrazeal.com/2020/01/05/code-wise-cloud-foolis...
• nemacol 6y ago
  This conversation sounds a lot like the debate around globalization and outsourcing manufacturing to me. It might be a stretch but there is something here. There is room for both Cloud and On-Prem to exist. This endless drive by industry to push everyone to cloud infrastructure and SaaS, in my humble opinion, will look exactly like the whole supply chain coming from the east during a pandemic. The economics of it look great in a lot of use cases, but putting our whole company at the mercy of a few providers sounds terrible to me. Even more so when I see posts on HN about folks getting locked out of their accounts with little notice. It does not take much to bring our modern cloud to a grinding halt. For example, a mistake by an mostly unheard of ISP lead to a massive outage not less than a year ago(1). It was amazing to see the interconnections turn to cascading issues. 1 ISP goofs. 1-2 major providers have issues and the trickle down effect was such that even services that thought they were immune from cloud issues were realizing that they rely on a 3rd party that relies on a different 3th party that uses cloudflare or AWS. So, even though I think the cloud is (usually) secure, stable, resilient, etc... I still advocate for its use in moderation and for 2 main use cases. 1 - elastic demands. Those non-critical systems that add some value or make work easier. Things we could do without for several days and not hurt the business much. 2 - DR / Backup / redundancy. We have a robust 2 data center / DR fail over system. Adding cloud components to that seems reasonable to me. (1)https://slate.com/technology/2019/06/verizon-dqe-outage-internet-cloudflare-reddit-aws.html https://slate.com/technology/2019/06/verizon-dqe-outage-inte... Edit: Spelling and clarity Edit2: New reasons to stay on prem are happening all the time. https://www.bleepingcomputer.com/news/security/microsofts-github-account-allegedly-hacked-500gb-stolen/ https://www.bleepingcomputer.com/news/security/microsofts-gi...
• TuringNYC 6y ago
  Here is how we went about w/ CSPs (AWS, Azure, GC, Oracle). Thoughts welcome Getting Started --> Definitely go w/ CSPs. No need to worry about infra. Pre Product Market Fit + Steady Growth --> On Premise, because CSPs might be expensive until you find a consistently profitable business. Pre Product Market Fit + HyperGrowth --> CSPs since you wont be able to keep up [we never got to this stage] Product Market Fit w/ Sustainable Good Margins --> CSPs, pay to remove the headache [we never got to this stage] Side Note: w/ GPUs, CSPs rarely make sense
• cm2187 6y ago
  One good reason for your list: diversification. You don't want all the banking systems of a country all running on AWS. It's an unacceptable single point of failure risk for a country.
• gameswithgo 6y ago
  Our business has had greatly increased load due to COVID-19. It would have been very nice to buy a 128 core EPYC bare metal server to run our SQL Server on at this time, to buy us time to rearchitect to handle the load. Instead we are stuck with 96vCPUs because that is the most Amazon can do. Its also very very expensive to have a 96vcpu VM on amazon!
• maltelandwehr 6y ago
  Gigantic Elastic Search cluster - according to Elastic the largest in Europe (as of 2 years ago) - used in production. Broke again and again on AWS. We needed more shards than AWS supports. Moved to bare metal again.
• JoeAltmaier 6y ago
  My client's customers are geolocated (continental US) and their personal data is sensitive. So their server is in their own firewalled server closet.
• blaser-waffle 6y ago
  Big public clouds have a genuine purpose but there is a shit-ton of marketing and FUD being thrown around about them -- I'd bet my hat that's what this post is, given the phrasing up top. I'm not a fan. In short: Cost. CapEx and depreciation vs. OpEx. The numbers look amazing for ~3 years until the credits and discounts wear off. Then it's just high OpEx costs forever. Meanwhile I can depreciate my $10k server over time and get some cash back in taxes; plus it's paid for after a couple years -- $0 OpEx outside of licenses, and CentOS has no license cost. Once you have significant presence in someone's cloud, they're not going to just lower costs either -- they've got you now. What in American Capitalism circa 2020 makes you think they won't find a way to nickle and dime you to death? It's not going to reduce headcount, either. Instead of 14 devops/sysadmins, now I have 14 cloud admins, sitting pretty with their Azure or GCP certs. Automation is what's going to reduce those headcounts and costs, and Ansible+Jenkins+Kubernetes works fine just with VMware, Docker, and Cisco on-prem. Trust. The Google Cloud just had a 12-hour outage -- I first read about it here on HN. AWS and Azure have had plenty of outages too... usually they're just not as open as Google is about it. You also have to trust that they won't get back-doored like what happened to NordVPN's providers, and that they're not secretly MITM'ing everything or dup-ing your data. We (and some of our clients) compete with some of the cloud providers companies and their subsidiaries, and we know for a fact that they will investigate and siphon any data that could give them an advantage. Purpose. We just don't need hyper-scalable architecture. We've got a (mostly) fixed number of users in a fixed number of locations, with needs that are fairly easy to estimate / build for. Outside of a handful of sales & financial processing purposes, we will never scale up or down in any dramatic fashion. And for the one-off cases, we can either make it work with VMware, or outsource it to the software provider's SaaS cloud offering. If we were doing e-commerce -- absolutely. Some sort of android app? Sure, AWS or Azure would be great. But it's a lot of risk and cost with no benefit for the Enterprise orgs than can afford their own stuff.
• AdrianB1 6y ago
  Manufacturing plants across the world controlling production lines: no way to go in the cloud for that. We put the reporting data in the cloud, no problem there.
• gbasin 6y ago
  Cloud will continue to dominate, even if it's more expensive. Why? Because the best companies are able to focus on what makes them special, and outsource the rest. Cost and security are important, but they may not be most important. In a business, the scarcest commodity is FOCUS. By outsourcing anything that isn't core to your product, you can excel at what differentiates you.
• bpyne 6y ago
  My employer is a mid-sized university. Cost is the main issue. Our environment is a mixture of in-house developed apps and COTS. Until recently, our major COTS vendors didn't have cloud solutions. Now they have cloud solutions but they're far too costly for us to afford. So we need to keep them in-house and continue to employ the staff to support it. Our in-house apps integrate the COTS systems. Our newer apps are mostly in the cloud. But the older ones are in technologies that need to stay where the database server is, which is in our server room for the reason stated in the last paragraph. Rewriting the apps isn't on our radar due to new work coming in. Historically, outsource vs. in-source seems to ebb and flow. The clear path is usually muddied when new technologies come out to reduce cost on one side or the other.
• fpierfed 6y ago
  We do not use the cloud. We operate (24/7) facilities in remote locations where we do not have super reliable internet connection (we do have redundant links including three different fibres on distinct paths plus radio links but still). For this reason alone nothing critical can be in the cloud. In our experience, however, cloud offerings are not that cheap compared to purchasing and operating your own machines. Besides, one still needs sysadmins even when operating infrastructure in the cloud.
• leroy_masochist 6y ago
  Background: I consult extensively in the SaaS space and ask people this question all the time in the course of strategy reassessments, transactional diligence, etc. 1. Physical control over data is still a premium for many professional investors. As a hedge fund CIO told me recently when I asked her why she was so anti-cloud migration, "I want our data to live on our servers in our building behind our security guard and I want our lawyer, not AWS's, to read the subpoena if the SEC ever comes for us." 2. There are a lot of niche ERP- and CRM-adjacent platforms out there -- e.g., medical imaging software -- where the best providers are still on-prem focused, so customers in that space are waiting for the software to catch up before they switch. 3. A lot of people still fundamentally don't trust the security of the cloud. And I'd say this distrust isn't of the tinfoil hat, "I don't believe SSL really works" variety that existed a decade ago. Instead it's, "we'd have to transition to a completely different SysAdmin environment and we'd probably fuck up an integration and inadvertently cause some kind of horrendous breach".
• jll29 6y ago
  The smartest approach is to be able to run anywhere, which is increasingly practical due to VMs, Docker etc. (At least funded) startups should start with the cloud as speed to completion is key, but can later optimize for cost. Elasticity of the cloud is also great, dealing with peak demands dynamically without having to purchase hardware. I'd suggest larger companies to use at leas two cloud vendors to add resilience (when MS Teams went down, so did Slack - I was told they both use MSFT's cloud).
• aripickar 6y ago
  I work for AWS, so (in the most pedantic way possible) technically yes
• iamgopal 6y ago
  When cloud computing started, alternative software deployment was complicated, after a decade, much has been improved. So ease of management is also one of the factor. Not all need state of the art data and AI.
• sapir 6y ago
  what do you think about it? https://www.youtube.com/watch?v=G8Bt3jEhIwA https://www.youtube.com/watch?v=G8Bt3jEhIwA
• thorwasdfasdf 6y ago
  Well, for one thing: jobs. It takes a lot of IT man-power to manage an on-premise solution, especially when you run everything on-premise. Just imagine if the CTO were to switch the company to a cloud based solutions, it would save the company millions of dollars but also it would mean cutting a lot of jobs. Gov departments that use on-premise do so for security reasons and to maintain existing jobs.
• funny948 6y ago
  At least a few years ago, tons of company were afraid of the 'cloud'. It does change right now.
• shakkhar 6y ago
  Because we don't trust either amazon or google.
• whatsmyusername 6y ago
  Cloud falls over if you have a lot of data egress. I don't work on those types of workloads (mainly in PCI and a little bit of HIPAA) so I stick to cloud for the sheer convenience factor (it's easier to fix a security group than having to drive to the office and plug in somewhere like I had to earlier today). Dealing with hardware has become a very specific skill set. I have it, but I don't enjoy it, so I don't look for it. I still have to build physical networks occasionally (ex: we are building a small manufacturing facility in a very specific niche that's required to be onsite for compliance reasons) but the scale is so small that I can get away with a lot of open source components (pfsense netgates are great) and not have to use things that are obnoxious to deal with (if I never have to deploy cisco anything ever again I won't be upset).
• mister_hn 6y ago
  There's a fake believe in my company that all the data must be on premise because of privacy concerns. We've never used cloud services and we do not want to use it. Some are saying it's a matter of costs, but you know? For a dual node server (hot standby) we were asked 120K € + 50K € only for configuration fees.
• prirun 6y ago
  I did a startup with a co-founder in 1998, before cloud was a thing. We hosted at above.net first, then he.net following above.net's bankruptcy. Both were very good and we never had colo-related problems with either, though he.net was significantly cheaper. We started with 2 white-box PC's as servers, 2 mirrored RAID1 drives in each. We added a 3rd PC we built ourselves: total nightmare. The motherboard had a bug where, when using both IDE channels, it overwrote the wrong drive. We nearly lost our entire business. Putting both drives on the same IDE channel fixed it, but that's dangerous for RAID1. A few years in, we needed an upgrade and bought 5 identical SuperMicro 2U servers with hardware RAID1 for around $10K. Those things were beasts: rock solid, fast, and gave us plenty of capacity. We split our services across machines with DNS and the 5 machines were in a local LAN to talk to each other for access to the database server. The machines' serial ports were wired together in a daisy-chain so we could get direct console access to any machine that failed, and we had watchdog cards installed on each so that if one ever locked up, it automagically rebooted itself. When I left in 2005, we were handling 100's of request/s, every page dynamically generated from a search engine or database. Of course it took effort to set all this up. But the nice thing is, you control and understand _everything_. Some big company doesn't just do things to you, you have no idea what is happening, and they're not talking. And if things do go south, you can very quickly figure out why, because you built it. The biggest mistake we made was in the first few years, where we used those crappy white-box PCs. Sure, we saved a couple thousand dollars, but we had the money and it was a terrible trade. Night and day difference between those and real servers.
• dryst 6y ago
  If your product doesn't run connected to the internet, it is difficult to make the case for cloud development. You need developers who understand hardware, and abstraction layer cloud provides is a handicap instead of an enabler.
• ai_ja_nai 6y ago
  It's the people cost, not the hardware. Hardware is super cheap: -A 40 slots rack, with gigabit fiber, dual power and a handful of public IP addresses, costs on average 10000€/y. -A reconditioned server on eBay with 16 cores and 96GB of RAM costs 500€ (never seen them break in 3 years). -A brand new Dell Poweredge with AMD EPYC 32 core and 64GB of RAM will cost 3000€. -Storage is super cheap: 500GB of SSD costs 80€ (consumer stuff is super fine as long as you wisely plan between redundancy and careful load) and rotational disks are even cheaper. Never seen a rotational disk break. Once bought, all of this is yours forever, not for a single month. You can pack very remarkable densities in a rack and have MUCH more infrastructure estate at disposal than you would ever afford on AWS. The flip side of the coin is that you need operation expertise. If it's always you, then ok (although you won't be doing much more than babysitting the datacenter). Otherwise, if you need to hire a dedicated person, people is the most expensive resource yet and that should definitely be added to the cost of operations.
• blodkorv 6y ago
  Am the CTO of a small company making payroll software. We don't have on premise servers but we currently are moving away from azure to renting VPSs from a local provider. The cost benefits are huge and since our app is mostly a normal web app we dont need that many fancy cloud things. And i dont see us needing it in the future. I really dont understand why a company doing similar things would wanna go the could route. Its so damn expensive and its not always easy to use and setup.
• Abukamel 6y ago
  Yes online.net and the like companies save costs
• rjgonza 6y ago
  I work for a US Stock Exchange, and some of the technologies that we rely on are not permitted in the cloud. The performance metrics we need are usually only achievable on highly tuned bare metal deployments as well so cloud is usually not an option. I guess it really depends on your workload, but I think there is a very healthy amount of production being deployed and worked on businesses own datacenter/private clouds.