16 ms·

Move Back to Datacenters

78 points by rakkhi 3y ago

randomdigitz 3y ago
Try telling teams they’ll have to go back to old procurement timelines and procedures.
politelemon 3y ago
The conclusion completely handwaves the massive overheads that come with not only owning your own infrastructure, but also having to manage a complex platform stack and its security. The security points, which the main points hinges on, seem hyperfocused and in most cases misguided. Redshift is not internet facing. If your Redshift is internet facing, you've messed up somewhere. The CPU attack example given is for AMD ZEN, which the most common AWS workload CPUs aren't. Further, the benefit of using a cloud provider is that they put mitigations in place for most exploits, whereas running your own stack means it's now on you; running your own stack does not excuse you from having to put mitigations in place. In the examples again, the speed to market's problems are a reflection of your organisation, not the cloud. Stepping back a little, I'm thinking (as terrible as it is) that this is a case of blaming the tools, but never yourself; a lot of the problems the author is seeing seem very specific to their observations but are a poor use and poor understanding of AWS in general. Overall not a great article, with a headline designed for people who already dislike AWS/GCP/Azure.
- deleted 3y ago
  [deleted]
- pas 3y ago
  not to mention that the obvious/incremental move is not to completely exclusive DC, but to simply cheaper providers with managed VMs, then dedicated (but still managed) HW, and there's a long tail of that before breaking ground for a new DC makes any sense. somewhere along there's Oxide's "cloud in a rack" thing.
- Volume999 3y ago
  I agree on the security part, and what about the cost argument? I think I've seen a few companies switching over to data centers due to the costs they can save money on moving from the clouds, and I also think mature orgs prefer cap ex and depreciation to op ex (although I am not far from clueless on this point)?
- tmpX7dMeXU 3y ago
  Genuinely feels like there’s a constant trickle of arguments from people looking to do all they can to shift the tides such that their skill set is more in demand. Or maybe it’s just an ideological obsession. I don’t know. I feel the same way whenever the “use C for everything” people catch a whiff of performance mattering and very strongly suggest that the only way to dig ourselves out of this mess is to move back to the One True Language.
  - stavros 3y ago
    In my experience The Cloud™ requires a much larger skillset, ensuring that your ops department will never stop growing.
    - acqq 3y ago
      I have an impression that is "avoided" in many companies by demanding from the newly hired programmers to have "cloud experience". My problem is that that usually ends up being "programmers have taken courses about the way cloud vendor Y does the job" which then actually means "the programmers had drunk the propaganda of the cloud vendor Y" which is not a directly reusable knowledge for anything than "here's how we adjust our work to be charged by the cloud vendor Y". Which I think is guaranteed to have a company which can't even imagine doing it any other way.
    - duggan 3y ago
      It also depend on what requirements your business has and who you have available. The skillets are different and they overlap in places, but one way or the other if your business is growing your operations requirements will grow. I am hugely in favour of AWS for web startups I am involved with because 1) I know AWS very well, 2) they provide hundreds of thousands of euros/dollars of credit to startups (I think all the big cloud vendors do at this stage), 3) it's useful to have architectural flexibility, and 4) they can take on a bunch of the security and availability burden that can otherwise be very labour intensive in a small company. Also you can pry S3 and RDS Aurora out of my cold dead hands!
  - deleted 3y ago
    [deleted]
- antonvs 3y ago
  In general when people mention "cost" as an issue for cloud, they're narrowly comparing what a CPU core or GB of RAM or disk costs on paper between cloud and datacenter/onprem. This completely misses the business case for cloud, and is why these sorts of analyses don't convince that many business folk, and why cloud use continues to grow fast. The business case is that someone else is doing the infrastructure management for you, at commodity prices, but you still have complete, direct control over "your" resources and ability to easily automate everything. This tends to save much more money than any % of your CPU, memory, and disk budget could possibly achieve. It's sort of like saying, instead of taking cars from your corporate fleet to a repair shop when they break, repair them yourself! Just hire some mechanics, build or rent a garage and equipment, and voila! And also, why are you buying electricity from the electric company anyway? Just build a generator next to your office building! Maybe make it nuclear just like some of the big electric companies! For many companies, the "just host it in a datacenter" argument is no less silly than these scenarios.
  - deleted 3y ago
    [deleted]
  - rhaway84773 3y ago
    The real business case for the cloud is it replaces capex with opex. Which is great if you have limited capital (bootstrapping, new startup, etc), or are the CEO of a large company that wants to show better cash flow to earn a larger bonus. In reality, the idea that the cloud somehow means you can outsource your infrastructure team rarely works out in practice. At best you’ll get much more expensive developers having to spend time on infrastructure which might look like a couple of fewer salaries to pay, but you’re likely now paying for it through dev time anyways.
    - datavirtue 3y ago
      The REAL business case is cargo culting.
  - ericyan 3y ago
    > The business case is that someone else is doing the infrastructure management for you. Many business people believe that moving to cloud can reduce the headcount needed for managing the infra, but that is usually not what's happening. You will still need more or less the same amount of people to patch the OS and configure the networking, but with a slightly different skill set -- instead of Cisco IOS commands, they now need to deal with AWS transit gateway. Every data center I worked with offers on-site remote hands that will rack new servers or replace hard drives or PSUs for you. There are also third-party companies that offers this service. Redundant power and air conditioning are the responsibility of the colocation provider and those are covered by the SLA. Those data centres do have generators on-site. That said, configuring you top-of-rack switch is usually not covered by the co-location contract. This need to be done by a network engineer, usually the same person that would manage VPCs.
- di4na 3y ago
  The other thing they keep missing. Where are the people with the skills to do this work? I wrote a whole post about it, but it is not like people with that knowledge exist in troves just waiting to be hired. There are no path to learn this stuff either. Do we really expect every single company out there to build teams of sysadmin and operators from a non existing pool of talents? It is easy to say that people can move to onprem, but orgs can barely hire the talents they need already. Sure let put even more pressure on a limited supply.
  - deleted 3y ago
    [deleted]
  - otabdeveloper4 3y ago
    AWS and Azure is a much more niche skill set. E.g., there's probably three orders of magnitude more engineers familiar with Postgres out there in the world than with DynamoDB.
    - di4na 3y ago
      To the point of running it smoothly at scale in prod with backup? Nope. Hence you learn the RDS api and use it.
- PartiallyTyped 3y ago
  It handwaves all the things EC2 and all services built on top do. The amount of engineering needed just to migrate droplets live is enormous - it’s an art (according to a friend in hypervisors), and that ignores everything else regarding hypervisors and security. Scalability is also a thing you get “for free” with serverless computing.
- rakkhi 3y ago
  Thanks all for your engagement and comments. I have updated the article to address many of these points.
- jsiepkes 3y ago
  > The conclusion completely handwaves the massive overheads that come with not only owning your own infrastructure Even if you feel that is a serious obstacle (which I personally don't) you can buy a managed solution (such as an Oxide rack) and use that on prem.
johncolanduoni 3y ago
Naturally the first vulnerability they point out was never exploitable for cross-VM attacks on any major cloud provider at any point…
- qaq 3y ago
  Thats a strong statement we only know about breaches that have a) being identified b) publicly disclosed
  - johncolanduoni 3y ago
    It requires concurrent execution on the same core of multiple VMs, which none of the major cloud providers do (at least on machines new enough to have Zen cores).
    - qaq 3y ago
      Sure they do for "serverless workloads"
      - johncolanduoni 3y ago
        I can’t find specific documentation that indicates that AWS deploys firecracker with core isolation (though I would be pretty surprised if they didn’t given their documented caution around shared-core VMs). Google uses gVisor with Linux Core Scheduling to prevent this for GKE sandbox, but I can’t find any documentation that they do the same for Cloud Functions. In any case ”don’t use serverless functions” is a very far goalpost from “move back to the data center”.
        qaq 3y ago
        That is not the only possible vulnerability.
neilv 3y ago
> I used to “sell” computer leases about 20 years ago saying hey don’t buy a computer, rent it and upgrade it in a year. Turns out the fine print was terrible. This prompts the question of "So, what are you selling this year?" I'm sympathetic to on-prem and datacenters, but maybe all the reaction GIFs are distracting the CIO/CTO from the new fine print?
orkaa 3y ago
The analogy of this article would be: Stop using airbnbs when traveling, just build a house.
- brabel 3y ago
  I think the correct analogy would be more like "stop living in Airbnb, just build a house and in the rare occasion you must travel, then use Airbnb".
  - maratc 3y ago
    Or "stop renting your main office from WeWork, just get an office, and when you need short-term expansions, then use WeWork".
- dist-epoch 3y ago
  No, the analogy would be: stop staying in the same airbnb for a year, just build a house.
  - albert_e 3y ago
    Or: You have a lot of business travel wherein your employees currently stay in hotels whenever they visit major cities? Don't use hotels, they are expensive! Build and operate your own company owned full-service apartments in those cities.
    - rhaway84773 3y ago
      The traveling analogies make no sense. For most companies their software is their business. Take away the software and the company stops functioning. Their employees aren’t just visiting their software every few weeks or so. They’re using it every day from morning to evening. There is no travel involved. Your software is your home. So really a better analogy would be renting vs buying your home. The rental comes with the furniture and for several things that might go wrong the landlord will come and fix it, whereas when you own the house, you have to buy the furniture and arrange it, and if anything goes wrong you are responsible for fixing it. Of course, the analogy breaks down in many ways because when you’re on Prem you are buying established software with a support license so if something does fail the support license should provide as much, if not more service than you would get on AWS. The entire analogy is completely pointless because it adds a layer of indirection for no reason other than for some reason tech people absolutely love making analogies to physical objects.
chrisandchris 3y ago
Is there some plugin that removes GIFs from blog posts? IMHO an article looses a lot of credibility if I also have to look at some unrelated GIF in an otherwise very well written article.
- RockRobotRock 3y ago
  Ublock origin rule: .gif^$domain=*.substack.com,image .gif^$domain=*.medium.com,image this would probably work, right?
raffraffraff 3y ago
Best of luck running your own data center properly. It's a massive pain in the hoop.
- atoav 3y ago
  Most university IT departments I know manage just fine to do this. Allthough I get the idea of paying others to run your servers (I do so myself), I am somewhat puzzled that people so readily want to move on from the option of selfhosting. Knowing how to do things yourself has its own value. This reminds me a bit of developers who can't write anything useful without using a framework. There is life outside of frameworks and life outside of the cloud. These are the technological foundations of being able to do anything we are doing. Granted, it can be useful to outsource these parts, but that doesn't come without it's very own short and long term costs, it strengthens some market asymmetries that you maybe don't wanna strengthen, it let's your organization forget to do things that are the basis of your business and so on. I don't say everybody has to run their own infra, just that the work of doing so can pay off in different ways — and whether these trade offs are worth it depends entirely on the nature of your endevour.
  - raffraffraff 3y ago
    It depends on what level of "run your own infra" were talking about here. Like, purchasing hardware, racking and connecting, coming up with a nice way to PXE build your fleet and a way to manage the OS and config (chef/ansible/puppet/nix?), before implementing a complete monitoring stack that works on network equipment, raid controllers etc, and then bootstrapping your own Kubernetes cluster or hashistack (or pretending that you don't need complexity, but over time cobbling together your own unique undocumented mess). It's a huge amount of work that startups usually can't afford. The time and effort to do it properly is monstrous. You need to be a company of a certain size to devote the required engineering and support staff to it.
hdjjhhvvhga 3y ago
> Serverless functions – still running on someone’s server… but if you have a function that you need to evoke infrequently and cold start times don’t matter that much, yeah good for cloud. While I agree in principle, these functions don't exist in thin air. You will often want to store the results somewhere. And then protect them. So with time you are almost replicating a lot of infrastructure. And if you are a big org you will probably want to enable SecurityHub and other security/governance services like AWS Config and things get expensive again.
jbverschoor 3y ago
So many negative comments. Makes you wonder who works for aws…
- throwaway85858 3y ago
  That's why they call it Big Data
- andrewstuart 3y ago
  Everyone here talks up the cloud. They’ve all drunk the kool aid and are cloud true believers.
  - rhaway84773 3y ago
    Also, a lot of people here work for startups or have their own startups. The cloud makes a lot of sense for startups for a variety of reasons. Low capex, easier to get started, you’re willing to pay a premium to not have to deal with certain issues, at least in the beginning, and your costs are likely not very high anyways because your usage is relatively low.
- duggan 3y ago
  Reasonable professionals can disagree – strongly – without being part of a conspiracy.
RedShift1 3y ago
Move stuff back on-prem. Great. But aside from all the usual practical matters (power, backups, spare parts, service contracts, etc...), software is rapidly becoming a roadblock to move back to on-prem. If everything you have is running on Kubernetes or other open source software, great. If not, then there's an increasing amount of roadblocks being put up: * Some software is only available as SaaS anymore * Jacking up prices to ridiculous levels for on-prem licenses (to favor their SaaS offering of course) * Intentionally knee capping on-prem software feature wise * Stifle development of the on-prem product * Force you to use some parts of their cloud services to make other things work * Dark patterns and endless nags in software to push people to use some part of their cloud services * Poor documentation how to install/use/maintain the software on-prem or make it needlessly complex * Slower response from the software vendor in case of security issues * Exporting data from the cloud and importing it on-prem is impossible And I can make this list go on and on and on... My point being, for small time firms that don't have the resources and solely rely on commercial software, moving back might not even be an option anymore.
- antonvs 3y ago
  > Intentionally knee capping on-prem software feature wise Or unavoidably. We have a SaaS product that's also available onprem - because we have some big enterprise customers who won't accept anything else - but there's a bunch of stuff we take advantage of in the cloud that we can't easily replicate in arbitrary onprem environments. So the onprem software ends up "kneecapped" just out of sheer pragmatism. There's not a single missing feature onprem that isn't because it would be too much effort to implement and support it. > Stifle development of the on-prem product Similar issue here. We have finite engineering resources, and onprem is a fraction of revenue. > Force you to use some parts of their cloud services to make other things work See first point. > Poor documentation how to install/use/maintain the software on-prem or make it needlessly complex See second point. In short, many of the things you're saying are indeed disadvantages of onprem products today, but it's often not the result of some sort of "intentional kneecapping" or "dark patterns". In my direct experience, it has never been the result of that.
  - lifeisstillgood 3y ago
pachico 3y ago
I couldn't read it all because the number of GIFs started to make me nervous :(
throwaway85858 3y ago
In my experience small firms will struggle to hire excellent system engineers that can manage a bare metal setup, it's easier however to hire okay-ish SWE's that can design applications appropriate for cloud.
throwaway2990 3y ago
[flagged]
Havoc 3y ago
I'd say the attack surface on rolling your own tech in a datacenter is substantially higher. Yes DynamoDB has an API, but I'd wager a AWS engineer with good security skills has looked at it carefully. Do you have an equally skilled security expert on hand to look at the datacenters stack and then same for whatever you're deploying on it? Not all internet exposure is equal. Moving out of cloud often makes sense, but security isn't the right motivation for it.
- rhaway84773 3y ago
  > Yes DynamoDB has an API, but I'd wager a AWS engineer with good security skills has looked at it carefully. Do you have an equally skilled security expert on hand to look at the datacenters stack and then same for whatever you're deploying on it? Is that AWS security engineer looking specifically at how your systems interact with DynamoDB? And how you have specifically configured your DynamoDB instances? And someone rolling their own infrastructure isn’t building out their own custom software to do it. They’re using software from other companies which also have excellent security engineers that have looked into how you can safely deploy and use their software. Now, it may be true that in recent times with the amount of money made in the cloud a lot of these vendors are using carrots and sticks to push you to their cloud offerings (often by making their on Prem offerings worse), but let’s not pretend that on-prem software is two cups tied together with a string or that moving to the cloud somehow alleviates the need to have in house expertise to ensure your cloud software is running correctly and securely.
  - Havoc 3y ago
    Again, not all configuration is equal. In hosted services a lot of the decisions have been made for you, that reduces customizability and flexibility but also footguns. >They’re using software from other companies which also have excellent security engineers that have looked into how you can safely deploy and use their software. Of course. Even with well designed software the risk is much higher for the end user though. Deploying a cloud load balancer is pretty idiot proof. If I do the same via local nginx...configuration files, local firewall, file permission, maybe my distro repackaged it, did I set up the user right, am I updating it regularly, what about my underlying operating system security etc. And on top of that it's a once off configuration...it isn't battle tested by thousands running the same load balancer. It's not that it can't be done, but you do pay the price in complexity and engineering hours. >And how you have specifically configured your DynamoDB instances? Well yeah - I'd expect them to have thought through the possible permutations in reasonable detail, which is feasible given the limited customizability. If you're self deploying the possible combinations are endless. It's a bit like testing on android vs iphone. You can reasonably test against the 30 or whatever common iphones, but you're never testing against the 1000s of funky android hardware combinations out there.
fnordpiglet 3y ago
With IAM you can restrict all dynamodb endpoint access to a VPCE/private link. But the insecurity of the public facing endpoints is vastly overstated even without using VPCE. The rest is a bunch of FUD - I spent years going through these points with some of the worlds best security teams to secure some of the most systemically important workloads. These arguments are fairly tired. I’ll tackle another one - speculative attacks. First, you certainly can get bare metal exclusive access to hosts. But instances move around the broader infrastructure of an AZ, even if you’re using something like placement groups which only assure a local affinity. The chance a bad actor can colocate in the same physical device as your workload and successfully attack through side channels is vanishingly low in larger regions. To target anyone specific you would need to do such an enormous fishing expedition that it’s impractical. Further cloud providers aren’t insensate to such attacks and accounts that are doing that sort of topological mapping are easily detected. A better solution is to simply cycle your instances periodically to migrate your workloads around. For very sensitive workloads where the extraordinary unlikelihood isn’t sufficient, just get a bare metal instance. I don’t dissuade anyone from running data centers. But I’ve yet to find anyone running back.
- trallnag 3y ago
  > worlds best security teams > the most systemically important workloads Big claims require big evidence. How did you end up with these superlatives?
  - fnordpiglet 3y ago
    A long career working hard and building a reputation for delivering hard stuff well and building the right network that I used to good effect.
mvkel 3y ago
No. We rolled our own stack for 10 years, until 2018, well after AWS and Azure were around. We switched to Azure in 2018 and never looked back. Sure you trade security (do you really, though?) in exchange for: - not needing to head to the DC because a power supply failed and a rando who was in the cage never plugged in the redundant one - not having to be way over-capacity in scalability, or suddenly under-capacity and emergency ordering some more 1Us - not sifting through eBay to buy a spare hard drive that one of your boxes from 2011 needs but dell no longer makes and the list goes on ad infinitum. This problem has been solved. It's time to move on. Any time a company spends tinkering with their stack is less time spent delivering something of value.
- NorwegianDude 3y ago
  Then there are options that are probably better for you. I've been doing colocation for many years, but I've never seen the inside of the datacenter. Remote hands set up the hardware for me and replace failed hardware. The service is so rarely needed that it's basically free. In recent years I've started to rent the hardware I want as I want more frequent changes. I tell my host what hardware I want and they'll acquire it and install it. If you're running very old hardware that you can't even buy replacement drives for then you are doing something very wrong. Need to scale fast? Then I don't have to wait for servers to arrive, I use their cloud offering that's in the same network. I agree that it's a solved problem, but the solution to anything somewhat permanent is not the cloud. It's not really any harder to run your own hardware.
  - mvkel 3y ago
    All of those things still require management, which is the issue at hand here. Also if you're renting your hardware... for a subscription ... that sounds an awful lot like the cloud with extra steps.
jgb1984 3y ago
Why is the choice always portrayed as if it's either cloud, or on-prem / colocation ? Those are two extremes. At work I use dedicated physical machines from Hetzner. If I need an extra one they deliver it in a few minutes, and thanks to Ansible it's provisioned within a few minutes more. Hetzner keeps an eye on the hardware and replaces disks, PSU's and the like if needed. I wouldn't often advocate to do colocation or on-prem since that indeed comes with a whole set of headaches, but renting dedicated physical servers like we do offers a lot of flexibility, with very little overhead, at a price/performance ratio that makes AWS look like pure extortion.
- _xnmw 3y ago
  Yep, that's the compromise I came to as well at my solo bootstrapped SaaS. However, there are still some benefits of colocation that I would love to have: 1. ability to use M1 Mac minis as servers for specialized workloads, while having an identical developer setup for testing. This would really enable me to finetune, test, and squeeze out performance to the max and have the dream of a "single box" deployment serving hundreds of thousands of customers.
- candiddevmike 3y ago
  I think your example falls under the cloud use case, it's a form of IaaS. The reason it doesn't scale is at some point it becomes cheaper to own the hardware or you need better scaling/redundancy and switch to managed cloud VMs/containers/functions
- riku_iki 3y ago
  > at a price/performance ratio that makes AWS look like pure extortion. I think if you try to host in US to have lower latency to many customers, and with some less cheap and more reliable server grade network/hardware, it is likely cost will be higher than for hetzner.
RantyDave 3y ago
Oh, "just" run Kubernetes. Gotcha.
qaq 3y ago
Important point they miss is that unless your monthly spend with AWS is couple of mil a month you are nothing to them.
jauntywundrkind 3y ago
I thought for a second I was hearing a.suggestion that by running your own hardware you can turn off all mitigations. Which should save you a ton of money; for the same core you may be getting more than double the performance. And since you're not running any unknown foreign workloads, no one would have access to be running timing attacks on you. The article pretty quickly moved off that point. And it seems to at points be saying that making it hard to setup infrastructure is a feature, will help the bottom line. Which has some truth but at enormous emotional cost to your teams. I do think this shift needs to happen and I appreciate such lengthy sets of concerns being brought out. But I'm pretty lukewarm on this analysis.
Matthias247 3y ago
> in traditional datacenters, with Infrastructure and Support teams separate from Development (anti DevOps), there are/were strong human checks and balances. If your Devs wanted to make an API Internet accessible and connect it to what they thought was a “sanitized” database, they probably had to raise a change, submit some firewall rules, maybe talk to a DBA to get credentials. This is just romanticizing things. In every deployment - whether in it's "in cloud" or anywhere else - there is always the quick on off change that someone makes. Probabably for a valid reason, e.g. solving a production problem quickly. Chances are high that will go unnoticed, until another problem manifests (and that is hopefully not a security issue!). I would argue that in cloud setups the chances for that to happen are actually slightly lower, because teams are incentivized to use immutable and declarative infrastructure. And that there might be an audit log in place which tells what changes have been made - although that still requires people to look at it, which again happens only in the case problems show up.
lijok 3y ago
[flagged]
deleted 3y ago
[deleted]

78 points by rakkhi 3y ago

Try telling teams they’ll have to go back to old procurement timelines and procedures.

The conclusion completely handwaves the massive overheads that come with not only owning your own infrastructure, but also having to manage a complex platform stack and its security. The security points, which the main points hinges on, seem hyperfocused and in most cases misguided. Redshift is not internet facing. If your Redshift is internet facing, you've messed up somewhere. The CPU attack example given is for AMD ZEN, which the most common AWS workload CPUs aren't. Further, the benefit of using a cloud provider is that they put mitigations in place for most exploits, whereas running your own stack means it's now on you; running your own stack does not excuse you from having to put mitigations in place. In the examples again, the speed to market's problems are a reflection of your organisation, not the cloud. Stepping back a little, I'm thinking (as terrible as it is) that this is a case of blaming the tools, but never yourself; a lot of the problems the author is seeing seem very specific to their observations but are a poor use and poor understanding of AWS in general. Overall not a great article, with a headline designed for people who already dislike AWS/GCP/Azure.

deleted 3y ago
[deleted]
pas 3y ago
not to mention that the obvious/incremental move is not to completely exclusive DC, but to simply cheaper providers with managed VMs, then dedicated (but still managed) HW, and there's a long tail of that before breaking ground for a new DC makes any sense. somewhere along there's Oxide's "cloud in a rack" thing.
Volume999 3y ago
I agree on the security part, and what about the cost argument? I think I've seen a few companies switching over to data centers due to the costs they can save money on moving from the clouds, and I also think mature orgs prefer cap ex and depreciation to op ex (although I am not far from clueless on this point)?
tmpX7dMeXU 3y ago
Genuinely feels like there’s a constant trickle of arguments from people looking to do all they can to shift the tides such that their skill set is more in demand. Or maybe it’s just an ideological obsession. I don’t know. I feel the same way whenever the “use C for everything” people catch a whiff of performance mattering and very strongly suggest that the only way to dig ourselves out of this mess is to move back to the One True Language.
- stavros 3y ago
  In my experience The Cloud™ requires a much larger skillset, ensuring that your ops department will never stop growing.
  - acqq 3y ago
    I have an impression that is "avoided" in many companies by demanding from the newly hired programmers to have "cloud experience". My problem is that that usually ends up being "programmers have taken courses about the way cloud vendor Y does the job" which then actually means "the programmers had drunk the propaganda of the cloud vendor Y" which is not a directly reusable knowledge for anything than "here's how we adjust our work to be charged by the cloud vendor Y". Which I think is guaranteed to have a company which can't even imagine doing it any other way.
  - duggan 3y ago
    It also depend on what requirements your business has and who you have available. The skillets are different and they overlap in places, but one way or the other if your business is growing your operations requirements will grow. I am hugely in favour of AWS for web startups I am involved with because 1) I know AWS very well, 2) they provide hundreds of thousands of euros/dollars of credit to startups (I think all the big cloud vendors do at this stage), 3) it's useful to have architectural flexibility, and 4) they can take on a bunch of the security and availability burden that can otherwise be very labour intensive in a small company. Also you can pry S3 and RDS Aurora out of my cold dead hands!
- deleted 3y ago
  [deleted]
antonvs 3y ago
In general when people mention "cost" as an issue for cloud, they're narrowly comparing what a CPU core or GB of RAM or disk costs on paper between cloud and datacenter/onprem. This completely misses the business case for cloud, and is why these sorts of analyses don't convince that many business folk, and why cloud use continues to grow fast. The business case is that someone else is doing the infrastructure management for you, at commodity prices, but you still have complete, direct control over "your" resources and ability to easily automate everything. This tends to save much more money than any % of your CPU, memory, and disk budget could possibly achieve. It's sort of like saying, instead of taking cars from your corporate fleet to a repair shop when they break, repair them yourself! Just hire some mechanics, build or rent a garage and equipment, and voila! And also, why are you buying electricity from the electric company anyway? Just build a generator next to your office building! Maybe make it nuclear just like some of the big electric companies! For many companies, the "just host it in a datacenter" argument is no less silly than these scenarios.
- deleted 3y ago
  [deleted]
- rhaway84773 3y ago
  The real business case for the cloud is it replaces capex with opex. Which is great if you have limited capital (bootstrapping, new startup, etc), or are the CEO of a large company that wants to show better cash flow to earn a larger bonus. In reality, the idea that the cloud somehow means you can outsource your infrastructure team rarely works out in practice. At best you’ll get much more expensive developers having to spend time on infrastructure which might look like a couple of fewer salaries to pay, but you’re likely now paying for it through dev time anyways.
  - datavirtue 3y ago
    The REAL business case is cargo culting.
- ericyan 3y ago
  > The business case is that someone else is doing the infrastructure management for you. Many business people believe that moving to cloud can reduce the headcount needed for managing the infra, but that is usually not what's happening. You will still need more or less the same amount of people to patch the OS and configure the networking, but with a slightly different skill set -- instead of Cisco IOS commands, they now need to deal with AWS transit gateway. Every data center I worked with offers on-site remote hands that will rack new servers or replace hard drives or PSUs for you. There are also third-party companies that offers this service. Redundant power and air conditioning are the responsibility of the colocation provider and those are covered by the SLA. Those data centres do have generators on-site. That said, configuring you top-of-rack switch is usually not covered by the co-location contract. This need to be done by a network engineer, usually the same person that would manage VPCs.
di4na 3y ago
The other thing they keep missing. Where are the people with the skills to do this work? I wrote a whole post about it, but it is not like people with that knowledge exist in troves just waiting to be hired. There are no path to learn this stuff either. Do we really expect every single company out there to build teams of sysadmin and operators from a non existing pool of talents? It is easy to say that people can move to onprem, but orgs can barely hire the talents they need already. Sure let put even more pressure on a limited supply.
- deleted 3y ago
  [deleted]
- otabdeveloper4 3y ago
  AWS and Azure is a much more niche skill set. E.g., there's probably three orders of magnitude more engineers familiar with Postgres out there in the world than with DynamoDB.
  - di4na 3y ago
    To the point of running it smoothly at scale in prod with backup? Nope. Hence you learn the RDS api and use it.
PartiallyTyped 3y ago
It handwaves all the things EC2 and all services built on top do. The amount of engineering needed just to migrate droplets live is enormous - it’s an art (according to a friend in hypervisors), and that ignores everything else regarding hypervisors and security. Scalability is also a thing you get “for free” with serverless computing.
rakkhi 3y ago
Thanks all for your engagement and comments. I have updated the article to address many of these points.
jsiepkes 3y ago
> The conclusion completely handwaves the massive overheads that come with not only owning your own infrastructure Even if you feel that is a serious obstacle (which I personally don't) you can buy a managed solution (such as an Oxide rack) and use that on prem.

johncolanduoni 3y ago

Naturally the first vulnerability they point out was never exploitable for cross-VM attacks on any major cloud provider at any point…

qaq 3y ago
Thats a strong statement we only know about breaches that have a) being identified b) publicly disclosed
- johncolanduoni 3y ago
  It requires concurrent execution on the same core of multiple VMs, which none of the major cloud providers do (at least on machines new enough to have Zen cores).
  - qaq 3y ago
    Sure they do for "serverless workloads"
    - johncolanduoni 3y ago
      I can’t find specific documentation that indicates that AWS deploys firecracker with core isolation (though I would be pretty surprised if they didn’t given their documented caution around shared-core VMs). Google uses gVisor with Linux Core Scheduling to prevent this for GKE sandbox, but I can’t find any documentation that they do the same for Cloud Functions. In any case ”don’t use serverless functions” is a very far goalpost from “move back to the data center”.
      - qaq 3y ago
        That is not the only possible vulnerability.

neilv 3y ago

> I used to “sell” computer leases about 20 years ago saying hey don’t buy a computer, rent it and upgrade it in a year. Turns out the fine print was terrible. This prompts the question of "So, what are you selling this year?" I'm sympathetic to on-prem and datacenters, but maybe all the reaction GIFs are distracting the CIO/CTO from the new fine print?

orkaa 3y ago

The analogy of this article would be: Stop using airbnbs when traveling, just build a house.

brabel 3y ago
I think the correct analogy would be more like "stop living in Airbnb, just build a house and in the rare occasion you must travel, then use Airbnb".
- maratc 3y ago
  Or "stop renting your main office from WeWork, just get an office, and when you need short-term expansions, then use WeWork".
dist-epoch 3y ago
No, the analogy would be: stop staying in the same airbnb for a year, just build a house.
- albert_e 3y ago
  Or: You have a lot of business travel wherein your employees currently stay in hotels whenever they visit major cities? Don't use hotels, they are expensive! Build and operate your own company owned full-service apartments in those cities.
  - rhaway84773 3y ago
    The traveling analogies make no sense. For most companies their software is their business. Take away the software and the company stops functioning. Their employees aren’t just visiting their software every few weeks or so. They’re using it every day from morning to evening. There is no travel involved. Your software is your home. So really a better analogy would be renting vs buying your home. The rental comes with the furniture and for several things that might go wrong the landlord will come and fix it, whereas when you own the house, you have to buy the furniture and arrange it, and if anything goes wrong you are responsible for fixing it. Of course, the analogy breaks down in many ways because when you’re on Prem you are buying established software with a support license so if something does fail the support license should provide as much, if not more service than you would get on AWS. The entire analogy is completely pointless because it adds a layer of indirection for no reason other than for some reason tech people absolutely love making analogies to physical objects.

chrisandchris 3y ago

Is there some plugin that removes GIFs from blog posts? IMHO an article looses a lot of credibility if I also have to look at some unrelated GIF in an otherwise very well written article.

RockRobotRock 3y ago
Ublock origin rule: .gif^$domain=*.substack.com,image .gif^$domain=*.medium.com,image this would probably work, right?

raffraffraff 3y ago

Best of luck running your own data center properly. It's a massive pain in the hoop.

atoav 3y ago
Most university IT departments I know manage just fine to do this. Allthough I get the idea of paying others to run your servers (I do so myself), I am somewhat puzzled that people so readily want to move on from the option of selfhosting. Knowing how to do things yourself has its own value. This reminds me a bit of developers who can't write anything useful without using a framework. There is life outside of frameworks and life outside of the cloud. These are the technological foundations of being able to do anything we are doing. Granted, it can be useful to outsource these parts, but that doesn't come without it's very own short and long term costs, it strengthens some market asymmetries that you maybe don't wanna strengthen, it let's your organization forget to do things that are the basis of your business and so on. I don't say everybody has to run their own infra, just that the work of doing so can pay off in different ways — and whether these trade offs are worth it depends entirely on the nature of your endevour.
- raffraffraff 3y ago
  It depends on what level of "run your own infra" were talking about here. Like, purchasing hardware, racking and connecting, coming up with a nice way to PXE build your fleet and a way to manage the OS and config (chef/ansible/puppet/nix?), before implementing a complete monitoring stack that works on network equipment, raid controllers etc, and then bootstrapping your own Kubernetes cluster or hashistack (or pretending that you don't need complexity, but over time cobbling together your own unique undocumented mess). It's a huge amount of work that startups usually can't afford. The time and effort to do it properly is monstrous. You need to be a company of a certain size to devote the required engineering and support staff to it.

hdjjhhvvhga 3y ago

> Serverless functions – still running on someone’s server… but if you have a function that you need to evoke infrequently and cold start times don’t matter that much, yeah good for cloud. While I agree in principle, these functions don't exist in thin air. You will often want to store the results somewhere. And then protect them. So with time you are almost replicating a lot of infrastructure. And if you are a big org you will probably want to enable SecurityHub and other security/governance services like AWS Config and things get expensive again.

jbverschoor 3y ago

So many negative comments. Makes you wonder who works for aws…

throwaway85858 3y ago
That's why they call it Big Data
andrewstuart 3y ago
Everyone here talks up the cloud. They’ve all drunk the kool aid and are cloud true believers.
- rhaway84773 3y ago
  Also, a lot of people here work for startups or have their own startups. The cloud makes a lot of sense for startups for a variety of reasons. Low capex, easier to get started, you’re willing to pay a premium to not have to deal with certain issues, at least in the beginning, and your costs are likely not very high anyways because your usage is relatively low.
duggan 3y ago
Reasonable professionals can disagree – strongly – without being part of a conspiracy.

RedShift1 3y ago

Move stuff back on-prem. Great. But aside from all the usual practical matters (power, backups, spare parts, service contracts, etc...), software is rapidly becoming a roadblock to move back to on-prem. If everything you have is running on Kubernetes or other open source software, great. If not, then there's an increasing amount of roadblocks being put up: * Some software is only available as SaaS anymore * Jacking up prices to ridiculous levels for on-prem licenses (to favor their SaaS offering of course) * Intentionally knee capping on-prem software feature wise * Stifle development of the on-prem product * Force you to use some parts of their cloud services to make other things work * Dark patterns and endless nags in software to push people to use some part of their cloud services * Poor documentation how to install/use/maintain the software on-prem or make it needlessly complex * Slower response from the software vendor in case of security issues * Exporting data from the cloud and importing it on-prem is impossible And I can make this list go on and on and on... My point being, for small time firms that don't have the resources and solely rely on commercial software, moving back might not even be an option anymore.

antonvs 3y ago
> Intentionally knee capping on-prem software feature wise Or unavoidably. We have a SaaS product that's also available onprem - because we have some big enterprise customers who won't accept anything else - but there's a bunch of stuff we take advantage of in the cloud that we can't easily replicate in arbitrary onprem environments. So the onprem software ends up "kneecapped" just out of sheer pragmatism. There's not a single missing feature onprem that isn't because it would be too much effort to implement and support it. > Stifle development of the on-prem product Similar issue here. We have finite engineering resources, and onprem is a fraction of revenue. > Force you to use some parts of their cloud services to make other things work See first point. > Poor documentation how to install/use/maintain the software on-prem or make it needlessly complex See second point. In short, many of the things you're saying are indeed disadvantages of onprem products today, but it's often not the result of some sort of "intentional kneecapping" or "dark patterns". In my direct experience, it has never been the result of that.
- lifeisstillgood 3y ago

pachico 3y ago

I couldn't read it all because the number of GIFs started to make me nervous :(

throwaway85858 3y ago

In my experience small firms will struggle to hire excellent system engineers that can manage a bare metal setup, it's easier however to hire okay-ish SWE's that can design applications appropriate for cloud.

throwaway2990 3y ago

[flagged]

Havoc 3y ago

I'd say the attack surface on rolling your own tech in a datacenter is substantially higher. Yes DynamoDB has an API, but I'd wager a AWS engineer with good security skills has looked at it carefully. Do you have an equally skilled security expert on hand to look at the datacenters stack and then same for whatever you're deploying on it? Not all internet exposure is equal. Moving out of cloud often makes sense, but security isn't the right motivation for it.

rhaway84773 3y ago
> Yes DynamoDB has an API, but I'd wager a AWS engineer with good security skills has looked at it carefully. Do you have an equally skilled security expert on hand to look at the datacenters stack and then same for whatever you're deploying on it? Is that AWS security engineer looking specifically at how your systems interact with DynamoDB? And how you have specifically configured your DynamoDB instances? And someone rolling their own infrastructure isn’t building out their own custom software to do it. They’re using software from other companies which also have excellent security engineers that have looked into how you can safely deploy and use their software. Now, it may be true that in recent times with the amount of money made in the cloud a lot of these vendors are using carrots and sticks to push you to their cloud offerings (often by making their on Prem offerings worse), but let’s not pretend that on-prem software is two cups tied together with a string or that moving to the cloud somehow alleviates the need to have in house expertise to ensure your cloud software is running correctly and securely.
- Havoc 3y ago
  Again, not all configuration is equal. In hosted services a lot of the decisions have been made for you, that reduces customizability and flexibility but also footguns. >They’re using software from other companies which also have excellent security engineers that have looked into how you can safely deploy and use their software. Of course. Even with well designed software the risk is much higher for the end user though. Deploying a cloud load balancer is pretty idiot proof. If I do the same via local nginx...configuration files, local firewall, file permission, maybe my distro repackaged it, did I set up the user right, am I updating it regularly, what about my underlying operating system security etc. And on top of that it's a once off configuration...it isn't battle tested by thousands running the same load balancer. It's not that it can't be done, but you do pay the price in complexity and engineering hours. >And how you have specifically configured your DynamoDB instances? Well yeah - I'd expect them to have thought through the possible permutations in reasonable detail, which is feasible given the limited customizability. If you're self deploying the possible combinations are endless. It's a bit like testing on android vs iphone. You can reasonably test against the 30 or whatever common iphones, but you're never testing against the 1000s of funky android hardware combinations out there.

fnordpiglet 3y ago

With IAM you can restrict all dynamodb endpoint access to a VPCE/private link. But the insecurity of the public facing endpoints is vastly overstated even without using VPCE. The rest is a bunch of FUD - I spent years going through these points with some of the worlds best security teams to secure some of the most systemically important workloads. These arguments are fairly tired. I’ll tackle another one - speculative attacks. First, you certainly can get bare metal exclusive access to hosts. But instances move around the broader infrastructure of an AZ, even if you’re using something like placement groups which only assure a local affinity. The chance a bad actor can colocate in the same physical device as your workload and successfully attack through side channels is vanishingly low in larger regions. To target anyone specific you would need to do such an enormous fishing expedition that it’s impractical. Further cloud providers aren’t insensate to such attacks and accounts that are doing that sort of topological mapping are easily detected. A better solution is to simply cycle your instances periodically to migrate your workloads around. For very sensitive workloads where the extraordinary unlikelihood isn’t sufficient, just get a bare metal instance. I don’t dissuade anyone from running data centers. But I’ve yet to find anyone running back.

trallnag 3y ago
> worlds best security teams > the most systemically important workloads Big claims require big evidence. How did you end up with these superlatives?
- fnordpiglet 3y ago
  A long career working hard and building a reputation for delivering hard stuff well and building the right network that I used to good effect.

mvkel 3y ago

No. We rolled our own stack for 10 years, until 2018, well after AWS and Azure were around. We switched to Azure in 2018 and never looked back. Sure you trade security (do you really, though?) in exchange for: - not needing to head to the DC because a power supply failed and a rando who was in the cage never plugged in the redundant one - not having to be way over-capacity in scalability, or suddenly under-capacity and emergency ordering some more 1Us - not sifting through eBay to buy a spare hard drive that one of your boxes from 2011 needs but dell no longer makes and the list goes on ad infinitum. This problem has been solved. It's time to move on. Any time a company spends tinkering with their stack is less time spent delivering something of value.

NorwegianDude 3y ago
Then there are options that are probably better for you. I've been doing colocation for many years, but I've never seen the inside of the datacenter. Remote hands set up the hardware for me and replace failed hardware. The service is so rarely needed that it's basically free. In recent years I've started to rent the hardware I want as I want more frequent changes. I tell my host what hardware I want and they'll acquire it and install it. If you're running very old hardware that you can't even buy replacement drives for then you are doing something very wrong. Need to scale fast? Then I don't have to wait for servers to arrive, I use their cloud offering that's in the same network. I agree that it's a solved problem, but the solution to anything somewhat permanent is not the cloud. It's not really any harder to run your own hardware.
- mvkel 3y ago
  All of those things still require management, which is the issue at hand here. Also if you're renting your hardware... for a subscription ... that sounds an awful lot like the cloud with extra steps.

jgb1984 3y ago

Why is the choice always portrayed as if it's either cloud, or on-prem / colocation ? Those are two extremes. At work I use dedicated physical machines from Hetzner. If I need an extra one they deliver it in a few minutes, and thanks to Ansible it's provisioned within a few minutes more. Hetzner keeps an eye on the hardware and replaces disks, PSU's and the like if needed. I wouldn't often advocate to do colocation or on-prem since that indeed comes with a whole set of headaches, but renting dedicated physical servers like we do offers a lot of flexibility, with very little overhead, at a price/performance ratio that makes AWS look like pure extortion.

_xnmw 3y ago
Yep, that's the compromise I came to as well at my solo bootstrapped SaaS. However, there are still some benefits of colocation that I would love to have: 1. ability to use M1 Mac minis as servers for specialized workloads, while having an identical developer setup for testing. This would really enable me to finetune, test, and squeeze out performance to the max and have the dream of a "single box" deployment serving hundreds of thousands of customers.
candiddevmike 3y ago
I think your example falls under the cloud use case, it's a form of IaaS. The reason it doesn't scale is at some point it becomes cheaper to own the hardware or you need better scaling/redundancy and switch to managed cloud VMs/containers/functions
riku_iki 3y ago
> at a price/performance ratio that makes AWS look like pure extortion. I think if you try to host in US to have lower latency to many customers, and with some less cheap and more reliable server grade network/hardware, it is likely cost will be higher than for hetzner.

RantyDave 3y ago

Oh, "just" run Kubernetes. Gotcha.

qaq 3y ago

Important point they miss is that unless your monthly spend with AWS is couple of mil a month you are nothing to them.

jauntywundrkind 3y ago

I thought for a second I was hearing a.suggestion that by running your own hardware you can turn off all mitigations. Which should save you a ton of money; for the same core you may be getting more than double the performance. And since you're not running any unknown foreign workloads, no one would have access to be running timing attacks on you. The article pretty quickly moved off that point. And it seems to at points be saying that making it hard to setup infrastructure is a feature, will help the bottom line. Which has some truth but at enormous emotional cost to your teams. I do think this shift needs to happen and I appreciate such lengthy sets of concerns being brought out. But I'm pretty lukewarm on this analysis.

Matthias247 3y ago

> in traditional datacenters, with Infrastructure and Support teams separate from Development (anti DevOps), there are/were strong human checks and balances. If your Devs wanted to make an API Internet accessible and connect it to what they thought was a “sanitized” database, they probably had to raise a change, submit some firewall rules, maybe talk to a DBA to get credentials. This is just romanticizing things. In every deployment - whether in it's "in cloud" or anywhere else - there is always the quick on off change that someone makes. Probabably for a valid reason, e.g. solving a production problem quickly. Chances are high that will go unnoticed, until another problem manifests (and that is hopefully not a security issue!). I would argue that in cloud setups the chances for that to happen are actually slightly lower, because teams are incentivized to use immutable and declarative infrastructure. And that there might be an audit log in place which tells what changes have been made - although that still requires people to look at it, which again happens only in the case problems show up.

lijok 3y ago

[flagged]

deleted 3y ago

[deleted]