10 ms·

Don't build private clouds

66 points by mhausenblas 10y ago

alrs 10y ago
Private clouds wouldn't be the recurring disaster that they are if the people who built them had used AWS before. Instead of "don't build private clouds," I'd offer "If you think Openstack means 'free VMWare,' you've already failed."
whyagaindavid 10y ago
We are in the same boat. We tried opencloud and has severe issues with our unified login - mix of AIX,samba,ladp,AD systems. I wonder how to convince universities who have legacy data and staff (aka Prof) who are unwilling to change to cloud especially in these days of data leak. Any pointers?
- Freak_NL 10y ago
  > […] who are unwilling to change to cloud especially in these days of data leak. I would start by addressing that concern, but it may come down to a matter of beliefs; i.e., you trust that 'the cloud' will be secure this next iteration (or the next, or the next), and they remain sceptical. Can't blame them.
- adrianN 10y ago
  Either you find things that would work a lot better for them after the change or the change just doesn't make sense from their point of view. Just because you have less work with a public cloud that doesn't mean your users will be happier too.
- lykron 10y ago
  The only argument that is needed is cost. Is it cheaper to keep your data local or move it off site? Most of the time, it is cheaper to keep it on site because you don't pay for bandwidth,
deleted 10y ago
[deleted]
jtwaleson 10y ago
What is the ratio between public and private cloud data centers is in terms of server count / data center count etc? I think the amount of enterprise DC's / colocation facilities is pretty stable but that almost all of the growth is in public cloud. Don't have any data to back this up though.
- discodave 10y ago
  In terms of existing infra, private DC capacity is surely an order of magnitude more than public cloud. New capacity getting installed might be a bit more evenly split. How you differentiate between any private infra and private cloud I don't know. Public clouds often use colo facilities, especially for new/small regions. So colo != private.
  - kesor 10y ago
    Depends on the public cloud probider, but the leading providers are actively building their own buildings for expansion purposes. For example MSFT investing $15B in its datacenter infrastructure. https://azure.microsoft.com/en-us/overview/datacenters/ https://azure.microsoft.com/en-us/overview/datacenters/. GOOG naturally don't need to colo anything, they have their own https://www.google.com/about/datacenters/inside/ https://www.google.com/about/datacenters/inside/. This article shows an analysis of DC vs. Public/Private Cloud IT Infrastructure: http://www.datacenterknowledge.com/archives/2016/07/06/cloud-providers-will-spend-23b-on-data-center-hardware-in-2016-report/ http://www.datacenterknowledge.com/archives/2016/07/06/cloud...
Freak_NL 10y ago
If you are handling data that is subject to national regulations concerning personal data (e.g., health care) using a cloud provider for all your needs is risky and might not be legal. If you use one of the big-name (and therefore affordable) providers such as Amazon, your data may very well fall under America's Patriot Act — there is a lot of legal uncertainty about this. If I go with a local party instead, there is also no guarantee that this company in turn won't branch out to the US, or get bought up by an American company, again putting citizen's private data in reach of (in my case) foreign governments.
- rcarmo 10y ago
  Erm, no. All major cloud providers have clear statements regarding data sovereignty, compliance and whatnot. There is no uncertainty, only lack of awareness (but I'll freely grant that it is not a simple matter). Look for the equivalent of https://www.microsoft.com/en-us/TrustCenter/Transparency#Transparent https://www.microsoft.com/en-us/TrustCenter/Transparency#Tra... (Full disclosure: I work at Microsoft)
  - eveningcoffee 10y ago
    Sorry but, I am afraid that this train has already left the station.
  - thinkMOAR 10y ago
    So just because mickeysoft and other 'major' clouds providers as you say, have some certificates and a webpage where it says to play nice it's all good? Wow...
  - cyberpunk 10y ago
    Do you honesty believe that? Maybe I'm too cynical these days, but I find it pretty laughable that a statement regarding 'data sovereignty' has any real meaning at all these days. Unless you control the hardware, you have no idea what the hypervisors are doing on your rented vm's. The idea that national actors don't / wouldn't pull anything they want from them seems comically naive to me. Christ, we can barely trust our own hardware anymore with backdoored firmware crap, IME and such.. Realistically though, most companies don't care so it's not a problem if you keep the blinkers on; their compliance folks just want the document which says it's not happening and they'll never know when it does so who cares. Their ops folks can barely use the APIs probably, let alone contemplate the impact of someone reading the memory of my databases from the hypervisors. As an individual and someone who cares for my data or that of my clients though, that's not enough for me, and I'd never trust tnese folks with any data, only mindless compute and serving public content...
pasta 10y ago
What about privacy? Our company doesn't use a cloud provider for that exact reason.
- kesor 10y ago
  Have you investigated the issues with privacy in public clouds, or do you just "assume" that it is much worse than your own mismanaged hardware without doing any due diligence? From exprience, most organisations manage the privacy of their customers much much worse on their own hardware than they would if they used a public cloud provider. Want to know how "privacy" really looks like? Take a look here - http://www.informationisbeautiful.net/visualizations/worlds-biggest-data-breaches-hacks/ http://www.informationisbeautiful.net/visualizations/worlds-...
  - cyberpunk 10y ago
    The difference is simply though that they own the entire scope of the privacy concerns on prem; even if they manage them poorly. In the public cloud, as an engineer/employee/government I can just login to any given XEN/KVM/SmartOS/Whatever host that's running your instances and simply read your secrets and even if your opsec is perfect there's nothing you can do about it. 'privacy' in the cloud is not about mismanagement/good practice -- it's about who is able to act on it. There is no way that anything you have running on infra you don't completely own and control will ever be private; to even a casual observer it's obvious that there is absolutely no defense or privacy possible in such an environment.
  - pasta 10y ago
    I'm sorry I was not very clear. I meant privacy as in company critical data like finance, price agreements and so one. There are also a lot of companies who don't like storing data in a foreign country. What if the government was taken over by a Mr Trump who doesn't give about past agreements? And using a managed cloud doesn't mean no data breach. You can still write SQL injectable websites that read data from the 'safe' cloud.
a_imho 10y ago
I don't quite agree with the cost analysis. You either spend on building your infrastructure or integrating public cloud services. You will pay your devops team anyway. Owning stuff is usually cheaper than renting, there are many cases when it can make sense to have your own hardware.
Jaruzel 10y ago
Can anyone tell me what typeface is being used on this graph? https://d262ilb51hltx0.cloudfront.net/max/1546/1*6rkj6frEx5Bf8_s6sIi_9A.png https://d262ilb51hltx0.cloudfront.net/max/1546/1*6rkj6frEx5B...
- coroxout 10y ago
  Looks like Apple's "Chalkduster": http://www.identifont.com/find?font=Chalkduster&q=Go http://www.identifont.com/find?font=Chalkduster&q=Go
  - Jaruzel 10y ago
    It does! Thank you. :)
micheljansen 10y ago
This last point trumps all others: "[...] on-premise infrastructure on the other-hand brews a culture of mistrust, centralization, dependency and control. [...] These difference between on-premise data centers and public clouds influence how teams think, plan and execute. These are nothing but attributes of culture." I'm currently working with a large financial services organisation that is moving to the cloud. It's incredibly painful, because of all the challenges involved in guaranteeing the safety and security of private data etc., but once these hurdles are taken once, it's incredibly liberating. The move is not just a technical one though, it requires a change of culture as well.
- kesor 10y ago
  Unfrotuantely, all too often when a company is moving some workloads to a public cloud, they take along the same culture they use in their "private" datacenter.
  - micheljansen 10y ago
    Absolutely. Fortunately, in this case, we are actually moving because we want to change culture.
sfifs 10y ago
If you store your data with a third party cloud provider, government agencies can legally compel your cloud provider to give them your data while preventing them from telling you. If your data is on-premises, they'll have to come and take it from you. If you are a multinational company with entities in multiple geographies and related party transactions, you know what you'd prefer.
- kesor 10y ago
  The Pirate Bay
cyberpunk 10y ago
Am I missing something here? Is your point that everyone should move to the public cloud because you think it's "difficult" to build a "private" one? Sorry to be blunt, but are you joking, an idiot or just completely unaware of the reality our industry outside of startup land? Forgetting the compliance reasons, or that I have zero confidence that a combination of giant American corporations have any interest in protecting my data and that of my users, forgetting also that the performance differences are incredibly significant (take 8 15k sas's in raid 10 vs.. what? EBS with 2k 'provisioned IOPS' on a shared san over iscsi?), the cost savings are much more significant than you claim here. I recently bought a half rack load of dell boxes which have like 380GB of ram each and 52 threads, and they cost around £4k each. Even with a whole army of overpriced ops dudes to install openstack or kubernetes on all that, the cost we spent would be spanked in a month on AWS for similar capacity, and mine isn't a recurring one. Why does it have to be so black and white for you folks? A "private cloud" is absolutely the correct choice for a lot of people. If I were you I'd quietly delete that post, because I've absolutely judged you -- and probably unfairly -- on the content. On that, I wouldn't hire you. Listen, both these things have a place, but they're NOT competing in the minds of those of us running serious infra. AWS allows us to prototype apps rapidly, to put project work on a credit card associated with a specific project budget instead of spending the shared budget, and for sure there are some nice services which do a bit of heavy lifting for us which helps speed that up. After that phase though, very, very few people things actually need or make sense to stay there once it becomes a real product. Your scaling concerns are not being well served by 1000s of ec2 instances. Moving back in house is usually the smarter option. If you're pumping loads of data into the internet, then you should be behind a CDN and the origin shouldn't be smashed to the point where you're dependent on throwing compute into the void to cope. Akamai or CloudFlare are cheaper than deploying 100s of ec2's because some shitty code can't handle your traffic. If data is coming in, read a book on queues; you'd be pretty surprised what you can do with even two 1U boxes and a good software architecture. We could handle 100k+/sec of messages back in the java 1.5 and activemq days with 1U sparc boxes; how much do you recon it costs to do that today in AWS? tl;dr - urgh. lets' talk in 10 years. </rant>
- davidjgraph 10y ago
  > If I were you I'd quietly delete that post, because I've absolutely judged you -- and probably unfairly -- on the content. On that, I wouldn't hire you. Based on your reply, I suspect most people wouldn't want to be hired by you, either...
cyberpunk 10y ago
Okay, I wrote out a whole breakdown of everything that's wrong with this post, but after I re-read it, it came across as so harsh that I've just deleted the lot. TBH; the only people who would have agreed with me knew my points anyway so bother? I felt like as this is my specific area, that I have a kind of responsibility to people who read misinformation like this to help see through such stuff. It's hard to do for me without getting kind of angry though; opinions like this is why I end up having to deal with 800k/month aws bills and 3 hour deploys.. Either way; regardles of me -- please, people -- if you think that the 'typical journey' of implementing your own infrastructure and migrating a legacy application to run on it is ANYTHING like the description in this blog post, then please take it from me as someone who moves horrible legacy codebases into CD for a living, that you should perhaps seek some alternative advice. Suubu; I'm sorry, I'm not trying to be mean but your typical' private cloud journey is complete and utter horseshit.
- deleted 10y ago
  [deleted]
- oarsinsync 10y ago
  > TBH; the only people who would have agreed with me knew my points anyway so bother? Can you please actually elaborate, or it defeats the point of your entire post. While the rant may have helped you, I don't know you, I don't your history, and this post doesn't encourage me to go through your post history to find it. At the same time, OP's post filled me with a slight sense of dread as I'm about to go into a private cloud environment and wondered if I was making a mistake. Your post has certainly calmed those nerves. Actually having some meat with the mustard would help more. Thanks in advance!
  - cyberpunk 10y ago
    Well, I don't want to pick the whole thing apart because I've been a bit too mean already, but the 'typical' process of private cloud in that post is bullshit. Lets make some assumptions on this gig you're starting then, we'll imagine you're going into some job at a place which has a horrible legacy codebase (lets' go medieval assume struts and oracle 10, deployed with some ancient bamboo/rpm concoction which takes forever and always dies and the existing ops team barely keep the lights on, let alone are capable of keeping up with the rate dev is trying to let the business move..). So you land. They've got a fresh budget, a load of bearded scala devs who are rewriting the whole thing as microservices and words like "ISB" and "ASYNC" are being thrown around. Hell, someone even mentioned elixir in the bar on your welcome drinkies. You're going to drag this horrible monster into the new world, but to get there and to a better arch you're going to have to replace it a bit at a time, and while that happens the code that's making all the money needs to be maintained, and it needs to change faster. And you're going to do this on prem. So. You land. On prem, ooooeh baby. Realise: the pain you're going to have is not going to be how you instantiate vms to run the apps or spin up loadbalancer IPs or whatever the arse else infra wise -- AWS will do all that right away for you for sure, while working out how to get a bunch of tin to do it will take longer, but it's like getting into first gear slightly faster than the guy beside you on a F1 race -- in the long run, this isn't where the time is going to be spent. Lets say you're the ops dude instead, how do you move a few hundred hand built centos and java boxes to some kind of api based provisioning thingy with no config management? Yeah well.. There are a lot of options, if you're the one building it out, tbh the best choice for the infra part of the provisioning is just to pay someone to make this their problem. Redhat. Vmware. Whoever does Proxmox. Whatever. You'll lose E-Penis points but who cares your job is to help the business work faster. Even if you have do it yourself, there is a large ecosystem around things such as openstack or proxmox or smartos or whatever cobbler and so on, and this stuff is pretty friendly, it won't take you THAT long. Hire a contractor for a bit. Blammo. The post talks about doing 'compute' then 'storage' and 'network' then 'stateless monoliths' and so on. Whatever. That's not reality. Why is the business paying for this? They want to be able to iterate quicker! CD is a tough goal with a legacy codebase, but they got used to agile and hell if it's not going to get faster -- the job is to enable that in any way possible, the tricky part of that is not going to be how you deal with infra, it's how you get the code through a pipeline and deploying painlessy. Infra is a pain point. Legacy environments are fragile as hell and good operations folks are few and far between, and most of them are undervalued and a bit weird. -- but you personally, oarsinsync, are going to better off on prem for a few simple reasons in that world: 1) Total Ownership: Something goes titsup? Yeah, your team owns the entire show up until your ISP. Network borks? Storage slows down? It's all yours. Dig in. AWS develops cross-az latency? Log a ticket and 2 days later when they respond you'll have already forgotten it happened. Explain that to the business. 2) Perf: Don't believe the rubes; you'll be quicker on your own gear if only because you can move your noisy VMs to sep machines, set various limits/quotes/throttles that you can't on a public cloud, and you're not sharing spindles with 300 mongo vms. 3) Expierence: It's not exactly a business goal, but you'll learn more when you're having to stace some jvm at 3:00am and crying into a toilet than if you're just bouncing instances and getting away with shit code. If you just outsource all your problems forever you'll never be able to have an opinion on them, even if that opinion is just "I never want to use RAC again" or "mongodb is death" or whatever..
kkapelon 10y ago
The article does not address important topics such as privacy and security. For some organizations using a private cloud is not a technical decision but instead a political/ethics one. As an example I would bet that there are several companies out there who would not care if all services were down for a full day (in their private cloud), as long as they were certain that no government could easily access them.
- erelde 10y ago
  GovernmentS can't host their services on Google's or Amazon's cloud for example. (the US are not counting since this is THE government many others are not trusting and the home of the public clouds)
  - mritun 10y ago
    They already do: https://aws.amazon.com/govcloud-us/ https://aws.amazon.com/govcloud-us/
  - us0r 10y ago
    Amazon's largest customer is the CIA. http://www.theatlantic.com/technology/archive/2014/07/the-details-about-the-cias-deal-with-amazon/374632/ http://www.theatlantic.com/technology/archive/2014/07/the-de...
    - Retric 10y ago
      That's a private cloud built by Amazon for the CIA and does not run on Amazon's network.
  - kesor 10y ago
    The FUD is strong, but that was exactly the point of that article. Having old-fashioned people in goverments who disagree that they should ditch their datacenters and start using clouds, is backwards thinking. Just like goverments find ways to allow private companies to run their cellular networks, phone networks, tv cable networks, power grids - the same reasoning should allow private companies (public clouds) who do it much better than goverment employees, to run their compute services.
kitd 10y ago
I was musing yesterday whether a technologically-advanced nation state of the future might provide cloud infrastructure like it provides roads/rail/healthcare/etc. Obviously, this only makes sense in a world where individuals as well as corps need to present themselves online as a matter of course, but it might enable eg personal ID or data privacy tools that address many of the problems people have 'donating' their private data to commercial orgs.
- adrianN 10y ago
  A wet dream for police states. Distribute dumb terminals to government-run cloud servers for free and soon enough the only people who don't make their whole life transparent to the state are the criminals.
- nickpsecurity 10y ago
  It's possible. They'd probably do internet cafes and libraries first. That gives them visual surveillance on top of the Internet surveillance. Far as in U.S., there was one satirical poster someone made selling the NSA surveillance program as a free replacement for backup software. ;)
anon1253 10y ago
Yeah so we have a different story. We build Natural Language Processing and statistics software. We tried the cloud first, but it was just unworkable. Limited IOPS, outrageous prices for even a modest amount of RAM, slow virtualized CPUs, limited GPU options with terrible performance. So one day I hired half a rack, bought some machines with fast SSDs, 128GB RAM each, dual CPU with high clockspeed, and even some boxes with GPUs. The difference was incredible. Not just in performance, which was orders of magnitudes faster (going from barely workable to "pleasant user experience" without any software changes), but also in terms of cost. If we were to provision these machines on Azure or E2 we'd spend in a month what these machines cost, and our current current costs are maybe $500 in electricity + rack rental. Sure, it's a bit of a workout to put stuff in racks, but I actually quite enjoyed doing it. Sure stuff will break and things will go down. We can take the downtime at this point. And seriously, cost and performance wise there is no competition between cloud and rolling your own hardware (if you know how to do it). Also I'm perpetually surprised by how overstated the complexity of this is, yes it's tricky. But so is CSS, and Javascript these days. And unless you go "full cloud" (never go full cloud), you still have to manage fallover/redudancy/CI/provisioning yourself. It's not much different on "actual" hardware (there is no cloud, it's just somebody else's computer). If my firewall or switch catches fire, that might be problematic. But if that happens at an Amazon or Azure data center, that's also problematic. At the very least you'll suffer terrible latency because it needs to move the VMs dynamically.
- cyberpunk 10y ago
  Rock on anon' -- we feel the same way! Once you get above a few boxes though, you might need some pricey geeks for a month or two to rig up cobbler, ceph, openstack, proxmox, or whatever-takes-ya simply because it's quicker than getting up on all that yourself right away -- but really there is no magic involved here and it's just so you can have it totally manageable as you grow. The tooling is much more friendly these days; and to be totally honest -- provisioning probably isn't the problem you're trying to solve anyway (more likely packaging/deployment/ci is the pain after you got a bit of breathing space on perf). I've seen loads of folks coming back on prem spend months rigging up crazy vm provisioning things when their estate is basically static because they didn't realise they were solving the wrong problem. If your ops time is spent building machines, then sort that out. Probably and more likely it's spent on getting apps running and dealing with moving traffic about and deployments and such; so that's the real thing to work on. I think the 'complexity' argument seem so loud on HN as the opsy people are the minority here and it's not exactly a sexy/exciting/fun thing to pick up, and when it goes wrong it's brutal, especially when you can be nailing features instead. All I'd say is -- regardless of what you're doing, getting into containers at this point makes your life quite a lot easier; you get a standard deployable unit of code you can throw into aws, or on prem depending on your workload, and it removes a lot of the anti-patters of config mgt.
snarfy 10y ago
> Phase 3: Then deal with the stateful monoliths. These are your large monolithic databases. This is usually where private cloud journey hits the wall due to the risk and complexity in making such monoliths cloud native. We are tasked with moving to the cloud, and this is where everything stops. Over the years, instead of fixing the monolithic database design, the company instead kept beefing up the hardware for the database. A bigger cpu and more ram is cheaper up front than re-engineering the entire company infrastructure. And now we are paying for it. Our db hardware now is the highest end hardware the db software supports. None of the cloud providers db support can handle it. We are forced to fix the monolithic design if we are to move to the cloud. It will be a multi-year effort.
- mooreds 10y ago
  > It will be a multi-year effort And the company is willing to go through it because the public cloud offers so much? Just curious if this was going to stop the overall project or not.
  - snarfy 10y ago
    We were bought out by investors and this is their major initiative. I assume it is to reduce long term operating costs.
- cyberpunk 10y ago
  My current gig had a similar problem; changing the DB or putting a layer in front of it at a large scale or really er 'mature' codebase is basically a rewrite, and that's not really the smart way to play it and it's probably not possible anyway. Start from the top instead of the data. Moving various bits of the API to new implementations/microservices and modernize the stack -- new features, make 'em all standalone services and try to deal with your logic in the app instead of in the db. Move /send/giant_dildo_to?uzer=cyberpunk out of the monolith and to a sep service, and eventually you might get to the point where not a huge amount of the legacy code is actually being hit by anything besides your new code. At that point, the cost of redoing the persistence bit is way less than it was when you started. Depending on your app, you might even get away with creating a new DB and using the orig as a kind of 'feeder' which you run on batch -- all options are open right? But obv it depends what you're doing.

66 points by mhausenblas 10y ago

alrs 10y ago

Private clouds wouldn't be the recurring disaster that they are if the people who built them had used AWS before. Instead of "don't build private clouds," I'd offer "If you think Openstack means 'free VMWare,' you've already failed."

whyagaindavid 10y ago

We are in the same boat. We tried opencloud and has severe issues with our unified login - mix of AIX,samba,ladp,AD systems. I wonder how to convince universities who have legacy data and staff (aka Prof) who are unwilling to change to cloud especially in these days of data leak. Any pointers?

Freak_NL 10y ago
> […] who are unwilling to change to cloud especially in these days of data leak. I would start by addressing that concern, but it may come down to a matter of beliefs; i.e., you trust that 'the cloud' will be secure this next iteration (or the next, or the next), and they remain sceptical. Can't blame them.
adrianN 10y ago
Either you find things that would work a lot better for them after the change or the change just doesn't make sense from their point of view. Just because you have less work with a public cloud that doesn't mean your users will be happier too.
lykron 10y ago
The only argument that is needed is cost. Is it cheaper to keep your data local or move it off site? Most of the time, it is cheaper to keep it on site because you don't pay for bandwidth,

deleted 10y ago

[deleted]

jtwaleson 10y ago

What is the ratio between public and private cloud data centers is in terms of server count / data center count etc? I think the amount of enterprise DC's / colocation facilities is pretty stable but that almost all of the growth is in public cloud. Don't have any data to back this up though.

discodave 10y ago
In terms of existing infra, private DC capacity is surely an order of magnitude more than public cloud. New capacity getting installed might be a bit more evenly split. How you differentiate between any private infra and private cloud I don't know. Public clouds often use colo facilities, especially for new/small regions. So colo != private.
- kesor 10y ago
  Depends on the public cloud probider, but the leading providers are actively building their own buildings for expansion purposes. For example MSFT investing $15B in its datacenter infrastructure. https://azure.microsoft.com/en-us/overview/datacenters/ https://azure.microsoft.com/en-us/overview/datacenters/. GOOG naturally don't need to colo anything, they have their own https://www.google.com/about/datacenters/inside/ https://www.google.com/about/datacenters/inside/. This article shows an analysis of DC vs. Public/Private Cloud IT Infrastructure: http://www.datacenterknowledge.com/archives/2016/07/06/cloud-providers-will-spend-23b-on-data-center-hardware-in-2016-report/ http://www.datacenterknowledge.com/archives/2016/07/06/cloud...

Freak_NL 10y ago

If you are handling data that is subject to national regulations concerning personal data (e.g., health care) using a cloud provider for all your needs is risky and might not be legal. If you use one of the big-name (and therefore affordable) providers such as Amazon, your data may very well fall under America's Patriot Act — there is a lot of legal uncertainty about this. If I go with a local party instead, there is also no guarantee that this company in turn won't branch out to the US, or get bought up by an American company, again putting citizen's private data in reach of (in my case) foreign governments.

rcarmo 10y ago
Erm, no. All major cloud providers have clear statements regarding data sovereignty, compliance and whatnot. There is no uncertainty, only lack of awareness (but I'll freely grant that it is not a simple matter). Look for the equivalent of https://www.microsoft.com/en-us/TrustCenter/Transparency#Transparent https://www.microsoft.com/en-us/TrustCenter/Transparency#Tra... (Full disclosure: I work at Microsoft)
- eveningcoffee 10y ago
  Sorry but, I am afraid that this train has already left the station.
- thinkMOAR 10y ago
  So just because mickeysoft and other 'major' clouds providers as you say, have some certificates and a webpage where it says to play nice it's all good? Wow...
- cyberpunk 10y ago
  Do you honesty believe that? Maybe I'm too cynical these days, but I find it pretty laughable that a statement regarding 'data sovereignty' has any real meaning at all these days. Unless you control the hardware, you have no idea what the hypervisors are doing on your rented vm's. The idea that national actors don't / wouldn't pull anything they want from them seems comically naive to me. Christ, we can barely trust our own hardware anymore with backdoored firmware crap, IME and such.. Realistically though, most companies don't care so it's not a problem if you keep the blinkers on; their compliance folks just want the document which says it's not happening and they'll never know when it does so who cares. Their ops folks can barely use the APIs probably, let alone contemplate the impact of someone reading the memory of my databases from the hypervisors. As an individual and someone who cares for my data or that of my clients though, that's not enough for me, and I'd never trust tnese folks with any data, only mindless compute and serving public content...

pasta 10y ago

What about privacy? Our company doesn't use a cloud provider for that exact reason.

kesor 10y ago
Have you investigated the issues with privacy in public clouds, or do you just "assume" that it is much worse than your own mismanaged hardware without doing any due diligence? From exprience, most organisations manage the privacy of their customers much much worse on their own hardware than they would if they used a public cloud provider. Want to know how "privacy" really looks like? Take a look here - http://www.informationisbeautiful.net/visualizations/worlds-biggest-data-breaches-hacks/ http://www.informationisbeautiful.net/visualizations/worlds-...
- cyberpunk 10y ago
  The difference is simply though that they own the entire scope of the privacy concerns on prem; even if they manage them poorly. In the public cloud, as an engineer/employee/government I can just login to any given XEN/KVM/SmartOS/Whatever host that's running your instances and simply read your secrets and even if your opsec is perfect there's nothing you can do about it. 'privacy' in the cloud is not about mismanagement/good practice -- it's about who is able to act on it. There is no way that anything you have running on infra you don't completely own and control will ever be private; to even a casual observer it's obvious that there is absolutely no defense or privacy possible in such an environment.
- pasta 10y ago
  I'm sorry I was not very clear. I meant privacy as in company critical data like finance, price agreements and so one. There are also a lot of companies who don't like storing data in a foreign country. What if the government was taken over by a Mr Trump who doesn't give about past agreements? And using a managed cloud doesn't mean no data breach. You can still write SQL injectable websites that read data from the 'safe' cloud.

a_imho 10y ago

I don't quite agree with the cost analysis. You either spend on building your infrastructure or integrating public cloud services. You will pay your devops team anyway. Owning stuff is usually cheaper than renting, there are many cases when it can make sense to have your own hardware.

Jaruzel 10y ago

Can anyone tell me what typeface is being used on this graph? https://d262ilb51hltx0.cloudfront.net/max/1546/1*6rkj6frEx5Bf8_s6sIi_9A.png https://d262ilb51hltx0.cloudfront.net/max/1546/1*6rkj6frEx5B...

coroxout 10y ago
Looks like Apple's "Chalkduster": http://www.identifont.com/find?font=Chalkduster&q=Go http://www.identifont.com/find?font=Chalkduster&q=Go
- Jaruzel 10y ago
  It does! Thank you. :)

micheljansen 10y ago

This last point trumps all others: "[...] on-premise infrastructure on the other-hand brews a culture of mistrust, centralization, dependency and control. [...] These difference between on-premise data centers and public clouds influence how teams think, plan and execute. These are nothing but attributes of culture." I'm currently working with a large financial services organisation that is moving to the cloud. It's incredibly painful, because of all the challenges involved in guaranteeing the safety and security of private data etc., but once these hurdles are taken once, it's incredibly liberating. The move is not just a technical one though, it requires a change of culture as well.

kesor 10y ago
Unfrotuantely, all too often when a company is moving some workloads to a public cloud, they take along the same culture they use in their "private" datacenter.
- micheljansen 10y ago
  Absolutely. Fortunately, in this case, we are actually moving because we want to change culture.

sfifs 10y ago

If you store your data with a third party cloud provider, government agencies can legally compel your cloud provider to give them your data while preventing them from telling you. If your data is on-premises, they'll have to come and take it from you. If you are a multinational company with entities in multiple geographies and related party transactions, you know what you'd prefer.

kesor 10y ago
The Pirate Bay

cyberpunk 10y ago

Am I missing something here? Is your point that everyone should move to the public cloud because you think it's "difficult" to build a "private" one? Sorry to be blunt, but are you joking, an idiot or just completely unaware of the reality our industry outside of startup land? Forgetting the compliance reasons, or that I have zero confidence that a combination of giant American corporations have any interest in protecting my data and that of my users, forgetting also that the performance differences are incredibly significant (take 8 15k sas's in raid 10 vs.. what? EBS with 2k 'provisioned IOPS' on a shared san over iscsi?), the cost savings are much more significant than you claim here. I recently bought a half rack load of dell boxes which have like 380GB of ram each and 52 threads, and they cost around £4k each. Even with a whole army of overpriced ops dudes to install openstack or kubernetes on all that, the cost we spent would be spanked in a month on AWS for similar capacity, and mine isn't a recurring one. Why does it have to be so black and white for you folks? A "private cloud" is absolutely the correct choice for a lot of people. If I were you I'd quietly delete that post, because I've absolutely judged you -- and probably unfairly -- on the content. On that, I wouldn't hire you. Listen, both these things have a place, but they're NOT competing in the minds of those of us running serious infra. AWS allows us to prototype apps rapidly, to put project work on a credit card associated with a specific project budget instead of spending the shared budget, and for sure there are some nice services which do a bit of heavy lifting for us which helps speed that up. After that phase though, very, very few people things actually need or make sense to stay there once it becomes a real product. Your scaling concerns are not being well served by 1000s of ec2 instances. Moving back in house is usually the smarter option. If you're pumping loads of data into the internet, then you should be behind a CDN and the origin shouldn't be smashed to the point where you're dependent on throwing compute into the void to cope. Akamai or CloudFlare are cheaper than deploying 100s of ec2's because some shitty code can't handle your traffic. If data is coming in, read a book on queues; you'd be pretty surprised what you can do with even two 1U boxes and a good software architecture. We could handle 100k+/sec of messages back in the java 1.5 and activemq days with 1U sparc boxes; how much do you recon it costs to do that today in AWS? tl;dr - urgh. lets' talk in 10 years. </rant>

davidjgraph 10y ago
> If I were you I'd quietly delete that post, because I've absolutely judged you -- and probably unfairly -- on the content. On that, I wouldn't hire you. Based on your reply, I suspect most people wouldn't want to be hired by you, either...

cyberpunk 10y ago

Okay, I wrote out a whole breakdown of everything that's wrong with this post, but after I re-read it, it came across as so harsh that I've just deleted the lot. TBH; the only people who would have agreed with me knew my points anyway so bother? I felt like as this is my specific area, that I have a kind of responsibility to people who read misinformation like this to help see through such stuff. It's hard to do for me without getting kind of angry though; opinions like this is why I end up having to deal with 800k/month aws bills and 3 hour deploys.. Either way; regardles of me -- please, people -- if you think that the 'typical journey' of implementing your own infrastructure and migrating a legacy application to run on it is ANYTHING like the description in this blog post, then please take it from me as someone who moves horrible legacy codebases into CD for a living, that you should perhaps seek some alternative advice. Suubu; I'm sorry, I'm not trying to be mean but your typical' private cloud journey is complete and utter horseshit.

deleted 10y ago
[deleted]
oarsinsync 10y ago
> TBH; the only people who would have agreed with me knew my points anyway so bother? Can you please actually elaborate, or it defeats the point of your entire post. While the rant may have helped you, I don't know you, I don't your history, and this post doesn't encourage me to go through your post history to find it. At the same time, OP's post filled me with a slight sense of dread as I'm about to go into a private cloud environment and wondered if I was making a mistake. Your post has certainly calmed those nerves. Actually having some meat with the mustard would help more. Thanks in advance!
- cyberpunk 10y ago
  Well, I don't want to pick the whole thing apart because I've been a bit too mean already, but the 'typical' process of private cloud in that post is bullshit. Lets make some assumptions on this gig you're starting then, we'll imagine you're going into some job at a place which has a horrible legacy codebase (lets' go medieval assume struts and oracle 10, deployed with some ancient bamboo/rpm concoction which takes forever and always dies and the existing ops team barely keep the lights on, let alone are capable of keeping up with the rate dev is trying to let the business move..). So you land. They've got a fresh budget, a load of bearded scala devs who are rewriting the whole thing as microservices and words like "ISB" and "ASYNC" are being thrown around. Hell, someone even mentioned elixir in the bar on your welcome drinkies. You're going to drag this horrible monster into the new world, but to get there and to a better arch you're going to have to replace it a bit at a time, and while that happens the code that's making all the money needs to be maintained, and it needs to change faster. And you're going to do this on prem. So. You land. On prem, ooooeh baby. Realise: the pain you're going to have is not going to be how you instantiate vms to run the apps or spin up loadbalancer IPs or whatever the arse else infra wise -- AWS will do all that right away for you for sure, while working out how to get a bunch of tin to do it will take longer, but it's like getting into first gear slightly faster than the guy beside you on a F1 race -- in the long run, this isn't where the time is going to be spent. Lets say you're the ops dude instead, how do you move a few hundred hand built centos and java boxes to some kind of api based provisioning thingy with no config management? Yeah well.. There are a lot of options, if you're the one building it out, tbh the best choice for the infra part of the provisioning is just to pay someone to make this their problem. Redhat. Vmware. Whoever does Proxmox. Whatever. You'll lose E-Penis points but who cares your job is to help the business work faster. Even if you have do it yourself, there is a large ecosystem around things such as openstack or proxmox or smartos or whatever cobbler and so on, and this stuff is pretty friendly, it won't take you THAT long. Hire a contractor for a bit. Blammo. The post talks about doing 'compute' then 'storage' and 'network' then 'stateless monoliths' and so on. Whatever. That's not reality. Why is the business paying for this? They want to be able to iterate quicker! CD is a tough goal with a legacy codebase, but they got used to agile and hell if it's not going to get faster -- the job is to enable that in any way possible, the tricky part of that is not going to be how you deal with infra, it's how you get the code through a pipeline and deploying painlessy. Infra is a pain point. Legacy environments are fragile as hell and good operations folks are few and far between, and most of them are undervalued and a bit weird. -- but you personally, oarsinsync, are going to better off on prem for a few simple reasons in that world: 1) Total Ownership: Something goes titsup? Yeah, your team owns the entire show up until your ISP. Network borks? Storage slows down? It's all yours. Dig in. AWS develops cross-az latency? Log a ticket and 2 days later when they respond you'll have already forgotten it happened. Explain that to the business. 2) Perf: Don't believe the rubes; you'll be quicker on your own gear if only because you can move your noisy VMs to sep machines, set various limits/quotes/throttles that you can't on a public cloud, and you're not sharing spindles with 300 mongo vms. 3) Expierence: It's not exactly a business goal, but you'll learn more when you're having to stace some jvm at 3:00am and crying into a toilet than if you're just bouncing instances and getting away with shit code. If you just outsource all your problems forever you'll never be able to have an opinion on them, even if that opinion is just "I never want to use RAC again" or "mongodb is death" or whatever..

kkapelon 10y ago

The article does not address important topics such as privacy and security. For some organizations using a private cloud is not a technical decision but instead a political/ethics one. As an example I would bet that there are several companies out there who would not care if all services were down for a full day (in their private cloud), as long as they were certain that no government could easily access them.

erelde 10y ago
GovernmentS can't host their services on Google's or Amazon's cloud for example. (the US are not counting since this is THE government many others are not trusting and the home of the public clouds)
- mritun 10y ago
  They already do: https://aws.amazon.com/govcloud-us/ https://aws.amazon.com/govcloud-us/
- us0r 10y ago
  Amazon's largest customer is the CIA. http://www.theatlantic.com/technology/archive/2014/07/the-details-about-the-cias-deal-with-amazon/374632/ http://www.theatlantic.com/technology/archive/2014/07/the-de...
  - Retric 10y ago
    That's a private cloud built by Amazon for the CIA and does not run on Amazon's network.
- kesor 10y ago
  The FUD is strong, but that was exactly the point of that article. Having old-fashioned people in goverments who disagree that they should ditch their datacenters and start using clouds, is backwards thinking. Just like goverments find ways to allow private companies to run their cellular networks, phone networks, tv cable networks, power grids - the same reasoning should allow private companies (public clouds) who do it much better than goverment employees, to run their compute services.

kitd 10y ago

I was musing yesterday whether a technologically-advanced nation state of the future might provide cloud infrastructure like it provides roads/rail/healthcare/etc. Obviously, this only makes sense in a world where individuals as well as corps need to present themselves online as a matter of course, but it might enable eg personal ID or data privacy tools that address many of the problems people have 'donating' their private data to commercial orgs.

adrianN 10y ago
A wet dream for police states. Distribute dumb terminals to government-run cloud servers for free and soon enough the only people who don't make their whole life transparent to the state are the criminals.
nickpsecurity 10y ago
It's possible. They'd probably do internet cafes and libraries first. That gives them visual surveillance on top of the Internet surveillance. Far as in U.S., there was one satirical poster someone made selling the NSA surveillance program as a free replacement for backup software. ;)

anon1253 10y ago

Yeah so we have a different story. We build Natural Language Processing and statistics software. We tried the cloud first, but it was just unworkable. Limited IOPS, outrageous prices for even a modest amount of RAM, slow virtualized CPUs, limited GPU options with terrible performance. So one day I hired half a rack, bought some machines with fast SSDs, 128GB RAM each, dual CPU with high clockspeed, and even some boxes with GPUs. The difference was incredible. Not just in performance, which was orders of magnitudes faster (going from barely workable to "pleasant user experience" without any software changes), but also in terms of cost. If we were to provision these machines on Azure or E2 we'd spend in a month what these machines cost, and our current current costs are maybe $500 in electricity + rack rental. Sure, it's a bit of a workout to put stuff in racks, but I actually quite enjoyed doing it. Sure stuff will break and things will go down. We can take the downtime at this point. And seriously, cost and performance wise there is no competition between cloud and rolling your own hardware (if you know how to do it). Also I'm perpetually surprised by how overstated the complexity of this is, yes it's tricky. But so is CSS, and Javascript these days. And unless you go "full cloud" (never go full cloud), you still have to manage fallover/redudancy/CI/provisioning yourself. It's not much different on "actual" hardware (there is no cloud, it's just somebody else's computer). If my firewall or switch catches fire, that might be problematic. But if that happens at an Amazon or Azure data center, that's also problematic. At the very least you'll suffer terrible latency because it needs to move the VMs dynamically.

cyberpunk 10y ago
Rock on anon' -- we feel the same way! Once you get above a few boxes though, you might need some pricey geeks for a month or two to rig up cobbler, ceph, openstack, proxmox, or whatever-takes-ya simply because it's quicker than getting up on all that yourself right away -- but really there is no magic involved here and it's just so you can have it totally manageable as you grow. The tooling is much more friendly these days; and to be totally honest -- provisioning probably isn't the problem you're trying to solve anyway (more likely packaging/deployment/ci is the pain after you got a bit of breathing space on perf). I've seen loads of folks coming back on prem spend months rigging up crazy vm provisioning things when their estate is basically static because they didn't realise they were solving the wrong problem. If your ops time is spent building machines, then sort that out. Probably and more likely it's spent on getting apps running and dealing with moving traffic about and deployments and such; so that's the real thing to work on. I think the 'complexity' argument seem so loud on HN as the opsy people are the minority here and it's not exactly a sexy/exciting/fun thing to pick up, and when it goes wrong it's brutal, especially when you can be nailing features instead. All I'd say is -- regardless of what you're doing, getting into containers at this point makes your life quite a lot easier; you get a standard deployable unit of code you can throw into aws, or on prem depending on your workload, and it removes a lot of the anti-patters of config mgt.

snarfy 10y ago

> Phase 3: Then deal with the stateful monoliths. These are your large monolithic databases. This is usually where private cloud journey hits the wall due to the risk and complexity in making such monoliths cloud native. We are tasked with moving to the cloud, and this is where everything stops. Over the years, instead of fixing the monolithic database design, the company instead kept beefing up the hardware for the database. A bigger cpu and more ram is cheaper up front than re-engineering the entire company infrastructure. And now we are paying for it. Our db hardware now is the highest end hardware the db software supports. None of the cloud providers db support can handle it. We are forced to fix the monolithic design if we are to move to the cloud. It will be a multi-year effort.

mooreds 10y ago
> It will be a multi-year effort And the company is willing to go through it because the public cloud offers so much? Just curious if this was going to stop the overall project or not.
- snarfy 10y ago
  We were bought out by investors and this is their major initiative. I assume it is to reduce long term operating costs.
cyberpunk 10y ago
My current gig had a similar problem; changing the DB or putting a layer in front of it at a large scale or really er 'mature' codebase is basically a rewrite, and that's not really the smart way to play it and it's probably not possible anyway. Start from the top instead of the data. Moving various bits of the API to new implementations/microservices and modernize the stack -- new features, make 'em all standalone services and try to deal with your logic in the app instead of in the db. Move /send/giant_dildo_to?uzer=cyberpunk out of the monolith and to a sep service, and eventually you might get to the point where not a huge amount of the legacy code is actually being hit by anything besides your new code. At that point, the cost of redoing the persistence bit is way less than it was when you started. Depending on your app, you might even get away with creating a new DB and using the orig as a kind of 'feeder' which you run on batch -- all options are open right? But obv it depends what you're doing.