12 ms·

I think a strong argument can be made either way. There's definitely financial savings to be had by doing it in-house (either literally or utilizing existing d

I think a strong argument can be made either way. There's definitely financial savings to be had by doing it in-house (either literally or utilizing existing data centers as here). But what I've experienced is that companies are often attracted to Big Cloud™ not just for financial reasons but to flatten/simplify their corporate reporting structure. It is the ultimate delegation. This isn't simply about having fewer staff (although that can reduce internal politics and inefficiencies), but making fewer decisions and more importantly reducing the potential for making the wrong decisions (which can be career-costly, regardless of right decisions made previously). These "soft" costs/savings are rarely discussed because they cannot be measured, unlike a balance sheet. But there's definitely a whole other dimension to infrastructure outsourcing that is worthy of note. In this case BoA are saving money but in exchange retain additional layers of staffing, that require oversight, and ultimately decision makers willing to take risks.

mywittyname 7y ago
Also reducing friction in getting products to market. A small team with their own BigCloud account is probably going to deliver a product to market much faster than one that needs to go through the friction of dealing with the various IT departments which invariably exist at large companies. There is absolutely value to delivering a product to market a year or more sooner than otherwise.
- McAtNite 7y ago
  The friction from various IT departments are often beneficial. I’m probably speaking from a point of bias as I’m one of those departments, but we often have to sanity check the implementation of things and review the security implications. I’ve encountered quite a few devs whose only focus is on achieving the specified goal without considering the full security scope. I don’t blame them, that’s what they are there for. Stepping in for ensuring the customer security and regulation requirements is what I’m here for.
- testplzignore 7y ago
  But at a large company, how likely is it that a team is going to have their own AWS account AND have free rein over it? You'll still have all of the friction for compliance and security. And turf battles. It's like using package managers like Maven and npm. Sure, they make going to market much faster for small organizations. But at companies that have strict security procedures, downloading unvetted stuff from the internets and putting it into production does not fly. You'll have internal repos that work with your package manager tools, but nothing gets into those repos without at least a cursory review.
  - mywittyname 7y ago
    But at a large company, how likely is it that a team is going to have their own AWS account AND have free rein over it? The two companies I've used AWS at both operated this way. In fact, each team has multiple product accounts (for dev, testing, production). So I don't think it's unusual at all.
- zaroth 7y ago
  This is a false dichotomy. There is the issue of restrictive IT departments, legal review, compliance review, extended QA, documentation, product literature, etc. all extending a product launch timeframe... Separate from all of that there is a valid question whether the software stack at a big cloud provider gives your engineering department the ability to deliver features faster at scale and redundancy, and with lower ongoing maintenance overhead. I think this question is very much up for debate. At a small scale there are absolutely cloud frameworks that let you deploy services without an IT department at large scale. But then there’s a realistic argument of whether the high premium for theoretical scale—when in actuality you’re serving dozens not millions of customers—is worth the high cost of capital. E.g. several startups of mine have persistent on a couple colocated dedicated servers for over a decade at a cost of $300/mo. The equivalent deployment in the cloud is closer to $3,000/mo. I know this because I got $300k of Azure credits to play around with and ended up not using most of them. The long-run cost was too high to move to Azure. But there were months when I “spent” $6k of credits on a dozen VMs and associated storage and network that had equivalent performance to my rented bare metal.
stiray 7y ago
...but they are also acumulating know how which they might be able to sell in decades to come. Even more if the scenario of mainframes, IBM software, Oracle databases,... will happen and sooner or later I believe it will. Currently cloud providers are operating on absolute minimum, to get as much customers as possible into their trap. And for sure, it is highly beneficial for startups that cant afford to buy on premise hardware, but this wont last forever. History has shown that immediately when you are vendor locked, you will be charged for your dependance. Even more if people beeing able to build on premise infrastructure are becoming rare beasts. I would be glad to see I am wrong but history proves it is best teacher.
- chatmasta 7y ago
  Operating at a minimum? How do you come to that conclusion? From a pure cost of infrastructure perspective (ignoring all the human elements), the cloud is always more expensive than any other option. It’s got some of the most ridiculous margins and price gouging of any business out there. I think the opposite will happen. Cloud providers are becoming commoditized. Kubernetes, blah blah. Developers hate lock-in, and on a long time scale, developers drive tech decisions. Nowadays new software is cloud agnostic, and it will continue to be that way. This is commoditization, which generally drives down prices. That said, buying commodities from a market with four participants is more like buying from a cartel than a marketplace. I definitely worry about the consolidation of the world’s IT spending going into the coffers of three or four massively profitable and morally corrupt corporations.
  - blantonl 7y ago
    It’s got some of the most ridiculous margins and price gouging of any business out there. You and enterprise software should get together and have a introductory meeting...
  - birdyrooster 7y ago
    Great call out for Kubernetes. I’m a full believer that it will turn cloud providers into dumb aggregations of compute/network/storage (like simple utility companies) for the deployment of CRDs and Operators, which will obviate the need for AWS services like EBS, Route 53, RDS, and many many other offerings. This will lower the cost to entry for competitors and lower prices for everyone.
zelon88 7y ago
> In this case BoA are saving money but in exchange retain additional layers of staffing, that require oversight, and ultimately decision makers willing to take risks. I think some companies, like BOA, look at this as the safer alternative. We all assume that large Cloud providers are bulletproof. That's just because we haven't seen them taken down yet. Someday we will. Just wait until there's another major world conflict. These global companies with outsourced IT will be in ruins because politics and digital DMZ's will keep them from acting cohesively. For an organization like a bank that's worth over $400b that layer of staffing, equipment, oversight, and all the inefficiencies are a $400b insurance policy that you will be able to continue operating regardless of geographic or political events outside of your control.
- dwoozle 7y ago
  Exactly. Jeff Bezos tweets something that gets Xi’s panties in a ruffle and AWS is blocked in China and Bank of America can’t do cross border payments... no way should such a big company have their eggs in such a basket.
- Someone1234 7y ago
  I believe you misread the quoted remark. It was about decision maker's personal (career) risk when making decisions. Keeping things in-house means more decisions are taken in-house which increases personal risk to the management for getting things wrong. In context this is much clearer: > This isn't simply about having fewer staff (although that can reduce internal politics and inefficiencies), but making fewer decisions and more importantly reducing the potential for making the wrong decisions (which can be career-costly, regardless of right decisions made previously). The post above doesn't discuss cloud Vs. non-cloud security/safety at all.
- TuringNYC 7y ago
  > In this case BoA are saving money but in exchange retain additional layers of staffing, that require oversight, and ultimately decision makers willing to take risks. I've been at multiple Fortune 50 companies at multiple levels. Many times these types of infrastructure building are just forms of empire building and an attempt to embed ones' services in the organization at a very deep level. You convert one set of costs (AWS/GCP/Azure bills) for another set of costs (Infrastructure + Payroll). You also convert one set of headaches (CSP overcharging/auditing) for another (prima donna internal infra engineers.)
tyfon 7y ago
> This isn't simply about having fewer staff I've always wondered about this. In IT now instead of having 5 system operators one has 10 "cloud consultants" and nobody knows what they are doing. I've been through a couple of "migrations to the cloud" now and I've never seen a reduction in IT staff or the cost of running IT operations. About the wrong decisioning part, if you chose the wrong "cloud" for your business the cost to change can be enormous. The primary motivation for management is from what I've seen so far the ability to point to someone else if shit hits the fan, not cost reduction and all that other stuff.
- Aperocky 7y ago
  And nobody got fired for purchasing IBM. You start wondering if 90% of corporate world are there to do nothing if not erecting roadblocks for people who are actually doing something. But then you start wondering maybe it's a good thing that all of these people get paid and could spend their money so that majority of the country has purchasing power and not live in piss poor poverty.
trabant00 7y ago
> This isn't simply about having fewer staff (although that can reduce internal politics and inefficiencies), but making fewer decisions and more importantly reducing the potential for making the wrong decisions This is a myth directly from cloud providers marketing campaigns: that using the cloud simplifies infrastructure to the point that you don't need as much and/or as qualified staff. In reality it is quite the trap: without experienced system administrators AWS and the like are free to dig their hands deeper and deeper into the pockets of their unsuspecting customers.
- jedberg 7y ago
  Yeah but you don’t need data center engineers, real estate specialists. Power specialists. Etc.
  - deleted 7y ago
    [deleted]
  - briandear 7y ago
    Nor do you have to pay the rent on the buildings where those things happen.
  - ddorian43 7y ago
    You can rent dedicated server or onprem ? people always go from cloud to building your own cpu in these threads, never in the middle
    - rhizome 7y ago
      Well spotted.
    - twa927 7y ago
      Yes, this also seems weird to me. And I think it might be a cultural thing, I noticed that in Europe renting dedicated servers is far more popular than in the US.
brightball 7y ago
IMO there's just a lot of factors that will vary from business to business.
jmartrican 7y ago
I worked on the internal BOA cloud. My team wrote software to procure VM's via a website, versus the older (still current) way of doing it manually. There are two things worth considering. The BOA cloud is slow. That is to say, it is hard to procure machines to get your projects going. There is a lot of control on who will be paying for them. There is also A LOT of staff to manage it. So those 2 Billion could easily be tweaked downward when you consider the staff and lost of agility. On the flip side, AWS is hella expensive. To do the things that they do it would cost A LOT of money. Maybe even more than they think. I have seen (and maybe a lot of people here also) small companies with million dollar AWS bills. So this number can also be tweaked up when you consider dozens of teams each having their way (at BOA scale) on the AWS console.
- cosmodisk 7y ago
  "I have seen (and maybe a lot of people here also) small companies with million dollar AWS bills." Ultimately,it doesn't matter, if the value it provides is greater than the AWS bill.
  - jrockway 7y ago
    I am not sure I agree with that. Overpaying for something is still overpaying even if it provides value. Do you spend 100% of your salary on food? You will die if you don't eat. But somehow, the food required to keep you alive probably costs less than your salary.
- m463 7y ago
  Maybe it can make sense for a small company trying to scale where time is a factor using aws. But I would expect that a large established company would do better to create it's own reasonably priced reasonably managed infrastructure. Sort of rent vs build/own.
  - tekknik 7y ago
    not necessarily. for each of these “we build our own cloud” stories there’s quite a few more stories of large enterprises saving tons switching away from their own cloud. BOA themselves know their savings won’t last and they’ll be switching back. They’re already in negotiations
burtonator 7y ago
There's also the argument that can be made for rapid iteration. A company like BoA will eventually choke until this in the future. Also, this stuff sounds great until you need improved peering, or software-defined networks or massive bandwidth improvements, etc. ... and I get that the HN community is clever but banks and other institutions like this REALLY struggle to find this talent. $2B is a lot except when you're losing $10B a year in revenue because you can't compete in the market. In a way this would be like Amazon arguing it should build its own bank. It's not what they're good at... Focus on your strengths.
75dvtwin 7y ago
Would the cost/savings also depend on the 'predictability' of the infrastructure needs? To me, it seems like there is a dependency. For example if I can project steady growth or decline of my needs, it seems that I benefit less from the infrastructure-on-demand like AWS/Azure/GC Also, depending on the size of the organization it seems that there at least several options on data centers: - have my own data center - collocate my hardware in other's data center - rent hardware - use infrastructure-on-demand + software-as-a-service AWS/Azure/GC are in 4th category, but cost savings in terms of electricity/building capacity/peering points are available in the first 3 as well. Many organizations deploy Pivotal's https://pivotal.io/industries/financial-services https://pivotal.io/industries/financial-services or Redhat/OpenStack So they are not actually 'building' the infrastructure management layer from scratch. Also for investment banks (due to exchange connectivity) , telecoms (due to peering points, physical tower locations) always going to have their 'Front-office' applications in data centers that offer lowest latency for those apps. Midoffice applications (or BOSS as they are labeled in telecom) can live in data centers reasonably distant (in terms of ping time) from the front office. So the infrastructure for those has a different cost profile. Most of this applications touch trades or trade events that carry millions of dollars worth of contract value (not Facebook's likes or favorites... ) There is a lot of logging, reconciliation, post executing enrichment that's probably going on. Finally, from my 2nd hand understanding, Investment banks can have from 3K to 10K applications in production. With, perhaps 20% of them having more than million lines of code. Some of those apps are built in their own proprietray languages developed specifically to manage contracts and lifecycle of contracts (Banks had 'smart contracts' DSL and interpreters since 90s). https://news.efinancialcareers.com/uk-en/282097/slang-goldman-sachs https://news.efinancialcareers.com/uk-en/282097/slang-goldma...
deleted 7y ago
[deleted]

mywittyname 7y ago

Also reducing friction in getting products to market. A small team with their own BigCloud account is probably going to deliver a product to market much faster than one that needs to go through the friction of dealing with the various IT departments which invariably exist at large companies. There is absolutely value to delivering a product to market a year or more sooner than otherwise.

McAtNite 7y ago
The friction from various IT departments are often beneficial. I’m probably speaking from a point of bias as I’m one of those departments, but we often have to sanity check the implementation of things and review the security implications. I’ve encountered quite a few devs whose only focus is on achieving the specified goal without considering the full security scope. I don’t blame them, that’s what they are there for. Stepping in for ensuring the customer security and regulation requirements is what I’m here for.
testplzignore 7y ago
But at a large company, how likely is it that a team is going to have their own AWS account AND have free rein over it? You'll still have all of the friction for compliance and security. And turf battles. It's like using package managers like Maven and npm. Sure, they make going to market much faster for small organizations. But at companies that have strict security procedures, downloading unvetted stuff from the internets and putting it into production does not fly. You'll have internal repos that work with your package manager tools, but nothing gets into those repos without at least a cursory review.
- mywittyname 7y ago
  But at a large company, how likely is it that a team is going to have their own AWS account AND have free rein over it? The two companies I've used AWS at both operated this way. In fact, each team has multiple product accounts (for dev, testing, production). So I don't think it's unusual at all.
zaroth 7y ago
This is a false dichotomy. There is the issue of restrictive IT departments, legal review, compliance review, extended QA, documentation, product literature, etc. all extending a product launch timeframe... Separate from all of that there is a valid question whether the software stack at a big cloud provider gives your engineering department the ability to deliver features faster at scale and redundancy, and with lower ongoing maintenance overhead. I think this question is very much up for debate. At a small scale there are absolutely cloud frameworks that let you deploy services without an IT department at large scale. But then there’s a realistic argument of whether the high premium for theoretical scale—when in actuality you’re serving dozens not millions of customers—is worth the high cost of capital. E.g. several startups of mine have persistent on a couple colocated dedicated servers for over a decade at a cost of $300/mo. The equivalent deployment in the cloud is closer to $3,000/mo. I know this because I got $300k of Azure credits to play around with and ended up not using most of them. The long-run cost was too high to move to Azure. But there were months when I “spent” $6k of credits on a dozen VMs and associated storage and network that had equivalent performance to my rented bare metal.

stiray 7y ago

...but they are also acumulating know how which they might be able to sell in decades to come. Even more if the scenario of mainframes, IBM software, Oracle databases,... will happen and sooner or later I believe it will. Currently cloud providers are operating on absolute minimum, to get as much customers as possible into their trap. And for sure, it is highly beneficial for startups that cant afford to buy on premise hardware, but this wont last forever. History has shown that immediately when you are vendor locked, you will be charged for your dependance. Even more if people beeing able to build on premise infrastructure are becoming rare beasts. I would be glad to see I am wrong but history proves it is best teacher.

chatmasta 7y ago
Operating at a minimum? How do you come to that conclusion? From a pure cost of infrastructure perspective (ignoring all the human elements), the cloud is always more expensive than any other option. It’s got some of the most ridiculous margins and price gouging of any business out there. I think the opposite will happen. Cloud providers are becoming commoditized. Kubernetes, blah blah. Developers hate lock-in, and on a long time scale, developers drive tech decisions. Nowadays new software is cloud agnostic, and it will continue to be that way. This is commoditization, which generally drives down prices. That said, buying commodities from a market with four participants is more like buying from a cartel than a marketplace. I definitely worry about the consolidation of the world’s IT spending going into the coffers of three or four massively profitable and morally corrupt corporations.
- blantonl 7y ago
  It’s got some of the most ridiculous margins and price gouging of any business out there. You and enterprise software should get together and have a introductory meeting...
- birdyrooster 7y ago
  Great call out for Kubernetes. I’m a full believer that it will turn cloud providers into dumb aggregations of compute/network/storage (like simple utility companies) for the deployment of CRDs and Operators, which will obviate the need for AWS services like EBS, Route 53, RDS, and many many other offerings. This will lower the cost to entry for competitors and lower prices for everyone.

zelon88 7y ago

> In this case BoA are saving money but in exchange retain additional layers of staffing, that require oversight, and ultimately decision makers willing to take risks. I think some companies, like BOA, look at this as the safer alternative. We all assume that large Cloud providers are bulletproof. That's just because we haven't seen them taken down yet. Someday we will. Just wait until there's another major world conflict. These global companies with outsourced IT will be in ruins because politics and digital DMZ's will keep them from acting cohesively. For an organization like a bank that's worth over $400b that layer of staffing, equipment, oversight, and all the inefficiencies are a $400b insurance policy that you will be able to continue operating regardless of geographic or political events outside of your control.

dwoozle 7y ago
Exactly. Jeff Bezos tweets something that gets Xi’s panties in a ruffle and AWS is blocked in China and Bank of America can’t do cross border payments... no way should such a big company have their eggs in such a basket.
Someone1234 7y ago
I believe you misread the quoted remark. It was about decision maker's personal (career) risk when making decisions. Keeping things in-house means more decisions are taken in-house which increases personal risk to the management for getting things wrong. In context this is much clearer: > This isn't simply about having fewer staff (although that can reduce internal politics and inefficiencies), but making fewer decisions and more importantly reducing the potential for making the wrong decisions (which can be career-costly, regardless of right decisions made previously). The post above doesn't discuss cloud Vs. non-cloud security/safety at all.
TuringNYC 7y ago
> In this case BoA are saving money but in exchange retain additional layers of staffing, that require oversight, and ultimately decision makers willing to take risks. I've been at multiple Fortune 50 companies at multiple levels. Many times these types of infrastructure building are just forms of empire building and an attempt to embed ones' services in the organization at a very deep level. You convert one set of costs (AWS/GCP/Azure bills) for another set of costs (Infrastructure + Payroll). You also convert one set of headaches (CSP overcharging/auditing) for another (prima donna internal infra engineers.)

tyfon 7y ago

> This isn't simply about having fewer staff I've always wondered about this. In IT now instead of having 5 system operators one has 10 "cloud consultants" and nobody knows what they are doing. I've been through a couple of "migrations to the cloud" now and I've never seen a reduction in IT staff or the cost of running IT operations. About the wrong decisioning part, if you chose the wrong "cloud" for your business the cost to change can be enormous. The primary motivation for management is from what I've seen so far the ability to point to someone else if shit hits the fan, not cost reduction and all that other stuff.

Aperocky 7y ago
And nobody got fired for purchasing IBM. You start wondering if 90% of corporate world are there to do nothing if not erecting roadblocks for people who are actually doing something. But then you start wondering maybe it's a good thing that all of these people get paid and could spend their money so that majority of the country has purchasing power and not live in piss poor poverty.

trabant00 7y ago

> This isn't simply about having fewer staff (although that can reduce internal politics and inefficiencies), but making fewer decisions and more importantly reducing the potential for making the wrong decisions This is a myth directly from cloud providers marketing campaigns: that using the cloud simplifies infrastructure to the point that you don't need as much and/or as qualified staff. In reality it is quite the trap: without experienced system administrators AWS and the like are free to dig their hands deeper and deeper into the pockets of their unsuspecting customers.

jedberg 7y ago
Yeah but you don’t need data center engineers, real estate specialists. Power specialists. Etc.
- deleted 7y ago
  [deleted]
- briandear 7y ago
  Nor do you have to pay the rent on the buildings where those things happen.
- ddorian43 7y ago
  You can rent dedicated server or onprem ? people always go from cloud to building your own cpu in these threads, never in the middle
  - rhizome 7y ago
    Well spotted.
  - twa927 7y ago
    Yes, this also seems weird to me. And I think it might be a cultural thing, I noticed that in Europe renting dedicated servers is far more popular than in the US.

brightball 7y ago

IMO there's just a lot of factors that will vary from business to business.

jmartrican 7y ago

I worked on the internal BOA cloud. My team wrote software to procure VM's via a website, versus the older (still current) way of doing it manually. There are two things worth considering. The BOA cloud is slow. That is to say, it is hard to procure machines to get your projects going. There is a lot of control on who will be paying for them. There is also A LOT of staff to manage it. So those 2 Billion could easily be tweaked downward when you consider the staff and lost of agility. On the flip side, AWS is hella expensive. To do the things that they do it would cost A LOT of money. Maybe even more than they think. I have seen (and maybe a lot of people here also) small companies with million dollar AWS bills. So this number can also be tweaked up when you consider dozens of teams each having their way (at BOA scale) on the AWS console.

cosmodisk 7y ago
"I have seen (and maybe a lot of people here also) small companies with million dollar AWS bills." Ultimately,it doesn't matter, if the value it provides is greater than the AWS bill.
- jrockway 7y ago
  I am not sure I agree with that. Overpaying for something is still overpaying even if it provides value. Do you spend 100% of your salary on food? You will die if you don't eat. But somehow, the food required to keep you alive probably costs less than your salary.
m463 7y ago
Maybe it can make sense for a small company trying to scale where time is a factor using aws. But I would expect that a large established company would do better to create it's own reasonably priced reasonably managed infrastructure. Sort of rent vs build/own.
- tekknik 7y ago
  not necessarily. for each of these “we build our own cloud” stories there’s quite a few more stories of large enterprises saving tons switching away from their own cloud. BOA themselves know their savings won’t last and they’ll be switching back. They’re already in negotiations

burtonator 7y ago

There's also the argument that can be made for rapid iteration. A company like BoA will eventually choke until this in the future. Also, this stuff sounds great until you need improved peering, or software-defined networks or massive bandwidth improvements, etc. ... and I get that the HN community is clever but banks and other institutions like this REALLY struggle to find this talent. $2B is a lot except when you're losing $10B a year in revenue because you can't compete in the market. In a way this would be like Amazon arguing it should build its own bank. It's not what they're good at... Focus on your strengths.

75dvtwin 7y ago

Would the cost/savings also depend on the 'predictability' of the infrastructure needs? To me, it seems like there is a dependency. For example if I can project steady growth or decline of my needs, it seems that I benefit less from the infrastructure-on-demand like AWS/Azure/GC Also, depending on the size of the organization it seems that there at least several options on data centers: - have my own data center - collocate my hardware in other's data center - rent hardware - use infrastructure-on-demand + software-as-a-service AWS/Azure/GC are in 4th category, but cost savings in terms of electricity/building capacity/peering points are available in the first 3 as well. Many organizations deploy Pivotal's https://pivotal.io/industries/financial-services https://pivotal.io/industries/financial-services or Redhat/OpenStack So they are not actually 'building' the infrastructure management layer from scratch. Also for investment banks (due to exchange connectivity) , telecoms (due to peering points, physical tower locations) always going to have their 'Front-office' applications in data centers that offer lowest latency for those apps. Midoffice applications (or BOSS as they are labeled in telecom) can live in data centers reasonably distant (in terms of ping time) from the front office. So the infrastructure for those has a different cost profile. Most of this applications touch trades or trade events that carry millions of dollars worth of contract value (not Facebook's likes or favorites... ) There is a lot of logging, reconciliation, post executing enrichment that's probably going on. Finally, from my 2nd hand understanding, Investment banks can have from 3K to 10K applications in production. With, perhaps 20% of them having more than million lines of code. Some of those apps are built in their own proprietray languages developed specifically to manage contracts and lifecycle of contracts (Banks had 'smart contracts' DSL and interpreters since 90s). https://news.efinancialcareers.com/uk-en/282097/slang-goldman-sachs https://news.efinancialcareers.com/uk-en/282097/slang-goldma...

deleted 7y ago

[deleted]