16 ms·

Without a doubt, a future generation of executives will revisit and reverse the decision to rent all information infrastructure, but that will likely be many, m

by cs702 4y ago

Without a doubt, a future generation of executives will revisit and reverse the decision to rent all information infrastructure, but that will likely be many, many years down the road. In the meantime, the current generation of executives who made this decision will look very smart for saving the company lots of money for a good number of years. And they stand to benefit personally from it. They're doing the rational thing!

noobermin 4y ago
It's amazing there are no other voices in the room pushing back against this. It really makes you wonder how companies even function at all, much less make money. Being so short sided never ends well.
- jsiepkes 4y ago
  Convincing people future risks outweigh the short term gains is very hard. That goes for all things.
- numbsafari 4y ago
  There are very likely voices, but they didn’t win this battle.
- cs702 4y ago
  Actually, I don't think the decision is that shortsighted in this case. It could work out pretty well for a decade or longer. IMHO FedEx is likely to have a good run before the costs and risks of this decision start outweighing its benefits.
  - masswerk 4y ago
    However, when the situation starts to become unbearable, how long will it take to rebuild the infrastructure and source talent? Another decade? Adding the startup costs for any possible rebuild, you're apt for a substantial net loss. So you're probably going to shy away from such a reintegration, while losses accumulate further, until this becomes unsustainable, putting the entire corporation at risk. (The wise thing to do to mitigate financial impact may actually be starting the reintegration process right now.)
    - beckingz 4y ago
      $400m saved today in exchange for $500m 10 years from now sounds like a good deal financially! Sometimes tech debt can be used instead of financial debt!
      - masswerk 4y ago
        Mind that if you'd ever wanted to integrate again, this means buying land, planning infrastructure, building it, planning, obtaining and setting up hardware and software, hiring, training, defining and testing procedures, etc, as you're starting from zero again – and all this while the costs, which forced you to consider this move, are piling up. Odds are, you'll never do this and cloud providers will know this. The comparative costs are now not those of running your own infrastructure, but those of setting it up (again).
        scottyah 4y ago
        FedEx had very few on site people managing the hardware, it seemed like most work on hardware was done by the suppliers. There were also very few servers there and actually running, I think the software powering the enterprise took up a lot less than they expected. Either way, I think you're spot on with the "training" part. FedEx was one of the first companies to raise significant capital and pursue a business dependent on technology. It also treated its people very well so relatively few left since the 70's. I think they just didn't train the next generation enough and they realized that those skills are disappearing rapidly because no college grads want to learn old & boring stuff and everyone who does know it costs a lot to keep (FedEx still heavily relied on COBOL when I left ~5yrs ago).
        kragen 4y ago
        I deeply appreciate you sharing this context. What do you think about my wild predictions of disaster?
    - bluGill 4y ago
      Will it? If they do this right it won't. There is a large cost running a data center. So long as long as there is competition they are better off with experts doing computers while they focus on logistics which is what they do well. Of course they should make some effort to ensure there is competition. That means they are careful to ensure more than one provider exists even if it means taking a slightly higher priced option. Also contracts end early if the company is bought out. (that is if AWS, buys google cloud - see above about ensuring there are competitors in the market)
- oblio 4y ago
  The optimize for the near term and after a while some other company optimizing for their near terms as startups stumble upon an awesome new product-market fit and upend the previous companies. There is no grand plan, there's just evolution.
- lupire 4y ago
ithkuil 4y ago
Tragedy of the executives
tunap 4y ago
Sounds a lot like farming out NOC-Ops to Wipro. Somebody had to think it was a good idea & managed to sell it to the policy makers for a nice promotion. So, when reality hits, does the same person get demoted further down than their original ladder rung?
- goldenkey 4y ago
  They are gone by that point, with a padded resume to boot :-)
kragen 4y ago
I don't think it will be a future generation of executives; I think it's the current generation of executives at companies and agencies like Google, Amazon, Microsoft, and the NSA who will not rent all their information infrastructure. I agree that the executives who thus turn their companies into sharecroppers on land owned by Microsoft et al. will be richly rewarded despite the misfortune that befalls their stockholders; like Stephen Elop, if things go badly, they have an impeccable excuse.
- cs702 4y ago
  Thanks. I think it will have to be a future generation of executives whose egos won't be tied up in old debates and who therefore will be more willing to sacrifice old decisions. If things go badly, that future generation of executives will take over sooner.
goindeep 4y ago
I used to work with a very smart man that I'm sure was some kind of secret genius. He's was that sort of tech gofer. Hardware, software, didn't matter, if there was a problem he'd solve it. Sort of guy you'd see carrying a thick ass SQL book around because he 'needed to learn it' to solve just one little problem. He built whole entire solutions for the company I worked at in his spare time that the company once tried to sell for 500k and at a previous company I heard he figured out a way for the pain mixing machines to save on paint or recycle it or something saving them 1.3 Mil a year. When Raspberry Pis first came out he was one of the first people I saw tinkering with them and he was in his 50's doing it just for fun, I think he ended up using it to open and close his garage door from work or something just to scare his wife. That sort of guy. Well he once told me something about executives and upper managers working for corporations that I have never forgotten. He said to me, and of course I am paraphrasing: "Change gives the illusion of progress". I asked him what he meant and he responded with something to the effect of "They have the habit of changing big things every 5-10 years on purpose to make it look like they are productive, and to justify their own roles, one guy will come in and 'cut costs', the new guy after him will 'invest'".
- oblio 4y ago
  Never confuse movement for action :-)
  - hef19898 4y ago
    As everybody who saw people, and orgs, rotating in place at incredible speeds can confirm.
  - prometheus76 4y ago
    This isn't always easy to determine, especially if you are part of the movement. Often, we move by dead reckoning and only after some amount of time can we determine if what we did was "movement" or "progress".
- smitty1e 4y ago
  In this sense, Oscar Wilde was the executive's executive: "I have spent most of the day putting in a comma and the rest of the day taking it out."
VBprogrammer 4y ago
I think you are absolutely right, however when they do reverse the decision they will publish a great article about how bringing their datacentres in-house is going to save them $800m a year and a bunch of execs will grant themselves a bigger bonus for saving the company so much money. It's a win-win!
- prirun 4y ago
  It's quite possible that at today's prices, they will save $400M/yr, and in the future, as cloud vendors raise prices, bringing it in-house will save them $800M/yr.
  - VBprogrammer 4y ago
    Not only that, it's possible that cloud vendors don't charge an extra penny but that the their software grows to consume the virtually limitless computing resources available to them costing them significantly more than expected. It wouldn't take the most creative exec in the world to make a plausible looking case for changing in either direction even based on the same numbers. A bit of wishful thinking about which costs are included or excluded is probably more than enough.
pid-1 4y ago
That seems to be a common take on HN - cloud is too expensive. I'm curious whether the folks claiming that have any data center ops experience. Because, personally, I'd rather retire than deal with Dell, HP, Cisco, fibers, cooling issues, physical security, hardware failling... And that's just the hardware. Then you still need to pay VMWare for a decent virtualization platform, monitoring tools, etc.... Seriously, no amount of money would make me work in a DC again. I believe companies selling bare metal as a service are a happy compromise of cost and convenience, though.
- bluGill 4y ago
  To a company you need to pay those costs no matter what. If the AC breaks at 3am it neds to be fixed. It doesn't matter if you have your own HVAC people on sight 24x7, your own people on call to service it, a local HVAC service to come in, or you outsource the entire operations and so you have no idea how that is handled. In the end the important part of this story is that whatever you are doing with the AC continues to work. Different operations demand different levels of service (I doubt that anyone keeps HVAC techs on staff 24x7, but if the AC is that critical it is mandatory). The only case where the CEO is up at 3am is if the CEO is the owner of the local HVAC service company, not the CEO of the building with the problem. Once you realize that to management the cost is outsourced no matter what the only question is do it with your own people and HR, or outsource it. There are pros and cons to both approaches, but for most companies it isn't their business and so the only reason to do it in house is they can't trust any company they hire.
  - SnowHill9902 4y ago
    Not really. Time to service depends on SLA and redundancy. If you have no redundancy your time to service must be less or equal than your SLA. If you have redundancy it can be longer.
  - mschuster91 4y ago
    The thing is, the cost for the HVAC 24x7x365 support for a datacenter will be roughly the same for a given location... but it makes a difference if it is you paying the whole bill (=you're self-hosting in your own datacenter), you are splitting the bill with a bunch of other customers indirectly (=you're self-hosting in a colo DC), or if you're splitting the bill with a shitload of other customers (=you're using some service on one of the big public cloud providers). The downside for saving the costs is that you're losing control with every step taken away: as soon as you go into a datacenter of any kind, you simply cannot call up a HVAC company and offer them 100k in hard cash if they're showing up in the next 60 minutes and fix the issue. With a colo DC you can usually go and show up there to see if the HVAC, UPS and other systems are appropriate to your needs, but with one of the big cloud providers you have to trust their word that they are doing stuff correctly.
jes 4y ago
You are right about this. One of the visible signs of an unrecognized (and therefore, unresolved) dilemma is oscillation between two poles. I got this from the late Eli Goldratt and the problem-solving tools he created. One of those tools is the "Evaporating Cloud," which is a way to visualize a dilemma of some kind. I'd suggest that there is an unresolved dilemma here: Should we rent or own data centers? Over a period of years or decades, you can watch these things flip-flop between two extremes. It's interesting to me that this dilemma is kind of like a specialization (in OO terminology) of a more generic dilemma, which might be something like: "In general, do we want to own or rent the things we need to run our business?" If your turn your head a bit, close your eyes a bit and squint, you can see how a dilemma like this can apply to things like "What should be our policy regarding employees vs. contractors? Should we try to hire and retain over a long period of time, or should we rent the people we need to get the job done and release them as soon as we are done with them?" The overall point is that whenever you see flip-flopping between two poles, think "Unrecognized dilemma." Sorry if this is vague.
- hinkley 4y ago
  Often it’s simply the principle of the excluded middle. If not for the rent seeking behavior of AWS and its cohort, the answer to ”own or rent” would be clear. The answer is “yes”. Running a data center means you have your own sheep, which means you need shepherds. Shepherds are very useful to have when you have a question about sheep, especially when those questions are about how to manage or use sheep to best effect. An approachable shepherd can save the rest of the company a lot of money on missteps and bad assumptions, but if you start getting rid of all the sheep, the shepherd will leave too. We should be using cloud providers for DR, and for regional load balancing. But the company should be maintaining at least one data center of their own, in the same time zones as most of their developers. I mostly blame Dell and IBM for this. IBM experimented with making server rooms easier to maintain 15-20 years ago and didn’t make it stick. Others ran with some of those ideas. Dell… I don’t know what Dell has done but I know nobody has been writing about it, so from a visibility standpoint they have done nothing. If/when someone makes it easier (reduced labor) to manage your own servers, the pendulum will swing back.
- projektfu 4y ago
mbesto 4y ago
> rent all information infrastructure Most of them were renting most of the infra anyway: - Co-location (physical space rented) - Managed service to keep the lights on (power, back-up generators) - Leased hardware that gets replaced every 3 years - Managed service for switch, firewall, etc. monitoring - ISP, back-up generators, etc. > but that will likely be many, many years down the road They're going to explain to their future bosses that buying land, building a huge building, buying servers, figuring out cooling, setting up redundant ISPs, etc. etc. is somehow going to be smarter? Explain that one to me?
- ocdtrekkie 4y ago
  You're paying for all of that whether it's the cloud or on-prem. The only real difference if you're a large company is whether you're paying another company's profit margin as well. If you are big enough to have your own datacenter, you are paying Amazon enough to buy that much physical space, power, bandwidth, IT staff, etc. plus funding Bezos' trips to space. There's a justifiable niche where you're too small to justify running your own server/below the need of one full-time IT staff where the cloud makes sense, and startups temporarily benefit from the ability to rapidly scale on cloud platforms. But any Fortune 500 transitioning to the cloud is literally just taking their money and burning it, because the alleged cost savings of efficiencies of scale are being completely absorbed by the cloud provider's profit margins. And they're still going to end up paying their own IT staff to handle the cloud management in addition to (by proxy) paying the cloud provider's IT staff to manage the hardware.
  - nathanielarmer 4y ago
    Today I learned that economies of scale and specialization are not things.
    - noobermin 4y ago
      Please whenever you envoke any model, have a sense of scale or region of applicability of the model, no model is true in all cases unless you're religious. The point the GP is making is at the scale of a F500 it starts to make less sense especially if you already have the infrastructure, the expertise, the experience, and so on. AWS is going to manage your VM better than you at your 3 person start up, but the economy of scale / specialization arguments work less well when you have an experienced workforce and developed system in place already and you're one of the world's largest logistics companies.
synergy20 4y ago
Some might return to on-premise data center, but most will not. It's really just like the utilities these days, 98% people will not dig their own well and install some sewage system and buy a generator, but big companies can still opt to have them on-premise installed, even though some of those are just backup systems. there is no return for the cloud for 98% of us.
- fires10 4y ago
  You generally can not go from city water/sewage to on-prem for legal reasons. However, many are going on-prem with solar. I am going almost completely off-grid for cost and SLA requirements.
  - dehrmann 4y ago
    What's interesting is people seem really excited about installing rooftop solar, but I see fewer companies building collectors in places with cheap land and ample sun. That tells me part of the selling point of rooftop solar is it feels good, and the economics might not actually work.
    - lupire 4y ago
      Captured Solar (electricity in general) doesn't travel well over long distances. Generating solar for local use saves the cost of transport.
      - dehrmann 4y ago
        > electricity in general doesn't travel well over long distances. High-voltage line losses can be as low as 3% per 1,000 km. That should be easily offset by the better location and better angle/tracking.
    - marcosdumay 4y ago
      > but I see fewer companies building collectors in places with cheap land and ample sun You mean there are fewer companies making large investments than smaller ones?
WFHRenaissance 4y ago
Great concrete example of the principal-agent problem here.
gravypod 4y ago
> that will likely be many, many years down the road And this is where people in my age range (20-30) will get to swoop in if we invest the next 10-20 years learning about and tinkering with server hardware. I highly recommend it for everyone.
jollybean 4y ago
They are doing the rational thing and there is no going back. HNers need to get their heads around this: the economies of the could represent a fundamental, secular shift. Imagine if Fedex designed and made their own delivery vehicles. Then they 'outsourced' that to Ford/GM. You might say 'look at Ford's amazing margins, look at the money being left on the table' - but in reality, it's still more cost effective to 'buy vehicles' the to 'DIY' them. In the 'long run' those Azure/AWS margins will erode - and/or - the long tail will creep up on them. Basically data centers offering less, for less, and companies realizing they don't need the complexity of AWS do do basic things. What that will look like is hard to say. With hardware it meant 'pivot to Asia'. Will this happen again? Chinese companies creating large data centres in the US with minimal staffing, monitored and operated out of China? If we were not in a giant geostrategic kerfluffle with them, then yes, I would say that is the future. But given security issues etc. it's probably not. Can India do that? Maybe.
- kragen 4y ago
  Ford and GM and Freightliner are not in a position to use price discrimination to confiscate FedEx's entire profits; if they try to sell trucks to FedEx at an inflated price, FedEx can just buy the same trucks from dealers, or buy them used, or buy GM trucks instead of Freightliner trucks. By contrast, generally speaking, cloud providers are in an excellent position to use price discrimination to extract the entire surplus value of every transaction. And the flexibility and reliability provided by FedEx will to a significant extent simply be that of their computers. (As chrisseaton pointed out, Boeing is in a cloudier position.)
  - jollybean 4y ago
    So 100% yes, AWS has much more power in the value chain than Ford. But not really. Especially to the extent that special services are not leveraged, then there are cheaper alternatives. A lot of IT is just 'instances' not 'fancy cloud queues'. AWS pricing is very transparent and people can make the decisions they want. Generally speaking, AWS does not price discriminate on the basis of 'business model' and their margins are meaningless with respect to the overall business efforts of a Big Co. I mean - unless you are doing big AI crunching, or 'free hosting content' - then frankly the AWS bill is not going to be a huge line item relative to overall operating costs.
adamc 4y ago
This is one of the things bad about Wall Street: Short-term decision-making. Everybody cares what happens to stock prices in the near-term, rather than 10 years out.
dehrmann 4y ago
I disagree. This isn't FedEx's core competency or a differentiator for them. They're also not at a scale where it could make sense to colocate or build their own DCs. Fiddling with low-level tech infra would just be a distraction. Now, they might find that software is a differentiator for them, but that's different.
- dx034 4y ago
  For a logistics company, the IT system is the differentiator. Reliably routing and tracking packages to maximize utilization of trucks and planes allows them to be cheaper and faster. That all depends on both hardware and software. So I'd definitely think that running their own systems should be a given.
deleted 4y ago
[deleted]
systemvoltage 4y ago
Some are starting to question the almighty Cloud™: https://www.economist.com/business/2021/07/03/do-the-costs-of-the-cloud-outweigh-the-benefits https://www.economist.com/business/2021/07/03/do-the-costs-o... One of the things that bother me about AWS is that I still need to manage their risk of a data center going down by using multiple availability zones and managing the complexity of extra resilience. There is a conflict of interest: AWS has a perverse incentive in keeping a single AZ robust and unless there is a natural calamity, it should not go down. I thought one of the major reasons of going to cloud is that you need not manage risk and offloading it from on-prem.
bhewes 4y ago
At least in Oil and Gas. Servers moved back into the personal towers for the first time in a decade. Latency issues.
- smn1234 4y ago
  AWS Outposts ?
  - bhewes 4y ago
    Ah did not know about AWS outposts. They use Azure so hybrid setup and moved the latency sensitive servers back into the tower everything else lives on Azure.
walrus01 4y ago
reminds me a bit of: https://www.reddit.com/r/AskHistorians/comments/vqr30e/jack_welch_extracted_record_profits_from_ge_for/ https://www.reddit.com/r/AskHistorians/comments/vqr30e/jack_... "Jack Welch extracted record profits from GE for 20 years, but left it a hollowed-out "pile of shit," according to his successor. What exactly did Welch do that was so damaging, and how did he get away with it for so long?" from the post reply, in case it's not available from the URL: alecsliu · 2 days ago · edited 2 days ago Gold2Eureka!Bravo!Today I Learned Welch took over a GE that was at the time, a major company. At the time, he viewed GE as bloated and needing to change. While he might've been right about that, the approach he took was perhaps less ideal. One of the worst things he helped make commonplace among American companies is the concept of stack ranking. The way it worked was like this: people were divided into three groups: A, B and C. A's were the top performers who needed to be rewarded generously, B were adequate performers who should be allowed to stay, and then C were those who needed to be fired. So far so good right? Well, not exactly. For Welch, these three buckets could be separated into the top 20%, the middle 70%, and the bottom 10% (20/70/10). Based on the above points, the bottom 10% would thus need to be fired annually. In the short term, this helped in making the company lean and look more productive, increasing the bottom-line and portraying an image of success. However, the long term consequences of such a change was cultural degradation and the introduction of new bloat and waste (running all of those performance reviews and firing and rehiring so many people takes a lot of money.) Consider the case of a perfectly adequate team: the entire team has achieved their targets and has contributed to the company as per their job description. The issue? In stack ranking, 10% of this team would still have to be fired even though everyone did their job. Unsurprisingly, the introduction of this competitive atmosphere where it isn't enough to succeed, one must be better than their peers, results in backstabbing, competition, and a host of issues which eventually weaken a company's competitive edge. This was only a part of Welch's general treatment of workers as numbers rather than humans. Aggressive cost-cutting, offshoring, etc. were the norm under Welch's regime and he would destroy entire divisions. This again, was great in the short-term but bad in the long term. Welch clearly had a dim view of company culture and believed it to be unimportant. The other major issue of Welch's was the acquisition of hundreds and hundreds of businesses, as part of Welch's goal of acquiring his way to the top. On the surface, this is what Welch did; on paper, he didn't destroy the main profit makers of GE so much as create new ones, primarily in the form of its financial arm. The result was that this allowed GE to play with the numbers in a way that allowed him to make sure that GE was always meeting targets set by Wall Street; in order to generate the right earnings, simply buy or sell certain assets, write-off others, etc. and when your company is an acquisition machine, it's not too hard to find the right numbers. This process helped expand GE into a mega-conglomerate but again, ultimately left the company in a weaker than expected state. In particular, on paper the core business of GE became its financial arm, as that was where all the funny business with the numbers was happening (nothing explicitly illegal though). Ignoring the damage the Welch did to GE's profit centers through his horrible business practices, this was a huge part of why GE declined so rapidly in the years following. GE's valuation was based on an inaccurate picture of its profits and value, so when the truth started coming out (especially with the Great Recession collapsing financial services profits), GE was quick to follow. As for why Welch was able to get away with it all? Well, the answer was because he was delivering. He hit the earnings targets, he made the board and shareholders very happy, by all accounts GE was the paragon of success and everything was going right. What Welch did was unprecedented, and it's hard to really understate that. For all of his faults, he had America tricked into believing that what he could do was unique and that he could avoid the realities of the economy and cyclical markets, that no matter what was going on, GE was special. Welch died a rich, rich man and many, many people profited greatly off of GE during its 20 year bull-run. Edit: My off-the-cuff writing is always horrible wrt to grammar and structure so I'll probably edit it later haha
Jistern 4y ago
dhzhzjsbevs 4y ago
We have our own datacentres. Lucky enough to have got them up and running pre cloud era. If we were any later we would be full cloud but the reality is we outcompete everyone on price in our space because of it.
pauljurczak 4y ago
And they will gently depart under their beautiful golden parachutes. What a racket.

by cs702 4y ago

noobermin 4y ago

It's amazing there are no other voices in the room pushing back against this. It really makes you wonder how companies even function at all, much less make money. Being so short sided never ends well.

jsiepkes 4y ago
Convincing people future risks outweigh the short term gains is very hard. That goes for all things.
numbsafari 4y ago
There are very likely voices, but they didn’t win this battle.
cs702 4y ago
Actually, I don't think the decision is that shortsighted in this case. It could work out pretty well for a decade or longer. IMHO FedEx is likely to have a good run before the costs and risks of this decision start outweighing its benefits.
- masswerk 4y ago
  However, when the situation starts to become unbearable, how long will it take to rebuild the infrastructure and source talent? Another decade? Adding the startup costs for any possible rebuild, you're apt for a substantial net loss. So you're probably going to shy away from such a reintegration, while losses accumulate further, until this becomes unsustainable, putting the entire corporation at risk. (The wise thing to do to mitigate financial impact may actually be starting the reintegration process right now.)
  - beckingz 4y ago
    $400m saved today in exchange for $500m 10 years from now sounds like a good deal financially! Sometimes tech debt can be used instead of financial debt!
    - masswerk 4y ago
      Mind that if you'd ever wanted to integrate again, this means buying land, planning infrastructure, building it, planning, obtaining and setting up hardware and software, hiring, training, defining and testing procedures, etc, as you're starting from zero again – and all this while the costs, which forced you to consider this move, are piling up. Odds are, you'll never do this and cloud providers will know this. The comparative costs are now not those of running your own infrastructure, but those of setting it up (again).
      - scottyah 4y ago
        FedEx had very few on site people managing the hardware, it seemed like most work on hardware was done by the suppliers. There were also very few servers there and actually running, I think the software powering the enterprise took up a lot less than they expected. Either way, I think you're spot on with the "training" part. FedEx was one of the first companies to raise significant capital and pursue a business dependent on technology. It also treated its people very well so relatively few left since the 70's. I think they just didn't train the next generation enough and they realized that those skills are disappearing rapidly because no college grads want to learn old & boring stuff and everyone who does know it costs a lot to keep (FedEx still heavily relied on COBOL when I left ~5yrs ago).
        kragen 4y ago
        I deeply appreciate you sharing this context. What do you think about my wild predictions of disaster?
  - bluGill 4y ago
    Will it? If they do this right it won't. There is a large cost running a data center. So long as long as there is competition they are better off with experts doing computers while they focus on logistics which is what they do well. Of course they should make some effort to ensure there is competition. That means they are careful to ensure more than one provider exists even if it means taking a slightly higher priced option. Also contracts end early if the company is bought out. (that is if AWS, buys google cloud - see above about ensuring there are competitors in the market)
oblio 4y ago
The optimize for the near term and after a while some other company optimizing for their near terms as startups stumble upon an awesome new product-market fit and upend the previous companies. There is no grand plan, there's just evolution.
lupire 4y ago

ithkuil 4y ago

Tragedy of the executives

tunap 4y ago

Sounds a lot like farming out NOC-Ops to Wipro. Somebody had to think it was a good idea & managed to sell it to the policy makers for a nice promotion. So, when reality hits, does the same person get demoted further down than their original ladder rung?

goldenkey 4y ago
They are gone by that point, with a padded resume to boot :-)

kragen 4y ago

I don't think it will be a future generation of executives; I think it's the current generation of executives at companies and agencies like Google, Amazon, Microsoft, and the NSA who will not rent all their information infrastructure. I agree that the executives who thus turn their companies into sharecroppers on land owned by Microsoft et al. will be richly rewarded despite the misfortune that befalls their stockholders; like Stephen Elop, if things go badly, they have an impeccable excuse.

cs702 4y ago
Thanks. I think it will have to be a future generation of executives whose egos won't be tied up in old debates and who therefore will be more willing to sacrifice old decisions. If things go badly, that future generation of executives will take over sooner.

goindeep 4y ago

I used to work with a very smart man that I'm sure was some kind of secret genius. He's was that sort of tech gofer. Hardware, software, didn't matter, if there was a problem he'd solve it. Sort of guy you'd see carrying a thick ass SQL book around because he 'needed to learn it' to solve just one little problem. He built whole entire solutions for the company I worked at in his spare time that the company once tried to sell for 500k and at a previous company I heard he figured out a way for the pain mixing machines to save on paint or recycle it or something saving them 1.3 Mil a year. When Raspberry Pis first came out he was one of the first people I saw tinkering with them and he was in his 50's doing it just for fun, I think he ended up using it to open and close his garage door from work or something just to scare his wife. That sort of guy. Well he once told me something about executives and upper managers working for corporations that I have never forgotten. He said to me, and of course I am paraphrasing: "Change gives the illusion of progress". I asked him what he meant and he responded with something to the effect of "They have the habit of changing big things every 5-10 years on purpose to make it look like they are productive, and to justify their own roles, one guy will come in and 'cut costs', the new guy after him will 'invest'".

oblio 4y ago
Never confuse movement for action :-)
- hef19898 4y ago
  As everybody who saw people, and orgs, rotating in place at incredible speeds can confirm.
- prometheus76 4y ago
  This isn't always easy to determine, especially if you are part of the movement. Often, we move by dead reckoning and only after some amount of time can we determine if what we did was "movement" or "progress".
smitty1e 4y ago
In this sense, Oscar Wilde was the executive's executive: "I have spent most of the day putting in a comma and the rest of the day taking it out."

VBprogrammer 4y ago

I think you are absolutely right, however when they do reverse the decision they will publish a great article about how bringing their datacentres in-house is going to save them $800m a year and a bunch of execs will grant themselves a bigger bonus for saving the company so much money. It's a win-win!

prirun 4y ago
It's quite possible that at today's prices, they will save $400M/yr, and in the future, as cloud vendors raise prices, bringing it in-house will save them $800M/yr.
- VBprogrammer 4y ago
  Not only that, it's possible that cloud vendors don't charge an extra penny but that the their software grows to consume the virtually limitless computing resources available to them costing them significantly more than expected. It wouldn't take the most creative exec in the world to make a plausible looking case for changing in either direction even based on the same numbers. A bit of wishful thinking about which costs are included or excluded is probably more than enough.

pid-1 4y ago

That seems to be a common take on HN - cloud is too expensive. I'm curious whether the folks claiming that have any data center ops experience. Because, personally, I'd rather retire than deal with Dell, HP, Cisco, fibers, cooling issues, physical security, hardware failling... And that's just the hardware. Then you still need to pay VMWare for a decent virtualization platform, monitoring tools, etc.... Seriously, no amount of money would make me work in a DC again. I believe companies selling bare metal as a service are a happy compromise of cost and convenience, though.

bluGill 4y ago
To a company you need to pay those costs no matter what. If the AC breaks at 3am it neds to be fixed. It doesn't matter if you have your own HVAC people on sight 24x7, your own people on call to service it, a local HVAC service to come in, or you outsource the entire operations and so you have no idea how that is handled. In the end the important part of this story is that whatever you are doing with the AC continues to work. Different operations demand different levels of service (I doubt that anyone keeps HVAC techs on staff 24x7, but if the AC is that critical it is mandatory). The only case where the CEO is up at 3am is if the CEO is the owner of the local HVAC service company, not the CEO of the building with the problem. Once you realize that to management the cost is outsourced no matter what the only question is do it with your own people and HR, or outsource it. There are pros and cons to both approaches, but for most companies it isn't their business and so the only reason to do it in house is they can't trust any company they hire.
- SnowHill9902 4y ago
  Not really. Time to service depends on SLA and redundancy. If you have no redundancy your time to service must be less or equal than your SLA. If you have redundancy it can be longer.
- mschuster91 4y ago
  The thing is, the cost for the HVAC 24x7x365 support for a datacenter will be roughly the same for a given location... but it makes a difference if it is you paying the whole bill (=you're self-hosting in your own datacenter), you are splitting the bill with a bunch of other customers indirectly (=you're self-hosting in a colo DC), or if you're splitting the bill with a shitload of other customers (=you're using some service on one of the big public cloud providers). The downside for saving the costs is that you're losing control with every step taken away: as soon as you go into a datacenter of any kind, you simply cannot call up a HVAC company and offer them 100k in hard cash if they're showing up in the next 60 minutes and fix the issue. With a colo DC you can usually go and show up there to see if the HVAC, UPS and other systems are appropriate to your needs, but with one of the big cloud providers you have to trust their word that they are doing stuff correctly.

jes 4y ago

You are right about this. One of the visible signs of an unrecognized (and therefore, unresolved) dilemma is oscillation between two poles. I got this from the late Eli Goldratt and the problem-solving tools he created. One of those tools is the "Evaporating Cloud," which is a way to visualize a dilemma of some kind. I'd suggest that there is an unresolved dilemma here: Should we rent or own data centers? Over a period of years or decades, you can watch these things flip-flop between two extremes. It's interesting to me that this dilemma is kind of like a specialization (in OO terminology) of a more generic dilemma, which might be something like: "In general, do we want to own or rent the things we need to run our business?" If your turn your head a bit, close your eyes a bit and squint, you can see how a dilemma like this can apply to things like "What should be our policy regarding employees vs. contractors? Should we try to hire and retain over a long period of time, or should we rent the people we need to get the job done and release them as soon as we are done with them?" The overall point is that whenever you see flip-flopping between two poles, think "Unrecognized dilemma." Sorry if this is vague.

hinkley 4y ago
Often it’s simply the principle of the excluded middle. If not for the rent seeking behavior of AWS and its cohort, the answer to ”own or rent” would be clear. The answer is “yes”. Running a data center means you have your own sheep, which means you need shepherds. Shepherds are very useful to have when you have a question about sheep, especially when those questions are about how to manage or use sheep to best effect. An approachable shepherd can save the rest of the company a lot of money on missteps and bad assumptions, but if you start getting rid of all the sheep, the shepherd will leave too. We should be using cloud providers for DR, and for regional load balancing. But the company should be maintaining at least one data center of their own, in the same time zones as most of their developers. I mostly blame Dell and IBM for this. IBM experimented with making server rooms easier to maintain 15-20 years ago and didn’t make it stick. Others ran with some of those ideas. Dell… I don’t know what Dell has done but I know nobody has been writing about it, so from a visibility standpoint they have done nothing. If/when someone makes it easier (reduced labor) to manage your own servers, the pendulum will swing back.
projektfu 4y ago

mbesto 4y ago

> rent all information infrastructure Most of them were renting most of the infra anyway: - Co-location (physical space rented) - Managed service to keep the lights on (power, back-up generators) - Leased hardware that gets replaced every 3 years - Managed service for switch, firewall, etc. monitoring - ISP, back-up generators, etc. > but that will likely be many, many years down the road They're going to explain to their future bosses that buying land, building a huge building, buying servers, figuring out cooling, setting up redundant ISPs, etc. etc. is somehow going to be smarter? Explain that one to me?

ocdtrekkie 4y ago
You're paying for all of that whether it's the cloud or on-prem. The only real difference if you're a large company is whether you're paying another company's profit margin as well. If you are big enough to have your own datacenter, you are paying Amazon enough to buy that much physical space, power, bandwidth, IT staff, etc. plus funding Bezos' trips to space. There's a justifiable niche where you're too small to justify running your own server/below the need of one full-time IT staff where the cloud makes sense, and startups temporarily benefit from the ability to rapidly scale on cloud platforms. But any Fortune 500 transitioning to the cloud is literally just taking their money and burning it, because the alleged cost savings of efficiencies of scale are being completely absorbed by the cloud provider's profit margins. And they're still going to end up paying their own IT staff to handle the cloud management in addition to (by proxy) paying the cloud provider's IT staff to manage the hardware.
- nathanielarmer 4y ago
  Today I learned that economies of scale and specialization are not things.
  - noobermin 4y ago
    Please whenever you envoke any model, have a sense of scale or region of applicability of the model, no model is true in all cases unless you're religious. The point the GP is making is at the scale of a F500 it starts to make less sense especially if you already have the infrastructure, the expertise, the experience, and so on. AWS is going to manage your VM better than you at your 3 person start up, but the economy of scale / specialization arguments work less well when you have an experienced workforce and developed system in place already and you're one of the world's largest logistics companies.

synergy20 4y ago

Some might return to on-premise data center, but most will not. It's really just like the utilities these days, 98% people will not dig their own well and install some sewage system and buy a generator, but big companies can still opt to have them on-premise installed, even though some of those are just backup systems. there is no return for the cloud for 98% of us.

fires10 4y ago
You generally can not go from city water/sewage to on-prem for legal reasons. However, many are going on-prem with solar. I am going almost completely off-grid for cost and SLA requirements.
- dehrmann 4y ago
  What's interesting is people seem really excited about installing rooftop solar, but I see fewer companies building collectors in places with cheap land and ample sun. That tells me part of the selling point of rooftop solar is it feels good, and the economics might not actually work.
  - lupire 4y ago
    Captured Solar (electricity in general) doesn't travel well over long distances. Generating solar for local use saves the cost of transport.
    - dehrmann 4y ago
      > electricity in general doesn't travel well over long distances. High-voltage line losses can be as low as 3% per 1,000 km. That should be easily offset by the better location and better angle/tracking.
  - marcosdumay 4y ago
    > but I see fewer companies building collectors in places with cheap land and ample sun You mean there are fewer companies making large investments than smaller ones?

WFHRenaissance 4y ago

Great concrete example of the principal-agent problem here.

gravypod 4y ago

> that will likely be many, many years down the road And this is where people in my age range (20-30) will get to swoop in if we invest the next 10-20 years learning about and tinkering with server hardware. I highly recommend it for everyone.

jollybean 4y ago

They are doing the rational thing and there is no going back. HNers need to get their heads around this: the economies of the could represent a fundamental, secular shift. Imagine if Fedex designed and made their own delivery vehicles. Then they 'outsourced' that to Ford/GM. You might say 'look at Ford's amazing margins, look at the money being left on the table' - but in reality, it's still more cost effective to 'buy vehicles' the to 'DIY' them. In the 'long run' those Azure/AWS margins will erode - and/or - the long tail will creep up on them. Basically data centers offering less, for less, and companies realizing they don't need the complexity of AWS do do basic things. What that will look like is hard to say. With hardware it meant 'pivot to Asia'. Will this happen again? Chinese companies creating large data centres in the US with minimal staffing, monitored and operated out of China? If we were not in a giant geostrategic kerfluffle with them, then yes, I would say that is the future. But given security issues etc. it's probably not. Can India do that? Maybe.

kragen 4y ago
Ford and GM and Freightliner are not in a position to use price discrimination to confiscate FedEx's entire profits; if they try to sell trucks to FedEx at an inflated price, FedEx can just buy the same trucks from dealers, or buy them used, or buy GM trucks instead of Freightliner trucks. By contrast, generally speaking, cloud providers are in an excellent position to use price discrimination to extract the entire surplus value of every transaction. And the flexibility and reliability provided by FedEx will to a significant extent simply be that of their computers. (As chrisseaton pointed out, Boeing is in a cloudier position.)
- jollybean 4y ago
  So 100% yes, AWS has much more power in the value chain than Ford. But not really. Especially to the extent that special services are not leveraged, then there are cheaper alternatives. A lot of IT is just 'instances' not 'fancy cloud queues'. AWS pricing is very transparent and people can make the decisions they want. Generally speaking, AWS does not price discriminate on the basis of 'business model' and their margins are meaningless with respect to the overall business efforts of a Big Co. I mean - unless you are doing big AI crunching, or 'free hosting content' - then frankly the AWS bill is not going to be a huge line item relative to overall operating costs.

adamc 4y ago

This is one of the things bad about Wall Street: Short-term decision-making. Everybody cares what happens to stock prices in the near-term, rather than 10 years out.

dehrmann 4y ago

I disagree. This isn't FedEx's core competency or a differentiator for them. They're also not at a scale where it could make sense to colocate or build their own DCs. Fiddling with low-level tech infra would just be a distraction. Now, they might find that software is a differentiator for them, but that's different.

dx034 4y ago
For a logistics company, the IT system is the differentiator. Reliably routing and tracking packages to maximize utilization of trucks and planes allows them to be cheaper and faster. That all depends on both hardware and software. So I'd definitely think that running their own systems should be a given.

deleted 4y ago

[deleted]

systemvoltage 4y ago

Some are starting to question the almighty Cloud™: https://www.economist.com/business/2021/07/03/do-the-costs-of-the-cloud-outweigh-the-benefits https://www.economist.com/business/2021/07/03/do-the-costs-o... One of the things that bother me about AWS is that I still need to manage their risk of a data center going down by using multiple availability zones and managing the complexity of extra resilience. There is a conflict of interest: AWS has a perverse incentive in keeping a single AZ robust and unless there is a natural calamity, it should not go down. I thought one of the major reasons of going to cloud is that you need not manage risk and offloading it from on-prem.

bhewes 4y ago

At least in Oil and Gas. Servers moved back into the personal towers for the first time in a decade. Latency issues.

smn1234 4y ago
AWS Outposts ?
- bhewes 4y ago
  Ah did not know about AWS outposts. They use Azure so hybrid setup and moved the latency sensitive servers back into the tower everything else lives on Azure.

walrus01 4y ago

reminds me a bit of: https://www.reddit.com/r/AskHistorians/comments/vqr30e/jack_welch_extracted_record_profits_from_ge_for/ https://www.reddit.com/r/AskHistorians/comments/vqr30e/jack_... "Jack Welch extracted record profits from GE for 20 years, but left it a hollowed-out "pile of shit," according to his successor. What exactly did Welch do that was so damaging, and how did he get away with it for so long?" from the post reply, in case it's not available from the URL: alecsliu · 2 days ago · edited 2 days ago Gold2Eureka!Bravo!Today I Learned Welch took over a GE that was at the time, a major company. At the time, he viewed GE as bloated and needing to change. While he might've been right about that, the approach he took was perhaps less ideal. One of the worst things he helped make commonplace among American companies is the concept of stack ranking. The way it worked was like this: people were divided into three groups: A, B and C. A's were the top performers who needed to be rewarded generously, B were adequate performers who should be allowed to stay, and then C were those who needed to be fired. So far so good right? Well, not exactly. For Welch, these three buckets could be separated into the top 20%, the middle 70%, and the bottom 10% (20/70/10). Based on the above points, the bottom 10% would thus need to be fired annually. In the short term, this helped in making the company lean and look more productive, increasing the bottom-line and portraying an image of success. However, the long term consequences of such a change was cultural degradation and the introduction of new bloat and waste (running all of those performance reviews and firing and rehiring so many people takes a lot of money.) Consider the case of a perfectly adequate team: the entire team has achieved their targets and has contributed to the company as per their job description. The issue? In stack ranking, 10% of this team would still have to be fired even though everyone did their job. Unsurprisingly, the introduction of this competitive atmosphere where it isn't enough to succeed, one must be better than their peers, results in backstabbing, competition, and a host of issues which eventually weaken a company's competitive edge. This was only a part of Welch's general treatment of workers as numbers rather than humans. Aggressive cost-cutting, offshoring, etc. were the norm under Welch's regime and he would destroy entire divisions. This again, was great in the short-term but bad in the long term. Welch clearly had a dim view of company culture and believed it to be unimportant. The other major issue of Welch's was the acquisition of hundreds and hundreds of businesses, as part of Welch's goal of acquiring his way to the top. On the surface, this is what Welch did; on paper, he didn't destroy the main profit makers of GE so much as create new ones, primarily in the form of its financial arm. The result was that this allowed GE to play with the numbers in a way that allowed him to make sure that GE was always meeting targets set by Wall Street; in order to generate the right earnings, simply buy or sell certain assets, write-off others, etc. and when your company is an acquisition machine, it's not too hard to find the right numbers. This process helped expand GE into a mega-conglomerate but again, ultimately left the company in a weaker than expected state. In particular, on paper the core business of GE became its financial arm, as that was where all the funny business with the numbers was happening (nothing explicitly illegal though). Ignoring the damage the Welch did to GE's profit centers through his horrible business practices, this was a huge part of why GE declined so rapidly in the years following. GE's valuation was based on an inaccurate picture of its profits and value, so when the truth started coming out (especially with the Great Recession collapsing financial services profits), GE was quick to follow. As for why Welch was able to get away with it all? Well, the answer was because he was delivering. He hit the earnings targets, he made the board and shareholders very happy, by all accounts GE was the paragon of success and everything was going right. What Welch did was unprecedented, and it's hard to really understate that. For all of his faults, he had America tricked into believing that what he could do was unique and that he could avoid the realities of the economy and cyclical markets, that no matter what was going on, GE was special. Welch died a rich, rich man and many, many people profited greatly off of GE during its 20 year bull-run. Edit: My off-the-cuff writing is always horrible wrt to grammar and structure so I'll probably edit it later haha

Jistern 4y ago

dhzhzjsbevs 4y ago

We have our own datacentres. Lucky enough to have got them up and running pre cloud era. If we were any later we would be full cloud but the reality is we outcompete everyone on price in our space because of it.

pauljurczak 4y ago

And they will gently depart under their beautiful golden parachutes. What a racket.