32 ms·

FedEx to close data centers, retire mainframes

492 points by taubek 4y ago

Thorentis 4y ago
Is there any gain here aside from cost saving? Will this saving be passed on to the customer? (laughs). What happens when the inevitable AWS outage then causes a global shipping bottleneck? Have we still not learnt that having sovereignty is better than saving money? Has COVID taught us nothing? sigh
- deleted 4y ago
  [deleted]
- jupp0r 4y ago
  What makes you think that a cloud provider outage is more likely than an existing-Fedex-datacenter outage?
  - rbanffy 4y ago
    Mainframes themselves are extremely reliable. When configured as a cluster they get five nines. I'd say a power or communications failure is much more likely than mainframe downtime. With cloud you can rely on more datacenters and, if your application is built right, it may end up being more resilient than what a mainframe setup would give you. Downtime is a fuzzy thing. Whenever possible, I design my apps for graceful degradation - if the database becomes read-only, we can still operate in degraded mode. If we lose queues, some things will not work, but others will continue normally and many users will be completely oblivious to the alarm bells at the NOC (just kidding, there's no such thing). My SLA for full operation is much more relaxed than for degraded operation.
    - quickthrower2 4y ago
      I think a lot of sites would benefit from offline-first and cache a bit of information, perhaps encrypted with the same password you used. There is a lot you can still do if you can a functioning site (served from a service worker) with some cached data. You could read HN articles you read before, for example! Not to rely on this for the nines, but as a nice way to keep things useful when there is an outage or just slowness.
- sokoloff 4y ago
  What’s to gain other than $400M in savings every year indeed? If it drops all the way to the bottom line, that would boost their 2022 profits by 35% and 2021 profits by about 45%. Seems like a decision you have to consider, even with possible failure modes.
  - oblio 4y ago
    https://en.wikipedia.org/wiki/FedEx https://en.wikipedia.org/wiki/FedEx Wikipedia says their net profit for 2021 was $75 billion, so your math seems off.
    - sokoloff 4y ago
      You are correct that I was wrong. (I searched "fedex profit 2021" and incorrectly reported based on a quarter's profit, which is sloppy as hell on my part.) That said, it seems like $75B is wildly too high as well. Page 43 of their most recent annual report shows a consolidated net income for the year ending May 31, 2021 as $5.2B, making $400M still a 7.5% boost (if it all passes through). https://investors.fedex.com/financial-information/annual-reports/default.aspx https://investors.fedex.com/financial-information/annual-rep... https://s21.q4cdn.com/665674268/files/doc_financials/annual/2021/389437(1)_12_FedEx_AR_WR.pdf https://s21.q4cdn.com/665674268/files/doc_financials/annual/...
    - lotsofpulp 4y ago
      Wikipedia is wrong, see page 43: https://www.sec.gov/Archives/edgar/data/1048911/000156459021037031/fdx-10k_20210531.htm#ITEM_6_SELECTED_FINANCIAL_DATA https://www.sec.gov/Archives/edgar/data/1048911/000156459021... I am not even sure where the $75.231B erroneous net income figure comes from, it is nowhere in the 2021 report: https://investors.fedex.com/home/default.aspx https://investors.fedex.com/home/default.aspx
    - mmiyer 4y ago
      That Wikipedia figure is vandalism from a couple weeks ago that I reverted.
  - ClumsyPilot 4y ago
- dchftcs 4y ago
  I'm not sure having mainframes counts as sovereignty. Yearly savings suggests it doesn't.
- iso1631 4y ago
  Fedex will continue to charge as much as they can to boost their profits. This move will increase their profits, making $1.50 on each of 100 million deliveries instead of $1, boosting profits from $100m to $150m However if they were to undercut UPS and others they might be able snap up UPS business, dropping the profit to $1, but on 200 million deliveries, and thus they'll make $200m instead, you the customer will have a cheaper delivery, and everybody wins
  - nly 4y ago
    Until it becomes a race to the bottom and UPS drop their prices as well. Then both companies suffer
    - JumpCrisscross 4y ago
      > becomes a race to the bottom and UPS drop their prices This is competition.
    - iso1631 4y ago
      And all consumers win. This is the beauty of capitalism, it's how it's supposed to work. It fails when you get high barriers to entry and company colluding
    - yuliyp 4y ago
      This is how the "passing the savings on to their customers" happens.
    - dx034 4y ago
      They're not really competing on price but on quality. They make the most money with super urgent deliveries. If they offer later pick up times or earlier guaranteed deliveries, customers will be happy to pay more. And their IT system is crucial for that.
- jon-wood 4y ago
  The immediate gains I can see are: * In-house development teams have more agility in standing up new services, since they can just use EC2/ECS/one of the other 1500 container runtimes AWS has, rather than having to wait for someone to provision new physical servers for them. * An extensive suite of complementary products are available from cloud providers, it's not just about being able to start a VM, in many cases you don't even need to because you've got Lambda, and the various managed services. * Less training/hiring overhead, since people who can use AWS are pretty much a commodity at this point, whereas people who can manage a mainframe are an increasingly rare breed. * Redundancy becomes easier. As does data locality, let's say Singapore declare all services delivered in Singapore must be hosted there. That's a lot easier to handle if you don't have to go and build/acquire a data centre first. * The aforementioned 400 million dollars per year.
  - throw0101a 4y ago
    > In-house development teams have more agility in standing up new services, since they can just use EC2/ECS/one of the other 1500 container runtimes AWS has, rather than having to wait for someone to provision new physical servers for them. How many people run physical servers 'raw' anymore? I would hazard their current stack involves VMware, Hyper-V, Open Stack, or a combination of the three. You can have an API system spin-up just as effectively in a private cloud as a public one.
    - sofixa 4y ago
      That handles IaaS, now do all databases (relational, NoSQL, NewSQL, KV, etc.), message brokers, object storage and a hundred other services. OpenStack handles some of those, with varying degrees of success, but with VMware and Hyper-V you have to DIY from scratch with IMHO the wrong level of abstraction.
    - Jochim 4y ago
      I worked somewhere with a private cloud. It was permanently over-subscribed and the VMs were nearly unusable because of it. Not saying it can't be done well but there's less incentive for a company whose main business isn't providing infrastructure to ensure they've provisioned enough resources to meet demand.
- knorker 4y ago
  You can take part of the savings by buying NYSE:FDX stock. I'm not being facetious.
- aliswe 4y ago
  These hyper-ideological takes on finance are so tiring. If you think they are bound to make more profits, I suggest you invest in them, they are called FDX on NY stock exchange. EDIT: I used to be like that, too.
sschueller 4y ago
So are the moving to AWS? Are we going to have a complete breakdown of packages delivery next time there is an outage at AWS?
- badgers 4y ago
  This was from 2019: https://www.switch.com/fedex-signs-10-year-data-center-infrastructure-agreement-with-switch/ https://www.switch.com/fedex-signs-10-year-data-center-infra...
- joekrill 4y ago
  > The company is a known Oracle Cloud and Microsoft Azure customer.
  - ehnto 4y ago
    Hard to say for certain. Companies that size often have many very independent operations run by separate internal departments. I have worked places with internal, AWS and Azure operations underway.
    - jon-wood 4y ago
      Where I am various parts of the business run workloads on AWS, Google Cloud, and Azure. To some extent I think this sort of behaviour is encouraged by large corporates because it makes contract negotiations easier if you're able to wave in the direction of the other cloud providers you're using and suggest it wouldn't be too hard to migrate over to them.
  - andyjohnson0 4y ago
    I read the parent comment as questioning the wisdom of relying on a cloud provider (but not specifically AWS, as you indicated) for a global operation life FedEx. Even with service zoning, in extreme cases that provider constitutes a potential single point of failure. Unless FedEx are able to run their core systems on Azure and Oracle‡ then I think the point stands. It would be interesting to know how Fedex sees the risk/reward equation, and how Azure (for example) compares to a (IBM?) mainframe data centre in terms of reliability and uptime ‡ I'm not sure if there are any cloud neutrality solutions that permit this, but it is possible that FedEx have rolled their own.
baud147258 4y ago
Are they really going to save 400 m$, or was it the figure "promised" by the shop that will handle the migration?
- afavour 4y ago
  Either that or it’s an accurate figure for the first year or so while the cloud provider gives a ton of credits. By the time that all runs out the senior exec behind the migration will have moved on and someone else can do the maths.
- kurupt213 4y ago
  $400M sounds huge, but why do I think it’s trivial next to FedEx’s operating budget? A couple years ago they were making 18 Billion* a quarter. * edited from million (meant to type billion)
  - oblio 4y ago
    If think you mean $90 *b*illion a quarter.
    - lotsofpulp 4y ago
      $90B per year. https://www.macrotrends.net/stocks/charts/FDX/fedex/revenue https://www.macrotrends.net/stocks/charts/FDX/fedex/revenue
- makeitdouble 4y ago
  Is there any reasonable answer to this question ?
- arghwhat 4y ago
  It certainly seems unlikely. Maybe it's the current operational cost, not subtracted the new operational cost. Maybe it's due to already adopting cloud and thus having paying double. Maybe it's due to the use of mainframes, rather than conventional servers. Either way, it's definitely not the cost difference between on-premise and cloud. Cloud providers are not charities, and their buildings and staff are not cheaper than yours.
benj111 4y ago
What's the 8MW in this context? Is that power. Or something else?
- bcraven 4y ago
  "We presently own and operate four primary campus locations (“Primes”), encompassing 12 active multi-tenant data center facilities with an aggregate of up to 14 million gross square feet (GSF) of space. Our existing facilities are equipped to provide up to 490 megawatts (MW) of power, with the potential to scale to more than 1,300 MW upon full build out of our existing footprint."[0] It sounds like Switch Inc. advertise their datacentres by Wattage. I am not sure how common that is. [0]https://investors.switch.com/company-profile/default.aspx https://investors.switch.com/company-profile/default.aspx
  - tgsovlerkhgsel 4y ago
    Very common.
    - benj111 4y ago
      Do you buy by the watt though? It also advertises the square footage but you don't buy by the sq ft either
      - theideaofcoffee 4y ago
        > Do you buy by the watt though? Usually in larger DC deployments, yes, you are quoted, metered, and billed by total power. The amount of space is (usually) an afterthought. There is more space than you can fill and remain under the power commit, which also correlates to total cooling load too.
rbanffy 4y ago
Reverend Mother Mohiam: Many men have tried Paul: They tried and failed? Reverend Mother Mohiam: They tried and died.
- bryanrasmussen 4y ago
  context is everything, reading that I thought it would be hilariously funny as an exchange in Father Ted.
  - deleted 4y ago
    [deleted]
  - sackerhews 4y ago
    Wow, this both a fantastic joke and could be used as a school book example of the importance of context. Given of course that everyone would have watched Father Ted. But the world would be a better place if everyone did.
    - rbanffy 4y ago
      Indeed. Hadn't heard of it before moving to Ireland, I credit my loving neighbors for introducing us to this national tradition.
bradfa 4y ago
It would be interesting to see the actual numbers on how much such a transition will cost. Can their existing mainframe software and databases simply migrate into a cloud offering? If not, then if their planned transition has delays or technical hurdles, how much of that cost savings is affected? If their cloud providers change pricing structures in future contract negotiations or if the amount of data they transfer in/out of each cloud provider location changes dramatically in order to better serve their customers, how much of that cost savings is affected? It's clear that you can run a business with minimal amount of physical plant for servers. It's still not entirely clear to me that large established businesses can actually save money this way.
kragen 4y ago
Ultimately companies that abdicate their informatics operations like this will give their profits to their data-center operators, who will be empowered to charge them whatever price they want. Because what's their BATNA? Migrating from Azure to AWS when Microsoft doesn't want to let them? Renting your information infrastructure is a great way to reduce startup costs, but down the road, that information infrastructure runs your company. Trying to outsource it is like trying to outsource upper management. To be clear, I'm not saying that the optimal amount of cloud services for an established company like FedEx to buy is 0. They bring in management consultants, too. But it sure isn't 100%.
- cs702 4y ago
  Without a doubt, a future generation of executives will revisit and reverse the decision to rent all information infrastructure, but that will likely be many, many years down the road. In the meantime, the current generation of executives who made this decision will look very smart for saving the company lots of money for a good number of years. And they stand to benefit personally from it. They're doing the rational thing!
  - noobermin 4y ago
    It's amazing there are no other voices in the room pushing back against this. It really makes you wonder how companies even function at all, much less make money. Being so short sided never ends well.
    - jsiepkes 4y ago
      Convincing people future risks outweigh the short term gains is very hard. That goes for all things.
    - numbsafari 4y ago
      There are very likely voices, but they didn’t win this battle.
    - cs702 4y ago
hnarn 4y ago
Making accurate calculations of data center cost is of course complicated and rarely manages to take everything into account, but the common knowledge I’ve heard is that there comes a point when a company is so large that going on-prem is what actually saves them a lot of money. If FedEx is not large enough to be one of those companies, how large do you actually have to be? Or has cloud pricing changed to the point where this is no longer true, and even huge corporations can save money in moving away from their own premises?
- Out_of_Characte 4y ago
  Technology scales faster than fedex's data needs. Maybe somewhere in the future the reverse trend happens as their needs can be met with very small servers. Though I've always been wary of putting your data somewhere because getting your data to a competitor costs alot of money.
  - ClumsyPilot 4y ago
    > Technology scales faster than fedex's data needs. I don't think this sentence has meaning
    - geodel 4y ago
      I mean it is like executive statement e.g. Real time analytics in Cloud IoT on Edge.
  - throwawaymaths 4y ago
    I don't think it's the server costs they care about it's stuff like db maintenance, security, security, and security. Security professionals are expensive, and it's very hard to get right in house on metal.
    - hrunt 4y ago
      Moving away from on-prem data centers does not reduce security costs. At best, it's a lateral move, and depending on the cloud setup, may actually be more expensive. Ultimately, this is an opex vs. capex decision. Data centers are expensive, and require a lot of up front capital to go into the ground. FedEx is worldwide and moving more technology to the edge, so buying land, designing and constructing buildings, and then operating them for 10 years or more in a large number of locales requires a huge investment. They can take that money now and solve their problems immediately. They are also not saying what they mean by "closing its data centers". FedEx announced a 10-year partnership with Switch last year to build and operate edge facilities. FedEx may be moving out of data centers it operates on its own into facilities built and run by others, and using "cloud-native" designs deployed on hardware in those facilities.
Aeolun 4y ago
Cue 10 years down the line, when everyone realizes cloud comes with tradeoffs in flexibility and reliability and they all move back to on-premise for business critical stuff.
- cs702 4y ago
  More like 20+ years -- after a new generation of executives and managers has taken over. Their egos won't be tied up in old debates and they will be more willing to sacrifice old decisions.
Ekaros 4y ago
Where are they using all of these data centers and mainframes? That is they are spending at least 400 million to do what exactly? To me that seems like rather high figure in any case for their sites and logistics...
- ajsnigrutin 4y ago
  Yep... A couple of million of packages daily + tracking a fleet of vehicles, client facing site, apis for their tracking and courrier services/devices plus some internal services (billing etc...).. this seems like it would fit on a few racks, not nearly 400million, or whatever the full number is (since 400m are just the savings). But yeah... old company, they might still have stuff written in cobol virtualized somewhere.
  - jeltz 4y ago
    But even virtualized Cobol should fit in a few racks.
  - hoofhearted 4y ago
    Their Memphis air hub processes 2 million packages by itself during the season. You also forgot to include FedEx Freight in your estimate.
    - hoofhearted 4y ago
      edit - Autocorrect was terrible on my phone today!... The Fedex Memphis air hub processes more than 2 million packages a day itself during the winter holiday peak season.
  - dzhiurgis 4y ago
    With 300k employees you can expect them to have thousands of disparate services that will likely get rewritten part of such migration
- oblio 4y ago
  With (most likely) 0 knowledge about the internal details of their business, but with the confidence of the average programmer: "they spend too much". I imagine Twitter could also be built over a weekend, right? :-)
kurupt213 4y ago
What price is acceptable for ownership of your data?
fanf2 4y ago
”FedEx first opened an on-premise data center at 250 Spectrum Loop in Colorado Springs, Colorado, in 2008” That seems implausible: where did they keep their mainframes before 2008?
- perlgeek 4y ago
  Colocated in another company's data center.
  - dx034 4y ago
    They could get back to colocation. I think even AWS & Co don't own all physical buildings, they also rent parts of data centers in some locations. You don't need to own real estate or a physical building to run servers.
dna_polymerase 4y ago
Maybe, just maybe FedEx had a bunch of employees making a lot of calculations before going this route. Just maybe they made an educated decision, based on their inside knowledge of the company and it’s needs. Or maybe a bunch of people on HN just know better.
langsoul-com 4y ago
How did Dropbox end up saving tens of millions bt making their own cloud infrastructure but FedEx loses money? Surely FedEx would have a metric shit ton of data?
- jeffbee 4y ago
  Ask yourself what probably happened a few years later when they had to negotiate new leases from their datacenter landlords. Funny how they never put out a follow up press release about the enduring value of on-prem. Also a company that basically can’t grow and hardly turns a profit. Really, not the first example I’d reach for.
- humanistbot 4y ago
  I'd bet Dropbox has way more raw bytes to store than FedEx. Plus, the Dropbox business model is based on data egress to clients, which is really expensive on cloud servers. FedEx needs data centers more for internal logistics, which is probably why they're going with Oracle.
fithisux 4y ago
Everything will merge to one BIG company because of cost-effectiveness.
thedougd 4y ago
From an executive viewpoint, this would be a reasonable forcing function to overcome long, deep rooted stasis in your IT department. Let's assume for a moment that modernization is good. It probably is wrt mainframes. For every team that refused to modernize, you can now drive them to the 2024 date, absolutely no exceptions.
bythckr 4y ago
I never understood the logic behind companies abandoning their own data centers and moving their gold (Data is the new gold) into somebody else hand. I see that many here are also thinking in the same line. But when I spoke to a CTO of a F500 company. This was the take he had. He feels one of the major driving force is to reduce the carbon foot print of the company. His company is designing the system to have the crucial data on their own infrastructure but keep they other data & do the processing of cloud service. Instead of relying on 1 provider, they are using multiple. Also supporting some small local cloud providers. This seems very relevant for a logistics company with a huge of fleet of air craft. Especially considering the upcoming carbon tax.
- hotpotamus 4y ago
  I'm skeptical of the carbon footprint thing for a few reasons, but the other point is interesting and something I think about once in awhile. As we build out more and more bandwidth, compute, storage, and APIs to tie them all together, it seems that the world more resembles some sort of large computer and most of the work is just getting the data and compute closer together when needed and minimizing costs when it's not. I know I'm far from the first to have this realization.
TheRealDunkirk 4y ago
I've seen two mainframe "replacements" fail to either retire the mainframe, OR produce an alternative system which people like. Has any Fortune 500 company successfully "retired" their mainframe?
- Someone1234 4y ago
  I've seen it. It was essentially four steps: - Build middleware (REST API endpoint) in front of the mainframe that was 1:1 mainframe functions. It initially does nothing except auth and routing. - Re-write/migrate all other software from using the mainframe directly to using the new middleware. - Work to further restrict anything directly connecting to the mainframe aside from the middleware, including staff's terminal access by building management solutions (e.g. new UIs, new management middleware). - Once the mainframe is completely isolated, start building drop-in replacements for business segments and then switch the middleware(s) to point at the new solution from the mainframe. This should be invisible to all external consumers. You can "hot test" it by having the API execute on the mainframe + new solution, and checking for 1:1 responses. The hardest part about the above isn't the tech, it is the internal norms that you're fighting against (e.g. staff have had terminal access for tens of years, and know the commands to get their work done by memory). So you have to create very compelling alternatives using the new API middleware/web/apps, "export to Microsoft Excel," graphing, mobile support, and similar that a terminal cannot directly offer is clutch here.
  - malfist 4y ago
    This is exactly what Hilton did in the 2010s. We stood up an ESB (wso2) between the modern front ends and the legacy mainframe backend, and then broke down the mainframe's functions into microservice domains and wrote them. Had the ESB route traffic to the microservices until the mainframe wasn't doing any work anymore and was able to discontinue it. There was a lot more complexity that that, but that's the gist of it. By the time I left, my team had carved out the bulk of the mainframe's services and had stood up over 50 microservices.
  - kjs3 4y ago
    This is (essentially) how I've seen it done as well, the very few times I've seen it done successfully. But did you guys do it in 2 years, like these folks think they are? The hardest part about the above isn't the tech, it is the internal norms that you're fighting against This is so true it's almost painful.
ryanmercer 4y ago
I left FedEx (Trade Networks) after 15 and a half years earlier this year. The entirety of that 15 1/2 years was spent using IBM AS/400 terminal to interact with various systems and was also used to submit paperwork to Customs and any other applicable government agencies to clear international freight. It... was an experience. I imagine that is going to take quite a bit of work to migrate over to someone else's servers, as I doubt they're going to be like "sure, we can accommodate your decade and a half old, mission-critical, systems"
mensetmanusman 4y ago
And then AWS will charge $399M…
dontbenebby 4y ago
There's not really a market for three big airline shipping things. Amazon air + UPS are tag teaming them in a bad way. This isn't about IT, it's about that FedEx is not in great shape.
lettergram 4y ago
Lol “saving $400m” really equates to “moving onto AWS or Azure and paying hundreds of millions to them”
say_it_as_it_is 4y ago
This is an ideal project for managers whose end of year bonus is linked to immediate cost savings. Who cares if it costs FedEx more in the long run?
chevman 4y ago
I'm in corp land and this to me reads as the CIO using external PR to push internal agendas. Are they really going to close all their on-prem datacenters in the next 2 years? Maybe. More likely, this is a way to build momentum around the transition to cloud. Not a bad strategy, and not uncommon from what I've seen in the enterprise.
shrubble 4y ago
'Within the next two years we’ll close the last few remaining data centers that we have, we’ll eliminate the final 20 percent of the mainframe footprint' Does that truly sound achievable, when they have the most difficult to migrate part of their mainframe applications still on their mainframe? The easy stuff they have already moved off.
treesknees 4y ago
I wonder how much of this is motivated by the lack of staff. The market of available mainframe operators has been drying up for decades. When your entire business depends on a technology nobody is willing to learn or go to school to understand, that creates a problem. Also, from my understanding, it can take a tremendous number of distributed x86 systems to supplement the scale, speed and reliability of a mainframe. It's possible that unless FedEx gets away from managing storage arrays and hypervisor farms, they wouldn't have enough staff to operate the datacenters once they're full of enough servers to replace the mainframes.
- thr0wawayf00 4y ago
  > The market of available mainframe operators has been drying up for decades. Just like many other industries, the market for mainframe operators that want to earn less than technologists in modern stacks has been drying up. A lot of mainframe operators refuse to bring their pay scales in line with what developers in modern stacks are earning. Why take a pay cut to go work on a 40 year-old system when you can work on modern stuff with more money? They keep the pay low and complain about a labor shortage to deflect from the fact that they're simply choosing to ignore market forces.
- kamaal 4y ago
  Would be curious to learn the specs of current day mainframes, and cost. What sort of work is typically done on such machines these days.
  - treesknees 4y ago
    Disclaimer: I have not worked directly on mainframes themselves. They've been doing the same type of work for decades. Batch processing of records and transactions (payroll, insurance, banking, airline booking, etc.) IBM releases new mainframe models all the time, and typically companies will lease them and upgrade every few years. When I worked at an insurance company, the mainframes were the core of their financials. They had around 8 of them in an active/standby DR site. The machines could do instruction-level error checking, meaning even a weird processor failure/error wouldn't impact the end transaction. Basically if you want to guarantee accurate handling of your money, you use a mainframe. One common thing to see nowadays is the mainframe becomes abstracted by some other layer of API, such as a rest API. This has reduced the need somewhat for mainframe operators, but in the end you're still interacting with some COBOL that was written 30 years ago and somebody needs to understand it and the hardware. And the hardware is so different from traditional servers, you can't just pick it up and wing it. You really need someone who knows what they're doing.
filereaper 4y ago
Its weird to have companies say they're moving from Mainframes to the Cloud. The Mainframe is a sort of cloud, each one has usually double the spare capacity and you call IBM to unlock it. The cost of running a Mainframe is measured in terms of MIPS based on amount of compute used and there's compute reserved for offloading things like JITs (zAAPS). Essentially a version of cloud compute costs. I can understand if you're locked into COBOL with EBCDIC and can't find talent, but Z runs freaking Ubuntu now! I know IBM works furiously on modern ecosystem support. Not shill for IBM, but what exactly is it that they're expecting to be so different from their cloud migrations? Or is everyone here caught up in the "Mainframe old" falsehood.
- mhh__ 4y ago
  Mainframe = locked in to IBM, is probably the logic.
  - JamesAdir 4y ago
    How is it any different than being locked to AWS/Azure/Oracle/Google?
    - notyourwork 4y ago
      Its not be most C-level execs are these companies can present it as IT evolution and progress. Balance sheets look different because accounting has costs in different places. It all looks good.
    - safaci2000 4y ago
      Well your also hardware locked. The last time I worked on a mainframe (about a decade), 8 mb of RAM was $1000, a network card was $800. Commodity hardware around the same time 8mb was probably $50 and a network card even a good one was most $100. You're paying a premium for both the hardware and the lack of develops knowing EBSIDIC, JCL, Cobol, etc.. Actually, from what I remember of Oracle, that might be very similar. I remember having to pay license fees per core.
bosveld 4y ago
Another way of looking at at: Where own data-center cost gets to high it might be due to lack of responsibility to act or inability to manage. A company then have two options: (a) replace/pressure top/middle management to get costs down and processes streamlined or, (b) move to the cloud. Its easier (and possibly good for your resume) to move to the cloud and then a few yours later -- decide again. So moving to the cloud is sometimes an indicator of an inflexible/stale/mature company.
jchook 4y ago
Interesting story as I recall Bank of America saving 5x that amount per year by doing the exact opposite move. https://www.businessinsider.com/bank-of-americas-350-million-internal-cloud-bet-striking-payoff-2019-10 https://www.businessinsider.com/bank-of-americas-350-million...
- Spivak 4y ago
  Doesn’t matter what you actually do, you get the benefits of shaking off all the barnacles doing any rewrite or rearchitectire.
- jeroenhd 4y ago
  The difference may be that they're sticking with their own data center but they don't have a mainframe setup. Mainframes come with expensive, locked out hardware, expensive maintenance, expensive experts, you name it. With modern open source tooling, ridiculously powerful graphics cards, easily accessible FPGAs and free data center hosting software like OpenStack, Kubernetes, and glusterfs you can replicate many of the features that made mainframes enticing many years ago. It'll require a heck of a lot of work to get the same performance out of a custom built solution, but long term it's probably cheaper than relying on IBM. I doubt that they'll be saving any money by moving to the cloud unless they're horribly inefficient with their contracts right now. Just moving away from mainframes alone should save a significant buck, but if they were planning on restructuring their digital infrastructure anyway then now is probably the best time.
- mbesto 4y ago
  This is 100% reasonable and plausible. It's perfectly acceptable that one F500 company can see savings by using cloud and others see increased costs.
- dx034 4y ago
  That's how you do it. You don't stick with on-prem but just build your own cloud. At that size, AWS won't really buy hardware much cheaper than you. And while you can't build every cloud product that AWS has, the most popular ones are open source anyway.
fsociety 4y ago
The thing I see missing in the comments is that sourcing hardware over the past few years has been incredibly difficult. It will get better, yes, but that may have been the last straw to convince them to switch.
- bradfa 4y ago
  Past few years? Maybe the past 18 months it's been slightly more difficult but even at the beginning of COVID it wasn't hard to source most anything. And even during these more recent supply-constrained times, you can still source hardware it just might take a little longer for delivery or you might pay a bit more than you did before.
tgflynn 4y ago
I wonder how much of that savings comes from closing data centers and how much from getting off IBM mainframes. It's hard for me to see why a company like FedEx would need to be spending anything like $400m a year on data centers unless a big chunk of that is paying IBM to rent their mainframes.
- tgflynn 4y ago
  It would be interesting if those down-voting my comment would explain where my intuition is wrong. FedEx delivers 12 million packages per day. If each package requires 10 transactions that works out to about 40k transactions per second. You should be able to handle that with 1 IBM mainframe or much less than 1000 servers and I don't believe either of those would approach $400 million per year to operate.
sbf501 4y ago
I'm surprised at the amount of vitriol being spewed at FedEx in this post for their decision, mostly about the doom-and-gloom that awaits FedEx. Either there are a lot of Fortune-500 CTO's on Hacker News that know better than FedEx, or there is a deep-seated fear of the cloud, specifically PaaS.
simonw 4y ago
"FedEx first opened an on-premise data center at 250 Spectrum Loop in Colorado Springs, Colorado, in 2008" - what were they doing before that?
epberry 4y ago
Surprising to see a company claim massive cost savings for moving _off_ the cloud. But perhaps they're talking about engineering velocity, reliability, etc. Of course everyone knows Dropbox's story with this: https://www.datacenterknowledge.com/manage/dropbox-s-reverse-migration-cloud-own-data-centers-five-years https://www.datacenterknowledge.com/manage/dropbox-s-reverse...
dend 4y ago
This makes a lot of sense if you think through the angle that their core value prop does not come from the network/data latency optimization. It's not Netflix and Dropbox - this is FedEx. Their core business value proposition comes from logistics and being able to react to the information quickly, but extra millisecond latency is unlikely to make or break the business. In light of this, it's easier to outsource the network and data infrastructure to a provider that actually does it day in and day out and put the limited engineering resources to something that is more impactful. Just reaffirms that quite a few companies do not want to run their own servers unless they absolutely have to.
- peteradio 4y ago
  > through the angle that their core value prop Surely this is just one ingredient to a successful company and there are myriad factors involved. The core business of Boeing isn't to make planes its to sell them.
  - dend 4y ago
    That's the wrong analogy. In your example, the former is required for the latter. FedEx wants network stability and a data store, most likely, but the success of their business does not depend on them hosting their own infrastructure.
imperialdrive 4y ago
https://www.switch.com/colocation/ https://www.switch.com/colocation/ The Switch colos all appear to be rendered concepts, but impressively so.
- dx034 4y ago
  I always wondered what latency requirements they must have in a distribution center that has to route packages dynamically and make those routing decisions within milliseconds. Surely there must be a lot of logic at the distribution center, they probably are their own data centers. Or can you really have 10+ms latency for each decision?
winddude 4y ago
Too bad it's not sooner, could use a flood of decent servers on the second hand market.
fuzzfactor 4y ago
Along with the data, They. Own. The. Data center(s). Well, in some sense or another. They might not be completely paid for but you get the idea, there's got to be a certain amount of worthwhile assets that could be leveraged. Too bad there's not any business executives who have any experience at making money from a data center itself.
formerkrogemp 4y ago
From an managerial accounting perspective it makes sense. Reduce cost centers and focus on core competencies. But many things make more sense from a short-term, myopic, improve this quarter accounting sense that are actually bad ideas.
exabrial 4y ago
"save" In the very short term. Data centers are expensive as hell to operate, but so is the cloud. I think colo would have been the way to go.
- intrasight 4y ago
  Most financial analysis are companies are short-term. But labor costs keep going up and cloud computing costs keep going down. So I don't see how this is short-termism.
  - exabrial 4y ago
    I see most places raising their prices as seed funding goes out.
  - hinkley 4y ago
    > cloud computing costs keep going down. For now. I also have to keep reminding my coworkers that AWS gives us faster cores every few years but so far they always have the exact same amount of memory. So caching things (especially in process, but also on loop back) to save computation isn’t really a winning long term strategy. It’s never going to get better than what it does for you in the beginning. It will only decline in value.
    - intrasight 4y ago
      They want you to use other services for caching, right?
wvh 4y ago
Here we are, meticulously crafting open-source programming languages, libraries, operating and distributed systems, only for people to squander the knowledge and freedom on closed ecosystems like Apple's and Google's, and, in the data-center world, the top three cloud behemoths. Assuming typical lock-in, this won't save anything in the long run, nor help with resilience and freedom of choice in the market.
Shorel 4y ago
Not saving, just giving those $400m to AWS or Azure, over some amount of time =)
- citizenpaul 4y ago
  Long enough for this exec team to "prove" on paper they saved money during their tenure. Than hit the road and the next exec team to move it back because 'ooops'
jordemort 4y ago
Is there a website where I can bet on the outcome of this? I've seen lots of companies announce their intentions to move off of their mainframes, but I'm not sure I've ever actually seen one pull it off.
- schmeckleberg 4y ago
  I think the thing you're looking for is a "prediction market". https://en.wikipedia.org/wiki/Prediction_market https://en.wikipedia.org/wiki/Prediction_market Related to that, one funny thing I remember from Robert Heinlein's "The Moon Is A Harsh Mistress" is that the Lunie protagonist didn't understand why Terrans bothered with insurance companies. Are there no bookies?
rzz3 4y ago
It’s kindof shocking to see it take so long to move away from mainframes. I understand it a bit with regard to banking specifically, but I’m shocked to learn FedEx is such a dinosaur.
sithadmin 4y ago
Guessing they're not counting stuff at their flight sim training center (12 bays) as a 'data center'. There is zero % chance that the workloads supporting simulators are gonna run in the cloud. They're incredibly latency sensitive, extremely finicky, and just getting the stuff virtualized continues to be a struggle for aviation OEMs and flight training centers (though FedEx was a pioneer in this regard). Granted, the footprint per sim bay isn't huge (1-2 racks, usually, sometimes 3), but it can add up at the larger sim facilities. Several of the major airlines are running 30+ bays simultaneously, with United poised to come in with the most at 40+ in the near future.
- dx034 4y ago
  They have up to three racks for a flight simulator? I would've never thought they're that complex. Do they need that much computing power or is it other devices that are needed?
  - sithadmin 4y ago
    2/3rd's ish compute, 1/3 other stuff from what I've seen. A good chunk of the compute are appliance-type boxes supporting simulation of particular embedded systems in a traditional environment. Most of the complexity comes from integration of these misc accessory systems. In general aside from the boxes handling the graphics projection, most of the 'compute' is idle a lot of the time, even when the simulator is running. There's usually a big burst of work to prep and load the simulation into the simulator, then minimal (but very latency sensitive) chatter back and forth between the various simulator components and the backend compute as the simulation runs.
stjohnswarts 4y ago
I think they will regret this in the long run when they have disagreements with their providers or their providers go bankrupt and fold overnight because of corporate mismanagement/thievery (ala Enron). Even Amazon and Google will go under eventually...
- dx034 4y ago
  Or the amazon cloud becomes the new IBM mainframe and in 20 years they all work on getting off it, but can't because they are so deeply integrated.
  - stjohnswarts 4y ago
    it would be hilarious if IBM is the immortal one and eventually buy out a limping AWS and keep the software running for the next N decades.
Blackstrat 4y ago
Extremely shortsighted. But the cloud is still the big trend. It’s a “brilliant” move until it isn’t. And then the pain is monstrous.
qwertyuiop_ 4y ago
This is because companies like AWS are able to perform Undifferentiated Heavy Lifting. There is no need for companies with undifferentiated workloads to keep investing to Capex that will depreciate as soon as they are stacked.

492 points by taubek 4y ago

Is there any gain here aside from cost saving? Will this saving be passed on to the customer? (laughs). What happens when the inevitable AWS outage then causes a global shipping bottleneck? Have we still not learnt that having sovereignty is better than saving money? Has COVID taught us nothing? sigh

deleted 4y ago
[deleted]
jupp0r 4y ago
What makes you think that a cloud provider outage is more likely than an existing-Fedex-datacenter outage?
- rbanffy 4y ago
  Mainframes themselves are extremely reliable. When configured as a cluster they get five nines. I'd say a power or communications failure is much more likely than mainframe downtime. With cloud you can rely on more datacenters and, if your application is built right, it may end up being more resilient than what a mainframe setup would give you. Downtime is a fuzzy thing. Whenever possible, I design my apps for graceful degradation - if the database becomes read-only, we can still operate in degraded mode. If we lose queues, some things will not work, but others will continue normally and many users will be completely oblivious to the alarm bells at the NOC (just kidding, there's no such thing). My SLA for full operation is much more relaxed than for degraded operation.
  - quickthrower2 4y ago
    I think a lot of sites would benefit from offline-first and cache a bit of information, perhaps encrypted with the same password you used. There is a lot you can still do if you can a functioning site (served from a service worker) with some cached data. You could read HN articles you read before, for example! Not to rely on this for the nines, but as a nice way to keep things useful when there is an outage or just slowness.
sokoloff 4y ago
What’s to gain other than $400M in savings every year indeed? If it drops all the way to the bottom line, that would boost their 2022 profits by 35% and 2021 profits by about 45%. Seems like a decision you have to consider, even with possible failure modes.
- oblio 4y ago
  https://en.wikipedia.org/wiki/FedEx https://en.wikipedia.org/wiki/FedEx Wikipedia says their net profit for 2021 was $75 billion, so your math seems off.
  - sokoloff 4y ago
    You are correct that I was wrong. (I searched "fedex profit 2021" and incorrectly reported based on a quarter's profit, which is sloppy as hell on my part.) That said, it seems like $75B is wildly too high as well. Page 43 of their most recent annual report shows a consolidated net income for the year ending May 31, 2021 as $5.2B, making $400M still a 7.5% boost (if it all passes through). https://investors.fedex.com/financial-information/annual-reports/default.aspx https://investors.fedex.com/financial-information/annual-rep... https://s21.q4cdn.com/665674268/files/doc_financials/annual/2021/389437(1)_12_FedEx_AR_WR.pdf https://s21.q4cdn.com/665674268/files/doc_financials/annual/...
  - lotsofpulp 4y ago
    Wikipedia is wrong, see page 43: https://www.sec.gov/Archives/edgar/data/1048911/000156459021037031/fdx-10k_20210531.htm#ITEM_6_SELECTED_FINANCIAL_DATA https://www.sec.gov/Archives/edgar/data/1048911/000156459021... I am not even sure where the $75.231B erroneous net income figure comes from, it is nowhere in the 2021 report: https://investors.fedex.com/home/default.aspx https://investors.fedex.com/home/default.aspx
  - mmiyer 4y ago
    That Wikipedia figure is vandalism from a couple weeks ago that I reverted.
- ClumsyPilot 4y ago
dchftcs 4y ago
I'm not sure having mainframes counts as sovereignty. Yearly savings suggests it doesn't.
iso1631 4y ago
Fedex will continue to charge as much as they can to boost their profits. This move will increase their profits, making $1.50 on each of 100 million deliveries instead of $1, boosting profits from $100m to $150m However if they were to undercut UPS and others they might be able snap up UPS business, dropping the profit to $1, but on 200 million deliveries, and thus they'll make $200m instead, you the customer will have a cheaper delivery, and everybody wins
- nly 4y ago
  Until it becomes a race to the bottom and UPS drop their prices as well. Then both companies suffer
  - JumpCrisscross 4y ago
    > becomes a race to the bottom and UPS drop their prices This is competition.
  - iso1631 4y ago
    And all consumers win. This is the beauty of capitalism, it's how it's supposed to work. It fails when you get high barriers to entry and company colluding
  - yuliyp 4y ago
    This is how the "passing the savings on to their customers" happens.
  - dx034 4y ago
    They're not really competing on price but on quality. They make the most money with super urgent deliveries. If they offer later pick up times or earlier guaranteed deliveries, customers will be happy to pay more. And their IT system is crucial for that.
jon-wood 4y ago
The immediate gains I can see are: * In-house development teams have more agility in standing up new services, since they can just use EC2/ECS/one of the other 1500 container runtimes AWS has, rather than having to wait for someone to provision new physical servers for them. * An extensive suite of complementary products are available from cloud providers, it's not just about being able to start a VM, in many cases you don't even need to because you've got Lambda, and the various managed services. * Less training/hiring overhead, since people who can use AWS are pretty much a commodity at this point, whereas people who can manage a mainframe are an increasingly rare breed. * Redundancy becomes easier. As does data locality, let's say Singapore declare all services delivered in Singapore must be hosted there. That's a lot easier to handle if you don't have to go and build/acquire a data centre first. * The aforementioned 400 million dollars per year.
- throw0101a 4y ago
  > In-house development teams have more agility in standing up new services, since they can just use EC2/ECS/one of the other 1500 container runtimes AWS has, rather than having to wait for someone to provision new physical servers for them. How many people run physical servers 'raw' anymore? I would hazard their current stack involves VMware, Hyper-V, Open Stack, or a combination of the three. You can have an API system spin-up just as effectively in a private cloud as a public one.
  - sofixa 4y ago
    That handles IaaS, now do all databases (relational, NoSQL, NewSQL, KV, etc.), message brokers, object storage and a hundred other services. OpenStack handles some of those, with varying degrees of success, but with VMware and Hyper-V you have to DIY from scratch with IMHO the wrong level of abstraction.
  - Jochim 4y ago
    I worked somewhere with a private cloud. It was permanently over-subscribed and the VMs were nearly unusable because of it. Not saying it can't be done well but there's less incentive for a company whose main business isn't providing infrastructure to ensure they've provisioned enough resources to meet demand.
knorker 4y ago
You can take part of the savings by buying NYSE:FDX stock. I'm not being facetious.
aliswe 4y ago
These hyper-ideological takes on finance are so tiring. If you think they are bound to make more profits, I suggest you invest in them, they are called FDX on NY stock exchange. EDIT: I used to be like that, too.

sschueller 4y ago

So are the moving to AWS? Are we going to have a complete breakdown of packages delivery next time there is an outage at AWS?

badgers 4y ago
This was from 2019: https://www.switch.com/fedex-signs-10-year-data-center-infrastructure-agreement-with-switch/ https://www.switch.com/fedex-signs-10-year-data-center-infra...
joekrill 4y ago
> The company is a known Oracle Cloud and Microsoft Azure customer.
- ehnto 4y ago
  Hard to say for certain. Companies that size often have many very independent operations run by separate internal departments. I have worked places with internal, AWS and Azure operations underway.
  - jon-wood 4y ago
    Where I am various parts of the business run workloads on AWS, Google Cloud, and Azure. To some extent I think this sort of behaviour is encouraged by large corporates because it makes contract negotiations easier if you're able to wave in the direction of the other cloud providers you're using and suggest it wouldn't be too hard to migrate over to them.
- andyjohnson0 4y ago
  I read the parent comment as questioning the wisdom of relying on a cloud provider (but not specifically AWS, as you indicated) for a global operation life FedEx. Even with service zoning, in extreme cases that provider constitutes a potential single point of failure. Unless FedEx are able to run their core systems on Azure and Oracle‡ then I think the point stands. It would be interesting to know how Fedex sees the risk/reward equation, and how Azure (for example) compares to a (IBM?) mainframe data centre in terms of reliability and uptime ‡ I'm not sure if there are any cloud neutrality solutions that permit this, but it is possible that FedEx have rolled their own.

baud147258 4y ago

Are they really going to save 400 m$, or was it the figure "promised" by the shop that will handle the migration?

afavour 4y ago
Either that or it’s an accurate figure for the first year or so while the cloud provider gives a ton of credits. By the time that all runs out the senior exec behind the migration will have moved on and someone else can do the maths.
kurupt213 4y ago
$400M sounds huge, but why do I think it’s trivial next to FedEx’s operating budget? A couple years ago they were making 18 Billion* a quarter. * edited from million (meant to type billion)
- oblio 4y ago
  If think you mean $90 *b*illion a quarter.
  - lotsofpulp 4y ago
    $90B per year. https://www.macrotrends.net/stocks/charts/FDX/fedex/revenue https://www.macrotrends.net/stocks/charts/FDX/fedex/revenue
makeitdouble 4y ago
Is there any reasonable answer to this question ?
arghwhat 4y ago
It certainly seems unlikely. Maybe it's the current operational cost, not subtracted the new operational cost. Maybe it's due to already adopting cloud and thus having paying double. Maybe it's due to the use of mainframes, rather than conventional servers. Either way, it's definitely not the cost difference between on-premise and cloud. Cloud providers are not charities, and their buildings and staff are not cheaper than yours.

benj111 4y ago

What's the 8MW in this context? Is that power. Or something else?

bcraven 4y ago
"We presently own and operate four primary campus locations (“Primes”), encompassing 12 active multi-tenant data center facilities with an aggregate of up to 14 million gross square feet (GSF) of space. Our existing facilities are equipped to provide up to 490 megawatts (MW) of power, with the potential to scale to more than 1,300 MW upon full build out of our existing footprint."[0] It sounds like Switch Inc. advertise their datacentres by Wattage. I am not sure how common that is. [0]https://investors.switch.com/company-profile/default.aspx https://investors.switch.com/company-profile/default.aspx
- tgsovlerkhgsel 4y ago
  Very common.
  - benj111 4y ago
    Do you buy by the watt though? It also advertises the square footage but you don't buy by the sq ft either
    - theideaofcoffee 4y ago
      > Do you buy by the watt though? Usually in larger DC deployments, yes, you are quoted, metered, and billed by total power. The amount of space is (usually) an afterthought. There is more space than you can fill and remain under the power commit, which also correlates to total cooling load too.

rbanffy 4y ago

Reverend Mother Mohiam: Many men have tried Paul: They tried and failed? Reverend Mother Mohiam: They tried and died.

bryanrasmussen 4y ago
context is everything, reading that I thought it would be hilariously funny as an exchange in Father Ted.
- deleted 4y ago
  [deleted]
- sackerhews 4y ago
  Wow, this both a fantastic joke and could be used as a school book example of the importance of context. Given of course that everyone would have watched Father Ted. But the world would be a better place if everyone did.
  - rbanffy 4y ago
    Indeed. Hadn't heard of it before moving to Ireland, I credit my loving neighbors for introducing us to this national tradition.

bradfa 4y ago

It would be interesting to see the actual numbers on how much such a transition will cost. Can their existing mainframe software and databases simply migrate into a cloud offering? If not, then if their planned transition has delays or technical hurdles, how much of that cost savings is affected? If their cloud providers change pricing structures in future contract negotiations or if the amount of data they transfer in/out of each cloud provider location changes dramatically in order to better serve their customers, how much of that cost savings is affected? It's clear that you can run a business with minimal amount of physical plant for servers. It's still not entirely clear to me that large established businesses can actually save money this way.

kragen 4y ago

Ultimately companies that abdicate their informatics operations like this will give their profits to their data-center operators, who will be empowered to charge them whatever price they want. Because what's their BATNA? Migrating from Azure to AWS when Microsoft doesn't want to let them? Renting your information infrastructure is a great way to reduce startup costs, but down the road, that information infrastructure runs your company. Trying to outsource it is like trying to outsource upper management. To be clear, I'm not saying that the optimal amount of cloud services for an established company like FedEx to buy is 0. They bring in management consultants, too. But it sure isn't 100%.

cs702 4y ago
Without a doubt, a future generation of executives will revisit and reverse the decision to rent all information infrastructure, but that will likely be many, many years down the road. In the meantime, the current generation of executives who made this decision will look very smart for saving the company lots of money for a good number of years. And they stand to benefit personally from it. They're doing the rational thing!
- noobermin 4y ago
  It's amazing there are no other voices in the room pushing back against this. It really makes you wonder how companies even function at all, much less make money. Being so short sided never ends well.
  - jsiepkes 4y ago
    Convincing people future risks outweigh the short term gains is very hard. That goes for all things.
  - numbsafari 4y ago
    There are very likely voices, but they didn’t win this battle.
  - cs702 4y ago

hnarn 4y ago

Making accurate calculations of data center cost is of course complicated and rarely manages to take everything into account, but the common knowledge I’ve heard is that there comes a point when a company is so large that going on-prem is what actually saves them a lot of money. If FedEx is not large enough to be one of those companies, how large do you actually have to be? Or has cloud pricing changed to the point where this is no longer true, and even huge corporations can save money in moving away from their own premises?

Out_of_Characte 4y ago
Technology scales faster than fedex's data needs. Maybe somewhere in the future the reverse trend happens as their needs can be met with very small servers. Though I've always been wary of putting your data somewhere because getting your data to a competitor costs alot of money.
- ClumsyPilot 4y ago
  > Technology scales faster than fedex's data needs. I don't think this sentence has meaning
  - geodel 4y ago
    I mean it is like executive statement e.g. Real time analytics in Cloud IoT on Edge.
- throwawaymaths 4y ago
  I don't think it's the server costs they care about it's stuff like db maintenance, security, security, and security. Security professionals are expensive, and it's very hard to get right in house on metal.
  - hrunt 4y ago
    Moving away from on-prem data centers does not reduce security costs. At best, it's a lateral move, and depending on the cloud setup, may actually be more expensive. Ultimately, this is an opex vs. capex decision. Data centers are expensive, and require a lot of up front capital to go into the ground. FedEx is worldwide and moving more technology to the edge, so buying land, designing and constructing buildings, and then operating them for 10 years or more in a large number of locales requires a huge investment. They can take that money now and solve their problems immediately. They are also not saying what they mean by "closing its data centers". FedEx announced a 10-year partnership with Switch last year to build and operate edge facilities. FedEx may be moving out of data centers it operates on its own into facilities built and run by others, and using "cloud-native" designs deployed on hardware in those facilities.

Aeolun 4y ago

Cue 10 years down the line, when everyone realizes cloud comes with tradeoffs in flexibility and reliability and they all move back to on-premise for business critical stuff.

cs702 4y ago
More like 20+ years -- after a new generation of executives and managers has taken over. Their egos won't be tied up in old debates and they will be more willing to sacrifice old decisions.

Ekaros 4y ago

Where are they using all of these data centers and mainframes? That is they are spending at least 400 million to do what exactly? To me that seems like rather high figure in any case for their sites and logistics...

ajsnigrutin 4y ago
Yep... A couple of million of packages daily + tracking a fleet of vehicles, client facing site, apis for their tracking and courrier services/devices plus some internal services (billing etc...).. this seems like it would fit on a few racks, not nearly 400million, or whatever the full number is (since 400m are just the savings). But yeah... old company, they might still have stuff written in cobol virtualized somewhere.
- jeltz 4y ago
  But even virtualized Cobol should fit in a few racks.
- hoofhearted 4y ago
  Their Memphis air hub processes 2 million packages by itself during the season. You also forgot to include FedEx Freight in your estimate.
  - hoofhearted 4y ago
    edit - Autocorrect was terrible on my phone today!... The Fedex Memphis air hub processes more than 2 million packages a day itself during the winter holiday peak season.
- dzhiurgis 4y ago
  With 300k employees you can expect them to have thousands of disparate services that will likely get rewritten part of such migration
oblio 4y ago
With (most likely) 0 knowledge about the internal details of their business, but with the confidence of the average programmer: "they spend too much". I imagine Twitter could also be built over a weekend, right? :-)

kurupt213 4y ago

What price is acceptable for ownership of your data?

fanf2 4y ago

”FedEx first opened an on-premise data center at 250 Spectrum Loop in Colorado Springs, Colorado, in 2008” That seems implausible: where did they keep their mainframes before 2008?

perlgeek 4y ago
Colocated in another company's data center.
- dx034 4y ago
  They could get back to colocation. I think even AWS & Co don't own all physical buildings, they also rent parts of data centers in some locations. You don't need to own real estate or a physical building to run servers.

dna_polymerase 4y ago

Maybe, just maybe FedEx had a bunch of employees making a lot of calculations before going this route. Just maybe they made an educated decision, based on their inside knowledge of the company and it’s needs. Or maybe a bunch of people on HN just know better.

langsoul-com 4y ago

How did Dropbox end up saving tens of millions bt making their own cloud infrastructure but FedEx loses money? Surely FedEx would have a metric shit ton of data?

jeffbee 4y ago
Ask yourself what probably happened a few years later when they had to negotiate new leases from their datacenter landlords. Funny how they never put out a follow up press release about the enduring value of on-prem. Also a company that basically can’t grow and hardly turns a profit. Really, not the first example I’d reach for.
humanistbot 4y ago
I'd bet Dropbox has way more raw bytes to store than FedEx. Plus, the Dropbox business model is based on data egress to clients, which is really expensive on cloud servers. FedEx needs data centers more for internal logistics, which is probably why they're going with Oracle.

fithisux 4y ago

Everything will merge to one BIG company because of cost-effectiveness.

thedougd 4y ago

From an executive viewpoint, this would be a reasonable forcing function to overcome long, deep rooted stasis in your IT department. Let's assume for a moment that modernization is good. It probably is wrt mainframes. For every team that refused to modernize, you can now drive them to the 2024 date, absolutely no exceptions.

bythckr 4y ago

I never understood the logic behind companies abandoning their own data centers and moving their gold (Data is the new gold) into somebody else hand. I see that many here are also thinking in the same line. But when I spoke to a CTO of a F500 company. This was the take he had. He feels one of the major driving force is to reduce the carbon foot print of the company. His company is designing the system to have the crucial data on their own infrastructure but keep they other data & do the processing of cloud service. Instead of relying on 1 provider, they are using multiple. Also supporting some small local cloud providers. This seems very relevant for a logistics company with a huge of fleet of air craft. Especially considering the upcoming carbon tax.

hotpotamus 4y ago
I'm skeptical of the carbon footprint thing for a few reasons, but the other point is interesting and something I think about once in awhile. As we build out more and more bandwidth, compute, storage, and APIs to tie them all together, it seems that the world more resembles some sort of large computer and most of the work is just getting the data and compute closer together when needed and minimizing costs when it's not. I know I'm far from the first to have this realization.

TheRealDunkirk 4y ago

I've seen two mainframe "replacements" fail to either retire the mainframe, OR produce an alternative system which people like. Has any Fortune 500 company successfully "retired" their mainframe?

Someone1234 4y ago
I've seen it. It was essentially four steps: - Build middleware (REST API endpoint) in front of the mainframe that was 1:1 mainframe functions. It initially does nothing except auth and routing. - Re-write/migrate all other software from using the mainframe directly to using the new middleware. - Work to further restrict anything directly connecting to the mainframe aside from the middleware, including staff's terminal access by building management solutions (e.g. new UIs, new management middleware). - Once the mainframe is completely isolated, start building drop-in replacements for business segments and then switch the middleware(s) to point at the new solution from the mainframe. This should be invisible to all external consumers. You can "hot test" it by having the API execute on the mainframe + new solution, and checking for 1:1 responses. The hardest part about the above isn't the tech, it is the internal norms that you're fighting against (e.g. staff have had terminal access for tens of years, and know the commands to get their work done by memory). So you have to create very compelling alternatives using the new API middleware/web/apps, "export to Microsoft Excel," graphing, mobile support, and similar that a terminal cannot directly offer is clutch here.
- malfist 4y ago
  This is exactly what Hilton did in the 2010s. We stood up an ESB (wso2) between the modern front ends and the legacy mainframe backend, and then broke down the mainframe's functions into microservice domains and wrote them. Had the ESB route traffic to the microservices until the mainframe wasn't doing any work anymore and was able to discontinue it. There was a lot more complexity that that, but that's the gist of it. By the time I left, my team had carved out the bulk of the mainframe's services and had stood up over 50 microservices.
- kjs3 4y ago
  This is (essentially) how I've seen it done as well, the very few times I've seen it done successfully. But did you guys do it in 2 years, like these folks think they are? The hardest part about the above isn't the tech, it is the internal norms that you're fighting against This is so true it's almost painful.

ryanmercer 4y ago

I left FedEx (Trade Networks) after 15 and a half years earlier this year. The entirety of that 15 1/2 years was spent using IBM AS/400 terminal to interact with various systems and was also used to submit paperwork to Customs and any other applicable government agencies to clear international freight. It... was an experience. I imagine that is going to take quite a bit of work to migrate over to someone else's servers, as I doubt they're going to be like "sure, we can accommodate your decade and a half old, mission-critical, systems"

mensetmanusman 4y ago

And then AWS will charge $399M…

dontbenebby 4y ago

There's not really a market for three big airline shipping things. Amazon air + UPS are tag teaming them in a bad way. This isn't about IT, it's about that FedEx is not in great shape.

lettergram 4y ago

Lol “saving $400m” really equates to “moving onto AWS or Azure and paying hundreds of millions to them”

say_it_as_it_is 4y ago

This is an ideal project for managers whose end of year bonus is linked to immediate cost savings. Who cares if it costs FedEx more in the long run?

chevman 4y ago

I'm in corp land and this to me reads as the CIO using external PR to push internal agendas. Are they really going to close all their on-prem datacenters in the next 2 years? Maybe. More likely, this is a way to build momentum around the transition to cloud. Not a bad strategy, and not uncommon from what I've seen in the enterprise.

shrubble 4y ago

'Within the next two years we’ll close the last few remaining data centers that we have, we’ll eliminate the final 20 percent of the mainframe footprint' Does that truly sound achievable, when they have the most difficult to migrate part of their mainframe applications still on their mainframe? The easy stuff they have already moved off.

treesknees 4y ago

I wonder how much of this is motivated by the lack of staff. The market of available mainframe operators has been drying up for decades. When your entire business depends on a technology nobody is willing to learn or go to school to understand, that creates a problem. Also, from my understanding, it can take a tremendous number of distributed x86 systems to supplement the scale, speed and reliability of a mainframe. It's possible that unless FedEx gets away from managing storage arrays and hypervisor farms, they wouldn't have enough staff to operate the datacenters once they're full of enough servers to replace the mainframes.

thr0wawayf00 4y ago
> The market of available mainframe operators has been drying up for decades. Just like many other industries, the market for mainframe operators that want to earn less than technologists in modern stacks has been drying up. A lot of mainframe operators refuse to bring their pay scales in line with what developers in modern stacks are earning. Why take a pay cut to go work on a 40 year-old system when you can work on modern stuff with more money? They keep the pay low and complain about a labor shortage to deflect from the fact that they're simply choosing to ignore market forces.
kamaal 4y ago
Would be curious to learn the specs of current day mainframes, and cost. What sort of work is typically done on such machines these days.
- treesknees 4y ago
  Disclaimer: I have not worked directly on mainframes themselves. They've been doing the same type of work for decades. Batch processing of records and transactions (payroll, insurance, banking, airline booking, etc.) IBM releases new mainframe models all the time, and typically companies will lease them and upgrade every few years. When I worked at an insurance company, the mainframes were the core of their financials. They had around 8 of them in an active/standby DR site. The machines could do instruction-level error checking, meaning even a weird processor failure/error wouldn't impact the end transaction. Basically if you want to guarantee accurate handling of your money, you use a mainframe. One common thing to see nowadays is the mainframe becomes abstracted by some other layer of API, such as a rest API. This has reduced the need somewhat for mainframe operators, but in the end you're still interacting with some COBOL that was written 30 years ago and somebody needs to understand it and the hardware. And the hardware is so different from traditional servers, you can't just pick it up and wing it. You really need someone who knows what they're doing.

filereaper 4y ago

Its weird to have companies say they're moving from Mainframes to the Cloud. The Mainframe is a sort of cloud, each one has usually double the spare capacity and you call IBM to unlock it. The cost of running a Mainframe is measured in terms of MIPS based on amount of compute used and there's compute reserved for offloading things like JITs (zAAPS). Essentially a version of cloud compute costs. I can understand if you're locked into COBOL with EBCDIC and can't find talent, but Z runs freaking Ubuntu now! I know IBM works furiously on modern ecosystem support. Not shill for IBM, but what exactly is it that they're expecting to be so different from their cloud migrations? Or is everyone here caught up in the "Mainframe old" falsehood.

mhh__ 4y ago
Mainframe = locked in to IBM, is probably the logic.
- JamesAdir 4y ago
  How is it any different than being locked to AWS/Azure/Oracle/Google?
  - notyourwork 4y ago
    Its not be most C-level execs are these companies can present it as IT evolution and progress. Balance sheets look different because accounting has costs in different places. It all looks good.
  - safaci2000 4y ago
    Well your also hardware locked. The last time I worked on a mainframe (about a decade), 8 mb of RAM was $1000, a network card was $800. Commodity hardware around the same time 8mb was probably $50 and a network card even a good one was most $100. You're paying a premium for both the hardware and the lack of develops knowing EBSIDIC, JCL, Cobol, etc.. Actually, from what I remember of Oracle, that might be very similar. I remember having to pay license fees per core.

bosveld 4y ago

Another way of looking at at: Where own data-center cost gets to high it might be due to lack of responsibility to act or inability to manage. A company then have two options: (a) replace/pressure top/middle management to get costs down and processes streamlined or, (b) move to the cloud. Its easier (and possibly good for your resume) to move to the cloud and then a few yours later -- decide again. So moving to the cloud is sometimes an indicator of an inflexible/stale/mature company.

jchook 4y ago

Interesting story as I recall Bank of America saving 5x that amount per year by doing the exact opposite move. https://www.businessinsider.com/bank-of-americas-350-million-internal-cloud-bet-striking-payoff-2019-10 https://www.businessinsider.com/bank-of-americas-350-million...

Spivak 4y ago
Doesn’t matter what you actually do, you get the benefits of shaking off all the barnacles doing any rewrite or rearchitectire.
jeroenhd 4y ago
The difference may be that they're sticking with their own data center but they don't have a mainframe setup. Mainframes come with expensive, locked out hardware, expensive maintenance, expensive experts, you name it. With modern open source tooling, ridiculously powerful graphics cards, easily accessible FPGAs and free data center hosting software like OpenStack, Kubernetes, and glusterfs you can replicate many of the features that made mainframes enticing many years ago. It'll require a heck of a lot of work to get the same performance out of a custom built solution, but long term it's probably cheaper than relying on IBM. I doubt that they'll be saving any money by moving to the cloud unless they're horribly inefficient with their contracts right now. Just moving away from mainframes alone should save a significant buck, but if they were planning on restructuring their digital infrastructure anyway then now is probably the best time.
mbesto 4y ago
This is 100% reasonable and plausible. It's perfectly acceptable that one F500 company can see savings by using cloud and others see increased costs.
dx034 4y ago
That's how you do it. You don't stick with on-prem but just build your own cloud. At that size, AWS won't really buy hardware much cheaper than you. And while you can't build every cloud product that AWS has, the most popular ones are open source anyway.

fsociety 4y ago

The thing I see missing in the comments is that sourcing hardware over the past few years has been incredibly difficult. It will get better, yes, but that may have been the last straw to convince them to switch.

bradfa 4y ago
Past few years? Maybe the past 18 months it's been slightly more difficult but even at the beginning of COVID it wasn't hard to source most anything. And even during these more recent supply-constrained times, you can still source hardware it just might take a little longer for delivery or you might pay a bit more than you did before.

tgflynn 4y ago

I wonder how much of that savings comes from closing data centers and how much from getting off IBM mainframes. It's hard for me to see why a company like FedEx would need to be spending anything like $400m a year on data centers unless a big chunk of that is paying IBM to rent their mainframes.

tgflynn 4y ago
It would be interesting if those down-voting my comment would explain where my intuition is wrong. FedEx delivers 12 million packages per day. If each package requires 10 transactions that works out to about 40k transactions per second. You should be able to handle that with 1 IBM mainframe or much less than 1000 servers and I don't believe either of those would approach $400 million per year to operate.

sbf501 4y ago

I'm surprised at the amount of vitriol being spewed at FedEx in this post for their decision, mostly about the doom-and-gloom that awaits FedEx. Either there are a lot of Fortune-500 CTO's on Hacker News that know better than FedEx, or there is a deep-seated fear of the cloud, specifically PaaS.

simonw 4y ago

"FedEx first opened an on-premise data center at 250 Spectrum Loop in Colorado Springs, Colorado, in 2008" - what were they doing before that?

epberry 4y ago

Surprising to see a company claim massive cost savings for moving _off_ the cloud. But perhaps they're talking about engineering velocity, reliability, etc. Of course everyone knows Dropbox's story with this: https://www.datacenterknowledge.com/manage/dropbox-s-reverse-migration-cloud-own-data-centers-five-years https://www.datacenterknowledge.com/manage/dropbox-s-reverse...

dend 4y ago

This makes a lot of sense if you think through the angle that their core value prop does not come from the network/data latency optimization. It's not Netflix and Dropbox - this is FedEx. Their core business value proposition comes from logistics and being able to react to the information quickly, but extra millisecond latency is unlikely to make or break the business. In light of this, it's easier to outsource the network and data infrastructure to a provider that actually does it day in and day out and put the limited engineering resources to something that is more impactful. Just reaffirms that quite a few companies do not want to run their own servers unless they absolutely have to.

peteradio 4y ago
> through the angle that their core value prop Surely this is just one ingredient to a successful company and there are myriad factors involved. The core business of Boeing isn't to make planes its to sell them.
- dend 4y ago
  That's the wrong analogy. In your example, the former is required for the latter. FedEx wants network stability and a data store, most likely, but the success of their business does not depend on them hosting their own infrastructure.

imperialdrive 4y ago

https://www.switch.com/colocation/ https://www.switch.com/colocation/ The Switch colos all appear to be rendered concepts, but impressively so.

dx034 4y ago
I always wondered what latency requirements they must have in a distribution center that has to route packages dynamically and make those routing decisions within milliseconds. Surely there must be a lot of logic at the distribution center, they probably are their own data centers. Or can you really have 10+ms latency for each decision?

winddude 4y ago

Too bad it's not sooner, could use a flood of decent servers on the second hand market.

fuzzfactor 4y ago

Along with the data, They. Own. The. Data center(s). Well, in some sense or another. They might not be completely paid for but you get the idea, there's got to be a certain amount of worthwhile assets that could be leveraged. Too bad there's not any business executives who have any experience at making money from a data center itself.

formerkrogemp 4y ago

From an managerial accounting perspective it makes sense. Reduce cost centers and focus on core competencies. But many things make more sense from a short-term, myopic, improve this quarter accounting sense that are actually bad ideas.

exabrial 4y ago

"save" In the very short term. Data centers are expensive as hell to operate, but so is the cloud. I think colo would have been the way to go.

intrasight 4y ago
Most financial analysis are companies are short-term. But labor costs keep going up and cloud computing costs keep going down. So I don't see how this is short-termism.
- exabrial 4y ago
  I see most places raising their prices as seed funding goes out.
- hinkley 4y ago
  > cloud computing costs keep going down. For now. I also have to keep reminding my coworkers that AWS gives us faster cores every few years but so far they always have the exact same amount of memory. So caching things (especially in process, but also on loop back) to save computation isn’t really a winning long term strategy. It’s never going to get better than what it does for you in the beginning. It will only decline in value.
  - intrasight 4y ago
    They want you to use other services for caching, right?

wvh 4y ago

Here we are, meticulously crafting open-source programming languages, libraries, operating and distributed systems, only for people to squander the knowledge and freedom on closed ecosystems like Apple's and Google's, and, in the data-center world, the top three cloud behemoths. Assuming typical lock-in, this won't save anything in the long run, nor help with resilience and freedom of choice in the market.

Shorel 4y ago

Not saving, just giving those $400m to AWS or Azure, over some amount of time =)

citizenpaul 4y ago
Long enough for this exec team to "prove" on paper they saved money during their tenure. Than hit the road and the next exec team to move it back because 'ooops'

jordemort 4y ago

Is there a website where I can bet on the outcome of this? I've seen lots of companies announce their intentions to move off of their mainframes, but I'm not sure I've ever actually seen one pull it off.

schmeckleberg 4y ago
I think the thing you're looking for is a "prediction market". https://en.wikipedia.org/wiki/Prediction_market https://en.wikipedia.org/wiki/Prediction_market Related to that, one funny thing I remember from Robert Heinlein's "The Moon Is A Harsh Mistress" is that the Lunie protagonist didn't understand why Terrans bothered with insurance companies. Are there no bookies?

rzz3 4y ago

It’s kindof shocking to see it take so long to move away from mainframes. I understand it a bit with regard to banking specifically, but I’m shocked to learn FedEx is such a dinosaur.

sithadmin 4y ago

Guessing they're not counting stuff at their flight sim training center (12 bays) as a 'data center'. There is zero % chance that the workloads supporting simulators are gonna run in the cloud. They're incredibly latency sensitive, extremely finicky, and just getting the stuff virtualized continues to be a struggle for aviation OEMs and flight training centers (though FedEx was a pioneer in this regard). Granted, the footprint per sim bay isn't huge (1-2 racks, usually, sometimes 3), but it can add up at the larger sim facilities. Several of the major airlines are running 30+ bays simultaneously, with United poised to come in with the most at 40+ in the near future.

dx034 4y ago
They have up to three racks for a flight simulator? I would've never thought they're that complex. Do they need that much computing power or is it other devices that are needed?
- sithadmin 4y ago
  2/3rd's ish compute, 1/3 other stuff from what I've seen. A good chunk of the compute are appliance-type boxes supporting simulation of particular embedded systems in a traditional environment. Most of the complexity comes from integration of these misc accessory systems. In general aside from the boxes handling the graphics projection, most of the 'compute' is idle a lot of the time, even when the simulator is running. There's usually a big burst of work to prep and load the simulation into the simulator, then minimal (but very latency sensitive) chatter back and forth between the various simulator components and the backend compute as the simulation runs.

stjohnswarts 4y ago

I think they will regret this in the long run when they have disagreements with their providers or their providers go bankrupt and fold overnight because of corporate mismanagement/thievery (ala Enron). Even Amazon and Google will go under eventually...

dx034 4y ago
Or the amazon cloud becomes the new IBM mainframe and in 20 years they all work on getting off it, but can't because they are so deeply integrated.
- stjohnswarts 4y ago
  it would be hilarious if IBM is the immortal one and eventually buy out a limping AWS and keep the software running for the next N decades.

Blackstrat 4y ago

Extremely shortsighted. But the cloud is still the big trend. It’s a “brilliant” move until it isn’t. And then the pain is monstrous.

qwertyuiop_ 4y ago

This is because companies like AWS are able to perform Undifferentiated Heavy Lifting. There is no need for companies with undifferentiated workloads to keep investing to Capex that will depreciate as soon as they are stacked.