7 ms·
The Great Unracking: Saying goodbye to the servers at our physical datacenter
- coderintherye 1y agoRelated: https://news.ycombinator.com/item?id=3248911 https://news.ycombinator.com/item?id=3248911 - Why Stack Exchange Isn’t in the Cloud (2011) The original blog post is down but available here: http://web.archive.org/web/20120120201529/http://blog.serverfault.com/2011/11/17/why-stack-exchange-isn’t-in-the-cloud/ http://web.archive.org/web/20120120201529/http://blog.server...
- jauntywundrkind 1y agoWorthy of a re-submit / retry, but also: posted 15h ago: https://news.ycombinator.com/item?id=44604696 https://news.ycombinator.com/item?id=44604696
- layla5alive 1y ago"We like burning money, and hardware, on fire."
- frankzander 1y ago... and like fall for marketing pitches
- abrookewood 1y ago"For security reasons (and to protect the PII of all our users and customers), everything was being shredded and/or destroyed" - what!? That is ridiculous. The only possible thing you need to destroy would be the hard drives. Why on earth would you shred everything?
- treve 1y agoAlso thought that was quite wasteful. Even ethernet cables are cut. Why not just put them online for free pickup? Also I can't help wondering if the switch to cloud makes sense for stack overflow now again because their traffic collapsed. I took the whole post as something that should be mourned a bit, not gleefully destroyed.
- ZeroSolstice 1y agoFree pickup? at a secure data center? Lets use some logic here. The disposal company is taking the cables with them to recycle them for the copper wire. Same with power cables.
- treve 1y agoThey're cables, someone can pick them up and move them somewhere else. Some more logic for ya
- ZeroSolstice 1y agoWhere? Should they drop them at a starbucks out front? Have an employee volunteer their home address? Put them out front at the office? How long should they hold onto the "free" cables that people are going to ask ridiculous questions such as "Do they work? When were they tested?" Are you going to force people to take all the cable or allow random selections? Are you going to waste the cost savings of moving to the cloud to have one of your tech people monitor requests for pickup? What if no one picks them up are you calling the recycler out again to pick up the cables you could have just given them? Anyone who has posted "free" things online knows it comes a cost, thats the logic part I was referring too. When you work through the scenarios the "logical" conclusion is to give them to the recycler that you already have out at the datacenter for the systems you are decommissioning.
- sebstefan 1y agoTotally unsolvable problems! Let's give up (Put them in a box that says "free" in the lobby of the NY office then throw what hasn't been taken a month later)
- chneu 1y agoThis sounds like laziness disguised as security, lol.
- thomascountz 1y agoIt's as if they believe some bytes of PII might be recoverable from residual capacitance in twisted pairs of copper...
- poulpy123 1y agoI would say they evaluated it would cost more for them to remove the HDD and sell the machine than just shred them. And they would not risk to forget a HDD inside
- msgodel 1y agoThe last couple places I've worked at did this with their old equipment too. It always drove me a little nuts.
- ZeroSolstice 1y agoThis is common compliance nomenclature. The only people paying the high cost to have a full sized piece of equipment destroyed are governments or R&D companies with unique prototypes. The hard drives are most likely being shredded since that is a common practice and certification feature offered by most disposal companies. The servers are being "destroyed" because thats how they will be accounted for in inventory and tax purposes to account for full depreciation. The company isn't "selling" the servers to the disposal company so they are marked as "destroyed." Unless specified in the contract the disposal company will sell the chassis without the drives to a reseller or if they are being paid to dispose of the system, they will separate the components and recycle the metal. The same goes for the power and network cables, they will go off to a recycler, its how disposal companies off-set their pricing.
- abrookewood 1y agoOK that's much more reasonable. Hoping you are correct.
- kingnothing 1y agoThis is correct. Once upon a time, I worked for a company doing this type of disposal. They had 18 wheelers full of equipment show up a couple of times per week. Drives were pulled and put in a pile to be shredded. Everything else was tested and either restored to working order and sold in bulk to organizations in need of cheaper computers, parted out on eBay, or scrapped.
- x______________ 1y agoI've seen the innards of data centers but not what happens afterwards... What happens with the shredded material? Is it recycled? Sent to heavy industries?
- kingnothing 1y agoI believe the place I worked at sold it as scrap metal.
- jraedisch 1y agoThe reasoning could be that this makes reliably scaling down (and thus keep making a profit) easier, starting with getting rid of SREs. We have similar movement going on with Xing here in Hamburg, Germany (once conceived as a LinkedIn competitor). Great names that still have a lot of momentum, but are expected by ownership to slow down. Reminds me of Scott Galloway’s most profitable investment having been a yellow pages company. Yes, the market shrunk, but they could shrink running costs as fast or even faster.
- realxrobau 1y agoAny company that switches from real hardware to "the cloud" is going to triple their compute costs. They're obviously making too much money. With them not even selling their old hardware, they are doing the equivalent of setting large piles of money on fire
- robertlagrant 1y agoIt could just be that they think the increase in their cloud bill will be low enough to justify switching and stopping employing people who currently run things in their datacentre.
- rcxdude 1y agoAnd yet people don't really consider the cost of employing people to manage the cloud infrastructure... In most companies I know of that are all in on cloud that's a larger team than what manages physical infrastructure in the companies that still have that.
- robertlagrant 1y agoYeah, I don't know how it will work out.
- ErrorNoBrain 1y ago"physical" datacenter ? whats the alternative? a datacenter that exists only in my imagination?
- gashmol 1y agoThat would be imperfect.
- sneak 1y agoThis is a terrible waste. Wiping all storage would take a day at best; this hardware is still worth $10-50k. They could donate it. Then again, they’re migrating to Azure and the whole thing ran for years on SQL Server; being good at tech was never these ex-MS guys’ strong suit. This kind of forklifting is expected from this specific type of corporate droid, it’s how they’ve always done it. Entire industries run just like this, and it’s terrible and stupid.
- dangus 1y agoNobody likes to hear it but hardware just doesn’t last that long. Even if you reuse the hardware by selling it off to homelab types or donating it, it’ll get faulty in a few very short years. It’s already been running 24/7 for a long time. It’ll also be far less energy efficient in operation compared to newer hardware generations. I also don’t really see what your beef with SO is for using SQL Server. Are you suggesting that companies do a major refactor every time their software stack goes out of style? There really isn’t anything wrong with SQL Server nor is there anything wrong with SO at a technical level. You might be upset that entire industries run on imperfect tech stacks but value isn’t driven by tech stack choice. I’m not going to buy a sandwich from a different restaurant because they are using better tech than the restaurant with bad food.
- wahern 1y ago> Nobody likes to hear it but hardware just doesn’t last that long. My experience completely contradicts this, both at work and otherwise. The typical longevity of servers has always astounded me. Though my current biggish server (not including routers, etc) is a Zen 1 on 14nm built pre-pandemic; I've read 12nm and smaller might be more susceptible to degradation. RAM feature size has remained fairly constant, but I wouldn't be surprised if SSD MLC longevity has degraded. The real issue, IMO, is that when you have a small set of servers any hardware issues become a bigger hassle. It's precisely because you can go years with a rack of servers humming along that when there is an issue, it feels quite intrusive and annoying.
- dangus 1y ago
- charcircuit 1y agoHave they never heard of remote hands? The cloud will be much more expensive than than what you would pay for someone to replace a hard drive for you.
- hamdouni 1y agoThe cloud is just someone else's datacenter.
- Doohickey-d 1y agoBlog post from 2016, detailing the setup: https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/ https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar... It's a bit of a shame, but I guess also with declining traffic and revenue, they're also downsizing.
- tra3 1y agoSO was the prime example for a well architected monolith. And on windows no less!
- southernplaces7 1y ago"Stack Overflow no longer has any physical datacenters or offices; we are fully in the cloud and remote!" Am I misunderstanding something here? They're just transferring from a physical datacenter owned and managed directly by them, to a small rented/leased part of those owned and managed by someone bigger. Since digital data can't just exist in a magical, airy fairy realm, it has to physically be somewhere either way. Wouldn't it have been safer to control their own physical servers, considering how they mention protecting user information?
- datadrivenangel 1y agoIt means that Stack Overflow as a company doesn't need to think about the exact location, except in so far as they need the actual operator to satisfactorily assert that it's secure and operating in the right jurisdiction with enough backup redundancies.
- adityaathalye 1y agoAs a Private Equity holding, it's safer to outsource regulatory/compliance/security risk and liability to a reputable third party, where reputable is the proverbial "nobody gets fired for buying IBM" of the cloud era.
- jeffbee 1y ago"our datacenter vendor in NJ decided to shut down that location, and we needed to be out by July 2025." Ah, so you're saying that going "on-prem" does not in fact give you total control over the situation? How peculiar! Has AWS ever shut down an EC2 region and forced everyone out?
- adityaathalye 1y agoThey are saying that "on-prem" does not give the website's Private Equity owner full financial control over the situation.
- senko 1y agoThat's a red herring: unless you generate your own electricity and have at least a few uplinks, you're never going to have "total control over the situation". On-prem still may be preferable to cloud for some use cases.
- jeffbee 1y agoThe amount of petty bullshit a "repatriation" shop has to endure is multiples of what cloud customers would put up with.
- adamcharnock 1y agoI think this is more akin to shutting down an AZ, rather than a region, which certainly does happen. Except AZ lettering is randomised per AWS account, so your 'us-east1-a' isn't my 'us-east1-a'. Which means AWS can migrate people away, over time. I believe older accounts which still use the old AZ are given notice that it is closing. Plus, there was the whole closure of AWS EC2 Classic, replaced with AWS VPC. This another reason why deploying over multiple AZs has its benefit. Not just for technical failure, but it means you can still move should one region close down. I suppose an interesting questions is: would I prefer to move a single-AZ deployment such as this in the cloud, or in the real world? And honestly I can see the pros and cons of each. In the cloud it involves a bunch of engineering time (possibly minimal, likely a lot more given reality). In the real-world it involves a temporary fibre connection to the next DC over, and a gradual or rapid move of hardware with the help of some specialist contractors (for example). But at least the state and implementation quirks move with the compute. I can see it either way, but I can feel myself wanting to believe in the latter. There is something about trucking servers across town that appeals me.
- seneca 1y agoHasn't stack overflow been in steady decline for years? How can they justify the huge increase in hardware cost that going to a cloud provider brings? I suppose it makes it easier to rapidly scale down your footprint to meet lower demand.
- robertlagrant 1y agoI imagine they want to replace skilled workers with a cloud bill.
- swiftcoder 1y agoif traffic has been declining, their physical datacenter is likely to have been over provisioned for current usage levels. Laying off most of their SREs is just another bonus for the bottom line
- vachina 1y agoOn my small business, it never sense to move to the cloud. My service cost $0+electricity to run, forever.
- pixl97 1y agoIf you have the knowledge and time it doesn't make sense. That said it's def not a zero cost setup. You are playing sysadmin which isn't a low paid position in itself. You just amortize the actual cost.
- vachina 1y agoIf you choose the right stack, and plan before you setup, sysadmin becomes easy. I could go months without touching the thing running the application.
- pixl97 1y agoYou could. Or lighting could blow the UPS. Or the mainboard could flake out. Or the RAID could drop a disk. Or you could get hit by a bus. Again, there is never any free, at best it is cost deferred.
- rcxdude 1y agoA full replacement of a physical server every few years is still cheaper than the equivalent in the cloud.
- deleted 1y ago[deleted]
- rcxdude 1y agoCloud is not a replacement for a sysadmin, though.
- jujube3 1y agoGrok, what was Stack Exchange?
- lesser-shadow 1y ago...man I wonder whether you have those for sale, I'd love to purchase a piece of histo- >"For security reasons (and to protect the PII of all our users and customers), everything was being shredded and/or destroyed. Nothing was being kept." ... are you retarded? >"As Ellora Praharaj, our Director of Reliability Engineering, said, “No need to be gentle anymore.” Of course it's an Indian Director