11 ms·
Overheating datacenter stopped 2.5M bank transactions
- BaiCheng 3y ago[dead]
- jessiedarryl8 3y ago[dead]
- thedrbrian 3y agoSo that’s like a month of bitcoin transactions?
- someplaceguy 3y agoTwo nodes in the Lightning network [1] could right now be performing 5M transactions per day and you'd be none the wiser (unless they'd tell you, that is). And that's just 2 nodes out of the current 15K-20K. [1] i.e. Bitcoin second layer.
- CyberDildonics 3y agoExcept that those transactions don't count until they get on the main chain and no one needs to send 1000 transactions of single penny, then have the exact same problems as everyone else when trying to get that to actually be usable.
- someplaceguy 3y ago> those transactions don't count They don't? Then surely you wouldn't mind sending me a few bitcoins over a lightning channel? Pretty please? > no one needs to send 1000 transactions of single penny There is no such "single penny" limit. > then have the exact same problems as everyone else What problems?
- eli 3y agoI sent them over but I used a third layer network (a spreadsheet on my desktop)
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- chrischattin 3y agoCan't tell if sarcastic or not. But, that's perfectly legitimate. You could do that and if everything is signed correctly, the network will accept it. I could write down a seed phrase on a piece of paper and hand it to you, and that's a completely offline transaction.
- someplaceguy 3y agoBut just to be clear, unlike the hypothetical "spreadsheet on a desktop" (even with signed transactions), Lightning has more cryptographic assurances than that. It prevents the parties of a transaction from double-spending balances behind each other's backs. It also doesn't require trusting the other party once a transaction has (almost instantaneously) completed.
- eli 3y agoYou don't need a network at all. Just email me who you want to transact with and I'll create a row for them.
- someplaceguy 3y agoOk, so can I now redeem the BTC that you sent me earlier please, so that I can spend them in the first layer (or the second one)? Given that nobody else seems to want to use your third layer. The fact that I even have to ask you already demonstrates how shitty your third layer is compared to Lightning, not to mention all the other obvious ways in which it's infinitely worse.
- u32480932048 3y agoWorse, Lightning won't let you send pennies, so you have to send 10,000,000 transactions of 0.0000000001 BTC - each transaction consuming the equivalent energy of a big lightning bolt the size of a small lightning bolt (hence the name).
- someplaceguy 3y agoWhat the heck are you talking about? None of what you just said makes any sense.
- u32480932048 3y agoPennies need converted to BTC first, which means many more transactions, which means that Lightning destroys the planet even faster than BTC. It's simple math. BTC has been consuming all of the planets energy since at least 2020[1]. This increased transaction rate of Lightning just means that BTC will soon be consuming over twice the total energy produced by humanity, which is unsustainable. Of course, people who don't understand the scientificness of this all are still defending bitcoin as if we don't already have enough payment apps. [1] https://www.weforum.org/agenda/2017/12/bitcoin-consume-more-power-than-world-2020/
- snapcaster 3y agoI'm not a bitcoin zealot by any means but none of what you're saying is correct
- krisoft 3y ago> BTC has been consuming all of the planets energy since at least 2020[1]. That is plainly false[2], and the linked article does not say what your sentence purports it to say. :D 2: as evidence I can point out that I have seen at least one car moving since 2020. That was clearly using energy BTC did not consume. Same with observing at least one light bulb illuminating, or at least one person breathing.
- throw0101c 3y agoSeems like more testing is in order: > Upon the outage, both banks immediately activated IT disaster recovery and business continuity plans. > "However," according to Tan, "both banks encountered technical issues which prevented them from fully recovering their affected systems at their respective backup datacenters – DBS due to a network misconfiguration and Citibank due to connectivity issues." Perhaps regularly flipping between them on a regular basis, at least for a short time, so the 'passive' part of the active-passive pair gets some work, is worth considering.
- dylan604 3y ago"No recovery plan is worth anything if it is not tested regularly." "An untested recovery plan is worth less than the paper on which it is printed." I'm sure there are many other versions of the quote
- pphysch 3y agoBut think of all the boxes you cannot check if you are honest about your recovery plan
- user3939382 3y agoIf you haven’t tested your backup you don’t have one.
- hinkley 3y agoHave you seen this video of the guy who on some sort of “adventure” outing? They are way up in the air on some sort of platform where you are supposed to jump across these planks from one platform to another. Someone is filming from either a platform or plateau next to the rig. There’s a harness connected to a rail above you to catch you if you fall. In the video the man gets halfway across and the cable attached to his harness just pops off. The instructor behind him instinctively reaches out as if he can use The Force to catch him if he biffs, but he makes it to the other side unharmed, and is all smiles until the cable catches up and bops him in the back. At which point he turns around and connects the dots. It’s a bit like that. And this is the image that comes into my mind every time I discover we have untested recovery processes while we are in the middle of a recovery.
- jl6 3y ago> Question: Anything else besides multicultural tolerance that enabled Singapore's success? > Answer: Air conditioning. Air conditioning was a most important invention for us, perhaps one of the signal inventions of history. It changed the nature of civilization by making development possible in the tropics. https://www.vox.com/2015/3/23/8278085/singapore-lee-kuan-yew-air-conditioning https://www.vox.com/2015/3/23/8278085/singapore-lee-kuan-yew...
- hknapp 3y agoI'd believe it. Ever try doing desk work in a hot room? I get discomfort from sweating quickly.
- deleted 3y ago[deleted]
- forgingahead 3y agoThe article seems to be talking about an outage that happened in Oct 2023, but then references a LinkedIn post that was written in 2021? One of the banks, DBS, has had several outages in recent years, but this seems like sloppy writing.
- xyst 3y agoBlaming contractors/subcontractors is so hot right now. Cloudflare. Equinix. No accountability.
- ToucanLoucan 3y agoDoes this surprise you? What is a corporation if not a legal entity designed to launder responsibility? Have a problem with a service and you find out the company you bought from subcontracts to an LLC you've never heard of which in turn subcontracts to four other LLCs one of which is actually liable but only if you can figure out which one, and oh by the way yours has two employees and it's headquarters is a UPS Store box, good fucking luck.
- gottorf 3y ago> What is a corporation if not a legal entity designed to launder responsibility? Corporations (and other legal entities used for business) are legal fictions that allow people to pool capital for productive endeavors and yes, with limited liability on the parts of the owners, because we as a society decided that encouraging productive endeavors this way is better than the alternative. In a world with no limited liability, you could happily wring the owner's neck if they screwed up, but I'd imagine society as a whole would be a lot poorer off, because very few people would take that risk to engage in productive endeavors. > Have a problem with a service and you find out the company you bought from subcontracts to an LLC you've never heard of which in turn subcontracts to four other LLCs one of which is actually liable It doesn't matter, because the company you actually paid money to is the one that's responsible to you.
- ip26 3y agoI'd imagine society as a whole would be a lot poorer off, because very few people would take that risk Oh, people would make risky ventures alright, but you’d see a lot more private security of the Pinkerton variety.
- lawlessone 3y ago>In a world with no limited liability, you could happily wring the owner's neck if they screwed up I'd say more than a few people would like that opportunity with Boeing
- dan_can_code 3y agoI wonder what the cost of a datacenter within Singapore is? The cost per sq/m in that country, which is the most expensive in the world, must be huge.
- BirAdam 3y agoWell, considering you’d have most of the needed utilities already connected and present, a large population of well educated people, and extremely high bandwidth connections in close proximity, it’s probably only slightly higher than elsewhere.
- latchkey 3y agoI've run equipment in dc's in multiple places in the US, Germany, and Singapore. By far, the most expensive was Singapore.
- Scoundreller 3y agoIs it because it costs the operator a lot to run one or because they can charge a lot? (I guess over time those should converge, but sometimes takes a while or doesn’t)
- latchkey 3y agoAll of the above. Space/Power/Staff is just more expensive there. We also had a lot of trouble because a couple of our Dell servers had issues and it took some time to figure out how to get them repaired. There was just a lack of urgency there too.
- c_o_n_v_e_x 3y agoComparatively speaking, I would expect SG DC staff to be lower cost than western countries. Power is the killer, and it's doubly bad. Not only is it expensive, you're using a lot of power to keep things cool given ambient temp fluctuates typically between 24-33C, 60%-90%Rh 24/7. There was a moratorium on new DC builds in Singapore, so there's also a huge demand/premium on rack space in SG.
- psychlops 3y agoThe overheating datacenter didn't stop the transactions. It was the technical inability of the banks to fail over which should have been seamless. Outages happen.
- jimt1234 3y agoIMHO, "fail over" will always fail. Any multi-data-center architecture should actively utilize all data centers, all the time, maybe dedicate more traffic to one or the other. But, if you've got a "passive" data center that you wanna activate during an outage, you're gonna have a bad day. Every time.
- ms_frizzle 3y agoMy experience contradicts your hyperbole. Over the past two decades I've been part of several fail overs with active/passive and build from scratch DR solutions. Most succeeded. The ones that didn't succeed on the first attempt did so on the second one. Not a perfect track record, but it does happen with proper documentation, testing, and skilled staff to handle it.
- cm2187 3y agoHow to make this problem even worse: make all the banks in the street use the same datacenters, namely either AWS or Azure.
- lawlessone 3y agoAWS version "Overheating employee stopped 2.5M bank transacti..."
- bobbiechen 3y agoIn the EU, the Digital Operational Resilience Act (DORA) mandates that financial services companies must manage their cloud risk - https://www.digital-operational-resilience-act.com/ https://www.digital-operational-resilience-act.com/ Could lead to more multi-cloud adoption among financial services companies who are in scope. (no relation to the other DORA, DevOps Research and Assessment, by the way)
- johgoy 3y agoCrypto does not overheat.
- devaiops9001 3y agoWe are no longer in an era where this should be allowed to happen, even for say... a legacy application that runs on Windows 2000 and needs NTFS on a network block device. After 9/11, the "zero nines" whitepaper was released to push for cloud (never let a disaster go to waste). I have hands-on deployed many K8s+Ceph clusters. Ceph's block storage can be deployed multi-AZ where multiple datacenters each with their own power systems are in close proximity. There are methods like Raft (used by K8s and Patroni) and a really great method demonstrated by Google's CloudSQL that uses a SQL table to infer which SQL server should be a "leader". So like wtf guys.
- bandyaboot 3y agoOf course they blame the mistake of the single contractor rather than their own inadequate system monitoring.
- deleted 3y ago[deleted]
- Arnold07 3y ago[flagged]