21 ms·
> the people with physical access is separate from the people with knowledge of [...] Welcome to the brave new world of troubleshooting. This will seriously bi
by pmlnr 5y ago
> the people with physical access is separate from the people with knowledge of [...]
Welcome to the brave new world of troubleshooting. This will seriously bite us one day.
- suyash 5y agofolks with physical access are also denied. source - https://twitter.com/YourAnonOne/status/1445100431181598723 https://twitter.com/YourAnonOne/status/1445100431181598723
- prox 5y agoLet me guess, it is tied to FB systems which are down. That would be hilarious.
- MauranKilom 5y agoFWIW that's not the original source, just some twitter account reposting info shared by someone else. See this sub-thread: https://news.ycombinator.com/item?id=28750888 https://news.ycombinator.com/item?id=28750888
- drdeadringer 5y agoIT: "Please do this fix." Person 1: "I can't, I don't have physical access." IT: "Please do this fix." Person 2: "I can't, I don't have digital access." Why? It's [IT's?] policy.
- RobRivera 5y agolike today! xD
- dsr_ 5y agoIt just bit FB.
- rvnx 5y agoI like how FB decided to send "ramenporn" as their spokesperson.
- huevosabio 5y agoA particular facet I love of the internet era is journalists reporting serious events while having to use the completely absurd usernames... "A Facebook engineer in the response team, ramenporn..."
- noir_lord 5y agoI got quoted as noir_lord in the press. My bbs handle from 30 years ago.
- Apocryphon 5y agoImmortality.
- ivanmontillam 5y ago"Discussed in Hacker News, the user that goes by the 'huevosabio' handle, stated as a fact that..."
- runawaybottle 5y ago‘He was then subsequently attacked by “OverTheCounterIvermectin” for his tweets on transgender bathrooms from several months ago’.
- hdjjhhvvhga 5y agoThe problem with tweets on transgender bathrooms is that you can be attacked for them by either side at any point in the future, so the user OverTheCounterIvermectin should have known better.
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]
- 5y ago
- gbil 5y agothis is not new, this is everyday life with helping hands, on duty engineers, l2-l3 levels telling people with physical access which commands to run etc. etc. etc.
- Scoundreller 5y agoThen you have security issues like this where someone impersonates a client with helping hands and drains your exchanges hot wallet: https://www.huffpost.com/archive/ca/entry/canadian-bitcoins-hacked-rogers_n_4995030 https://www.huffpost.com/archive/ca/entry/canadian-bitcoins-...
- gbil 5y agoThis is not a real datacenter case but normal social hacking. On the datacenter side you have many more security checks plus many of the times the helping hands and engineers are part of the same company, using internal communication tools etc. so they are on the same logical footprint anyhow
- t0mas88 5y agoThe places I've seen this at had specific verification codes for this. One had a simple static code per person that the hands-on guys looked up in a physical binder on their desk. Very disaster proof. The other ones had a system on the internal network in which they looked you up, called back on your company phone and asked for a passphrase the system showed them. Probably more secure but requires those systems to be working.
- deleted 5y ago[deleted]
- munk-a 5y agoTelecommunication satellite communication issues might seriously shut down whole regions if they occur.
- jfrunyon 5y agoI can't fathom how they didn't plan for this. In any business of size, you have to change configuration remotely on a regular basis, and can easily lock yourself out on a regular basis. Every single system has a local user with a random password that we can hand out for just this kind of circumstance...
- chasd00 5y agoAnother Monday morning at a boring datacenter job, i bet they weren't even there yet at 830 when the phones started ringing.
- bink 5y agoYou mean the VOIP phones that could no longer receive incoming calls?
- mro_name 5y agophones? how lame.
- radomir_cernoch 5y agoIt certainly wasn't the Messenger.
- pmlnr 5y agoPhones - the old, analogue, direct cable ones - were self-sustaining, and kept running even when there was a power cut in the house.
- mro_name 5y agoyes, indeed. Reliability. That's so 20th century. #lame. (Actually not lame at all in my eyes)
- steelframe 5y agoAssuming anyone can actually look up the phone numbers to call.
- formerly_proven 5y agoThis sounds like something that might have been done with security in mind. Although generally speaking, remote hands don't have to be elite hackors.
- pmlnr 5y agoHave you ever tried to remotely troubleshoot THROUGH another person?!
- jl6 5y agoYes, and it works if both parties are able to communicate using precise language. The onus is on the remote SME to exactly articulate steps, and on the local hands to exactly follow instructions and pause for clarifications when necessary.
- ikiris 5y agoYeah. Do what you have to. Sometimes the DR plan isn't so much I have to have a working key, I just have to know who gets their first with a working key, and break glass might be literal.
- jfrunyon 5y agoYes. Depending on the person, it can either go extremely well or extremely poorly. Getting someone else to point a camera at the screen helps.
- hamburglar 5y agoMy company runs copies of all our internal services in air-gapped data centers for special customers. The operators are just people with security clearance who have some technical skills. They have no special knowledge of our service inner workings. We (the dev team) aren’t allowed to see screenshots or get any data back. So yeah, I have done that sort of troubleshooting many times. It’s very reminiscent of helping your grandma set up her printer over the phone.
- touisteur 5y ago
- lmilcin 5y agoI don't think so. I bet nobody is ever going to make that mistake at FB again after today.
- cheese_van 5y agoThis is why so many teams fight back against the audit findings: "The information systems office did not enforce logical access to the system in accordance with role-based access policies." Invariably, you want your best people to have full access to all systems.
- Accujack 5y agoWell, you want the right people to have access. If you're a small shop or act like one, that's your "top" techs. If you're a mature larger company, that's the team leads in your networking area on the team that deal with that service area (BGP routing, or routers in general). Most likely Facebook et. al. management never understood this could happen because it's "never been a problem before".
- est 5y agoI think it's the same with supply chains.