9 ms·
Disclaimer: I'm CIO @ Namecheap 1. The credentials were resent to an already compromised email account 2. This is an isolated case 3. Established procedure w
by matthewdrussell 10y ago
Disclaimer: I'm CIO @ Namecheap
1. The credentials were resent to an already compromised email account
2. This is an isolated case
3. Established procedure was not followed
4. With thissaid, we've used this as a learning example and additional training has been provided to the individual involved
5. Anyone with any self-managed server with ANY provider should always keep their own multiple backups
- Khao 10y ago> Established procedure was not followed Why have a procedure if your support doesn't follow it? Even if you have a procedure, everything falls apart when it isn't followed. This is the same as having no procedure at all.
- tamar 10y agoKhao - the matter was addressed and the staff was retrained to close the gaps in procedure.
- eximius 10y agoHopefully because the majority will follow it? We have no statistics on this, just this one case that failed.
- deleted 10y ago[deleted]
- SeanDav 10y agoPeople make mistakes. The customer support person was probably just trying to be helpful and not fully aware of all the ramifications. This is unfortunate but presumably there has been some retraining. Note: I have no direct or indirect relationship with Namecheap at all.
- chinathrow 10y ago3. Established procedure was not followed Wouldn't it make sense that support staff can only generate and send out password reset mails if the PIN/password has been entered into a form? I don't know the term for this - like "coded procedure". In this case, the support staff wouldn't even needed to be trusted in the first case.
- vblord 10y agoThis is a great point. The software should be modified to not allow the employee to even make any modifications to the account without the correct credentials.
- matthewdrussell 10y agoAgreed, and we're looking to improve this area too.
- kelukelugames 10y agoI love namecheap but 5 sounds like victim blaming. Come on. EDIT: My use of the term is a bit strong. I feel frustrated that company execs cannot explicitly admit a mistake or apologize. I should have worded it differently. EDIT2: just for Tamar. By explicit I mean literally using the words "sorry", "apologize", or "mistake". What we have is the standard corporate nonapology. EDIT3: congrats to Tamar for being promoted to a Namecheap executive!
- matthewdrussell 10y agoEveryone should practice a good backup routine and take responsibility for backups.
- kelukelugames 10y agoThis is not good damage control/PR. You are letting ego get in the way.
- matthewdrussell 10y agoI respectfully disagree. I'm here, along with Tamar, reviewing and considering each point posted. There's some good suggestions and we're listening. The opposite of what I'm suggesting is that people - individuals/companies - do not look after their own backups. That's a dangerous precedent.
- peterwwillis 10y agoImagine you just lost two servers you can't replace, or you're a potential customer reading this thread, and are afraid of the same. This is what they read as the company's response to this loss: "Anyone with any self-managed server with ANY provider should always keep their own multiple backups. Dumbass." Note the change I made at the end to reflect how some people [who are empathizing with someone who was attacked and lost their property] will interpret that statement. Did any of that statement help the situation at all? Did it help customers feel better? Or did it have the opposite effect? Would this be considered a good way to engender goodwill for your brand? Now consider this reinterpretation of the statement: "With self-managed servers, it is good best practice to keep multiple backups for yourself, no matter who your service provider is."
- sneak 10y agoIf 3 is possible, how are we to believe 2?
- tamar 10y agoI guess take Matt up on his offer where he said this: "Also let me reiterate this is an isolated event. We handle over 10,000 chat sessions every day without a glitch. I invite people to use our live chat service and see what is and what is not possible, as well as the security precautions we have in place."
- jc4p 10y agoI love this. Am I wrong or is it "if you don't believe me, try the social engineering hack yourself!"
- coalescence 10y agoWith #3 - ideally your systems should not allow you to break established procedure. Mitigate the risk by not giving the support staff tools to shoot yourself in the foot so easily. This could be achieved with peer verification or some other mechanism (lots of ways if you think it through).
- tamar 10y agoYes, learning experiences are had when things happen like this. We look to the future, not to the past, to ensure the same mistakes do not recur.
- snowwolf 10y ago> 4. With thissaid, we've used this as a learning example and additional training has been provided to the individual involved This is not the correct solution. What's to prevent the next new person from making the same mistake? If it shouldn't happen, don't make it possible to happen. Put in place a technical solution that doesn't allow it happen. And if there is some special case where it still needs to be possible, make it that it needs a secondary signoff from a senior team member. People will always be fallible.
- tamar 10y agoThat's part of the whole issue. It wasn't simply an issue of retraining. The entire company is well aware of this issue and is using it to improve, not simply to reprimand a single person.
- fencepost 10y agoThis is the kind of 'learning experience' that becomes part of corporate culture and future training. When someone says 'Why bother with all this?' the response can now be 'Read this writeup of how ONE person NOT doing this correctly cost the company a ton of marketing $$$ and STILL left us with a black eye with our more technically-savvy customers.' And how many people at Namecheap do you think aren't aware of this by now? But yes, technical solutions should go in but those take longer to implement. Among other things, it seems to me that re-prompting for the account password might be a good idea before any VPS reinstall/reinitialization that's going to wipe an existing VPS (not that it would've helped much here).
- swanson 10y agoMy hobby: role-playing how I would respond as the CEO if my company was getting skewered on HN. Here is my version! --- Disclaimer: I'm [not] CIO @ Namecheap We messed up, big time. While we handle 1000s of live chat sessions everyday without issue, I realize that even one breakdown in security protocol can cause huge problems and a loss of trust for our customers. In response to this isolated case (in which our established procedure was not followed), we will be creating additional training material for all our live support staff. Additionally, we will be exploring technical solutions to try to make this kind of breakdown much harder. Mistakes happen, but if we can prevent them, it is worth doing. We also would like to take this opportunity to remind folks that any self-managed server (regardless of provider) should always be backed up in multiple places. For information on how to do this with Namecheap, we've published a guide here: <link> I've reached out to author of the post already by email and we are working to help them resolve any outstanding issues.
- tamar 10y ago^^ we'll take it. We've been responding for the last 1+ hour to things in real time across several social networks, so we're a little rushed. But thanks for the role play :)
- helloguille 10y agoYou are really good at it
- biot 10y agoPreviously, IFTTT: https://news.ycombinator.com/item?id=11379475 https://news.ycombinator.com/item?id=11379475
- alasano 10y agoI like your response! If only for the fact that I'm seeing people taking apart the real CIOs response like it's code because it's in a numbered list.
- SolarNet 10y agoIf I ever run a company that screws up I'm calling you.
- opendomain 10y agoMatthew, It sounds like you are confirming that this incident did happen and it was your fault for not following your procedures. I am not a lawyer, but since there was signification loss, it would probably be in your best interest to offer better reparations. OpenDomain has several domains that are on NameCheap - I will transfer them immediately since it appears you do not care about customers.
- tamar 10y agoOpenDomain - We have apologized, admitted mistakes, and made tremendous internal change to move on for the better. We would not do that or even post here if we didn't care.
- alainv 10y agoThe offer of a free year of hosting is nonetheless a paltry joke. The high road here is to acknowledge that customer may choose never to host with you again and still go above and beyond in attempting to make it right to them, e.g. by offering a full refund for the last year of hosting they'd paid for or the like.
- tobltobs 10y agoTell me one registrar where you can be sure that this will not happen and I will move my domains today. I wouldn't even care if I have to pay 100$ per year for a domain.
- deleted 10y ago
- vehementi 10y agoPutting #1 as #1 looks like bitter deflection. You do it elsewhere in the thread too, saying that lack of 2fa on the email account opened the door to this. You should be well aware both that most security issues end up being perfect storm of circumstances, and that attackers can and will target multiple points in the chain. Relying on #1 as the spearhead of your apparent defense here is tantamount to admitting that you are relying on the security of people's email accounts as part of your own security process, which is wild. You also didn't mention all the terrible things the OP pointed out that someone can do with just your password even when 2fa is enabled.
- rudolf0 10y agoIt is petty to list it as #1. However, it's relevant to the story because there's a huge difference between sending a password reset to the email already listed on an account vs. resetting it for any random person who starts a chat. This doesn't excuse their other issues, but it makes the customer support rep's behavior a bit less awful, even if they still violated protocol.
- mtgx 10y agoRegardless, to fix this PR disaster, I suggest you add some strong and perhaps just as importantly modern security features in the future that would regain you good will with HN types (and therefore everyone else). And although it's not you area, can I just say that Namecheap's website is just way too slow since the redesign? I appreciate that you even did a redesign, but for some reason it's one of the slowest websites around. I don't know if it's because of the large images you use on your pages or what's the problem, but I suggest you fix it. It may be losing you customers. A web services company's site should be snappy.
- Patriotspade 10y agoWelcome to the jungle. Kumbaya.