4 ms·
Keybase CEO here. Let me tell a quick story. January 2019. I was loading the car to leave for a short family ski vacation when I got a truly horrifying email: t
by maxtaco 7y ago
Keybase CEO here. Let me tell a quick story. January 2019. I was loading the car to leave for a short family ski vacation when I got a truly horrifying email: that my slack account had been accessed from a distant land (that I hadn't been to).
There goes my weekend!
When we first started Keybase, we used Slack as other teams did, but were gradually moving all Slack-based workflows over to Keybase. As such, we didn't use it for anything beyond communicating when Keybase was down. But I was very worried. I knew I used a good, random, one-time password for Slack, so it couldn't have been that the password was stolen from somewhere else. Had my computer been rooted? Had my side-hustle password manager been compromised (oneshallpass.com). I immediately contacted Slack security and asked them if this issue was on their side, and they neglected to point me to the relevant blog article from 2015 (which didn't detail the extent of the compromise, we now know). They just said they take security very seriously and hinted I was at fault.
In the subsequent few weeks, I reset all of my passwords, threw away all my computers, bought new computers, factory-reset my phone, rotated all of my Keybase devices, and reestablished everything from the ground up. It cost me a lot of time, money and stress. In the end, I was pretty sure but not 100% convinced that if I had been rooted, that the attackers couldn't follow me to my new setup. But with these things, you can really never know for sure.
I got the email today that my account might have been compromised in the attack. I would say for sure that it was compromised, and I can breathe a big sigh of relief, that was the explanation I wanted to hear all along.
It was great to know throughout this ordeal that the product we're building --- Keybase --- solves this problem in a fundamental way, not with adding further workarounds (2FA while better than just password alone, reminds me a bit of the 3-digit verification code on the back of your credit card; and if Slack's credential database is compromised again, 2FA won't help at all). With Keybase, all of your data is E2E encrypted, and your encryption keys never leave your device. A would-be attacker who compromised our database would have no ability to access any important user data.
Summary: estimated cost to me:
- $5000 worth of hardware
- 60 hours of labor
- 25 hours of lost sleep
- 10 additional hours of team effort
Fortunately:
* Keybase does not communicate sensitive information in slack such as cap tables, financials, employment decisions or compensation discussions, team reviews, company devops secrets, stupid memes that could be taken out of context, or private DM'ing. Basically we just use a `#breaking` channel in Slack, for when we break Keybase.
* Keybase itself is immune from this kind of break-in.
Edits: wordsmithing and improvements
Also: Import your Slack team to Keybase: https://keybase.io/slack-importer https://keybase.io/slack-importer
- jhiesey 7y agoWhy throw away your computers? Why not remove/disconnect the batteries (if portable) and just store them somewhere in case you eventually no longer suspect a hardware hack, as in this case?
- maxtaco 7y agohttps://www.welivesecurity.com/2018/09/27/lojax-first-uefi-rootkit-found-wild-courtesy-sednit-group/ https://www.welivesecurity.com/2018/09/27/lojax-first-uefi-r...
- malgorithms 7y agoFurther: Keybase is a security product and it wasn't deemed worth the risk for the CEO. And while Keybase isn't made of money, the $5k was roughly irrelevant compared to the other costs mentioned here and the _magnitude of the risk_. If you haven't been through this kind of thing, it's hard to understand how scary it is to have a break-in of unknown origin. If you use strong, unique passwords as Max did, then you're almost certain it's a server break in (and again, this is why Slack is scary for sensitive info)...but being 99% certain isn't enough. Removing that computer permanently from the team gave peace of mind.
- frivolous 7y agoSorry, but you're just being wasteful. You were looking for an excuse and you know it.
- voiper1 7y agotl;dr: UEFI rootkits can survive operating system reinstallation and even a hard disk replacement. That's why he needed a new physical computer.
- ubercow13 7y agoI think if this happened to me I would just assume the company was hacked due to poor security practices. It seems much more likely than my password being stolen from a device of mine considering that a very significant percentage of companies I have account with seem to have had breaches at some point. But maybe I am just naive.