12 ms·
FTX stored private keys to crypto assets in plaintext, without access controls
- cs702 4y agoWait, what? Private keys were stored in unprotected plaintext files regularly opened by multiple people at the company? WTF? That crosses the line and goes deep into "willful negligence" territory, in my view. The physical equivalent would be stacking customer assets like dollar bills and gold bars in big piles inside a heavily trafficked room that has no lock. The term "irresponsible" doesn't quite do justice to it. Unbelievable.
- albatross13 4y agoThe fact that you're surprised by this is the most alarming thing in all of this.
- berkle4455 4y agoYet FTX wasn’t hacked. Their own irresponsible bets lost it all instead.
- danielvf 4y agoI mean FTX had over 300 million dollars moved out of company funds, without company authorization, by parties unknown, and with insufficient monitoring to even know it happened until third parties let them know. So kind of depends on your definition of hacked, I guess.
- tough 4y agoSounds like really nice plausible deniability for whomever came up with such a blatant wrong way of storing secrets/value
- notfed 4y agoNov. 11 — Friday: SBF resigns, FTX goes bankrupt Nov. 12 — Saturday: FTX hacked for most of its remaining crypto Y'all be the judge.
- tough 4y agoLet's hope he doesn't buy the judge with the stolen funds...
- HDThoreaun 4y agoThey may have already been hacked and the hackers were laying low. Seems more likely that insiders stole it though yea.
- panarky 4y agoWhy not both? https://www.bbc.com/news/business-64313624 https://www.bbc.com/news/business-64313624
- matthewdgreen 4y agoMy understanding from that document is that the bankruptcy managers are just beginning to piece together an accounting of what happened to the money at FTX, and even that accounting is incomplete because personal laptops belonging to executives are being held back by Bahamian authorities. So “FTX wasn’t hacked” is just a hypothesis at this point. Hacking probably wasn’t the major contributing factor to the exchange’s financial problems would probably be a more accurate statement.
- AlexandrB 4y agoOne way to look at FTX is as a horse race between multiple catastrophic failure modes. In this case the financial malfeasance "won". Maybe in another universe where interest rates stayed low for another year we'd see FTX go down to a hack instead.
- causality0 4y agoPersonally I believe there's a strong correlation between how crazy you act on social media and how competent your internal security is.
- codyb 4y agoThat does not bode well for Twitter....
- clueless 4y agoman, then look out for elon's companies
- deleted 4y ago[deleted]
- michaelsshaw 4y agoTrue competent professionals tend to keep quiet
- WeylandYutani 4y agoThere's a reason why all bankers from Nigeria to Japan wear boring gray suits in commercials. Never understood what people saw in SBF.
- arcticbull 4y ago> That crosses the line and goes deep into "willful negligence" territory, in my view. Er, that's the thing that pushed you over the line? Not all the fraud and crime?
- mnky9800n 4y agoi was okay with the fraud and the crime. it was the hierarchical polyamory that pushed me over the line.
- h2odragon 4y ago> the hierarchical polyamory that pushed me over the line. Really. At that point one should have the decency to declare your outfit a religion, and stop paying taxes.
- mnky9800n 4y agoyou are obviously too clever to have been employed at FTX.
- EGreg 4y agoThat was just ironic I am sure Just like the “very easy math” that they all touted that was all that was needed to manage the entire thing
- mrguyorama 4y agoThe math gets REALLY easy when you do none of it.
- Waterluvian 4y agoHierarchical polyamory? I thought they meant spreadsheets, not spreading the sheets.
- A4ET8a8uTh0 4y agoBoth are bad. Crime is bad, but this is an argument for making software engineering more like a medical doctor's guild. Some things simply should not be done. There is an expectation of competence for some things like finance and medicine.
- logicalmonster 4y ago> That crosses the line and goes deep into "willful negligence" territory, in my view. A lot of people are making the assumption that gross incompetence reigned supreme with FTX, and that does seem like the likeliest explanation, but another potential explanation is deeply devious criminal activity. They could have preplanned this behavior. If they were ever caught doing anything really bad, they had "plausible deniability" by being so loosy-goosy with security and best practices. Everybody from a rogue employee to hackers could be blamed by them if the shit ever hit the fan and they could have some kind of defense that they were so disorganized that they didn't really know what happened. You might be able to distinguish this by looking at how SBF's own personal crypto funds were managed. If he knew enough to manage his own crypto in a saner way, then you could probably make a case that dysfunction in FTX was by design because he knew better and didn't do it.
- gleenn 4y agoWhile I agree with most of this, keeping a small number of things secure for yourself is far easier than doing it for thousands/millions of accounts in an automated way. That's true of almost everything in software. For instance, just because I know how to use a password manager doesn't mean it's easy to get my whole family using a password manager. They were clearly dysfunctional and there may be some of this at play but Occam's Razor says it was just easier to store less securely.
- roundandround 4y agoI think that is what is so hilarious about the situation. The US is forever creating situations where companies want to be first in and fastest scaling at any cost, offering free everything to begin. If they were a bank they would realize their growth is beyond their competence and bring in boring big bank security experts with some of the huge profits on the even more massive holdings.. But there were no legal profits on holding all these assets because they promised to be something better than a bank, making its money from risking your assets, so that was just done illegally leaving no above the table accounts for legitimate operation costs. (A friend of SBF with an illegal loan will obviously keep your keys safe.) How could an honest company that takes negligence seriously compete? It is like the opposite of regulations as barrier to entry. How do you sell things for less than the Mafia's laundering operation?
- AnimalMuppet 4y ago"Sufficiently advanced incompetence is indistinguishable from malice." I don't recall who said it, but it seems to fit.
- unaesthetic 4y agoHanlon's razor is an adage or rule of thumb that states, "Never attribute to malice that which is adequately explained by stupidity."
- AnimalMuppet 4y agoRight. My comment is a reverse of that, expressed in a similar form to Clarke's Law that "Sufficiently advanced technology is indistinguishable from magic." But I stole it. I didn't make it up.
- burnished 4y agoYeah, this is sort of the teleological view. I think Hanlon's razor is a great first guess when trying to figure out people's motivations, but this inverse is probably more appropriate when thinking about what your defenses/reactions should be.
- panarky 4y ago> Wait, what? ... WTF? ... Unbelievable. You sound shocked! shocked! to find gross incompetence going on in a place where the accounting system is an Excel spreadsheet manually maintained by the CEO himself, with entries like "Hidden, poorly internally labled fiat@ account" (sic) purportedly worth $8 billion. Private keys in plaintext in the shared Google Drive that the entire company has access to? That is the least surprising news I've heard today.
- AlexandrB 4y agoI wonder if this reaction is so strong because the readership here are generally more knowledgeable on software and software security than banking. If there was a Hacker News for banking experts, they probably had this reaction to the "accounting" spreadsheet SBF released previously.
- paulcole 4y ago> That crosses the line and goes deep into "willful negligence" territory This is FTX we're talking about. That line is far far far in the rear-view mirror.
- nawgz 4y agoRemember when these guys were buying stadium names and sponsoring esports teams and everything? Pure incompetence was the shining jewel of cryptocurrency. Hilarious.
- iameli 4y agoI get such an incredible volume of malicious crypto spam on Discord and Twitter every single day. They're constantly creating new sites as things get banned, new accounts with legitimate-looking traffic, custom art and even videos for the fake "airdrops" they're planning. And FTX is just keeping their billions of dollars unencrypted in S3 buckets. Hey scammers, wouldn't hacking into that be a better use of your time???
- skyechurch 4y agoWhen reading crypto clownworld stories like this, it is easy and fun to observe that cryptocurrency is a satire of the real (or "fiat", if you prefer) financial system. Less fun, but far more important, is to note how incredibly (infinitely?) subtle this satire is: https://news.ycombinator.com/item?id=22352840 https://news.ycombinator.com/item?id=22352840
- reisse 4y agoOne thinks about crypto as a clownworld only until one had to work with or inside the real financial system. Techincally, it is in no way better than crypto. The only difference is that in real financial system there is a strong legal cover for all the technical and security fuckups. Like, stealing from bank by exploiting their 10-years old Windows XP ATM connected to the internet is 10-years-in-jail offence, while stealing crypto may be hard or impossible to prosecute in many jurisdictions.
- xirdstl 4y agoThat only difference is an extremely important difference
- rossdavidh 4y agoMy impression is that there is a lot more auditing going on in the conventional financial system. Not to say that it's not bad, but there are at least some (legit) outside eyeballs on your system.
- piyh 4y agoMy financial company has amazing enforcement around code quality, deployments strategies, separation of concerns, testing enforcement, escalation, approvals, backups, minimum security standards, vulnerability remediation and so much more. All of this is aimed at being able to keep compliant at scale. It's a large burden, but it's one you take when you hold people's money.
- VHRanger 4y ago
- startupsfail 4y ago[flagged]
- clownpepe 4y agoRight….
- verytrivial 4y agoHey, and just like legacy/fiat currencies, it has made a tiny subset of often nefarious actors inexcusably rich while doing nothing for society, but for Good Reasons, right? Such an improvement!
- Tepix 4y agoI'm pretty sure that some Venezuelans and Chinese people have used it successfully to smuggle wealth outside their countries.
- romseb 4y agoThat sounds like a perfect clone of Mt.Gox. They did the same 12 years ago.
- kickaha 4y agoI’m starting to think that what’s keeping me off the cover of Forbes is my weak-ass sociopathy and my tiny tiny crumbs of competence.
- wnevets 4y agoMaybe it's time we stop conflating net worth with intelligence and competency.
- deleted 4y ago[deleted]
- willio58 4y agoWhile this is flatout insane, it does not speak toward crypto's security directly. If you personally decide that you want a company to hold your crypto that's your decision and a poor one at that. You have the ability to create your own wallet and hold your funds in it securely. Some exchanges like Coinbase even have wallet apps so the transition is super easy to make.
- krunck 4y agoAnd then there are the M of N keys schemes where one can have multiple parties holding the keys. Example: Three keys, any two can sign transactions. You have one somewhere safe(in a safe) and one on your hardware wallet. And the bank has one for your account. The bank can't do anything without you also signing the transaction. You can always sign transactions on your own with both your keys.
- themagician 4y agoWallet "apps" are not secure. It's an illusion of security. Unless the source is open AND verifiable there is no reason to trust it more than trusting FTX. I 100% guarantee that there will be a major hack, from a major app, at some point in the future where it turns out that all seed generation was not in fact random. Honestly, it's probably already happened multiple times and they just haven't gotten caught. Those stories where people claim that all their crypto got stolen even though they never shared their seed—you know the stories that everyone always dismisses as, "You must have let someone else see it." Well, some of those are probably true. When you generate a private key through an app you are 100% trusting that the person who published that version of that app did not do something trivially easy like decide to generate all seeds from a known incremented input. You'd never know. And if that person caught caught internally the company is far more likely to cover it up, because otherwise they will collapse overnight.
- pcthrowaway 4y agoI've never heard of a crypto wallet that was not open source. The coinbase one certainly is. That doesn't mean there won't be hacks and backdoors though. You also need to build them from the codebase yourself if you want to be completely sure that you're running the code that's visible (though maybe there's a better way to verify this with Android/iOS apps?)
- leetrout 4y ago> The Forbes survey also revealed that FTX did not have a SOC audit and was hoping to get these certificates in Q4 2022 or Q1 2023 from Prescient Assurance LLC, but given the firm’s collapse in November, it is unlikely they got them or will do so. https://www.forbes.com/sites/javierpaz/2022/12/02/crypto-exchanges-a-la-carte-approach-to-audits-a-recipe-for-disaster/ https://www.forbes.com/sites/javierpaz/2022/12/02/crypto-exc...
- tough 4y agoPrescient Audits never gone well
- caycep 4y agothere are crypto companies and then there are "crypto" companies....
- RegularOpossum 4y agoAlso, they advertised on fortune cookies at my local Chinese takeout place, which I think told me all I needed to know.
- whalesalad 4y agoI asked a girl to prom in HS with a fortune cookie. It didn't work, but it was a fun experiment.
- SilasX 4y agoI was confused, since this is kind of old news, but it looks to be news because it's part of a new report, which you can see if you scroll wayyyy up to the beginning of White's twitter thread and the link to the court document filed by John J. Ray's team at FTX: https://twitter.com/molly0xFFF/status/1645197258873270276 https://twitter.com/molly0xFFF/status/1645197258873270276 https://www.courtlistener.com/docket/65748821/1242/1/ftx-trading-ltd/ https://www.courtlistener.com/docket/65748821/1242/1/ftx-tra...
- stametseater 4y agoProbably incompetence. Or maybe, done this way to create doubt if/when they decided to take the money and run. "Who did it? Who had access to the keys?" "Uh, everybody.."
- whalesalad 4y agothis is how 95% of startups operate
- renewiltord 4y agoIt wasn't/isn't just them. It wasn't a massive secret either. https://news.ycombinator.com/item?id=32077583 https://news.ycombinator.com/item?id=32077583 The test of all these security exploits are in the exploiting. In practice, you can run wild and nothing will happen. My HN password was 000000 for years.
- RSZC 4y agoI mean I have a yahoo chess account with a password of like abc123, that's not the point. Your HN password doesn't provide access to your money, never mind other people's money.
- renewiltord 4y agoSure, but look at the timestamp on the linked comment. No one took the money for the next 4 months after that. And they had a lot of crypto to take.
- CodeWriter23 4y agoDo we even have a specific term for the opposite of "Defense in Depth"?
- rossdavidh 4y ago"Vulnerability in Depth"?
- AlexandrB 4y agoI'd suggest "Vulnerability in Breadth", since increasing your attack surface and number of points of failure is a good way to ensure something goes wrong.
- whalesalad 4y ago"fake it till you make it"
- Quarrelsome 4y agoAs the tweet points out the bad practice and doesn't comment on the good practice how _should_ one store their keys? Specifically the tweet states that using a secret manager or password vault is a problem, so what is the solution?
- HL33tibCe7 4y agoHardware security modules.
- datadata 4y agoA normal pattern is to tier storage into a hot wallet and a cold wallet. Hot wallet is used for daily operations and can have lower security, but has a very low percentage of value, so that if hacked the exchange and eat the loss. The cold wallet can have very very high security measures such as multisig, physical security, geographic distribution, etc, and only needs to be periodically accessed. Analogy in a old bank with cash or gold is hot wallet = cash that tellers have on hand, cold wallet = vault in the back that has everything else.
- pcthrowaway 4y agoYep. You can also put cold funds behind a multisig, which to use the vault in the back of the bank example, would require two managers to turn a key at the same time to open the vault.
- mithr 4y agoI think that if you, an individual who owns some crypto, wants to store your key in a password vault, that's probably fine, as long as you accept the risks (you've done your due diligence and trust your password manager is secure enough for your expected risk factor, etc). But if you're an exchange handling billions of dollars of customer assets, the requirements should probably be higher. The text implies that many employees at the company had access to the password vault, for example. Also, shared password vaults that I've seen tend to have functionality like the ability to share a password externally (something you probably don't want!), relatively low logging abilities (while it would probably be a good idea to track each and every time a crypto key was accessed and who acccesed it), etc. At least that's my guess at what they meant — perhaps someone had deeper knowledge and can share that.
- datadata 4y agoIt is fantastic that a company operating with such horrific practices is dead. While we are at it, when can we fix similar issues below with mainstream financial systems that millions of people are still using? - Social security numbers are used as a secret for identification, despite being in plaintext and having so low entropy as to be guessable, and originally issued on a card literally saying "Not for Identification". - Every bank check lists the bank account number, which serves as the only information needed for a party to issue a request to withdraw money from that account. - Credit card numbers are similarly a private number used as a public number, and printed on plaintext on the card. Is there any effort working on bringing asymmetric encryption to these systems or to replace them that has a reasonable chance of working?
- janaagaard 4y ago> Every bank check lists the bank account number, which serves as the only information needed for a party to issue a request to withdraw money from that account. Just here point out that the issue is that it's enough to have the bank account number to issue a request to withdraw money from that account, and not that the bank accounts numbers are listed.
- deleted 4y ago[deleted]
- misterprime 4y agoMe, many years ago setting up electronic payment. -OK, it's asking me for my bank account and routing number. These must be pretty secret. -Oh wow, they're both printed on my checks. Uh...this system works?
- themagician 4y agoTraditional banking is sufficiently slow enough that things can be reversed before permanently settled. The "slow" speed of moving money is a feature. It's designed for humans who make mistakes all the time.
- datadata 4y ago
- endisneigh 4y agoHow should the keys be stored? Not excusing them, but this isn't as straightforward to resolve as people think. That being said, not having any access controls is cringe. Really?
- kerblang 4y agoStupid & Evil are best friends - if you're looking for one, the other is often nearby. There's going to be a mix of incredibly stupid and incredibly crooked behavior all through this. It's not a surprise that this "genius" is a delusional, pathetic dolt who couldn't operate a frozen banana stand properly, much less a multi-million dollar company. It's also not a surprise that he truly thinks he's innocent on all charges, because he's that much of an idiot. Crooks typically believe and disbelieve their own bullshit simultaneously, which is even more nonsensical, but that's just criminal thinking at work. None of this is particularly confusing to criminal prosecutors, just the same ol' same ol'. SBF is going to jail, and if he doesn't switch his plea soon he's really going to jail...
- hef19898 4y agoIt is hard to accept, at least for me sometimes, that the answer to the question "Are those people stupid or fraud?" is more often than not is "both". It does explain a lot so.
- akira2501 4y ago> who couldn't operate a frozen banana stand properly, A lot of column inches are dedicated to making it seem that way; however, I have a sneaking suspicion that there's _always_ money in the banana stand.
- qeternity 4y agoMr Banana Grabber royalties
- kibwen 4y agoThis is why seeing people attempt to excuse behavior via Hanlon's Razor makes me fume. For any entity with a sufficient amount of power, stupidity is indistinguishable from malice.
- mabbo 4y agoImagine if you were an employee at this company with access to these keys. They're so disorganized, you could have stolen tens or millions of dollars in crypto and even right now after auditors have gone through everything, still no one would know you had done it. That anyone had done it. How ethical are you really? Could you actually resist that temptation? Do you think all your co-workers could too? What a fiasco.
- burnished 4y agoYeah? Being well compensated makes it much, much easier to resist temptation. Plus the nagging suspicion that you'd fuck up or never be able to use it in a meaningful fashion. I'm not saying I'd be surprised if some one took advantage, but I don't think it would be most folks natural inclination.
- physicles 4y agoExactly. If you have the skills to get hired at a place like FTX, a below-the-radar amount like ten million dollars isn’t an utterly life-changing amount of money. Not something worth risking years of freedom.
- bestbuyer__ 4y agoThat's most probably what occurred during the "hack" that FTX experienced shortly after entering bankruptcy.
- JustSomeNobody 4y agoWould this fall under "fake it till you make it" or "move fast and break things"?
- dbmikus 4y agoI think the general rule is "move fast and break things" unless you are handling money or healthcare. That said, some financial and health-tech firms do seem to fly by the seat of their pants.
- jrockway 4y agoThis sounds like your standard startup-y security stack. Bonus points for trying to use your cloud provider's hardware key storage and keeping secrets in 1password. It ain't great but you could do worse. (I've worked at startups and we did better. No access to the cloud provider without a time-based escalation. Secrets in secrets managers. Passwords rotated regularly. Mandatory 2FA. Signed commits. But it would probably still look god-awful if we were a finance company!)
- amanj41 4y agoWhat's particularly astounding about the case of FTX is that it was rolling in cash (unlike many startups), and yet never cared enough to throw money at hiring tons of security-minded staff and engineers
- jrockway 4y agoVery good point. Spending the money on security engineers at a crypto company sounds like a no-brainer to me. They have a lot of good ideas and the work is essential. (Demanding that crypto keys be stored in Google Drive is the kind of suggestion you'd make if you were planning on stealing all the money, I guess.)
- dieselgate 4y agoYeah have often thought the same but presumably they wouldn’t have been able to keep pulling all these shenanigans/fraud if they had proper security staff.
- japhyr 4y agoIf you read through some of the other parts of the thread, there are stories where they fired competent people as soon as they called attention to these kinds of issues.
- deleted 4y ago[deleted]
- commandersaki 4y agoWhat's wrong with Secrets Manager? I'm sure FTX didn't - but you could setup RBAC using IAM.
- more_corn 4y agoAlso they stole all the money their depositors entrusted to them. Doesn’t matter if you forgot to lock the vault if you sent all the money to alameda anyway.
- poppafuze 4y ago[dead]
- throwaway5959 4y agoSo an attacker could have created a totally anonymous wallet, transferred the assets there and then nuked the wallet? Then those assets would be made irretrievable?
- knifeinhead 4y agoShocker