10 ms·
Alameda lost tens of millions because of a fat fingering mistake
- kristjansson 3y agoSo like ~0.5% of their total losses?
- pharmakom 3y agoThis happens in “proper” firms too
- c7DJTLrn 3y agoCase in point: https://en.wikipedia.org/wiki/Knight_Capital_Group https://en.wikipedia.org/wiki/Knight_Capital_Group
- HDThoreaun 3y agoI believe that knight was not a fat finger
- c7DJTLrn 3y agoSorry, you are correct. It does demonstrate how brittle things can be at large funds though.
- xyzelement 3y agoThe number of things that had to go wrong at the same time in the Knight case was large as far as I remember.
- lokar 3y agoI’ve seen straight up attempts to sell a massive amount of a major stock at ~0 due to a single bug and failure of backup checks.
- blibble 3y agoon a sensible exchange that attempted match would trigger a circuit breaker and either halt the market for a short period or put it into an auction (and members that create orders to do that a lot either get their membership removed or pay fines)
- CaffeinatedDev 3y ago"the utility we gained by moving fast outweighed the occasional costs we paid due to poor risk checks, hacks, and the like" I think this may have potentially been true when everything was going up volume-wise, but as competition improves there's less margin for error.
- paulpauper 3y agoit's not about the number of mistakes but the size of the mistake
- dragonwriter 3y agoIt was true from Sam's perspective, where all of those holes provided cover for fraud, embezzlement, and self-dealing (possibly up to and including the large “oops, unattributable hack” loss right as Sam was about to be shoved out the airlock.) But that's because the holes were actually directed added utility from that perspective.
- ftxbro 3y agoIt's wild to think that just one fat fingering mistake was responsible for the collapse of the whole FTX and Alameda ecosystem.
- paulpauper 3y agoi don't recall this crash. must have been nanoseconds. I guess in hindsight that was the least of its problem. billions dollars missing makes ten million a drop in the bucket.
- oooyay 3y ago> According to SBF, the utility we gained by moving fast outweighed the occasional costs we paid due to poor risk checks, hacks, and the like. This was SBF's work philosophy, and it drove the culture he created at Alameda and FTX. Moving fast is great, but you need structure to support it. If you front load your development with guard rails that ensure you're always on track, then you "aim small, miss small" so to speak. If you do none of that and just hope that you can respond to problems as they arise then you've really not aimed at all and can potentially hit a target, but likely also hit a house.
- cameldrv 3y agoAnd this is why this sort of thing will always keep happening, especially in new fields. It's like an anthropic principle of competition. If you're willing to cut corners in a way that could blow up the whole company, you can get ahead, therefore the leaders in a new field most likely cut a lot of corners and may be at risk of blowing up.
- latchkey 3y agoCelsius lost millions in dumb mistakes like that too. Someone at Celsius manually approved a bogus transaction on a hacked website... https://www.coindesk.com/markets/2021/12/03/crypto-lender-celsius-admits-losses-in-120m-badgerdao-hack/ https://www.coindesk.com/markets/2021/12/03/crypto-lender-ce... Badger later reimbursed Celsius with inflation on their token that would be released slowly over several years. "restitution". Someone at Celsius thought that they could sell that restitution token and ended up forfeiting it... which was another $22m loss... https://www.thismorningonchain.com/articles/defi/celsius-made-all-the-wrong-moves-and-now-its-dying/ https://www.thismorningonchain.com/articles/defi/celsius-mad... Then they tried to get the DAO to vote to give them back their lost tokens, without even admitting who it was that lost the tokens... and everyone voted against it.
- paulpauper 3y agoyeah a lot of "dumb" mistakes. whoops accidentally sent to wrong address, which i happen to own the private key too...
- catchnear4321 3y agorookie mistake. have to send it to a family member’s address so you can claim it isn’t yours - truthfully.
- lacrimacida 3y agoYeah, easy steal, no traces, Id never believe it’s just a mistake but the burden of proof is on me.
- latchkey 3y agoIn this case, it wasn't anything like that. Someone exploited a weakness in CloudFlare and was able to replace the Badger website. When someone clicked on the site to execute an approval transaction, it went to the attacker first, which gave the attacker full control over their wallet. It did take a manual step on the part of Celsius though... which should have been checked more closely. The UX around that checking is really terrible though and when someone is trying to do something quickly, they aren't always going to check. This is a big failure of wallets these days. Balancer.fi just had a similar attack happen to them where the .fi registry allowed a nameserver change. https://twitter.com/Balancer/status/1704552288201883809 https://twitter.com/Balancer/status/1704552288201883809 It is also clear that the frontends really need to be hosted in a way that they can't be modified.
- andirk 3y agoA lot of the "genius" credit given to FTX, Alameda, Sam are all during a time when crypto was soaring, probably partially because people were home more and got free checks in the mail. The fact that a single person could absolutely tank the entire company with a single trade shows how reckless they all were. But it worked because the market was going up. That's it. The market slumped for like a week and their behavior dropped them to 0.
- finite_depth 3y ago[dead]
- johndhi 3y agoThis is my sense too. What's kind of funny is we weirdly get caught up and excited by the hype... every single time
- constantly 3y ago“We” is doing a lot of heavy lifting. I am proud that I have never gotten burned by this sort of thing because I understand that if there’s a lot of hype it’s probably too good to be true.
- johndhi 3y agoFair enough. I guess media cycles get caught up. And for me personally, I will say every time the economy booms for a few years I start to ask: gosh is this ever going to stop? For example with real estate, I haven't bought any because the prices always seem insane to me. But I'd be far wealthier (on paper at least) if I'd bitten the bullet and bought into the real estate market a few years ago.
- dotnet00 3y agoI always felt like SBF was a massive fraud. When someone is inflated as a genius and "the kindest billionaire" for superficial things like not driving an expensive car and not dressing appropriately, it sends alarm bells ringing to me. Acting like the exaggerated TV stereotype of a tech nerd screams "I'm covering something up", just like what happened with Theranos.
- josu 3y agoHere is a good analysis of the flash crash from that day: https://twitter.com/austerity_sucks/status/1452332472725819401 https://twitter.com/austerity_sucks/status/14523324727258194... - It lasted 12 seconds. - Total BTC traded was 290.54 - Only 2% of that was at price of 10K or below. - Most volume happened in $20-30K range, over 50% discount to mkt. Here is a tweet from someone (rightly) speculating that it had been Alameda: https://twitter.com/TheoryBitcoin/status/1452345411759398923 https://twitter.com/TheoryBitcoin/status/1452345411759398923
- NelsonMinar 3y agoThe good news is that the world's Bitcoin network can only process ~100 transactions in 12 seconds.
- Vecr 3y ago(Most) trading's not done on chain, it's settled later. Alameda could absolutely trade faster than that.
- 0cf8612b2e1e 3y agoDoesn’t that just mean the trading is done by a trusted third party? Fine until the third party goes belly up.
- vgatherps 3y agoThis is how most asset trading works, but outside of crypto these third parties are actually trustworthy
- JumpCrisscross 3y ago> outside of crypto these third parties are actually trustworthy With multiple layers of mutualised failsafes.
- abxytg 3y ago[flagged]
- tibbar 3y agoSam Bankman-Fried was associated with effective altruism, not e/acc (although there's a link between the two).
- abxytg 3y ago[flagged]
- HDThoreaun 3y agoThe OP of this post has e/acc on his twitter bio
- noduerme 3y agoHonest question: How do you determine that something like this was an honest mistake? Once, when I was much younger, I had a side gig dealing poker at an underground club in NYC. One morning at the end of 10 hours dealing, I accidentally exposed a burn card which turned out to be something one of the players was representing (bluffing), with about $10k on the table. It was an honest mistake - literally a fat finger mistake. I was tired and my dexterity slipped. The player jumped up, upset the chips and started shouting that I was cheating and in league with his opponent. The manager came over to calm things down. He was the young son of a mob boss, whose portfolio encompassed a variety of entertainment franchises. In order to calm the player, he pulled me off the table, told me he believed it was an honest mistake, and then threatened my life if it ever happen again. I quit then and there. Only after I had quit and refused to come back do I think he actually knew for certain that I wasn't pulling a scam. [edit] I'm really enjoying that people are enjoying this post, but I want to clarify what I was getting at: The original article assumes that SBF et. al. concluded by some internal mechanism that this was an "honest mistake", and that this is sufficient. That assumption belies the way criminal organizations actually work, which is to say, they wouldn't assume it was an honest mistake, but they'd happily say it was one and let off a patsy if they had orchestrated it. I kind of hope I can tilt the conversation toward this aspect of the OP's revelation?
- keyle 3y agoGreat story! Thanks for sharing.
- noduerme 3y agoHeh. I love being able to pseudonymously tell HN my crazy stories from time to time. It's the only online chatter I indulge in. Sadly I have no colleagues who I can both talk shop with and also talk about my previous lives. "Build what makes you happy..." is a darn good philosophy, btw.
- johndhi 3y agoI've played one of those spots by union square. Was fun to feel like a gangster for an evening but ultimately not a crowd I want to spend a lot of time around.
- nightpool 3y agoThis happens all the time at traditional finance firms also. For example, last May an investor in the Nordic region sold off hundreds of thousands of shares of an index fund leading to an 8% drop in dozens of stocks https://www.bloomberg.com/opinion/articles/2022-05-03/citi-did-a-flash-crash https://www.bloomberg.com/opinion/articles/2022-05-03/citi-d...
- lokar 3y agoAnd if you are waaaay off the exchange just shuts down your port for the day.
- patio11 3y agoInterestingly the Tokyo Stock Exchange did not countermand an order from Mizuho in 2005 which flipped order quantity and price in yen, a rather catastrophic thing to do when the intended trade was one share for about 600,000 yen. Regulators were… not pleased. (Quick English summary of incident: https://www.foxnews.com/story/typing-error-causes-225m-loss-at-tokyo-stock-exchange https://www.foxnews.com/story/typing-error-causes-225m-loss-... )
- xoranth 3y agoUS exchanges tend to either bust or price-adjust clearly erroneous trades retroactively. That is good for unsophisticated customers, but it creates a disincentive for market makers to provide liquidity when there is a fat finger event. That in turn leads to larger price dislocations on such events. (The reason is that the market maker faces adverse selection. Providing liquidity is a bet from the market maker that the price dislocation will revert. Price adjustment reduces the profit the market maker makes from mean reversion. But if the price dislocation was caused by a real news, the market maker will eat the full losses.)
- keyle 3y agoReading stories like this makes me wonder why I don't have a long term trade in place for a low amount and hope to fish a good one during a flash crash. Seen them on the stock market too. Back in my day I wrote a few trading bots, they never went anywhere positive after about a week of profit. Seems to me now that the smartest robo-trader algorithm is... 10 wait for human mistake... 20 goto 10
- thatguysaguy 3y ago[dead]
- mertd 3y agoI believe, the traditional exchanges either won't fill outrageous trades or would undo them if the counterparty can prove it was an honest mistake.
- artursapek 3y agoYes we call this a stink bid, and it’s a valid strategy. Keep one open just in case. Most exchanges have slippage protection in place but some are better at it than others.
- deleted 3y ago[deleted]
- gabcoh 3y agoIf you aren’t paying attention you just might end up trading on a real crash too.
- Tao3300 3y agoWhich I guess is why it has to be so low you don't care.
- paxys 3y agoHFT algos are going to make quick work of the opportunity well before your broker can even think about executing your limit order.
- abxytg 3y ago[flagged]
- az226 3y agoInteresting to see Binance lie and say it was an algo mistake as opposed to a human one.
- hnfong 3y agoCould be Alameda misrepresenting the situation. Doubtful that Binance would be able to tell whether Alameda's accounts were running an algorithm or manually trading
- paulusthe 3y agoRunning a system that allows a fat fingering mistake like that to go through is yet another sign of their gross incompetence.
- 1970-01-01 3y ago>The tricky thing about risk is that it's usually invisible, right up until it comes around and bites you in the ass. Saved!
- vgatherps 3y agoInvisible can equally mean “actually invisible” but often means “we told the people shouting and screaming about this risk to shut up”
- cwalv 3y ago> we had sanity checks in place to make sure that the orders being sent were reasonable relative to current market prices. Not so for manual trades, which were by nature discretionary. I guess we have different definitions of "invisible"
- artursapek 3y ago[flagged]
- mvdtnz 3y ago> The story of how a misplaced decimal point at Alameda Research caused a market crash that echoed around the world Hang on a second. It caused a dip in the price of Bitcoin, that is not a "market crash". Bitcoin is a toy and not a real market, its price is set almost entirely by bullshit wash trades and other forms of fraud.
- paxys 3y agoBitcoin has a market cap of $530B and at its peak was over $1.3T. That is not a "toy" but a very real financial market.
- Tao3300 3y ago> the total (ostensible) value of all coins that have been mined in a given cryptocurrency. These values should be taken with a hefty grain of salt, as they are considerably larger than the total value that could be realized if holders of a currency decided to try to cash out.
- paxys 3y agoYes, that's how all stocks and every other market in the world works. Still doesn't mean the value isn't valid.
- NSMutableSet 3y agoYou realize that this is how valuations for publicly-traded companies are calculated, right? Also the networths of people whose vast majority of wealth is tied to publicly-traded shares.
- polygamous_bat 3y agoPutting on a white coat doesn't make you a doctor, the years you spend studying and practicing does. Similarly, printing a billion tokens and trading one for a dollar doesn't give you a billion dollar market cap, even though "all the stocks do it!" Those stocks also back productive systems, have financial statements, forecasts, and a million other things that make them a billion dollar (or larger) market capped company.
- sharadov 3y agoIt's not a mistake - it shows the sheer lack of safeguards at a terrible company with a narcissistic child-leader and his cohort of children at the helm! Hope he rots in prison and is made an example of.
- labrador 3y ago> According to SBF, the utility we gained by moving fast outweighed the occasional costs we paid due to poor risk checks, hacks, and the like. When I was young I "moved fast and broke things", then had brushes with the law that fortunately didn't result in jail. Silicon Valley prides itself on second and third order thinking, but do people ever consider what might come after "moving fast and breaking things"
- anonymoushn 3y agoContacts who worked at Alameda claim that this guy did not work at Alameda, or if he did, they never met.
- anonymoushn 3y agoTurns out that the author of TFA did work at Alameda as confirmed by court documents.
- Dwedit 3y agoThey managed to successfully get money for cryptocurrency.
- cwalv 3y agoIs that a difficult thing to do?
- case 3y agoAlameda is a city (and an island) in Alameda County, California. Perhaps the title could be updated to reflect that this article is not referring to the city?
- komali2 3y agoI used to do some work for a company called Captech, then called Capital, I think it's called Hum Capital or something now, but I was always waiting to see how they'd handle it in the news if someone did an article on it when it was called Capital.
- nodesocket 3y ago> What they missed was the decimal point was off by a few spaces. Rather than selling BTC at the current market price, they sold it for pennies on the dollar. The result was immediate. The price of BTC shot from a high of $65k to as low as $8k on some venues. Wouldn't this sell just be gobbled up quickly by buyers? Why would it move the BTC price so dramatically to the downside? Surely this couldn't have been that much volume at near all-time highs of BTC in 2021?
- usefulcat 3y agoYes, it would, but at what price(s)? Think about what would happen if the size of the (very low priced) sell order was >= the size of all resting buy orders priced >= the price of the sell order.
- usefulcat 3y ago> So naturally, we had sanity checks in place to make sure that the orders being sent were reasonable relative to current market prices. Not so for manual trades, which were by nature discretionary. This seems to imply that risk checks are somehow either unnecessary or impractical for manual trades, which is completely untrue. This is 100% a case of 'we chose not to implement that'.
- trogdor 3y agoI used to work in operational risk management at a major investment bank. This kind of thing is not at all unusual. Fat finger errors would often result in unintentional gains. We incorporated them into our models as if they were losses of the same magnitude.
- ponorin 3y agoIn personal banking, at least of those I've experienced, the numbers are displayed in both plain language ("forty-two thousand sixty-nine") as well as numerals ("42,069"). Wonder if this would help prevent this kind of mistake as well.
- desaiguddu 3y agoI have a fair share of our story with Fat Fingers: The Athlete's Glitch Our client team had an amazing NFL season. But with the start of a new NFL season, but our NFL team was buzzing for a different reason – the unveiling of our off-season updates to the beloved iPad app used by one of the major NFL teams. Our app, which was primarily used for training and past game analysis, had undergone a major overhaul, and we were proud of the sleek new UI/UX designs. But just as the athletes started to get into the grind of their training, our office started receiving some unexpected feedback. "Hey! The app is taking me to the wrong sections!" one message read. "I keep hitting the wrong button. Something's off," said another. Confusion took over our team. We had spent months meticulously planning, designing, and testing these updates. Automated tests had been executed to perfection, internal testing hadn’t shown a single glitch. What was happening? Then, a video came in. It was from a coach, showing one of the top athletes trying to use our app. We watched intently as his fingers moved over the screen. The problem became immediately apparent: his fingers, sculpted by years of athletic training and naturally larger than the average, were simply too big for our redesigned interface. Every time he tried to tap a specific function, his finger would unintentionally touch the adjacent ones. Turns out, in our endeavor to create a sleeker, modern UI, we had inadvertently shrunk the size of the clickable buttons and packed them tightly in a grid. This might have looked aesthetically pleasing and worked perfectly for our testers, but for the athletes with their robust fingers, it was a recipe for frustration. We convened an emergency meeting. Our lead designer, Marcus, broke the silence, "Our primary users are these athletes. We should've considered their physical attributes in our designs. It's my oversight." Our project manager, Clara, nodded in agreement. "We need to fix this and roll out an update ASAP. We can't have the team struggling with this during their crucial training period." The next few days were a blur of coding, designing, and testing. With feedback from some of the players, we reintroduced larger buttons and ensured enough space between them, all while maintaining the sleek look of our new design. It was a lesson learned the hard way, but it reiterated the importance of understanding our users' needs and physical attributes. The next feedback we got was from the star quarterback, "Perfect! Back in the game with this. Thanks, TechTouch." The NFL season kicked off with roaring crowds, and our app, now more user-friendly than ever, was right there with the athletes, assisting them every step of the way.
- 5350-uiop-1130 3y agoNever attribute to stupidity that which is adequately explained by malice.
- dustypotato 3y agoisn't it fat finger and not fat fingering .... which sounds something nsfw?
- jjslocum3 3y agoThis description of SBF does evoke a Netflix "I was in a cult" documentary. A larger-than-life leader, with huge vision. I wonder how much of his success was due to his charisma and the ability to steamroll (and sidetrack) people with over-the-top world-changing vision.
- bluenomatterwho 3y agoSeems like a bunch of kids who were born into borderline fraudulent culture... and now damage control pieces (ordered from SBF's jail cell maybe, ha) are being tolerated... maybe with the tacit support of a culture which would rather think of these as things that 'everyone does.' Somewhat akin to the Stanford or Harvard data integrity issues, maybe? This is just business as usual and sometimes it spills over. Who cares as long as you are headquartered in the Caymans, right? To be sure, all of these 'oopsies! SBF is such a sweet, gentle soul' articles of late look a little weird. Dude threw his GF under the bus in obvious fashion.