11 ms·

OptiFi Program Incident Report

229 points by gscott 4y ago

krageon 4y ago
Dev edits on live, makes mistake. Nothing to really see here.
- sph 4y ago
  But he's a crypto dev, so let's all point and laugh. Otherwise why the edited title that explicitly points this out?
- nishs 4y ago
  I think there are some lessons that can learned from this. For example, the command line: solana program close didn't seem to sufficiently represent the destructive nature of the action. The program's designers should likely have chosen the verb "shutdown", instead of "close".
  - bdcravens 4y ago
    While it may not accurately represent what it does, "destroy" may be a better fit.
    - returnInfinity 4y ago
      this is better solana program lose-all-money
    - hbbio 4y ago
      Even "irreversible_destroy"
  - vicnicius 4y ago
    A simple confirmation step with a relevant warning could possibly make developers think twice before proceeding.
    - ReptileMan 4y ago
      From personal experience - no it won't. I have made mistakes with overriding warnings - when under pressure, hyper focused and already in the mental mode of - of course I am sure I want to do it - your slow brain is just shut down so you are moving a bit like on muscle memory.
      - hypertele-Xii 4y ago
        Simply add a 1 hour unskippable countdown timer to the warning. Now you have to think about it.
        ReptileMan 4y ago
        If you have one hour you shouldn't be editing live anyway.
    - krageon 4y ago
      From the many depressing years I've done UX work I can promise you no amount of changes they make will protect users from their own mistakes. It is far better for your own sanity to accept people make mistakes. This is what development and staging environments are for.
      - wzwy 4y ago
        Accepting that people make mistakes is one thing, but reducing those mistakes is another. If it’s a simple change, why not do it? Though, I can see your point if the change is monumental but barely reduces rate of mistakes.
dan-robertson 4y ago
Can you change the title of this submission to match the title of the post please? Normally people change titles when the original is sensationalised however the opposite is true in this case.
- SkyMarshal 4y ago
  Sounds like they only lost $661k worth, which is peanuts by comparison to most other crypto hacks/losses, probably didn't destroy the entire company since they're compensating their users, and not really worthy of the sensational title.
  - bhaak 4y ago
    Apparently 95% of that money was by the company itself or employees of the company. So the loss for normal users is very limited.
    - nishs 4y ago
      source: https://twitter.com/OptifiLabs/status/1564367455220219904 https://twitter.com/OptifiLabs/status/1564367455220219904
    - radicalbyte 4y ago
      I can take a box containing 1001 conkers, sell one to you for $1000, then claim that I have $1m in my bucket. What was lost was not money, but electronic conkers of which a small number people paid real money for.
      - avvt4avaw 4y ago
        The currency was USDC, which is a stablecoin pegged at $1 by Circle (www.circle.com) who are generally held to be reputable, so it very much was real money.
- benreesman 4y ago
  It sounds like a bunch of people are going to pay for the cost to users of their own mistake out of their own pocket, so if they follow through with that we should probably applaud the move? If they welch on it that’s another matter, but the stated time is tomorrow so…so far so good?
bhaak 4y ago
For all its flaws (and there are countless) the one thing about the show cryptocurrency space that stands out to me as a programmer is that programming errors can be suddenly very costly (granted, in this case it was more of a DevOps blunder). Being able to very easily put a price tag on sloppy programming is intriguing to me.
- pavlov 4y ago
  When the price tag is “everything you own”, how is this an improvement? Who wants to give over their financial life to a computer program? This aspect of crypto is a 1960s paranoid computer fear come to life, and somehow it’s often presented as an improvement over the existing system of human checks and balances.
  - askmike 4y ago
    > Who wants to give over their financial life to a computer program? Where do you think your income and bank balances are tracked and stored? On pieces of paper?
    - solumunus 4y ago
      This is missing the point to a degree that makes me think you're being intentionally obtuse, but maybe you're just ignorant so I'll bite. Banking computer errors can easily be rectified by humans, banks are regulated, your funds are at least partially guaranteed by the government (depending on where you live). The degree to which you're trusting computer programs with your finances is orders of magnitude less than with cryptocurrency where it's possible to lose any amount $ of asset value in an instant with absolutely no means of recourse. Your bank analogy is silly and nowhere near analogous.
      - bhaak 4y ago
        CitiBank can't get the money back they accidentally transferred to another company. https://www.bloomberg.com/news/articles/2022-08-15/citi-sues-revlon-over-lender-status-after-900-million-mistake https://www.bloomberg.com/news/articles/2022-08-15/citi-sues... You can say that's an edge case today but I and OP are saying, the future will look more like crypto looks today. Not a bright future.
Me163GermanSoup 4y ago
Man, if only there was a central authority to appeal those mistakes.
- moss2 4y ago
- whywhywhywhy 4y ago
  It’s Solana, so there kinda is. It’s pretty much centralized.
  - imtringued 4y ago
    I need an explanation for this because the only arguments I have seen are Reddit memes that disregard the centralisation of mining pools.
  - rvz 4y ago
    This sort of thing is not exclusive to Solana. Ethereum will also join Solana and be another highly centralized PoS network, so I expect more of the same centralization points to apply to Ethereum and many other chains. As for this issue, just look no further than the DAO hack and see if that has taught crypto anything about so-called decentralization; clearly it hasn't.
  - klabb3 4y ago
    Why is this called DeFi? If a dev can nuke an entire app (intentionally or accidentally) it's not decentralized. Contrary to the HN zeitgeist, I have a lot of excitment for decentralization medium- to long term. But it seems almost everything that's happening in the DeFi space today is about as decentralized as a traditional client server app with mysql.
    - birracerveza 4y ago
      Multiple DeFi platforms exist. Some are mostly decentralized, others only use DeFi as a marketing buzzword. Some of these are actually centralized, but it doesn't mean all of them are.
mellosouls 4y ago
From the article: Lessoned we learned harshly EVERY DEPLOYMENT NEEDS A RIGOROUS PROCESS AND SINGLE POINT FAILURE CAN BE AVOIDED. PLEASE DON’T RUSH LIKE WHAT WE DID, ESPECIALLY FOR DEFI PROJECTS. Best wishes to the company and individuals involved for recovery of assets and success going forward.
jjwtieke 4y ago
Just another example of how useless crypto is.
- ibz 4y ago
  rm -rf / Whoops, I lost everything! Computers are useless.
  - jiggawatts 4y ago
    You can recover from that by restoring from backups. In the crypto world all changes are permanent and immutable. There is no recourse. That’s the difference.
    - mgraczyk 4y ago
      But there is, even in cases like this. Circle could issue new USDC for example. No technical barrier to a remedy like that.
  - rjh29 4y ago
    I mean that won't work, because --preserve-root is on by default. In the same way, eventually crypto may acquire the same checks and balances that other industries have, but it'll take a lot of costly mistakes to get to that point.
    - selfhoster11 4y ago
      Preserve root exists now, but it didn’t exist in the past.
      - rjh29 4y ago
        That's my point. This is the early days for crypto too. I'm not a fan of crypto, but it's clearly going to learn from these mistakes and develop the same kind of safety guarantees over time. I mean look at online banking - in the UK it's only in the last few _years_ that we've been required to do 2fa when buying things online. The earliest online banks didn't even have 2fa to log in and move money around, and at some point HTTPS didn't even exist so everything was sent in plaintext. SMS is still mostly plaintext. Things evolve.
captn3m0 4y ago
So the keys to manage this specific program were kept on disk? (~/solana-keys as per the logs). Doesn't even look like the Solana cli supports an HSM for this usecase.
- renewiltord 4y ago
  Might just be a demonstration. Can just use process substitution to mimic files if necessary.
RedShift1 4y ago
Commands like that really need a confirmation prompt and a command line switch to override like --do-as-i-say (long form only). Good example of developers being put in an end user's shoes, I hope they learn from that mistake and update their programs.
- soco 4y ago
  I don't know about you folks but I can do stupid stuff also with confirmation. Maybe splitting the damaging actions in some substeps, so instead of a big bang you have a slow death?
  - imtringued 4y ago
    You are right, something like closing a program shouldn't be valid immediately, the program should be frozen for a day with the ability to cancel the closing at any time until the day is over.
  - synu 4y ago
    Also, for me when I make big mistakes like this I think I’m in a different environment or something. It’s not that I don’t realise what command I’m running, it’s that I miss the context. So I would just confirm but not realise what I was confirming.
- pbhjpbhj 4y ago
  An example of such a command came up on Linus Tech Tips (LTT), from apt (IIRC) on Pop_OS!. Basically (paraphrasing) "this will destroy your system and render it useless; type 'I'm an idiot and wish to proceed'". Our eponymous youtuber went right on, typed the phrase and blitzed the system, and then moaned about how the system shouldn't have let him break it ... I suspect such prompts only work for cautious people, who possibly don't need the prompts as they'll actually read the scary warnings.
  - rcxdude 4y ago
    In his defense, it was literally the first package he tried to install, which was in the official repositories but horribly broken, so he had no real way to tell whether this was the normal flow and linux just made it really irritating to install packages.
CGamesPlay 4y ago
Is there a good reason that the CLI is able to close a contract with money in it?
- jpgvm 4y ago
  The money within the contract wasn't the problem, that was transferred out as part of closing the program. However there were other assets on the chain that are locked using the program which are now inaccessible because its inoperable. Consider it like an operation that closes and deletes your user-account but returns you an archive of your home directory when it's complete. However all files that weren't in your home directory that were encrypted with your local user key now can't be read. It's likely infeasible to predicate program deletion on the program no longer being referenced in other programs/contracts. Disclaimer: I'm not a Solana expert (or a crypto advocate for that matter)
londons_explore 4y ago
This is why coins/tokens/contracts shouldn't have the code be updatable when they contain money... If you want something where some developer can change the rules anytime, then just pay the money direct to the developer, and hope he pays you back one day.
gjvc 4y ago
good news. we need more of this.
deleted 4y ago
[deleted]
benreesman 4y ago
I once deleted like, maybe a few petabytes of data valuable enough to store on FusionIO cards with a bad config change. I think it destroyed $50-100MM once the dust settled in all the various ways that it cost money were added up. I’m quite sure the hacker who pressed the button feels plenty bad enough already, and I hope the people around them were as kind as the people around me were.
- tinco 4y ago
  Was it Cambridge Analytica that was going to pay 50MM for that data? ;)
  - benreesman 4y ago
    No, it was features for a P(click|impression) model, which are surprisingly mundane: it is basically impossible to care much either way, let alone be outraged, if you look at the actual features used in these kinds of things and it’s not explicitly your field. It’s not a microphone listening on your smartphone! You can even read about it in this paper which publicly described the system contemporary to that event: http://www.quinonero.net/Publications/predicting-clicks-facebook.pdf http://www.quinonero.net/Publications/predicting-clicks-face... If you read it and are still steamed up, well, then you hold that opinion honestly, but I suspect most would be like, “cool story bro”.
    - tinco 4y ago
      Thanks that's interesting! Definitely not outraged, was just poking a bit for fun. 50MM is an insane amount of money/value that would bankrupt most companies and have a noticable impact on many industries. Funny that for Facebook it was probably just a small hiccup on the ad revenue gravy train.
      - eru 4y ago
        > 50MM is an insane amount of money/value that would bankrupt most companies [...] Well, it would probably bankrupt most companies, but it wouldn't bankrupt most people's employer. The average employee works at a larger than average company. Similar for the average customer or average investor. That's just because big companies have more employees / customers / investors to contribute to that kind of average.
spaceman_2020 4y ago
This is one of the biggest flaws in crypto. Small errors can erode hundreds of millions of value. That's when most crypto companies right now are very small. Can you imagine the chaos if crypto were to actually become big and dev count were to grow to 1k+ people spread across multiple offices? There's definite efficiency gains with crypto (a dex like Uniswap can do massive volume with very few developers for instance), but there needs to be a way to limit the cost of human errors.
- mpeg 4y ago
  It's not exactly unique to crypto... in adtech, a configuration mistake on a web server can cost 6-7 figures as ads are being paid for, but take people nowhere.
  - kelnos 4y ago
    Sure, but the company managing the ad network can choose to be gentle, and refund the money. With these sorts of crypto mistakes, the money is just gone, because humans are -- as a desired feature of the system -- unable to correct these sorts of mistakes.
  - blibble 4y ago
    and in a live trading system a small mistake can cost you 6-7 figures per second see: Knight Capital
    - mpeg 4y ago
      indeed! the fintech world is already full of these footguns, but HN is very anti-crypto, so everything wrong that happens there is proof the whole industry is doomed
  - LetMeInNowMate 4y ago
    - mpeg 4y ago
bschwindHN 4y ago
SOL, this comedy almost writes itself.
Havoc 4y ago
Solana ecosystem seems to have more than it’s fair share of mishaps. Still surprised anyone would prefer it over other options
thro388 4y ago
- rvz 4y ago
  Or what happens when the tech bros get their hands on radiography hardware [0]. They probably would write it in Javascript. [0] https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25
  - ClassyJacket 4y ago
    This is kind of a nitpick but it actually wasn't even radiography, it wasn't for imaging, it was for radiation therapy - blasting cancer with radiation to kill it. So it was even more dangerous than it might sound :( definitely something that should not have safety controlled by software. There was also a horrible disaster with someone taking the radioactive material from a machine like that in an abandoned hospital and messing around with it at home. People died in both incidents.
_el1s7 4y ago
it's not the dev's mistake, it's the whole unnecessary complicated and bullshit of this solana network and most other blockchain networks that are put up by incompetent developers.
netmonk 4y ago
dr_dshiv 4y ago
> Here it turned out that we didn’t really understand the impact and risk of this closing program command line. ‘solana program close’ is actually for closing the program permanently and sending the SOL tokens in the buffer account used by the program back to the recipient wallet.
pluc 4y ago
Crypto techs don't understand crypto tech - story at 11
t_mann 4y ago
Code executions reflect intentions, transactions are final. As promised on a blockchain. Let's see if it stays that way, wouldn't be the first time if not.
imwillofficial 4y ago
Could this be a used to launder money? As in, you have all this crypto with made up value, and we are going to “cash out” due to a mistake, and now you have whatever we stated the value is in cash?
rando14775 4y ago
And nothing of value was lost.
wzwy 4y ago
> Strictly execute *peer-surveillance approach*that requires at least 3 peers to engage in the deployment process > - We will adopt a peer-surveillance approach which requires at least 3 peers to engage in the deployment process. They have the responsibility to remind the main deployer of any potential risk, and make sure each step complies with the deployment guides and norms. > - In case anything abnormal happens during the deployment process, such as bad network status or insufficient deployment fee, we should calm down and have a discussion with peers to make sure each operation is safe. Meanwhile, we should mark down every command line and returned message for further reference. I wonder if that’s enough stack of Swiss Cheese to prevent such an accident from happening again. Hopefully they expand on the “we should calm down” to not be limited to saying “calm down please”. Calming down is quite difficult when you’re in the throes of something, and proper procedures tend to be put aside for the sake of pressure relief. I wish them swift recovery from their predicament.
targ65 4y ago
When are we going to admit that DeFi is a regression, not progress, versus the status quo financial system? Crypto is all about disintermediation. But that doesn't work when people's money / savings are on the line. So we can easily tell exactly where this leads: Once enough pain has been sustained through errors like this, DeFi code bugs, fraudulent transfers, etc... a whole industry of HUMANS will pop up that will (a) become an intermediary in crypto/DeFi transactions to ensure that fraud, errors, etc are prevented; (b) introduce mechanisms to revert transactions (controlled by said HUMANS); (c) perform the massive paperwork on compliance (KYC, register big transaction, check against blacklists, etc) Is there really a different outcome? Crypto fans claim finance is an overbloated industry and they are here to disrupt it. In reality all they are doing is recreating exactly the same system but with one more layer of indirection (instead of transferring dollars you transfer a stablecoin that points to a dollar). Will there be exciting use cases for crypto at the edges? maybe. Will it 'disrupt' the broader financial system, in the sense that it will make it better/faster/cheaper? I don't see it.
- nullc 4y ago
  Inevitably when people discuss Bitcoin someone shows up and calls it a ponzi scheme. Defi is just the market hearing that over and over again and saying, "OK. I heard you like ponzi schemes..." They all already have humans throughout--- if nothing else, in the form of the people walking off with the windfalls. The fact that the tech keeps blowing up is mostly just an artifact that the tech is just there as obfuscation for the fraud that underlies and motivates the enterprise. The technology is not well thought out because it doesn't need to be, it's not managed by people with high technical expertise because people with such expertise see through the schemes and can find better things to do with their time than to help rip people off. Given the economic or centrally trusted points of failure-- a competent and ethical engineer would usually tell you adding smart contract gunk to these schemes is an unacceptable source of risk without meaningful benefit (except perhaps as pretext to hide from law enforcement) and so the systems the world gets are the ones built by people who were less than competent and ethical. Apologies to the rare few things under that banner that aren't fraud -- but they're part of a lemon market where they can't be distinguished from their more fraudulent compatriots, so that kind of guilt by association is inevitable.
prepend 4y ago
I like that errors can be quantified as it removes some doom ambiguity from mistakes. The first company I worked for sold access to data on a per minute basis and it was like $1.50/minute or something (dialup access in the 90s to an interactive ClarionDB session). So any outage would be quantified by some curious programmer. I remember even small outages being quantified in hot washes as 10 minutes x 2000 users x $1.5 = $30k outage. I remember one of my bosses was nicknamed “Commander Crash” as he once crashed the whole system for hours based on a single live config file edit. No one was fired but it was actually liberating knowing how much mistakes cost.
dark-star 4y ago
I have not understood most of that article (optifi, mainnet, solana, PDAs, deployer address, program, tvl, ...what?) but what I get from it is that people who run crypto-anything make costly mistakes. Funny that people still ask me "hey, why do you trust those evil banks? you should do all in crypto instead!" and then laugh at me when I tell them I trust banks more than some random dev on the internet...
korginator 4y ago
The report reads like the OptiFi folks were trying to figure out how the different bits of the puzzle on the fly. Did these folks even know how to use their tools? Towards later in the article they say "Here it turned out that we didn’t really understand the impact and risk of this closing program command line. ‘solana program close’ is actually for closing the program permanently and sending the SOL tokens in the buffer account used by the program back to the recipient wallet." What?!
eloff 4y ago
I've watched crypto for years with interest, I've worked in the field, speculated in it with my savings, used it for international transfers, spent it, etc. That is to say I'm not some armchair critic. I find myself lately viewing crypto as mostly a bad idea, or a bad implementation of an idea with potential, or even outright as a Ponzi scheme. These smart contacts are really a bad idea. We have contracts in life with lawyers, these tend to work out pretty well because we can use our human intelligence to arbitrate them. The idea of code as law is really dumb. I say that as a programmer who works on code all day. Computers are dumb like your calculator. They do exactly what you tell them to. Which sounds great until you realize how hard it is to describe what you want in a way the computer can understand. That's why I make a good salary and people think my job is difficult. The reality is computers often do what you tell them, but it's not what you wanted. Rather you have the instructions wrong or you made a mistake. That's why software is riddled with bugs. Avoiding that is very, very, very difficult. Nobody gets it right. These crypto finance bros will definately not get it right. And when they get it wrong, there's no human, no court to appeal to. How's that an improvement?

229 points by gscott 4y ago

krageon 4y ago

Dev edits on live, makes mistake. Nothing to really see here.

sph 4y ago
But he's a crypto dev, so let's all point and laugh. Otherwise why the edited title that explicitly points this out?
nishs 4y ago
I think there are some lessons that can learned from this. For example, the command line: solana program close didn't seem to sufficiently represent the destructive nature of the action. The program's designers should likely have chosen the verb "shutdown", instead of "close".
- bdcravens 4y ago
  While it may not accurately represent what it does, "destroy" may be a better fit.
  - returnInfinity 4y ago
    this is better solana program lose-all-money
  - hbbio 4y ago
    Even "irreversible_destroy"
- vicnicius 4y ago
  A simple confirmation step with a relevant warning could possibly make developers think twice before proceeding.
  - ReptileMan 4y ago
    From personal experience - no it won't. I have made mistakes with overriding warnings - when under pressure, hyper focused and already in the mental mode of - of course I am sure I want to do it - your slow brain is just shut down so you are moving a bit like on muscle memory.
    - hypertele-Xii 4y ago
      Simply add a 1 hour unskippable countdown timer to the warning. Now you have to think about it.
      - ReptileMan 4y ago
        If you have one hour you shouldn't be editing live anyway.
  - krageon 4y ago
    From the many depressing years I've done UX work I can promise you no amount of changes they make will protect users from their own mistakes. It is far better for your own sanity to accept people make mistakes. This is what development and staging environments are for.
    - wzwy 4y ago
      Accepting that people make mistakes is one thing, but reducing those mistakes is another. If it’s a simple change, why not do it? Though, I can see your point if the change is monumental but barely reduces rate of mistakes.

dan-robertson 4y ago

Can you change the title of this submission to match the title of the post please? Normally people change titles when the original is sensationalised however the opposite is true in this case.

SkyMarshal 4y ago
Sounds like they only lost $661k worth, which is peanuts by comparison to most other crypto hacks/losses, probably didn't destroy the entire company since they're compensating their users, and not really worthy of the sensational title.
- bhaak 4y ago
  Apparently 95% of that money was by the company itself or employees of the company. So the loss for normal users is very limited.
  - nishs 4y ago
    source: https://twitter.com/OptifiLabs/status/1564367455220219904 https://twitter.com/OptifiLabs/status/1564367455220219904
  - radicalbyte 4y ago
    I can take a box containing 1001 conkers, sell one to you for $1000, then claim that I have $1m in my bucket. What was lost was not money, but electronic conkers of which a small number people paid real money for.
    - avvt4avaw 4y ago
      The currency was USDC, which is a stablecoin pegged at $1 by Circle (www.circle.com) who are generally held to be reputable, so it very much was real money.
benreesman 4y ago
It sounds like a bunch of people are going to pay for the cost to users of their own mistake out of their own pocket, so if they follow through with that we should probably applaud the move? If they welch on it that’s another matter, but the stated time is tomorrow so…so far so good?

bhaak 4y ago

For all its flaws (and there are countless) the one thing about the show cryptocurrency space that stands out to me as a programmer is that programming errors can be suddenly very costly (granted, in this case it was more of a DevOps blunder). Being able to very easily put a price tag on sloppy programming is intriguing to me.

pavlov 4y ago
When the price tag is “everything you own”, how is this an improvement? Who wants to give over their financial life to a computer program? This aspect of crypto is a 1960s paranoid computer fear come to life, and somehow it’s often presented as an improvement over the existing system of human checks and balances.
- askmike 4y ago
  > Who wants to give over their financial life to a computer program? Where do you think your income and bank balances are tracked and stored? On pieces of paper?
  - solumunus 4y ago
    This is missing the point to a degree that makes me think you're being intentionally obtuse, but maybe you're just ignorant so I'll bite. Banking computer errors can easily be rectified by humans, banks are regulated, your funds are at least partially guaranteed by the government (depending on where you live). The degree to which you're trusting computer programs with your finances is orders of magnitude less than with cryptocurrency where it's possible to lose any amount $ of asset value in an instant with absolutely no means of recourse. Your bank analogy is silly and nowhere near analogous.
    - bhaak 4y ago
      CitiBank can't get the money back they accidentally transferred to another company. https://www.bloomberg.com/news/articles/2022-08-15/citi-sues-revlon-over-lender-status-after-900-million-mistake https://www.bloomberg.com/news/articles/2022-08-15/citi-sues... You can say that's an edge case today but I and OP are saying, the future will look more like crypto looks today. Not a bright future.

Me163GermanSoup 4y ago

Man, if only there was a central authority to appeal those mistakes.

moss2 4y ago
whywhywhywhy 4y ago
It’s Solana, so there kinda is. It’s pretty much centralized.
- imtringued 4y ago
  I need an explanation for this because the only arguments I have seen are Reddit memes that disregard the centralisation of mining pools.
- rvz 4y ago
  This sort of thing is not exclusive to Solana. Ethereum will also join Solana and be another highly centralized PoS network, so I expect more of the same centralization points to apply to Ethereum and many other chains. As for this issue, just look no further than the DAO hack and see if that has taught crypto anything about so-called decentralization; clearly it hasn't.
- klabb3 4y ago
  Why is this called DeFi? If a dev can nuke an entire app (intentionally or accidentally) it's not decentralized. Contrary to the HN zeitgeist, I have a lot of excitment for decentralization medium- to long term. But it seems almost everything that's happening in the DeFi space today is about as decentralized as a traditional client server app with mysql.
  - birracerveza 4y ago
    Multiple DeFi platforms exist. Some are mostly decentralized, others only use DeFi as a marketing buzzword. Some of these are actually centralized, but it doesn't mean all of them are.

mellosouls 4y ago

From the article: Lessoned we learned harshly EVERY DEPLOYMENT NEEDS A RIGOROUS PROCESS AND SINGLE POINT FAILURE CAN BE AVOIDED. PLEASE DON’T RUSH LIKE WHAT WE DID, ESPECIALLY FOR DEFI PROJECTS. Best wishes to the company and individuals involved for recovery of assets and success going forward.

jjwtieke 4y ago

Just another example of how useless crypto is.

ibz 4y ago
rm -rf / Whoops, I lost everything! Computers are useless.
- jiggawatts 4y ago
  You can recover from that by restoring from backups. In the crypto world all changes are permanent and immutable. There is no recourse. That’s the difference.
  - mgraczyk 4y ago
    But there is, even in cases like this. Circle could issue new USDC for example. No technical barrier to a remedy like that.
- rjh29 4y ago
  I mean that won't work, because --preserve-root is on by default. In the same way, eventually crypto may acquire the same checks and balances that other industries have, but it'll take a lot of costly mistakes to get to that point.
  - selfhoster11 4y ago
    Preserve root exists now, but it didn’t exist in the past.
    - rjh29 4y ago
      That's my point. This is the early days for crypto too. I'm not a fan of crypto, but it's clearly going to learn from these mistakes and develop the same kind of safety guarantees over time. I mean look at online banking - in the UK it's only in the last few _years_ that we've been required to do 2fa when buying things online. The earliest online banks didn't even have 2fa to log in and move money around, and at some point HTTPS didn't even exist so everything was sent in plaintext. SMS is still mostly plaintext. Things evolve.

captn3m0 4y ago

So the keys to manage this specific program were kept on disk? (~/solana-keys as per the logs). Doesn't even look like the Solana cli supports an HSM for this usecase.

renewiltord 4y ago
Might just be a demonstration. Can just use process substitution to mimic files if necessary.

RedShift1 4y ago

Commands like that really need a confirmation prompt and a command line switch to override like --do-as-i-say (long form only). Good example of developers being put in an end user's shoes, I hope they learn from that mistake and update their programs.

soco 4y ago
I don't know about you folks but I can do stupid stuff also with confirmation. Maybe splitting the damaging actions in some substeps, so instead of a big bang you have a slow death?
- imtringued 4y ago
  You are right, something like closing a program shouldn't be valid immediately, the program should be frozen for a day with the ability to cancel the closing at any time until the day is over.
- synu 4y ago
  Also, for me when I make big mistakes like this I think I’m in a different environment or something. It’s not that I don’t realise what command I’m running, it’s that I miss the context. So I would just confirm but not realise what I was confirming.
pbhjpbhj 4y ago
An example of such a command came up on Linus Tech Tips (LTT), from apt (IIRC) on Pop_OS!. Basically (paraphrasing) "this will destroy your system and render it useless; type 'I'm an idiot and wish to proceed'". Our eponymous youtuber went right on, typed the phrase and blitzed the system, and then moaned about how the system shouldn't have let him break it ... I suspect such prompts only work for cautious people, who possibly don't need the prompts as they'll actually read the scary warnings.
- rcxdude 4y ago
  In his defense, it was literally the first package he tried to install, which was in the official repositories but horribly broken, so he had no real way to tell whether this was the normal flow and linux just made it really irritating to install packages.

CGamesPlay 4y ago

Is there a good reason that the CLI is able to close a contract with money in it?

jpgvm 4y ago
The money within the contract wasn't the problem, that was transferred out as part of closing the program. However there were other assets on the chain that are locked using the program which are now inaccessible because its inoperable. Consider it like an operation that closes and deletes your user-account but returns you an archive of your home directory when it's complete. However all files that weren't in your home directory that were encrypted with your local user key now can't be read. It's likely infeasible to predicate program deletion on the program no longer being referenced in other programs/contracts. Disclaimer: I'm not a Solana expert (or a crypto advocate for that matter)

londons_explore 4y ago

This is why coins/tokens/contracts shouldn't have the code be updatable when they contain money... If you want something where some developer can change the rules anytime, then just pay the money direct to the developer, and hope he pays you back one day.

gjvc 4y ago

good news. we need more of this.

deleted 4y ago

[deleted]

benreesman 4y ago

I once deleted like, maybe a few petabytes of data valuable enough to store on FusionIO cards with a bad config change. I think it destroyed $50-100MM once the dust settled in all the various ways that it cost money were added up. I’m quite sure the hacker who pressed the button feels plenty bad enough already, and I hope the people around them were as kind as the people around me were.

tinco 4y ago
Was it Cambridge Analytica that was going to pay 50MM for that data? ;)
- benreesman 4y ago
  No, it was features for a P(click|impression) model, which are surprisingly mundane: it is basically impossible to care much either way, let alone be outraged, if you look at the actual features used in these kinds of things and it’s not explicitly your field. It’s not a microphone listening on your smartphone! You can even read about it in this paper which publicly described the system contemporary to that event: http://www.quinonero.net/Publications/predicting-clicks-facebook.pdf http://www.quinonero.net/Publications/predicting-clicks-face... If you read it and are still steamed up, well, then you hold that opinion honestly, but I suspect most would be like, “cool story bro”.
  - tinco 4y ago
    Thanks that's interesting! Definitely not outraged, was just poking a bit for fun. 50MM is an insane amount of money/value that would bankrupt most companies and have a noticable impact on many industries. Funny that for Facebook it was probably just a small hiccup on the ad revenue gravy train.
    - eru 4y ago
      > 50MM is an insane amount of money/value that would bankrupt most companies [...] Well, it would probably bankrupt most companies, but it wouldn't bankrupt most people's employer. The average employee works at a larger than average company. Similar for the average customer or average investor. That's just because big companies have more employees / customers / investors to contribute to that kind of average.

spaceman_2020 4y ago

This is one of the biggest flaws in crypto. Small errors can erode hundreds of millions of value. That's when most crypto companies right now are very small. Can you imagine the chaos if crypto were to actually become big and dev count were to grow to 1k+ people spread across multiple offices? There's definite efficiency gains with crypto (a dex like Uniswap can do massive volume with very few developers for instance), but there needs to be a way to limit the cost of human errors.

mpeg 4y ago
It's not exactly unique to crypto... in adtech, a configuration mistake on a web server can cost 6-7 figures as ads are being paid for, but take people nowhere.
- kelnos 4y ago
  Sure, but the company managing the ad network can choose to be gentle, and refund the money. With these sorts of crypto mistakes, the money is just gone, because humans are -- as a desired feature of the system -- unable to correct these sorts of mistakes.
- blibble 4y ago
  and in a live trading system a small mistake can cost you 6-7 figures per second see: Knight Capital
  - mpeg 4y ago
    indeed! the fintech world is already full of these footguns, but HN is very anti-crypto, so everything wrong that happens there is proof the whole industry is doomed
- LetMeInNowMate 4y ago
  - mpeg 4y ago

bschwindHN 4y ago

SOL, this comedy almost writes itself.

Havoc 4y ago

Solana ecosystem seems to have more than it’s fair share of mishaps. Still surprised anyone would prefer it over other options

thro388 4y ago

rvz 4y ago
Or what happens when the tech bros get their hands on radiography hardware [0]. They probably would write it in Javascript. [0] https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25
- ClassyJacket 4y ago
  This is kind of a nitpick but it actually wasn't even radiography, it wasn't for imaging, it was for radiation therapy - blasting cancer with radiation to kill it. So it was even more dangerous than it might sound :( definitely something that should not have safety controlled by software. There was also a horrible disaster with someone taking the radioactive material from a machine like that in an abandoned hospital and messing around with it at home. People died in both incidents.

_el1s7 4y ago

it's not the dev's mistake, it's the whole unnecessary complicated and bullshit of this solana network and most other blockchain networks that are put up by incompetent developers.

netmonk 4y ago

dr_dshiv 4y ago

> Here it turned out that we didn’t really understand the impact and risk of this closing program command line. ‘solana program close’ is actually for closing the program permanently and sending the SOL tokens in the buffer account used by the program back to the recipient wallet.

pluc 4y ago

Crypto techs don't understand crypto tech - story at 11

t_mann 4y ago

Code executions reflect intentions, transactions are final. As promised on a blockchain. Let's see if it stays that way, wouldn't be the first time if not.

imwillofficial 4y ago

Could this be a used to launder money? As in, you have all this crypto with made up value, and we are going to “cash out” due to a mistake, and now you have whatever we stated the value is in cash?

rando14775 4y ago

And nothing of value was lost.

wzwy 4y ago

> Strictly execute *peer-surveillance approach*that requires at least 3 peers to engage in the deployment process > - We will adopt a peer-surveillance approach which requires at least 3 peers to engage in the deployment process. They have the responsibility to remind the main deployer of any potential risk, and make sure each step complies with the deployment guides and norms. > - In case anything abnormal happens during the deployment process, such as bad network status or insufficient deployment fee, we should calm down and have a discussion with peers to make sure each operation is safe. Meanwhile, we should mark down every command line and returned message for further reference. I wonder if that’s enough stack of Swiss Cheese to prevent such an accident from happening again. Hopefully they expand on the “we should calm down” to not be limited to saying “calm down please”. Calming down is quite difficult when you’re in the throes of something, and proper procedures tend to be put aside for the sake of pressure relief. I wish them swift recovery from their predicament.

targ65 4y ago

When are we going to admit that DeFi is a regression, not progress, versus the status quo financial system? Crypto is all about disintermediation. But that doesn't work when people's money / savings are on the line. So we can easily tell exactly where this leads: Once enough pain has been sustained through errors like this, DeFi code bugs, fraudulent transfers, etc... a whole industry of HUMANS will pop up that will (a) become an intermediary in crypto/DeFi transactions to ensure that fraud, errors, etc are prevented; (b) introduce mechanisms to revert transactions (controlled by said HUMANS); (c) perform the massive paperwork on compliance (KYC, register big transaction, check against blacklists, etc) Is there really a different outcome? Crypto fans claim finance is an overbloated industry and they are here to disrupt it. In reality all they are doing is recreating exactly the same system but with one more layer of indirection (instead of transferring dollars you transfer a stablecoin that points to a dollar). Will there be exciting use cases for crypto at the edges? maybe. Will it 'disrupt' the broader financial system, in the sense that it will make it better/faster/cheaper? I don't see it.

nullc 4y ago
Inevitably when people discuss Bitcoin someone shows up and calls it a ponzi scheme. Defi is just the market hearing that over and over again and saying, "OK. I heard you like ponzi schemes..." They all already have humans throughout--- if nothing else, in the form of the people walking off with the windfalls. The fact that the tech keeps blowing up is mostly just an artifact that the tech is just there as obfuscation for the fraud that underlies and motivates the enterprise. The technology is not well thought out because it doesn't need to be, it's not managed by people with high technical expertise because people with such expertise see through the schemes and can find better things to do with their time than to help rip people off. Given the economic or centrally trusted points of failure-- a competent and ethical engineer would usually tell you adding smart contract gunk to these schemes is an unacceptable source of risk without meaningful benefit (except perhaps as pretext to hide from law enforcement) and so the systems the world gets are the ones built by people who were less than competent and ethical. Apologies to the rare few things under that banner that aren't fraud -- but they're part of a lemon market where they can't be distinguished from their more fraudulent compatriots, so that kind of guilt by association is inevitable.

prepend 4y ago

I like that errors can be quantified as it removes some doom ambiguity from mistakes. The first company I worked for sold access to data on a per minute basis and it was like $1.50/minute or something (dialup access in the 90s to an interactive ClarionDB session). So any outage would be quantified by some curious programmer. I remember even small outages being quantified in hot washes as 10 minutes x 2000 users x $1.5 = $30k outage. I remember one of my bosses was nicknamed “Commander Crash” as he once crashed the whole system for hours based on a single live config file edit. No one was fired but it was actually liberating knowing how much mistakes cost.

dark-star 4y ago

I have not understood most of that article (optifi, mainnet, solana, PDAs, deployer address, program, tvl, ...what?) but what I get from it is that people who run crypto-anything make costly mistakes. Funny that people still ask me "hey, why do you trust those evil banks? you should do all in crypto instead!" and then laugh at me when I tell them I trust banks more than some random dev on the internet...

korginator 4y ago

The report reads like the OptiFi folks were trying to figure out how the different bits of the puzzle on the fly. Did these folks even know how to use their tools? Towards later in the article they say "Here it turned out that we didn’t really understand the impact and risk of this closing program command line. ‘solana program close’ is actually for closing the program permanently and sending the SOL tokens in the buffer account used by the program back to the recipient wallet." What?!

eloff 4y ago

I've watched crypto for years with interest, I've worked in the field, speculated in it with my savings, used it for international transfers, spent it, etc. That is to say I'm not some armchair critic. I find myself lately viewing crypto as mostly a bad idea, or a bad implementation of an idea with potential, or even outright as a Ponzi scheme. These smart contacts are really a bad idea. We have contracts in life with lawyers, these tend to work out pretty well because we can use our human intelligence to arbitrate them. The idea of code as law is really dumb. I say that as a programmer who works on code all day. Computers are dumb like your calculator. They do exactly what you tell them to. Which sounds great until you realize how hard it is to describe what you want in a way the computer can understand. That's why I make a good salary and people think my job is difficult. The reality is computers often do what you tell them, but it's not what you wanted. Rather you have the instructions wrong or you made a mistake. That's why software is riddled with bugs. Avoiding that is very, very, very difficult. Nobody gets it right. These crypto finance bros will definately not get it right. And when they get it wrong, there's no human, no court to appeal to. How's that an improvement?