6 ms·

Can someone explain how these guys managed to write their software in such a way that there was zero automatic verification that what their database said matche

by glesica 13y ago

Can someone explain how these guys managed to write their software in such a way that there was zero automatic verification that what their database said matched what was in the wallets, even just a cron job that ran once a week or something? I mean, I'm assuming there's a good reason, because I can't figure out how anyone could overlook something like that.

iSnow 13y ago
If the rumors are right, then Gox not even had unit tests or a version control system go figure. Bitcoin squarely is in the hand of amateurs - good ones like the dev team and bad ones like Karpeles et al.
drewcrawford 13y ago
It's pretty easy to miss "basics of a good design" like a cron job job when you are also missing actual basics like source control or any release process whatsoever [1]. I understand the Wired article to mean that for awhile they were actually editing source code on production servers in some cases. This really is a case of a company managing assets well in excess of the limitations of its experience or ability, literally a trading card company that found themselves operating as a bank. The thing I don't know was whether their customers had adequate notice that mtgox was grossly incompetent. Smart people have told me they did, but at the same time, the list of services I use where I have personally bothered to check if they use source control and have a test environment is not very long. [1] http://www.wired.com/wiredenterprise/2014/03/bitcoin-exchange/ http://www.wired.com/wiredenterprise/2014/03/bitcoin-exchang...
- gwern 13y ago
  > literally a trading card company that found themselves operating as a bank. Karpeles did not buy a trading carding company or even a former one. McCaleb did have a temporary Magic-related site at mtgox.com 4 years before he started the exchange, but literally the only thing the two shared in common was the domain name. I recently finished researching & clarifying the early history, if you are interested in the details: https://en.wikipedia.org/wiki/Mt._Gox#History https://en.wikipedia.org/wiki/Mt._Gox#History
sheetjs 13y ago
To be fair, even major financial firms and hedge funds have these types of problems. Lack of pre-trade risk controls and position checks (not too dissimilar to what happened with MtGox) contributed to the spectacular blow-up of Knight Capital a few years ago
deleted 13y ago
[deleted]
waterlesscloud 13y ago
Forget automatic verification, I'm extremely skeptical you wouldn't notice this just manually.
hibikir 13y ago
You'd be amazed by the large, boneheaded moves that even the smartest people make when they fail to think defensively. Any security holes, even when people see them, can be easily discounted by just 'eh, nobody will try to hack us'. I've seen that done in Fortune 500 companies whose main business is all about IP! If you can't identify how likely you are of being hacked, and how much is there to lose if that happens, you'll make terrible security decisions, regardless of the engineering skills available.
PeterisP 13y ago
If you have bitcoins in 'cold storage' (Gox supposedly had 99+% their BTC that way) - i.e., the keys are physically offline and not accessible remotely in any way - then I don't see any easy ways of checking if all those coins are still valid.
- prodigal_erik 13y ago
  Couldn't they keep the addresses online for private keys which are offline, and look up the balance for each address on the blockchain?
- moe 13y ago
  then I don't see any easy ways of checking if all those coins are still valid. Every client holds a copy of the blockchain, that is the full transaction history of the bitcoin network. Consequently it's not just easy to check the balance of a cold wallet, it is trivial. Here's the balance and history for a SatoshiDice address: https://blockchain.info/de/address/1dice8EMZmqKvrGE4Qc9bUFf9PX3xaYDp https://blockchain.info/de/address/1dice8EMZmqKvrGE4Qc9bUFf9...
- exit 13y ago
  "the keys" are the private component of a private key/public key pair. the public key can be shared with anyone / stored anywhere, and gives you the address which makes checking ones balance trivial.
  - PeterisP 13y ago
    Okay, you can detect open thefts that way and Gox should've implemented that. But you still can't detect if you've lost control of the private keys - if you've split all your cold balance into many addresses of X BTC each, then you can remotely check if the addresses are still filled, however, if an [inside] attacker takes those keys and replaces them with garbage, then no auditor is going to know that the addresses aren't in your control anymore; the solution would probably be periodic (automatic?) proof-of-ownership tests on those balances. In any case, securing BTC is hard and requires stricter controls and more discipline than securing general banking systems, as the nature of BTC makes it easier to get away with large amounts of funds. For example, you need a solid solution for multiple "write-only" offsite backups of 'cold keys' because otherwise you simply risk your assets being permanently destroyed due to a simple hardware failure, but if anyone in your company is able to singlehandedly recover&decrypt a single such backup then he can immediately abscond with all money.

by glesica 13y ago

iSnow 13y ago

If the rumors are right, then Gox not even had unit tests or a version control system go figure. Bitcoin squarely is in the hand of amateurs - good ones like the dev team and bad ones like Karpeles et al.

drewcrawford 13y ago

It's pretty easy to miss "basics of a good design" like a cron job job when you are also missing actual basics like source control or any release process whatsoever [1]. I understand the Wired article to mean that for awhile they were actually editing source code on production servers in some cases. This really is a case of a company managing assets well in excess of the limitations of its experience or ability, literally a trading card company that found themselves operating as a bank. The thing I don't know was whether their customers had adequate notice that mtgox was grossly incompetent. Smart people have told me they did, but at the same time, the list of services I use where I have personally bothered to check if they use source control and have a test environment is not very long. [1] http://www.wired.com/wiredenterprise/2014/03/bitcoin-exchange/ http://www.wired.com/wiredenterprise/2014/03/bitcoin-exchang...

gwern 13y ago
> literally a trading card company that found themselves operating as a bank. Karpeles did not buy a trading carding company or even a former one. McCaleb did have a temporary Magic-related site at mtgox.com 4 years before he started the exchange, but literally the only thing the two shared in common was the domain name. I recently finished researching & clarifying the early history, if you are interested in the details: https://en.wikipedia.org/wiki/Mt._Gox#History https://en.wikipedia.org/wiki/Mt._Gox#History

sheetjs 13y ago

To be fair, even major financial firms and hedge funds have these types of problems. Lack of pre-trade risk controls and position checks (not too dissimilar to what happened with MtGox) contributed to the spectacular blow-up of Knight Capital a few years ago

deleted 13y ago

[deleted]

waterlesscloud 13y ago

Forget automatic verification, I'm extremely skeptical you wouldn't notice this just manually.

hibikir 13y ago

You'd be amazed by the large, boneheaded moves that even the smartest people make when they fail to think defensively. Any security holes, even when people see them, can be easily discounted by just 'eh, nobody will try to hack us'. I've seen that done in Fortune 500 companies whose main business is all about IP! If you can't identify how likely you are of being hacked, and how much is there to lose if that happens, you'll make terrible security decisions, regardless of the engineering skills available.

PeterisP 13y ago

If you have bitcoins in 'cold storage' (Gox supposedly had 99+% their BTC that way) - i.e., the keys are physically offline and not accessible remotely in any way - then I don't see any easy ways of checking if all those coins are still valid.

prodigal_erik 13y ago
Couldn't they keep the addresses online for private keys which are offline, and look up the balance for each address on the blockchain?
moe 13y ago
then I don't see any easy ways of checking if all those coins are still valid. Every client holds a copy of the blockchain, that is the full transaction history of the bitcoin network. Consequently it's not just easy to check the balance of a cold wallet, it is trivial. Here's the balance and history for a SatoshiDice address: https://blockchain.info/de/address/1dice8EMZmqKvrGE4Qc9bUFf9PX3xaYDp https://blockchain.info/de/address/1dice8EMZmqKvrGE4Qc9bUFf9...
exit 13y ago
"the keys" are the private component of a private key/public key pair. the public key can be shared with anyone / stored anywhere, and gives you the address which makes checking ones balance trivial.
- PeterisP 13y ago
  Okay, you can detect open thefts that way and Gox should've implemented that. But you still can't detect if you've lost control of the private keys - if you've split all your cold balance into many addresses of X BTC each, then you can remotely check if the addresses are still filled, however, if an [inside] attacker takes those keys and replaces them with garbage, then no auditor is going to know that the addresses aren't in your control anymore; the solution would probably be periodic (automatic?) proof-of-ownership tests on those balances. In any case, securing BTC is hard and requires stricter controls and more discipline than securing general banking systems, as the nature of BTC makes it easier to get away with large amounts of funds. For example, you need a solid solution for multiple "write-only" offsite backups of 'cold keys' because otherwise you simply risk your assets being permanently destroyed due to a simple hardware failure, but if anyone in your company is able to singlehandedly recover&decrypt a single such backup then he can immediately abscond with all money.