5 ms·

As much as the joke about bitcoins "take the money and run" exchange business model is bandied about, how do people actually suggest securing exchange wallets?

As much as the joke about bitcoins "take the money and run" exchange business model is bandied about, how do people actually suggest securing exchange wallets? I'm sure it's talked about, but I've never really found a straightforward explanation of how it should be done - is there even an agreed upon set of best practices? If so, why are these supposedly sophisticated exchanges not able to successfully apply them?

ikeboy 10y ago
1. Have a cold wallet on an airgapped computer with 99% of your funds. 2. Have a warm wallet on a computer with a firewall that only connects to 3, with 1% of your funds 3. Have a web-facing server that sends back commands to 2 when it needs to send money. 2 alerts someone when it needs more money, who then goes to the cold wallet and transfers some, creating the transaction offline and using a new USB key to transfer it to a computer with internet. The last step can also require multiple people if desired.
- robryk 10y ago
  When that person goes to the cold storage wallet, the only trustworthy information they have (assuming compomise of the online system) is how much money they are transferring. This means that this arrangement succeeds at making a trustworthy rate limit and nothing more. If I wanted to make a trustworthy rate limit, I'd have the machine from (1) not airgapped, but connected by a serial (unidirectional) line to the machine from (2) along which it would periodically send signed transactions sending some of the money from "lukewarm" storage to hot storage, and sending change back to the very same "lukewarm" storage address. The machine from (1) can then implement a rate limit.
  - ikeboy 10y ago
    But then a compromise of 2 and a zero day remote attack can exploit 1. 1 is offline so no attack not involving insiders can affect it. Edit: actually nvm, didn't get the part about unidirectional. I don't deal with networking but I assume that's possible physically and there's no way to reverse the flow?
    - ch 10y ago
      Thats correct: https://en.wikipedia.org/wiki/Unidirectional_network https://en.wikipedia.org/wiki/Unidirectional_network Sometimes known as a Data Diode.
xorcist 10y ago
To start with, store most funds offline. A bank run (or hack) should be subject to intervention by a human. Contact customer if there is any doubt. Implement checks and limits for the remaining balance. Regularly audit and publish cryptographic proof you are in control of funds.
- Scirra_Tom 10y ago
  > Regularly audit and publish cryptographic proof you are in control of funds. What good is that proof of control? Literally the next second it could be drained and no one would know until the next scheduled proof of control. I've never been sure what publishing proof of control is meant to protect against exactly.
  - elevensies 10y ago
    It would protect against what happened with MtGox -- if, after losing some but not all customer funds, they tried to keep running the site to make back the money. It would protect you from putting money into a site that is already bankrupt.
    - Scirra_Tom 10y ago
      Ah good point. But then how do you verify their figure on total customer deposits?
      - elevensies 10y ago
        If privacy is not a concern, you can just publish a list of your liabilities at a given time. Each account holder can then verify the their own listing, calculate the total, and compare to the proven reserves. There are more sophisticated schemes that allow you to check the amount in your account, and verify that your account was included in the total, without seeing the size of other accounts: https://iwilcox.me.uk/2014/proving-bitcoin-reserves https://iwilcox.me.uk/2014/proving-bitcoin-reserves .
berberous 10y ago
"We don't use cold storage for bitcoin, since our implementation with bitgo we've used segregated customer wallets so that each user has their own bitcoin wallet." [0] This was hubris / stupidity. [0] https://www.reddit.com/r/Bitcoin/comments/4vtuxo/bitfinex_security_breach_trading_will_be_halted/d61pqrm https://www.reddit.com/r/Bitcoin/comments/4vtuxo/bitfinex_se...
- makomk 10y ago
  No, it was US futures trading regulations (though arguably that's the same thing). For regulatory reasons they had to provide "actual delivery" of the bitcoins in question, and the CTFC held that cold storage didn't count as "actual delivery". See http://www.cftc.gov/idc/groups/public/@lrenforcementactions/documents/legalpleading/enfbfxnaorder060216.pdf http://www.cftc.gov/idc/groups/public/@lrenforcementactions/...
- vamur 10y ago
  In that case they should have required users to also login to bitgo and approve any withdrawals. Otherwise, what's the point of Bitgo?
STRML 10y ago
Hey, Sam here, CTO of BitMEX - we're a crypto derivatives exchange that's been open since 2014. When we founded BitMEX, we went through a long period of discussion among the partners on this topic. We were just coming off the low of watching the MTGox heist and it it was obvious that wallets were becoming the #1 point of failure on any exchange. Ikeboy(sibling comment)'s explanation is as good as you'll get, in my opinion. And you can even skip steps 2 and 3, as we have done for two years on BitMEX. It's our opinion that customer funds are simply too important to expose to this kind of risk, so we process withdrawals only once a day from airgapped multisig wallets. It's a bear, and it's slow - we developed a few internal tools to make the review process easier and separate signal from noise - but aside from some manual work we've never had a problem. The #1 surprise, to me, was that customers don't mind the inconvenience. Sure, a few wonder why they can't simply get their funds immediately. But it's explained to them, they seem to appreciate the sensibility of managing this risk seriously. Compared to alternatives like ACH, at least it only takes one day and runs on weekends. I can't imagine a nightmare worse than losing millions of dollars of other people's money. I hope for Bitfinex's sake that they are able to recover a portion and continue operating. We've spoken personally with much of the team and enjoy the relationship. They - and their customers - don't deserve this loss.
- martinald 10y ago
  I know in the US you are used to ACH which is dreadful, but in the UK everyone is used to faster payments which allows you to send £100k to anywhere in the UK, often fee free, within 2 hours. Usually it's actually within a few seconds. Waiting a day for a transfer now seems hilariously outdated now to me. I understand there are other benefits to bitcoin but something seems inherently wrong if transfers have to be intentionally delayed so long to make the system more secure.
  - 300bps 10y ago
    I know in the US you are used to ACH which is dreadful, but in the UK everyone is used to faster payments which allows you to send £100k to anywhere in the UK, often fee free, within 2 hours. My understanding is that the inconvenience of ACH is by design - they are security delays. If you need to send money from one account to another instantly, you use a wire. If you have the amount of money in your account that is typical of people that need to send wires, they are often freely included with your higher-end checking account. Wires send the money instantly.
elevensies 10y ago
I've just run across this the other day, it is a set of guidelines for securing cryptocurrencies: https://cryptoconsortium.org/standards/CCSS https://cryptoconsortium.org/standards/CCSS . It is about the core crypto asset concerns, not about how to protect a web server. Here is a report which uses the guideline to assess a company's posture after a hack: https://www.scribd.com/doc/309591980/ShapeShift-Postmortem https://www.scribd.com/doc/309591980/ShapeShift-Postmortem (sorry about the scribd link, haven't been able to find pdf.) Here is a narrative description of the hack that the above report is about: http://moneyandstate.com/looting-of-the-fox/ http://moneyandstate.com/looting-of-the-fox/
Ironchefpython 10y ago
> how do people actually suggest securing exchange wallets? Why bother? It's much more profitable to pretend to secure the wallet, and then take the money and run.
vamur 10y ago
Maybe create a new address type on protocol level, where transactions from that address can be reversed within 36 hours with backup key(s)? Normal users wouldn't accept from these addresses but should be good enough for most exchanges for purposes of cold storage. Though in this case it looks like it was a hack on a third party provider, which is why relying on third parties for core functions is not a good idea. Why would Bitfinex use that provider instead of a multi-sig cold storage is anybody's idea.
- vamur 10y ago
  Also, looking at reddit, they moved to a new datacenter recently. So could be that a datacenter employee stole code and database and then submitted the withdrawal requests from the stolen code to Bitgo. Not even sure how to protect against that.
- robryk 10y ago
  See http://hackingdistributed.com/2016/02/26/how-to-implement-secure-bitcoin-vaults/ http://hackingdistributed.com/2016/02/26/how-to-implement-se... for a write-up on (I think) the exact thing you propose.
bachback 10y ago
The exchanges are not sophisticated at all. No single exchanges have opensourced their wallet technology. Kraken, Bitstamp and Bitfinex are using BitGo, a private source solution. The first would be to opensource a enterprise grade wallet which can be used for exchanges. And that's exactly what I'm working on. Then develop the procedures to secure funds with operational professionalism. Use of multi-sig is standard, but its not enough. Cold-wallet and transaction monitoring, deposit insurance, in combination with on-chain will develop over the next 1-2 years to keep deposits safe.

ikeboy 10y ago

1. Have a cold wallet on an airgapped computer with 99% of your funds. 2. Have a warm wallet on a computer with a firewall that only connects to 3, with 1% of your funds 3. Have a web-facing server that sends back commands to 2 when it needs to send money. 2 alerts someone when it needs more money, who then goes to the cold wallet and transfers some, creating the transaction offline and using a new USB key to transfer it to a computer with internet. The last step can also require multiple people if desired.

robryk 10y ago
When that person goes to the cold storage wallet, the only trustworthy information they have (assuming compomise of the online system) is how much money they are transferring. This means that this arrangement succeeds at making a trustworthy rate limit and nothing more. If I wanted to make a trustworthy rate limit, I'd have the machine from (1) not airgapped, but connected by a serial (unidirectional) line to the machine from (2) along which it would periodically send signed transactions sending some of the money from "lukewarm" storage to hot storage, and sending change back to the very same "lukewarm" storage address. The machine from (1) can then implement a rate limit.
- ikeboy 10y ago
  But then a compromise of 2 and a zero day remote attack can exploit 1. 1 is offline so no attack not involving insiders can affect it. Edit: actually nvm, didn't get the part about unidirectional. I don't deal with networking but I assume that's possible physically and there's no way to reverse the flow?
  - ch 10y ago
    Thats correct: https://en.wikipedia.org/wiki/Unidirectional_network https://en.wikipedia.org/wiki/Unidirectional_network Sometimes known as a Data Diode.

xorcist 10y ago

To start with, store most funds offline. A bank run (or hack) should be subject to intervention by a human. Contact customer if there is any doubt. Implement checks and limits for the remaining balance. Regularly audit and publish cryptographic proof you are in control of funds.

Scirra_Tom 10y ago
> Regularly audit and publish cryptographic proof you are in control of funds. What good is that proof of control? Literally the next second it could be drained and no one would know until the next scheduled proof of control. I've never been sure what publishing proof of control is meant to protect against exactly.
- elevensies 10y ago
  It would protect against what happened with MtGox -- if, after losing some but not all customer funds, they tried to keep running the site to make back the money. It would protect you from putting money into a site that is already bankrupt.
  - Scirra_Tom 10y ago
    Ah good point. But then how do you verify their figure on total customer deposits?
    - elevensies 10y ago
      If privacy is not a concern, you can just publish a list of your liabilities at a given time. Each account holder can then verify the their own listing, calculate the total, and compare to the proven reserves. There are more sophisticated schemes that allow you to check the amount in your account, and verify that your account was included in the total, without seeing the size of other accounts: https://iwilcox.me.uk/2014/proving-bitcoin-reserves https://iwilcox.me.uk/2014/proving-bitcoin-reserves .

berberous 10y ago

"We don't use cold storage for bitcoin, since our implementation with bitgo we've used segregated customer wallets so that each user has their own bitcoin wallet." [0] This was hubris / stupidity. [0] https://www.reddit.com/r/Bitcoin/comments/4vtuxo/bitfinex_security_breach_trading_will_be_halted/d61pqrm https://www.reddit.com/r/Bitcoin/comments/4vtuxo/bitfinex_se...

makomk 10y ago
No, it was US futures trading regulations (though arguably that's the same thing). For regulatory reasons they had to provide "actual delivery" of the bitcoins in question, and the CTFC held that cold storage didn't count as "actual delivery". See http://www.cftc.gov/idc/groups/public/@lrenforcementactions/documents/legalpleading/enfbfxnaorder060216.pdf http://www.cftc.gov/idc/groups/public/@lrenforcementactions/...
vamur 10y ago
In that case they should have required users to also login to bitgo and approve any withdrawals. Otherwise, what's the point of Bitgo?

STRML 10y ago

Hey, Sam here, CTO of BitMEX - we're a crypto derivatives exchange that's been open since 2014. When we founded BitMEX, we went through a long period of discussion among the partners on this topic. We were just coming off the low of watching the MTGox heist and it it was obvious that wallets were becoming the #1 point of failure on any exchange. Ikeboy(sibling comment)'s explanation is as good as you'll get, in my opinion. And you can even skip steps 2 and 3, as we have done for two years on BitMEX. It's our opinion that customer funds are simply too important to expose to this kind of risk, so we process withdrawals only once a day from airgapped multisig wallets. It's a bear, and it's slow - we developed a few internal tools to make the review process easier and separate signal from noise - but aside from some manual work we've never had a problem. The #1 surprise, to me, was that customers don't mind the inconvenience. Sure, a few wonder why they can't simply get their funds immediately. But it's explained to them, they seem to appreciate the sensibility of managing this risk seriously. Compared to alternatives like ACH, at least it only takes one day and runs on weekends. I can't imagine a nightmare worse than losing millions of dollars of other people's money. I hope for Bitfinex's sake that they are able to recover a portion and continue operating. We've spoken personally with much of the team and enjoy the relationship. They - and their customers - don't deserve this loss.

martinald 10y ago
I know in the US you are used to ACH which is dreadful, but in the UK everyone is used to faster payments which allows you to send £100k to anywhere in the UK, often fee free, within 2 hours. Usually it's actually within a few seconds. Waiting a day for a transfer now seems hilariously outdated now to me. I understand there are other benefits to bitcoin but something seems inherently wrong if transfers have to be intentionally delayed so long to make the system more secure.
- 300bps 10y ago
  I know in the US you are used to ACH which is dreadful, but in the UK everyone is used to faster payments which allows you to send £100k to anywhere in the UK, often fee free, within 2 hours. My understanding is that the inconvenience of ACH is by design - they are security delays. If you need to send money from one account to another instantly, you use a wire. If you have the amount of money in your account that is typical of people that need to send wires, they are often freely included with your higher-end checking account. Wires send the money instantly.

elevensies 10y ago

I've just run across this the other day, it is a set of guidelines for securing cryptocurrencies: https://cryptoconsortium.org/standards/CCSS https://cryptoconsortium.org/standards/CCSS . It is about the core crypto asset concerns, not about how to protect a web server. Here is a report which uses the guideline to assess a company's posture after a hack: https://www.scribd.com/doc/309591980/ShapeShift-Postmortem https://www.scribd.com/doc/309591980/ShapeShift-Postmortem (sorry about the scribd link, haven't been able to find pdf.) Here is a narrative description of the hack that the above report is about: http://moneyandstate.com/looting-of-the-fox/ http://moneyandstate.com/looting-of-the-fox/

Ironchefpython 10y ago

> how do people actually suggest securing exchange wallets? Why bother? It's much more profitable to pretend to secure the wallet, and then take the money and run.

vamur 10y ago

Maybe create a new address type on protocol level, where transactions from that address can be reversed within 36 hours with backup key(s)? Normal users wouldn't accept from these addresses but should be good enough for most exchanges for purposes of cold storage. Though in this case it looks like it was a hack on a third party provider, which is why relying on third parties for core functions is not a good idea. Why would Bitfinex use that provider instead of a multi-sig cold storage is anybody's idea.

vamur 10y ago
Also, looking at reddit, they moved to a new datacenter recently. So could be that a datacenter employee stole code and database and then submitted the withdrawal requests from the stolen code to Bitgo. Not even sure how to protect against that.
robryk 10y ago
See http://hackingdistributed.com/2016/02/26/how-to-implement-secure-bitcoin-vaults/ http://hackingdistributed.com/2016/02/26/how-to-implement-se... for a write-up on (I think) the exact thing you propose.

bachback 10y ago

The exchanges are not sophisticated at all. No single exchanges have opensourced their wallet technology. Kraken, Bitstamp and Bitfinex are using BitGo, a private source solution. The first would be to opensource a enterprise grade wallet which can be used for exchanges. And that's exactly what I'm working on. Then develop the procedures to secure funds with operational professionalism. Use of multi-sig is standard, but its not enough. Cold-wallet and transaction monitoring, deposit insurance, in combination with on-chain will develop over the next 1-2 years to keep deposits safe.