6 ms·

Open-Sourcing Subzero: Square’s Bitcoin Cold Storage Solution

165 points by mcpherrinm 8y ago

haneefmubarak 8y ago
This is neat, but I think it'd be interesting to see a writeup that talked more about why they needed a new solution and what they did differently that solved their particular problem. In particular, they mention an onion model - what does that look like for them? Hot/warm/cold can take a variety of forms...
- amenghra 8y ago
  The main motivation to build vs use an existing solution was a solid backup/DR story and the desire to be able to write custom business logic which is enforced inside the HSM. It’s things like controlling where funds can go, or requiring specific authorization based on amount being move around, etc. Being able to leverage hardware we are already familiar with or which has various certifications (such as Fips) is a neat bonus. If you look at the code, you’ll see that the funds flow from “anywhere -> hot -> cold -> warm -> anywhere”. This is the minimal setup and you can add more layers if needed.
  - tudorconstantin 8y ago
    With HD wallets you can generate receiving addresses based on the master public key of a wallet, without needing any private key. Is there a reason for not receiving funds straight into the cold wallet? Edit: in the docs it seems that you indeed receive funds directly in the cold wallet
    - amenghra 8y ago
      You are right. You can go from external entity to cold wallet directly. The important piece is how the funds leave the cold wallet.
aeternus 8y ago
>One specific customization we implemented is the ability to enforce that cold wallets can only send funds to a Square-owned hot wallet. This actually seems like a significant risk. Cold storage is based on the idea that hot wallets are more easily compromised. If someone does manage to compromise Square's hot wallet private keys, they can basically hold all cold storage funds hostage. Perhaps Square has some way to modify the allowed addresses, but that somewhat defeats the purpose of the restriction.
- mcpherrinm 8y ago
  The code which implements the restrictions on destination is just code. We can add new destinations if needed, but that requires a quorum of codesigners and a more arduous process than simply withdrawing funds. The goal here is the normal operation, cold withdrawal, is very low risk. If we ever did suspect our hot wallet keys were breached, we'd have to do that or else we would be stuck as you suggested.
  - aeternus 8y ago
    Makes sense, thanks for the explanation.
  - jstanley 8y ago
    So what's the material difference between the cold wallet and the hot wallet if funds from the cold wallet can be moved into the hot wallet freely?
    - sneak 8y ago
      Moving funds from multisig cold storage is not “freely” when it at the minimum requires two people to travel to two different key storage/signing locations. You’re generally looking at a 12-48h process at a minimum. “Cold” refers to it being entirely offline.
      - londons_explore 8y ago
tommoor 8y ago
I'd love to see a writeup on why they even went with the central storage model at all – I'm sure there were many considerations but it seems as though CashApp was perfectly positioned to become a mainstream Bitcoin wallet with the private keys in the hands of the users.
- pat2man 8y ago
  Users lose their keys all the time. The cost of support calls alone would probably be higher than the cost of developing a cold wallet solution, especially since it sounds like they already had HSMs and the expertise to use them.
deleted 8y ago
[deleted]
arthurcolle 8y ago
> cold wallets can only send funds to a Square-owned hot wallet. Amazing innovation here, truly. Maybe float an ERC20 token next!
- fizwhiz 8y ago
  Very sarcasm. wow. amaze.
dannyw 8y ago
Speaking of hardware security modules, are there any that can be easily and cheaply acquired by hobbyists?
- pmorici 8y ago
  As far as HSMs go the Yubico YubiHSM 2 is the cheapest and most approachable one I've found. [0] I don't have a good feeling for how it compares to some of the other products out there but it is also the only one that has a list price you don't have to call a sales person to get a quote on. [0] https://www.yubico.com/product/yubihsm-2/ https://www.yubico.com/product/yubihsm-2/
  - Darkstryder 8y ago
    I've used both Thales nShield and Bull Proteccio models at my previous work. Both are priced in the five figure euros/dollars range for any reasonable operational setup. (In the sense that you need at least a production HSM and an additional machine to be able to react quickly to incidents and experiment with). The YubiHSM v2 seems very promising to me too, but the thing you have to understand with HSMs is that their certifications are often as important, or even more important, than their actual security. (Usually you use HSMs because you legally have to and not just for fun) In that regard, both the nShield and the Proteccio are qualified at Level 3 of FIPS 140-2. This is a very expensive qualification to pass and the investment is reflected back in the price of those products. Yubico does not advertise any qualification level for the YubiHSM 2, therefore I infer they did not pass any certification process. Hence I feel they will target a different market than Thales and Bull. (People wanting more security but without needing certifications) A 650$ FIPS level 3, eIDAS qualified HSM would be groundbreaking. As certification processes are quite political, I'm not even sure it would be possible as it would basically decimate current HSMs vendors. Let's hope the YubiHSM v3 could be that though ? :)
    - pmorici 8y ago
      Yes, if you want FIPS 140-2 you are going to pay the big bucks. Yubico's Yubikey 4 has a FIPS certified version so they have been through the process with other products but you are right that their HSM is not certified. If you care about FIPS certification you can look up if a product is certified or not in NIST's database. [0] [0] https://csrc.nist.gov/projects/cryptographic-module-validation-program/validated-modules/search https://csrc.nist.gov/projects/cryptographic-module-validati...
SEJeff 8y ago
This project looks pretty interesting: https://github.com/square/beancounter/ https://github.com/square/beancounter/ Allows mapping a date to blocks or see what the wallet balance was at any given time.
Uptrenda 8y ago
I'm glad that exchanges are starting to take security more seriously. But the security of cold wallet funds is much easier to maintain. The issue here with centralization and hot wallets is that funds have to sit on publicly accessible servers. So really, any improvements made to cold wallet storage are nothing more than a red herring that distracts from the real issue that centralized exchanges were never designed to handle cryptocurrencies. There are far more problems to this approach than just wallet security too. With an inherent lack of transparency that comes from centralized exchanges how is the user to know that trades are being executed fairly? The exchange still has an edge over every trader in that market and can skim profits off of anyone. If we observe Mtgox it was the result of what can happen when everything goes wrong (hot and cold wallets mismanaged and order books manipulated.) So call me a skeptic but I don't see how this solves anything. I did enjoy the hand waving though. Very cool, and I'm sure Squares customers will be happy with it when their convoluted security protocols are inevitability broken by a rubber mallet.
- atomical 8y ago
  > With an inherent lack of transparency that comes from centralized exchanges how is the user to know that trades are being executed fairly? How do we know Nasdaq isn't screwing the buyers and sellers? It's only a problem with some exchanges for cryptocurrency. The trust issue is with cryptocurrency exchanges in general and not centralization. Check out the Unchained podcast. Laura Shin interviews the CEO of Binance. His answer on the question of whether tokens are securities is that some people say these are securities and some people say they aren't. Does that sound like someone you want to trust with your money? Even if you believe in cryptocurrency why would you trust a CEO that's taking huge risks by throwing up his hands and saying that there isn't regulatory clarity when the SEC says the opposite?
- pat2man 8y ago
  You can always trade on a decentralized exchange like https://www.radarrelay.com https://www.radarrelay.com
  - snissn 8y ago
o_nate 8y ago
If it's taught us nothing else, I'm grateful for the whole bitcoin mania for teaching us some humility about computer security. It seems the consensus is forming that a computer that's connected to the network is pretty much insecure by definition.

165 points by mcpherrinm 8y ago

This is neat, but I think it'd be interesting to see a writeup that talked more about why they needed a new solution and what they did differently that solved their particular problem. In particular, they mention an onion model - what does that look like for them? Hot/warm/cold can take a variety of forms...

amenghra 8y ago
The main motivation to build vs use an existing solution was a solid backup/DR story and the desire to be able to write custom business logic which is enforced inside the HSM. It’s things like controlling where funds can go, or requiring specific authorization based on amount being move around, etc. Being able to leverage hardware we are already familiar with or which has various certifications (such as Fips) is a neat bonus. If you look at the code, you’ll see that the funds flow from “anywhere -> hot -> cold -> warm -> anywhere”. This is the minimal setup and you can add more layers if needed.
- tudorconstantin 8y ago
  With HD wallets you can generate receiving addresses based on the master public key of a wallet, without needing any private key. Is there a reason for not receiving funds straight into the cold wallet? Edit: in the docs it seems that you indeed receive funds directly in the cold wallet
  - amenghra 8y ago
    You are right. You can go from external entity to cold wallet directly. The important piece is how the funds leave the cold wallet.

aeternus 8y ago

>One specific customization we implemented is the ability to enforce that cold wallets can only send funds to a Square-owned hot wallet. This actually seems like a significant risk. Cold storage is based on the idea that hot wallets are more easily compromised. If someone does manage to compromise Square's hot wallet private keys, they can basically hold all cold storage funds hostage. Perhaps Square has some way to modify the allowed addresses, but that somewhat defeats the purpose of the restriction.

mcpherrinm 8y ago
The code which implements the restrictions on destination is just code. We can add new destinations if needed, but that requires a quorum of codesigners and a more arduous process than simply withdrawing funds. The goal here is the normal operation, cold withdrawal, is very low risk. If we ever did suspect our hot wallet keys were breached, we'd have to do that or else we would be stuck as you suggested.
- aeternus 8y ago
  Makes sense, thanks for the explanation.
- jstanley 8y ago
  So what's the material difference between the cold wallet and the hot wallet if funds from the cold wallet can be moved into the hot wallet freely?
  - sneak 8y ago
    Moving funds from multisig cold storage is not “freely” when it at the minimum requires two people to travel to two different key storage/signing locations. You’re generally looking at a 12-48h process at a minimum. “Cold” refers to it being entirely offline.
    - londons_explore 8y ago

tommoor 8y ago

I'd love to see a writeup on why they even went with the central storage model at all – I'm sure there were many considerations but it seems as though CashApp was perfectly positioned to become a mainstream Bitcoin wallet with the private keys in the hands of the users.

pat2man 8y ago
Users lose their keys all the time. The cost of support calls alone would probably be higher than the cost of developing a cold wallet solution, especially since it sounds like they already had HSMs and the expertise to use them.

deleted 8y ago

[deleted]

arthurcolle 8y ago

> cold wallets can only send funds to a Square-owned hot wallet. Amazing innovation here, truly. Maybe float an ERC20 token next!

fizwhiz 8y ago
Very sarcasm. wow. amaze.

dannyw 8y ago

Speaking of hardware security modules, are there any that can be easily and cheaply acquired by hobbyists?

pmorici 8y ago
As far as HSMs go the Yubico YubiHSM 2 is the cheapest and most approachable one I've found. [0] I don't have a good feeling for how it compares to some of the other products out there but it is also the only one that has a list price you don't have to call a sales person to get a quote on. [0] https://www.yubico.com/product/yubihsm-2/ https://www.yubico.com/product/yubihsm-2/
- Darkstryder 8y ago
  I've used both Thales nShield and Bull Proteccio models at my previous work. Both are priced in the five figure euros/dollars range for any reasonable operational setup. (In the sense that you need at least a production HSM and an additional machine to be able to react quickly to incidents and experiment with). The YubiHSM v2 seems very promising to me too, but the thing you have to understand with HSMs is that their certifications are often as important, or even more important, than their actual security. (Usually you use HSMs because you legally have to and not just for fun) In that regard, both the nShield and the Proteccio are qualified at Level 3 of FIPS 140-2. This is a very expensive qualification to pass and the investment is reflected back in the price of those products. Yubico does not advertise any qualification level for the YubiHSM 2, therefore I infer they did not pass any certification process. Hence I feel they will target a different market than Thales and Bull. (People wanting more security but without needing certifications) A 650$ FIPS level 3, eIDAS qualified HSM would be groundbreaking. As certification processes are quite political, I'm not even sure it would be possible as it would basically decimate current HSMs vendors. Let's hope the YubiHSM v3 could be that though ? :)
  - pmorici 8y ago
    Yes, if you want FIPS 140-2 you are going to pay the big bucks. Yubico's Yubikey 4 has a FIPS certified version so they have been through the process with other products but you are right that their HSM is not certified. If you care about FIPS certification you can look up if a product is certified or not in NIST's database. [0] [0] https://csrc.nist.gov/projects/cryptographic-module-validation-program/validated-modules/search https://csrc.nist.gov/projects/cryptographic-module-validati...

SEJeff 8y ago

This project looks pretty interesting: https://github.com/square/beancounter/ https://github.com/square/beancounter/ Allows mapping a date to blocks or see what the wallet balance was at any given time.

Uptrenda 8y ago

I'm glad that exchanges are starting to take security more seriously. But the security of cold wallet funds is much easier to maintain. The issue here with centralization and hot wallets is that funds have to sit on publicly accessible servers. So really, any improvements made to cold wallet storage are nothing more than a red herring that distracts from the real issue that centralized exchanges were never designed to handle cryptocurrencies. There are far more problems to this approach than just wallet security too. With an inherent lack of transparency that comes from centralized exchanges how is the user to know that trades are being executed fairly? The exchange still has an edge over every trader in that market and can skim profits off of anyone. If we observe Mtgox it was the result of what can happen when everything goes wrong (hot and cold wallets mismanaged and order books manipulated.) So call me a skeptic but I don't see how this solves anything. I did enjoy the hand waving though. Very cool, and I'm sure Squares customers will be happy with it when their convoluted security protocols are inevitability broken by a rubber mallet.

atomical 8y ago
> With an inherent lack of transparency that comes from centralized exchanges how is the user to know that trades are being executed fairly? How do we know Nasdaq isn't screwing the buyers and sellers? It's only a problem with some exchanges for cryptocurrency. The trust issue is with cryptocurrency exchanges in general and not centralization. Check out the Unchained podcast. Laura Shin interviews the CEO of Binance. His answer on the question of whether tokens are securities is that some people say these are securities and some people say they aren't. Does that sound like someone you want to trust with your money? Even if you believe in cryptocurrency why would you trust a CEO that's taking huge risks by throwing up his hands and saying that there isn't regulatory clarity when the SEC says the opposite?
pat2man 8y ago
You can always trade on a decentralized exchange like https://www.radarrelay.com https://www.radarrelay.com
- snissn 8y ago

o_nate 8y ago

If it's taught us nothing else, I'm grateful for the whole bitcoin mania for teaching us some humility about computer security. It seems the consensus is forming that a computer that's connected to the network is pretty much insecure by definition.