8 ms·

Many Bitcoin wallets vulnerable to double-spending of confirmed transactions

188 points by aburan28 11y ago

eblume 11y ago
This seems to be further confirmation that the ONLY safe way to use bitcoin is to use a full blockchain wallet such as Bitcoin Core. Any software which attempts to 'fake' it by not keeping the full blockchain has issues, exactly like this one being announced today. This is a serious problem for Bitcoin, because right now it takes 2-3 days to download and verify the full block chain, and is fast approaching 10^H^H40 gigabytes in size. There's going to need to be some sort of mitigation for this at some point but I don't see how that's possible.
- programmernews3 11y ago
  It's 36GiB
- TD-Linux 11y ago
  If it was really that unsafe to use a SPV wallet, the announcement would have said so. It did say that pools need to run a full node to not lose money, but the 36GiB burden should be a lot less of a problem for them.
- zaroth 11y ago
  The unpruned blockchain is about 30GB. Of course, the slowest way to download it is through the P2P network using a standard client. There are torrents which let you bootstrap the full blockchain up to some recent date, then let the P2P client continue from that point, so it doesn't have to take "days" to download, you can pull most of that 30GB at full line rate. But more importantly, even to run as a "full node" all you need is the UTXO (unspent transactions) and you can throw away the rest. Bitcoin Core can be flagged to prune the blockchain as you go. Pruning just discards transactions after they are spent, keeping track of just enough headers to still provide "full node" validation strength. Basically all you need to know is what available "outputs" there are, aka the unspent outputs, because that is the universe of "coins" which can be used as inputs in new transactions. I think the only thing that may be missing from all this is a way to download a proven pre-pruned blockchain to get started. That is possible, and it would bring startup costs of a full node down to about 1GB. The trickier challenge is if you have a historical wallet and you want to download the full transaction history of the wallet, without any meta-data you are left to downloading the whole blockchain in order to discover them. But you would still be able to get your current wallet balance even with pruning. https://www.reddit.com/r/Bitcoin/comments/33qv3a/pruning_support_what_is_it_and_where_might_it/ https://www.reddit.com/r/Bitcoin/comments/33qv3a/pruning_sup...
  - the_mitsuhiko 11y ago
    Correct me if I'm wrong, but how would having just access to the UTXO helped here?
    - Buge 11y ago
      This fork could have still been prevented even if only UTXO were stored. The problem behind the fork here is that 950 out of the last 1000 blocks had the block version set to 3, but didn't follow through with their promise. By setting the block version to 3, they were voting to transition to BIP66, meaning that once 950 of the last 1000 blocks vote for version 3, then no more nonstandard DER signatures will be allowed in blocks. Even if you prune spent transactions (so only keep UTXO) you still keep the block version, so you can still measure if 950/1000 blocks vote for version 3. And you are still able to verify that new blocks don't have nonstandard DER signatures.
  - grubles 11y ago
    " As of Bitcoin Core version 0.10.0 or later, this torrent is slower than a direct download using the bitcoin P2P protocol & client. This torrent is now only maintained for research and development purposes as a secondary option. Please use Bitcoin Core directly to download the blockchain." https://bitcointalk.org/index.php?topic=145386.0 https://bitcointalk.org/index.php?topic=145386.0
- fanquake 11y ago
  Ever since headers-first sync was merged (https://github.com/bitcoin/bitcoin/pull/4468 https://github.com/bitcoin/bitcoin/pull/4468), the entire blockchain download & verification process can take as little as 3-4 hours. In regards to the size of the blockchain, you might be interested in the recently merged Autoprune functionality (https://github.com/bitcoin/bitcoin/pull/5863 https://github.com/bitcoin/bitcoin/pull/5863), which deletes block and undo files to keep the total space used by those files below a certain threshold.
  - makomk 11y ago
    Last time I looked you could only use autoprune if you completely disabled the wallet functionality, so it was useless to anyone that wanted to use Bitcoin Core as a wallet (and it probably couldn't act as a backend to any of the lightweight wallets either). I think it's basically only of use to miners in its current state.
    - fanquake 11y ago
      Wallet functionality has been re-enabled to work with autoprune, but it won't make it into the 0.11 release. https://github.com/bitcoin/bitcoin/pull/6057 https://github.com/bitcoin/bitcoin/pull/6057
- makomk 11y ago
  If the bug I reported a couple of years ago hadn't been fixed, then even running a full node wouldn't have been safe. Basically, some older versions of Bitcoin Core didn't validate signatures in blocks older than 24 hours in order to speed up the initial blockchain sync, so they could be tricked into accepting an invalid chain where someone spent bitcoins that weren't theirs if it was far enough back. This wasn't considered a major issue at the time because you'd have to do a 51% attack for 24 hours to exploit it. If a large proportion of mining nodes are no longer validating blocks, that assumption is no longer true - you'd just need to mine one block, plus enough to push the non-validating nodes over the 51% line if they weren't already, and wait. (Oh, and I think Bitcoin Core may still skip full validation of older blocks during initial sync even with the fix.) The idea that all these miners aren't validating blocks before building on top of them now is scary. Everything's designed around the assumption that they will unless they're trying to actively attack Bitcoin.
  - zaroth 11y ago
    These are valid blocks under the old consensus rule. AFAIK, there is no mining of fully invalid blocks going on here.
    - makomk 11y ago
      That's probably because no-one's produced a fully invalid block with sufficient proof of work, but supposedly the pools involved would quite happily have built on top of one because they're skipping block validation almost entirely.
    - maaku 11y ago
      BIP 66 changed the definition of validity. These blocks are fully invalid.
  - HappyTypist 11y ago
    No, any block building on a invalid block is automatically invalid.
- StephenFalken 11y ago
  There is an interesting graph on how the block chain size has progressed over the years since Jan 2009 [1]. [1] https://blockchain.info/charts/blocks-size?timespan=all&showDataPoints=false&daysAverageString=1&show_header=true&scale=0&address= https://blockchain.info/charts/blocks-size?timespan=all&show...
- freework 11y ago
  > This seems to be further confirmation that the ONLY safe way to use bitcoin is to use a full blockchain wallet such as Bitcoin Core This statement is false. Not all lightweight wallets (wallets that function without needing to download the blockchain) are SPV wallets. Some lightweight wallets use blockchain APIs (and therefore are immune to this problem) Downloading the full blockhain is not the only way to secure bitcoin.
  - runn1ng 11y ago
    > blockchain APIs What doest that even mean? You mean blockchain.info API? Well Blockchain.info is not immune to this problem, they were running an old bitcoin core version.
- crispweed 11y ago
  It seems like there should be cheaper ways to detect and mitigate these kind of fork event (than doing full block chain verification). It shouldn't be too hard 'just' to detect that there are multiple chains being actively extended, and mitigate based on this fact (e.g. big warning to user, disallow transactions temporarily, or whatever)..
deleted 11y ago
[deleted]
zaroth 11y ago
The bug here is interesting. There are three separate pieces of new code which was added. First, is code which implements a new ordering rule. The second is consensus code which actually requires the new ordering to be in place in order to consider a block valid. The third is a flag in the header. The consensus rule was pre-programmed to automatically activate after 995 of the last 1000 blocks (in theory 99.5% of the network) included a flag in the block indicating that they would auto-activate the new consensus rule. So think of it like green and red lights. If there is no set of 1000 blocks where at least 995 of them are green, then the old ordering is still valid. Important to note that the new ordering is a stricter subset of the old, so new ordering is perfectly fine under the old rules, just old ordering is not acceptable under the new rules. This is called a SOFT fork. As soon as there are any 1000 blocks where 995 are green, then every block from that point onward MUST be properly ordered. (BTW, if you change rules such that something new you are doing would be invalid under the old code, like say, increasing the block size limit, it is called a HARD fork. Hard forks are considered much more disruptive to the network, soft forks are supposed to be easier to orchestrate) By "showing green" the miners were supposed to be advertising that they would enforce the rule once the cutoff point was met. The problem is enough miners showed green, the cutoff was hit, but then when an "incorrectly" ordered block came up sometime after the threshold was met, too many miners still accepted the block as VALID and so now we have a significant (majority?) hashpower running on blocks which are "supposed" to be invalid. Of course consensus is, by definition, whatever the majority is doing. So now it's a battle of dev's trying to cajole enough hashpower into enforcing the rule, or somehow backing off the change. With every new block mined on each head, there's more money at stake for which way the decision ultimately falls. For Bitcoin to survive, "There Can Be Only One". It will be interesting to see how exactly this came about. Were there bugs in the trigger code which was supposed to activate the new consensus rule? Did ignoramus miners simply turn on the flag without actually priming the new consensus code for activation? Did miners mistakenly think they just needed to show "green" and then their blocks would be valid, but then they published a "green" block with the older ordering?
- Adlai 11y ago
  Two mining pools (Discus Fish and Antpool) were engaged in what's called "SPV Mining", where they assume a block broadcast by another miner is valid based solely upon the block header (essentially, just checking that the block was "difficult enough"), without verifying (or downloading!) the transactions in that block. This strategy lowers their orphan rate at the cost of the risk that they'd waste work by mining on an invalid chain. This low risk was increased by the BIP66 enforcement (activation happened once only 75% of the blocks had the version 3 "flag").
Adlai 11y ago
The title is a little sensationalist. While the majority of wallets do only have SPV security, and such wallets were vulnerable to double spending during the recent fork event. The fork is over, so the wallets are no more vulnerable now than they were before (this vulnerability is not new). I hope this incident has convinced other miners that the ~1% profit boost from "SPV Mining" is not worth the fork risk that it enables, although apparently one of the pools (Discus Fish aka F2Pool) has already been warned against SPV Mining in the past. Let's hope that coins lost through this fork event is the sterner warning they needed.
- joosters 11y ago
  Doesn't seem sensationalist to me. As you say, the majority of wallets were vulnerable. The only problem is that the warning is far too late to protect anyone. I doubt any potential F2Pool miner loss is enough to persuade them to not do SPV mining. Just look at how many blocks they've mined overall and consider how much extra they've earned over time thanks to this 1% boost...
  - Adlai 11y ago
    It's sensationalist to suggest this vulnerability is anything new. The warning is late, indeed... I'd suggest putting it in the Terms of Service of every SPV wallet, but then even fewer people would read it. Maybe a better approach would be a popup each time the wallet blindly trusts a new merkle root?
- maaku 11y ago
  The wallets are still vulnerable to future forks of the same nature. It is not a 1% profit boost. It is a ~1% revenue boost, and mining is a very thin profit margin business. 1% revenue can be 10-20% profit. Furthermore, that gives an edge over the competition which doesn't use the patch which may be enough to shut them out.
sysk 11y ago
Also, Electrum wallet and friends don't do proof of work validation, they merely check which is the longest chain.
- Buge 11y ago
  What do you mean by "don't do proof of work validation". They don't verify the has in the header is the hash of the previous block's header? They don't that the hash is sufficiently small? They don't calculate the difficulty correctly?
  - sysk 11y ago
    > They don't calculate the difficulty correctly? Yes. It's possible to create a long chain at low difficulty and the original bitcoin client protects against that by computing the total "work" that has gone in a chain. However, Electrum just assumes that the longest chain will be the one which has most "work" in it which is wrong.
    - zaroth 11y ago
      That would be a pretty lame bug! It's the difference between "chainLength++;" and "chainLength+=pow;"
- wcoenen 11y ago
  I don't think so. These wallets implement Simplified Payment Verification as described by Satoshi in section 8 of the original bitcoin paper[1]. They do validate the block headers, which contain the proof of work. [1] https://bitcoin.org/bitcoin.pdf https://bitcoin.org/bitcoin.pdf
  - sysk 11y ago
    The issue is not with SPV itself but with Electrum's implementation (by "and friends" I really meant Electrum forks, not all SPV clients). Electrum does validate block headers but it doesn't handle re-orgs correctly. It just assumes the longest chain has the most PoW in it which is wrong and makes it vulnerable to attacks by malicious Electrum servers. Block headers do not contain the cumulative work that has gone into the chain (only the difficulty of the current block). A chain's total work must be calculated and stored locally by clients. https://github.com/spesmilo/electrum/blob/master/lib/blockchain.py#L56 https://github.com/spesmilo/electrum/blob/master/lib/blockch...
HappyTypist 11y ago
To clarify, the issue isn't SPV mining - the issue is not validating your mined blocks with the latest version when submitting. Even if the pools were SPV mining, their invalid chains would only be 1 block deep as the core client will reject any further blocks.

188 points by aburan28 11y ago

This seems to be further confirmation that the ONLY safe way to use bitcoin is to use a full blockchain wallet such as Bitcoin Core. Any software which attempts to 'fake' it by not keeping the full blockchain has issues, exactly like this one being announced today. This is a serious problem for Bitcoin, because right now it takes 2-3 days to download and verify the full block chain, and is fast approaching 10^H^H40 gigabytes in size. There's going to need to be some sort of mitigation for this at some point but I don't see how that's possible.

programmernews3 11y ago
It's 36GiB
TD-Linux 11y ago
If it was really that unsafe to use a SPV wallet, the announcement would have said so. It did say that pools need to run a full node to not lose money, but the 36GiB burden should be a lot less of a problem for them.
zaroth 11y ago
The unpruned blockchain is about 30GB. Of course, the slowest way to download it is through the P2P network using a standard client. There are torrents which let you bootstrap the full blockchain up to some recent date, then let the P2P client continue from that point, so it doesn't have to take "days" to download, you can pull most of that 30GB at full line rate. But more importantly, even to run as a "full node" all you need is the UTXO (unspent transactions) and you can throw away the rest. Bitcoin Core can be flagged to prune the blockchain as you go. Pruning just discards transactions after they are spent, keeping track of just enough headers to still provide "full node" validation strength. Basically all you need to know is what available "outputs" there are, aka the unspent outputs, because that is the universe of "coins" which can be used as inputs in new transactions. I think the only thing that may be missing from all this is a way to download a proven pre-pruned blockchain to get started. That is possible, and it would bring startup costs of a full node down to about 1GB. The trickier challenge is if you have a historical wallet and you want to download the full transaction history of the wallet, without any meta-data you are left to downloading the whole blockchain in order to discover them. But you would still be able to get your current wallet balance even with pruning. https://www.reddit.com/r/Bitcoin/comments/33qv3a/pruning_support_what_is_it_and_where_might_it/ https://www.reddit.com/r/Bitcoin/comments/33qv3a/pruning_sup...
- the_mitsuhiko 11y ago
  Correct me if I'm wrong, but how would having just access to the UTXO helped here?
  - Buge 11y ago
    This fork could have still been prevented even if only UTXO were stored. The problem behind the fork here is that 950 out of the last 1000 blocks had the block version set to 3, but didn't follow through with their promise. By setting the block version to 3, they were voting to transition to BIP66, meaning that once 950 of the last 1000 blocks vote for version 3, then no more nonstandard DER signatures will be allowed in blocks. Even if you prune spent transactions (so only keep UTXO) you still keep the block version, so you can still measure if 950/1000 blocks vote for version 3. And you are still able to verify that new blocks don't have nonstandard DER signatures.
- grubles 11y ago
  " As of Bitcoin Core version 0.10.0 or later, this torrent is slower than a direct download using the bitcoin P2P protocol & client. This torrent is now only maintained for research and development purposes as a secondary option. Please use Bitcoin Core directly to download the blockchain." https://bitcointalk.org/index.php?topic=145386.0 https://bitcointalk.org/index.php?topic=145386.0
fanquake 11y ago
Ever since headers-first sync was merged (https://github.com/bitcoin/bitcoin/pull/4468 https://github.com/bitcoin/bitcoin/pull/4468), the entire blockchain download & verification process can take as little as 3-4 hours. In regards to the size of the blockchain, you might be interested in the recently merged Autoprune functionality (https://github.com/bitcoin/bitcoin/pull/5863 https://github.com/bitcoin/bitcoin/pull/5863), which deletes block and undo files to keep the total space used by those files below a certain threshold.
- makomk 11y ago
  Last time I looked you could only use autoprune if you completely disabled the wallet functionality, so it was useless to anyone that wanted to use Bitcoin Core as a wallet (and it probably couldn't act as a backend to any of the lightweight wallets either). I think it's basically only of use to miners in its current state.
  - fanquake 11y ago
    Wallet functionality has been re-enabled to work with autoprune, but it won't make it into the 0.11 release. https://github.com/bitcoin/bitcoin/pull/6057 https://github.com/bitcoin/bitcoin/pull/6057
makomk 11y ago
If the bug I reported a couple of years ago hadn't been fixed, then even running a full node wouldn't have been safe. Basically, some older versions of Bitcoin Core didn't validate signatures in blocks older than 24 hours in order to speed up the initial blockchain sync, so they could be tricked into accepting an invalid chain where someone spent bitcoins that weren't theirs if it was far enough back. This wasn't considered a major issue at the time because you'd have to do a 51% attack for 24 hours to exploit it. If a large proportion of mining nodes are no longer validating blocks, that assumption is no longer true - you'd just need to mine one block, plus enough to push the non-validating nodes over the 51% line if they weren't already, and wait. (Oh, and I think Bitcoin Core may still skip full validation of older blocks during initial sync even with the fix.) The idea that all these miners aren't validating blocks before building on top of them now is scary. Everything's designed around the assumption that they will unless they're trying to actively attack Bitcoin.
- zaroth 11y ago
  These are valid blocks under the old consensus rule. AFAIK, there is no mining of fully invalid blocks going on here.
  - makomk 11y ago
    That's probably because no-one's produced a fully invalid block with sufficient proof of work, but supposedly the pools involved would quite happily have built on top of one because they're skipping block validation almost entirely.
  - maaku 11y ago
    BIP 66 changed the definition of validity. These blocks are fully invalid.
- HappyTypist 11y ago
  No, any block building on a invalid block is automatically invalid.
StephenFalken 11y ago
There is an interesting graph on how the block chain size has progressed over the years since Jan 2009 [1]. [1] https://blockchain.info/charts/blocks-size?timespan=all&showDataPoints=false&daysAverageString=1&show_header=true&scale=0&address= https://blockchain.info/charts/blocks-size?timespan=all&show...
freework 11y ago
> This seems to be further confirmation that the ONLY safe way to use bitcoin is to use a full blockchain wallet such as Bitcoin Core This statement is false. Not all lightweight wallets (wallets that function without needing to download the blockchain) are SPV wallets. Some lightweight wallets use blockchain APIs (and therefore are immune to this problem) Downloading the full blockhain is not the only way to secure bitcoin.
- runn1ng 11y ago
  > blockchain APIs What doest that even mean? You mean blockchain.info API? Well Blockchain.info is not immune to this problem, they were running an old bitcoin core version.
crispweed 11y ago
It seems like there should be cheaper ways to detect and mitigate these kind of fork event (than doing full block chain verification). It shouldn't be too hard 'just' to detect that there are multiple chains being actively extended, and mitigate based on this fact (e.g. big warning to user, disallow transactions temporarily, or whatever)..

deleted 11y ago

[deleted]

zaroth 11y ago

The bug here is interesting. There are three separate pieces of new code which was added. First, is code which implements a new ordering rule. The second is consensus code which actually requires the new ordering to be in place in order to consider a block valid. The third is a flag in the header. The consensus rule was pre-programmed to automatically activate after 995 of the last 1000 blocks (in theory 99.5% of the network) included a flag in the block indicating that they would auto-activate the new consensus rule. So think of it like green and red lights. If there is no set of 1000 blocks where at least 995 of them are green, then the old ordering is still valid. Important to note that the new ordering is a stricter subset of the old, so new ordering is perfectly fine under the old rules, just old ordering is not acceptable under the new rules. This is called a SOFT fork. As soon as there are any 1000 blocks where 995 are green, then every block from that point onward MUST be properly ordered. (BTW, if you change rules such that something new you are doing would be invalid under the old code, like say, increasing the block size limit, it is called a HARD fork. Hard forks are considered much more disruptive to the network, soft forks are supposed to be easier to orchestrate) By "showing green" the miners were supposed to be advertising that they would enforce the rule once the cutoff point was met. The problem is enough miners showed green, the cutoff was hit, but then when an "incorrectly" ordered block came up sometime after the threshold was met, too many miners still accepted the block as VALID and so now we have a significant (majority?) hashpower running on blocks which are "supposed" to be invalid. Of course consensus is, by definition, whatever the majority is doing. So now it's a battle of dev's trying to cajole enough hashpower into enforcing the rule, or somehow backing off the change. With every new block mined on each head, there's more money at stake for which way the decision ultimately falls. For Bitcoin to survive, "There Can Be Only One". It will be interesting to see how exactly this came about. Were there bugs in the trigger code which was supposed to activate the new consensus rule? Did ignoramus miners simply turn on the flag without actually priming the new consensus code for activation? Did miners mistakenly think they just needed to show "green" and then their blocks would be valid, but then they published a "green" block with the older ordering?

Adlai 11y ago
Two mining pools (Discus Fish and Antpool) were engaged in what's called "SPV Mining", where they assume a block broadcast by another miner is valid based solely upon the block header (essentially, just checking that the block was "difficult enough"), without verifying (or downloading!) the transactions in that block. This strategy lowers their orphan rate at the cost of the risk that they'd waste work by mining on an invalid chain. This low risk was increased by the BIP66 enforcement (activation happened once only 75% of the blocks had the version 3 "flag").

Adlai 11y ago

The title is a little sensationalist. While the majority of wallets do only have SPV security, and such wallets were vulnerable to double spending during the recent fork event. The fork is over, so the wallets are no more vulnerable now than they were before (this vulnerability is not new). I hope this incident has convinced other miners that the ~1% profit boost from "SPV Mining" is not worth the fork risk that it enables, although apparently one of the pools (Discus Fish aka F2Pool) has already been warned against SPV Mining in the past. Let's hope that coins lost through this fork event is the sterner warning they needed.

joosters 11y ago
Doesn't seem sensationalist to me. As you say, the majority of wallets were vulnerable. The only problem is that the warning is far too late to protect anyone. I doubt any potential F2Pool miner loss is enough to persuade them to not do SPV mining. Just look at how many blocks they've mined overall and consider how much extra they've earned over time thanks to this 1% boost...
- Adlai 11y ago
  It's sensationalist to suggest this vulnerability is anything new. The warning is late, indeed... I'd suggest putting it in the Terms of Service of every SPV wallet, but then even fewer people would read it. Maybe a better approach would be a popup each time the wallet blindly trusts a new merkle root?
maaku 11y ago
The wallets are still vulnerable to future forks of the same nature. It is not a 1% profit boost. It is a ~1% revenue boost, and mining is a very thin profit margin business. 1% revenue can be 10-20% profit. Furthermore, that gives an edge over the competition which doesn't use the patch which may be enough to shut them out.

sysk 11y ago

Also, Electrum wallet and friends don't do proof of work validation, they merely check which is the longest chain.

Buge 11y ago
What do you mean by "don't do proof of work validation". They don't verify the has in the header is the hash of the previous block's header? They don't that the hash is sufficiently small? They don't calculate the difficulty correctly?
- sysk 11y ago
  > They don't calculate the difficulty correctly? Yes. It's possible to create a long chain at low difficulty and the original bitcoin client protects against that by computing the total "work" that has gone in a chain. However, Electrum just assumes that the longest chain will be the one which has most "work" in it which is wrong.
  - zaroth 11y ago
    That would be a pretty lame bug! It's the difference between "chainLength++;" and "chainLength+=pow;"
wcoenen 11y ago
I don't think so. These wallets implement Simplified Payment Verification as described by Satoshi in section 8 of the original bitcoin paper[1]. They do validate the block headers, which contain the proof of work. [1] https://bitcoin.org/bitcoin.pdf https://bitcoin.org/bitcoin.pdf
- sysk 11y ago
  The issue is not with SPV itself but with Electrum's implementation (by "and friends" I really meant Electrum forks, not all SPV clients). Electrum does validate block headers but it doesn't handle re-orgs correctly. It just assumes the longest chain has the most PoW in it which is wrong and makes it vulnerable to attacks by malicious Electrum servers. Block headers do not contain the cumulative work that has gone into the chain (only the difficulty of the current block). A chain's total work must be calculated and stored locally by clients. https://github.com/spesmilo/electrum/blob/master/lib/blockchain.py#L56 https://github.com/spesmilo/electrum/blob/master/lib/blockch...

HappyTypist 11y ago

To clarify, the issue isn't SPV mining - the issue is not validating your mined blocks with the latest version when submitting. Even if the pools were SPV mining, their invalid chains would only be 1 block deep as the core client will reject any further blocks.