9 ms·

The common adage seems to be "storage is cheap", meaning "multi-TB blockchains won't be a problem because you can buy that much storage for a few hundred dollar

by na85 12y ago

The common adage seems to be "storage is cheap", meaning "multi-TB blockchains won't be a problem because you can buy that much storage for a few hundred dollars". But I think the stupidly-large size of the blockchain is a major hurdle, given the concurrent trend of miniaturization. Embedded devices like one might find in the much-vaunted "internet of things" (eyeball roll) can't really afford to be toting six SSDs just for storing 3 or 4 different blockchains, or even one big one. There comes a point when it's just no longer practical to store the entire blockchain, but truncating the chain (or only storing some kind of "working set") isn't feasible either. Not to mention the gargantuan download that's required for that initial setup to get up to date with the latest network transactions. I think it's a great idea in theory but ultimately pretty annoying in practice.

deleted 12y ago
[deleted]
wanderingstan 12y ago
A friend of mine joined chain.com and, as I understand it, their value proposition is to handle the load of blockchain while providing trusted access to whatever needs to be known. I'm no expert, but this seems like the right sort of solution for the problem you pose. Blockchain management is handled in the cloud, and clients (including embedded devices) talk securely to a trusted cloud service to get the relevant details.
- rfrank 12y ago
  What defines "trusted access?" One of the points of bitcoin / blockchain is not having to trust anyone involved.
- na85 12y ago
  That seems self defeating. None of the advantages of a decentralized system are realized by centralizing the blockchain
  - sliverstorm 12y ago
    That's perhaps my favorite part of Bitcoin. For good or bad, it was started by people rallying against the big centralized banks, and now at every turn we are hitting reasons why centralization makes a lot of sense. I half suspect that eventually it will come to resemble the current centralized systems, whether people want it or not. For example, the number of complete blockchains is dwindling as more people individually elect to start using partial-blockchain clients because of space & traffic. It's especially endearing to me because it's my primary objection to anarchists (of whom I have several as friends)- that the system they envision just isn't stable, and we arrived at the system we have now for that reason.
    - vidarh 12y ago
      I find this line of thinking unconvincing. There is a vast difference between voluntary centralisation from which you can withdraw at any time, and forced centralisation from which there is no practical way to withdraw without huge sacrifices. Many socialist and anarchist ideologists are/were perfectly fine with centralisation of production, as long as decision making is decentralised, and participation is a result of free choice and can be withdrawn. The capability of decentralisation in Bitcoin is important because it acts as a safeguard against forced centralisation and coercion. As long as the capability remains, whether or not people for practical/efficiency reasons opts for centralisation ought not cause most anarchists any major concern.
      - jarek 12y ago
        > The capability of decentralisation in Bitcoin is important because it acts as a safeguard against forced centralisation and coercion. As long as the capability remains, whether or not people for practical/efficiency reasons opts for centralisation ought not cause most anarchists any major concern. I'm probably misunderstanding something, but isn't it more that blockchain technology allows decentralisation, while dominance of any particular deployment supports centralisation? As I understand it, once Bitcoin is centralised, the 51% can control it as they wish, and the only alternative is to set up a wholly separate Bitcoin2 chain, or another altcoin. (Again resulting in cryptocurrency being a bad store of value.)
mandrake-c-papi 12y ago
I recently replaced an old HDD with a new SSD for my win7 install. I had over 100g of blockchains for various coins in my user app data folder. To rebuild from scratch would have taken days and a big chunk of my broadband data allowance. I know it's centralisation and goes against one of the features of the blockchain, but having trusted hubs through which to interact would save a lot of wasted space, time and data. I'm not sure why inserting a check-point and resetting the blockchain size to zero would not be possible - can someone please explain this? If the status at the checkpoint were agreed upon by 51% of all nodes then why not?
- TylerE 12y ago
  You know what would save a lot more space, time, data, cpu power, hardware, and energy? Not using cryptocurrency.
  - baddox 12y ago
    Perhaps. And perhaps not using a computer or any electronic devices at all would further "save" those things. I don't see your point.
    - TylerE 12y ago
      Cryptocurrency is astoundingly bad at converting non-renewable natural resources into nothing of absolute value at all. Even (real) strip mining at least produces industrially useful ore. Blockchain technology as implemented in bitcoin is incredibly wasteful. The power used alone is in the tera-watt hours per year. All of those massive farms of custom miners...all to process an astounding ~90k transactions a day. The relative energy cost is astounding, considering that volume could easily be handled by a single server in a traditional architecture.
      - kens 12y ago
        A Raspberry Pi running a Postgres database can handle about 10 times the transaction rate of the Bitcoin network. (Obviously this isn't exactly apples-to-apples, but it gives a rough idea of how totally out of balance the resources consumed vs the transaction rate is.) (10 TPS for TPC-B: http://raspberrypg.org/2013/11/raspberry-pg-server-some-benchmark-results/ http://raspberrypg.org/2013/11/raspberry-pg-server-some-benc...)
anigbrowl 12y ago
You're dead right. A serious problem with the Bitcoin and Dogecoin clients (which I last used some time ago, but which changed relatively little over time) is taht they just installed themselves on the boot drive (if you're running windows) and had little in the way of configuration options. Cut to a few months later when I'm wondering why I'm running out of space on my SSD, only to realize that it's full of gigabytes of blockchain data that is a complete waste of space, with no convient way to put it somewhere else. Could I manually get it installed and running on another large-volume and slower speed drive where this wouldn't be an issue, of course. But why do I want to bother even dealing with something so user-hostile? The average Joe or Jill who isn't a computer nerd is just going to see this thing eating his/her hard drive and get rid of it.
roasbeef 12y ago
Some embedded "internet of things" like device wouldn't be a full-node. Rather it would be a lite-node relying on the SPV[1] model of security. Storing some type of "working set" is totally feasible. Technically you don't even need to keep around all blocks, simply the resultant state that you arrive at after processing all blocks in order. This is called the Unspent Transaction Output set or UTXO set. It's also possible to only keep around XGB of blocks, or the last 500 blocks, etc (a pruned log)[3]. The purpose of keeping around all blocks is to bootstrap the syncing of new nodes to the network, or optionally to full-chain wallets for re-scanning. Before pruned nodes are widely deployed, a system needs to be developed to allow a syncing node to efficiently identify and retrieve the relevant blockchain shards it needs so it can validate everything and arrive at the current state. Also, with BitcoinCore 0.10, syncing takes about 4 hours if you don't have shitty a hardware or internet connection [2]. [1]: https://en.bitcoin.it/wiki/Thin_Client_Security https://en.bitcoin.it/wiki/Thin_Client_Security [2]: https://github.com/bitcoin/bitcoin/pull/4468 https://github.com/bitcoin/bitcoin/pull/4468 [3]: https://github.com/bitcoin/bitcoin/pull/4701 https://github.com/bitcoin/bitcoin/pull/4701
- vertex-four 12y ago
  > Some embedded "internet of things" like device wouldn't be a full-node. Rather it would be a lite-node relying on the SPV[1] model of security. Well, it's a lot more likely in my eyes that it'd be a thin client operating against a server owned by its owner. Just as safe, and doesn't involve trying to figure out how to allocate enough money on a micro scale to all your individual devices so they don't run out.
IkmoIkmo 12y ago
> There comes a point when it's just no longer practical to store the entire blockchain Why would you have to? The point of decentralisation is not that everyone has a copy of the truth. It's that a sufficient number of nodes have a copy of the truth, such that you can check a number of them and easily verify the veracity of the data. Many blockchain applications run not by querying their own blockchain but rather querying a number of trusted sources via their API. Coinbase, Bitpay, Chain.com, Blockcypher etc all offer APIs to interface with their blockchain nodes and others. It's like saying Facebook will never work because you can't possible store the worlds' photos on everyone's smartphones. You don't have to, Facebook stores it on their server. Similarly we will see, and already see, dedicated servers that store the blockchain, and as long as you have enough independent ones you can maintain decentralisation just as if you were hosting your own node. There's absolutely no need to host the blockchain on small embedded devices any more than any other data, like music or photos we normally host on dedicated servers instead. > but truncating the chain (or only storing some kind of "working set") isn't feasible either. Yes it is. What makes you say it isn't? Pruning the blockchain is something that's currently being worked on. The idea is that if I send you $1 a million times and you send it back $1m times, every day, for the next 1 billion years, we could either store terabytes and terabytes of data of all these transactions, or we could store the end result (= no change occurred) and the last couple weeks of transaction data. The rest can be pruned. Because each block has a hash of all its transactions, and this hash is used to mine the next block (hence a chain of blocks, blockchain), we needn't store each individual transaction, we have the hash to verify the truth of transactions. We might have some universities saving all transactions for posterity & research, and some businesses for data mining, but the vast majorities of nodes that just need to secure the network can indeed, without security issues, not store long-spent outputs. (i.e. bitcoins that have already been spent many times over. e.g. if I give a dollar to you and you give it to the next guy, and he to the next girl, and so on 100 times, at that point there's absolutely no reason to store on every node the fact I gave a dollar to you for the security of the network.) > Not to mention the gargantuan download that's required for that initial setup to get up to date with the latest network transactions. Again, no need to have all redundant transactions. The pruned blockchain would currently be under a gigabyte large. Second, headers-first has already been implemented which made the initial setup much, much faster. Beyond that, we mustn't forget that the average American makes 2 transactions a day. And one such transaction is about 400 bytes or so, in bitcoin. If you compare this to tweets, or an hour of netflix, it's puny. If you count up all the data, it's about $8 to store one day of all of US consumer transactions on a retail harddrive. Imagine the cost of storage for Mastercard for one of their nodes was $8, and instead of just powering a fraction of US retail, they'd literally handle every single transaction by themselves and store them all for $8. Even if you wanted to have thousands of nodes and store the last year of data (and given the velocity of money, you wouldn't have to as again, you can prune data after money has been spent a couple times), that's a puny amount of money to power an entire consumer financial system of the world's biggest economy. And given storage, like CPU and bandwidth, on a 50-60% yoy growth rate (Kryders, Moore's and Nielsen's laws), I think by the time bitcoin becomes important enough to run 100% of US retail (which is likely never, but hypothetically speaking), that $8 will have dropped to below $0.10 easily. I know I'm taking big shortcuts here but this is just to illustrate the economics are extremely favorable today and in the future. I don't know if bitcoin will succeed or fail, but I'm pretty near certain it won't fail because storage/cpu/bandwidth isn't affordable enough for nodes to make it viable to run the network. I and many others have looked into this quite a bit and these aren't big problems. Bitcoin has big obstacles (no gigantic consumer benefits in a world of decent fintech like every OECD country to use bitcoin directly, is one), but storage probably isn't one of em.

by na85 12y ago

deleted 12y ago

[deleted]

wanderingstan 12y ago

A friend of mine joined chain.com and, as I understand it, their value proposition is to handle the load of blockchain while providing trusted access to whatever needs to be known. I'm no expert, but this seems like the right sort of solution for the problem you pose. Blockchain management is handled in the cloud, and clients (including embedded devices) talk securely to a trusted cloud service to get the relevant details.

rfrank 12y ago
What defines "trusted access?" One of the points of bitcoin / blockchain is not having to trust anyone involved.
na85 12y ago
That seems self defeating. None of the advantages of a decentralized system are realized by centralizing the blockchain
- sliverstorm 12y ago
  That's perhaps my favorite part of Bitcoin. For good or bad, it was started by people rallying against the big centralized banks, and now at every turn we are hitting reasons why centralization makes a lot of sense. I half suspect that eventually it will come to resemble the current centralized systems, whether people want it or not. For example, the number of complete blockchains is dwindling as more people individually elect to start using partial-blockchain clients because of space & traffic. It's especially endearing to me because it's my primary objection to anarchists (of whom I have several as friends)- that the system they envision just isn't stable, and we arrived at the system we have now for that reason.
  - vidarh 12y ago
    I find this line of thinking unconvincing. There is a vast difference between voluntary centralisation from which you can withdraw at any time, and forced centralisation from which there is no practical way to withdraw without huge sacrifices. Many socialist and anarchist ideologists are/were perfectly fine with centralisation of production, as long as decision making is decentralised, and participation is a result of free choice and can be withdrawn. The capability of decentralisation in Bitcoin is important because it acts as a safeguard against forced centralisation and coercion. As long as the capability remains, whether or not people for practical/efficiency reasons opts for centralisation ought not cause most anarchists any major concern.
    - jarek 12y ago
      > The capability of decentralisation in Bitcoin is important because it acts as a safeguard against forced centralisation and coercion. As long as the capability remains, whether or not people for practical/efficiency reasons opts for centralisation ought not cause most anarchists any major concern. I'm probably misunderstanding something, but isn't it more that blockchain technology allows decentralisation, while dominance of any particular deployment supports centralisation? As I understand it, once Bitcoin is centralised, the 51% can control it as they wish, and the only alternative is to set up a wholly separate Bitcoin2 chain, or another altcoin. (Again resulting in cryptocurrency being a bad store of value.)

mandrake-c-papi 12y ago

I recently replaced an old HDD with a new SSD for my win7 install. I had over 100g of blockchains for various coins in my user app data folder. To rebuild from scratch would have taken days and a big chunk of my broadband data allowance. I know it's centralisation and goes against one of the features of the blockchain, but having trusted hubs through which to interact would save a lot of wasted space, time and data. I'm not sure why inserting a check-point and resetting the blockchain size to zero would not be possible - can someone please explain this? If the status at the checkpoint were agreed upon by 51% of all nodes then why not?

TylerE 12y ago
You know what would save a lot more space, time, data, cpu power, hardware, and energy? Not using cryptocurrency.
- baddox 12y ago
  Perhaps. And perhaps not using a computer or any electronic devices at all would further "save" those things. I don't see your point.
  - TylerE 12y ago
    Cryptocurrency is astoundingly bad at converting non-renewable natural resources into nothing of absolute value at all. Even (real) strip mining at least produces industrially useful ore. Blockchain technology as implemented in bitcoin is incredibly wasteful. The power used alone is in the tera-watt hours per year. All of those massive farms of custom miners...all to process an astounding ~90k transactions a day. The relative energy cost is astounding, considering that volume could easily be handled by a single server in a traditional architecture.
    - kens 12y ago
      A Raspberry Pi running a Postgres database can handle about 10 times the transaction rate of the Bitcoin network. (Obviously this isn't exactly apples-to-apples, but it gives a rough idea of how totally out of balance the resources consumed vs the transaction rate is.) (10 TPS for TPC-B: http://raspberrypg.org/2013/11/raspberry-pg-server-some-benchmark-results/ http://raspberrypg.org/2013/11/raspberry-pg-server-some-benc...)

anigbrowl 12y ago

You're dead right. A serious problem with the Bitcoin and Dogecoin clients (which I last used some time ago, but which changed relatively little over time) is taht they just installed themselves on the boot drive (if you're running windows) and had little in the way of configuration options. Cut to a few months later when I'm wondering why I'm running out of space on my SSD, only to realize that it's full of gigabytes of blockchain data that is a complete waste of space, with no convient way to put it somewhere else. Could I manually get it installed and running on another large-volume and slower speed drive where this wouldn't be an issue, of course. But why do I want to bother even dealing with something so user-hostile? The average Joe or Jill who isn't a computer nerd is just going to see this thing eating his/her hard drive and get rid of it.

roasbeef 12y ago

Some embedded "internet of things" like device wouldn't be a full-node. Rather it would be a lite-node relying on the SPV[1] model of security. Storing some type of "working set" is totally feasible. Technically you don't even need to keep around all blocks, simply the resultant state that you arrive at after processing all blocks in order. This is called the Unspent Transaction Output set or UTXO set. It's also possible to only keep around XGB of blocks, or the last 500 blocks, etc (a pruned log)[3]. The purpose of keeping around all blocks is to bootstrap the syncing of new nodes to the network, or optionally to full-chain wallets for re-scanning. Before pruned nodes are widely deployed, a system needs to be developed to allow a syncing node to efficiently identify and retrieve the relevant blockchain shards it needs so it can validate everything and arrive at the current state. Also, with BitcoinCore 0.10, syncing takes about 4 hours if you don't have shitty a hardware or internet connection [2]. [1]: https://en.bitcoin.it/wiki/Thin_Client_Security https://en.bitcoin.it/wiki/Thin_Client_Security [2]: https://github.com/bitcoin/bitcoin/pull/4468 https://github.com/bitcoin/bitcoin/pull/4468 [3]: https://github.com/bitcoin/bitcoin/pull/4701 https://github.com/bitcoin/bitcoin/pull/4701

vertex-four 12y ago
> Some embedded "internet of things" like device wouldn't be a full-node. Rather it would be a lite-node relying on the SPV[1] model of security. Well, it's a lot more likely in my eyes that it'd be a thin client operating against a server owned by its owner. Just as safe, and doesn't involve trying to figure out how to allocate enough money on a micro scale to all your individual devices so they don't run out.

IkmoIkmo 12y ago

> There comes a point when it's just no longer practical to store the entire blockchain Why would you have to? The point of decentralisation is not that everyone has a copy of the truth. It's that a sufficient number of nodes have a copy of the truth, such that you can check a number of them and easily verify the veracity of the data. Many blockchain applications run not by querying their own blockchain but rather querying a number of trusted sources via their API. Coinbase, Bitpay, Chain.com, Blockcypher etc all offer APIs to interface with their blockchain nodes and others. It's like saying Facebook will never work because you can't possible store the worlds' photos on everyone's smartphones. You don't have to, Facebook stores it on their server. Similarly we will see, and already see, dedicated servers that store the blockchain, and as long as you have enough independent ones you can maintain decentralisation just as if you were hosting your own node. There's absolutely no need to host the blockchain on small embedded devices any more than any other data, like music or photos we normally host on dedicated servers instead. > but truncating the chain (or only storing some kind of "working set") isn't feasible either. Yes it is. What makes you say it isn't? Pruning the blockchain is something that's currently being worked on. The idea is that if I send you $1 a million times and you send it back $1m times, every day, for the next 1 billion years, we could either store terabytes and terabytes of data of all these transactions, or we could store the end result (= no change occurred) and the last couple weeks of transaction data. The rest can be pruned. Because each block has a hash of all its transactions, and this hash is used to mine the next block (hence a chain of blocks, blockchain), we needn't store each individual transaction, we have the hash to verify the truth of transactions. We might have some universities saving all transactions for posterity & research, and some businesses for data mining, but the vast majorities of nodes that just need to secure the network can indeed, without security issues, not store long-spent outputs. (i.e. bitcoins that have already been spent many times over. e.g. if I give a dollar to you and you give it to the next guy, and he to the next girl, and so on 100 times, at that point there's absolutely no reason to store on every node the fact I gave a dollar to you for the security of the network.) > Not to mention the gargantuan download that's required for that initial setup to get up to date with the latest network transactions. Again, no need to have all redundant transactions. The pruned blockchain would currently be under a gigabyte large. Second, headers-first has already been implemented which made the initial setup much, much faster. Beyond that, we mustn't forget that the average American makes 2 transactions a day. And one such transaction is about 400 bytes or so, in bitcoin. If you compare this to tweets, or an hour of netflix, it's puny. If you count up all the data, it's about $8 to store one day of all of US consumer transactions on a retail harddrive. Imagine the cost of storage for Mastercard for one of their nodes was $8, and instead of just powering a fraction of US retail, they'd literally handle every single transaction by themselves and store them all for $8. Even if you wanted to have thousands of nodes and store the last year of data (and given the velocity of money, you wouldn't have to as again, you can prune data after money has been spent a couple times), that's a puny amount of money to power an entire consumer financial system of the world's biggest economy. And given storage, like CPU and bandwidth, on a 50-60% yoy growth rate (Kryders, Moore's and Nielsen's laws), I think by the time bitcoin becomes important enough to run 100% of US retail (which is likely never, but hypothetically speaking), that $8 will have dropped to below $0.10 easily. I know I'm taking big shortcuts here but this is just to illustrate the economics are extremely favorable today and in the future. I don't know if bitcoin will succeed or fail, but I'm pretty near certain it won't fail because storage/cpu/bandwidth isn't affordable enough for nodes to make it viable to run the network. I and many others have looked into this quite a bit and these aren't big problems. Bitcoin has big obstacles (no gigantic consumer benefits in a world of decent fintech like every OECD country to use bitcoin directly, is one), but storage probably isn't one of em.