7 ms·

A protocol for storing up to 1277 bytes in the Bitcoin blockchain

50 points by williamcotton 11y ago

williamcotton 11y ago
We've been using this protocol underneath another Bitcoin protocol we've been working on called Open Publish. https://github.com/blockai/openpublish https://github.com/blockai/openpublish At some point soon we are going to be transitioning away from Blockcast and instead using a Protocol Buffer, but it has been really nice to be using a loose format based on JSON and Blockcast while we were figuring out all the requirements!
kang 11y ago
This is like finding unused bytes in TCP headers and using them to store data for messaging. This is not what the bitcoin's blockchain is designed for and if you want your data to persist across everyone's computer use something specifically designed for it like, say filecoin or ipfs, but please do not store data on the bitcoin's blockchain and burden everyone.
- Tinyyy 11y ago
  Doesn't that defeat the whole point of it?
- williamcotton 11y ago
  Yup, totally! The second paragraph of the README states: This protocol is intended for use while developing OP_RETURN based protocols. Mature protocols should switch to a custom OP_RETURN method that uses as few transactions as possible to store data. We're using Blockcast underneath Open Publish... we needed more than just the 80 bytes in a single OP_RETURN because we needed a few different content-addressable keys. We use a combination of IPFS, BitTorrent InfoHash, SHA-1, as well as a URI to where the ACTUAL content is posted. Open Publish is also a digital asset protocol so it very much benefits from the "colored coin" properties of the Bitcoin blockchain. But yes, content itself should be stored and distributed using IPFS, BitTorrent, or HTTP, it only makes sense to use the Bitcoin blockchain for things related to payments and ownership, like with the Open Publish protocol.
  - josu 11y ago
    And also: What about polluting the blockchain? We will move this protocol to a Bitcoin sidechain designed specifically for public data as soon as the technology for building sidechains becomes available. In the meantime we've been using this protocol while prototyping other protocols that rely on storing metadata in the Bitcoin blockchain. Woodsy Owl says "Give a Hoot! Don't Pollute!"
- suprfsat 11y ago
  Maybe you should regulate the problem away.
- zekevermillion 11y ago
  Bitcoin is permissionless. Whether you, or Gavin, or Nick Szabo, thinks it should be used for a particular application is not relevant, if it in fact works for that application. Maybe your use of bitcoin as a value transfer protocol is burdening me by driving up the costs of storing objects for multiplayer games! Altcoins are often not suitable because there is not a well-maintained software implementation of the protocol for a particular alt, or because there is insufficient mining on that chain to foster adequate security. As it becomes more expensive to send tx on bitcoin, of course, there are opportunities for alts to fill certain niches. Just as there are opportunities for people to find ways like sidechains, or other hacks, to get more out of bitcoin itself.
  - ryanlol 11y ago
    Until all of your data gets pruned from the blockchain in an update...
    - zekevermillion 11y ago
      Well, that is also fair game...
    - adrianmacneil 11y ago
      Pruning is/will be a client feature, so your data would never be pruned out of existence, it's simply up to clients whether or not they want to keep it around.
      - ryanlol 11y ago
        What's not a client feature? It'll be pretty annoying when you've only got a couple of custom clients hosting your version of the 10TB blockchain.
        adrianmacneil 11y ago
        Well, you can't prune the blockchain until you have fully downloaded and verified it anyway. So if no clients are hosting the full, complete blockchain then you won't be able to sync at all, which will be bad for everyone. You could theoretically get around this by starting sync from a known (hard coded) recent block + UXTO set, but at that point you are losing _all_ transaction history, not just your OP_RETURNs. So, I don't think OP_RETURN data added to the blockchain today will be going anywhere fast.
- arihant 11y ago
  That's shortsighted. Sometimes such practices eliminate the need for additional protocols for related tasks. Maybe instruction slips could be passed on here? This kind of approach can be very helpful in the wild. For delivery of SMS, for example.
  - ryanlol 11y ago
    How so? Blockchains would probably be the worst possible way to deliver SMS from a resource consumption point of view... And from a security POV it'd just be silly.
indutny 11y ago
Of course, no one remembers: https://github.com/indutny/bthread https://github.com/indutny/bthread
- jonpaul 11y ago
  Your statement reads as if you're bothered by the lack of usage / credit towards `bthread`? (Sincere question...) You write excellent code, most notably `elliptic`; your work on Node.js/io.js is much appreciated! I think the problem is that you don't prioritize documentation. The reality is that consumers of libraries need to know two things: 1) how said library can help save time / be a hero, etc 2) how to use said library. Keep writing the awesome code that you do. Next time, maybe spend bit more time on documentation, and I think you'll see more traction :)
spikexxx 11y ago
If only they could have found room for 60 more bytes...
aakilfernandes 11y ago
I would have loved this a few months ago. I've since switched to Ethereum and have been really happy with it. If you're trying to cram lots of data into OP_RETURNs, you should try looking at other blockchains.
- bduerst 11y ago
  How does Ethereum store data in a blockchain and what's the size?
  - NH_2 11y ago
    I'm curious about this too. Also, what's the retrieval rate of the data? How much of it can you query at once? Is it different/faster on Ethereum?
    - aakilfernandes 11y ago
      I don't think there's a meaningful difference between the two, they're both essentially instant. Bitcoin doesnt make the data queryable in any way. You need to know the transactions where the data is stored ahead of time. With Ethereum, you can create a "contract" that emits "events" which automatically get added to a queryable database.
      - williamcotton 11y ago
        You're right, Bitcoin doesn't natively have a way to query a database, but here's an example of a how to read Blockcast transactions from the blockchain: https://github.com/blockai/blockcast-state-engine/blob/master/test.js#L32 https://github.com/blockai/blockcast-state-engine/blob/maste... The next step would be storing the results in a SQL database as we do with our production servers that maintain a queryable database of Open Publish registrations and transactions (Open Publish uses Blockcast for storing metadata on digital media assets). https://github.com/blockai/openpublish-state https://github.com/blockai/openpublish-state This is definitely a more piecemeal approach to creating smart contract applications than with Ethereum's one-stop-shop, but there are benefits to building on top of Bitcoin, especially for Open Publish, which uses native Bitcoin transactions for payments and royalties.
ck2 11y ago
Bitcoin is still around? March 2013 it was worth less than $100 Currently it has dropped to $200
- grandalf 11y ago
  the price is not the measure of bitcoin's success. It suffered/benefitted from a highly speculative market.. but currencies are generally most effective when the price is relatively stable, which it has been lately.
- adrianmacneil 11y ago
  At the risk of feeding your trolling... Bitcoin the currency isn't seeing a huge amount of adoption (or even speculation) lately, leading to a fairly stable price. Bitcoin the protocol (a permissionless, decentralized database, or as the media calls it: "blockchain technology") is seeing a huge amount of innovation and a wide variety of applications coming out. I suggest you read up on it.
- brighton36 11y ago
  Bitcoin's price has been going down ever since it was $1. Similary, there's yet to be any competitor that's worth a damn - so it seems here to stay around for at least a bit longer.
jkot 11y ago
Bitcoin transaction logs already have over 60GB. If people start using it as a storage, its size will skyrocket and it will be unsustainable.
- aw3c2 11y ago
  No, Bitcoin is the solution to all financial transaction woes, totally decentralised and the way for a better world!
- IkmoIkmo 11y ago
  There's a price on storage. If you want to spend $0.05 for a few hundred bytes, you'll run out of money very quickly if you want to actually use it for typical file storage (e.g. your photo album or music library). But it's fine if you want to store or record for posterity something valuable, e.g. a $10m loan transaction [0] [0] http://www.wired.com/2015/10/hedge-fund-borrows-10m-in-stock-via-the-bitcoin-blockchain/ http://www.wired.com/2015/10/hedge-fund-borrows-10m-in-stock... There's been a lot of work going into a market for transaction fees based on a supply&demand model that'd make spam not just costly, but exponentially more costly if you want to scale it. It's not perfect but it's looking pretty sensible atm.
- ceejayoz 11y ago
  > If people start using it as a storage, its size will skyrocket and it will be unsustainable. If a misguided - not even malicious! - use case can make it unusable, maybe it's not destined to live all that long.
xirdstl 11y ago
Soon: cloud backup via the Bitcoin blockchain
asciimo 11y ago
Namecoin (http://namecoin.org http://namecoin.org) gives you 520 bytes out-of-the-box. It was designed to store arbitrary data on a blockchain. Being merge-mined with Bitcoin, it enjoys the highest security against a 51% attack of any altcoin.
- williamcotton 11y ago
  Namecoin doesn't match all specific use-cases as documents expire after ~200 days. It also lacks the infrastructure of exchanges, APIs, tools, and software that support Bitcoin. Ultimately we feel that Bitcoin sidechains are a better approach to crypto-currencies than having competing alt-coins. https://github.com/blockai/blockcast#what-about-an-alternative-currency-like-namecoin https://github.com/blockai/blockcast#what-about-an-alternati... Still waiting on those sidechains...

50 points by williamcotton 11y ago

We've been using this protocol underneath another Bitcoin protocol we've been working on called Open Publish. https://github.com/blockai/openpublish https://github.com/blockai/openpublish At some point soon we are going to be transitioning away from Blockcast and instead using a Protocol Buffer, but it has been really nice to be using a loose format based on JSON and Blockcast while we were figuring out all the requirements!

kang 11y ago

This is like finding unused bytes in TCP headers and using them to store data for messaging. This is not what the bitcoin's blockchain is designed for and if you want your data to persist across everyone's computer use something specifically designed for it like, say filecoin or ipfs, but please do not store data on the bitcoin's blockchain and burden everyone.

Tinyyy 11y ago
Doesn't that defeat the whole point of it?
williamcotton 11y ago
Yup, totally! The second paragraph of the README states: This protocol is intended for use while developing OP_RETURN based protocols. Mature protocols should switch to a custom OP_RETURN method that uses as few transactions as possible to store data. We're using Blockcast underneath Open Publish... we needed more than just the 80 bytes in a single OP_RETURN because we needed a few different content-addressable keys. We use a combination of IPFS, BitTorrent InfoHash, SHA-1, as well as a URI to where the ACTUAL content is posted. Open Publish is also a digital asset protocol so it very much benefits from the "colored coin" properties of the Bitcoin blockchain. But yes, content itself should be stored and distributed using IPFS, BitTorrent, or HTTP, it only makes sense to use the Bitcoin blockchain for things related to payments and ownership, like with the Open Publish protocol.
- josu 11y ago
  And also: What about polluting the blockchain? We will move this protocol to a Bitcoin sidechain designed specifically for public data as soon as the technology for building sidechains becomes available. In the meantime we've been using this protocol while prototyping other protocols that rely on storing metadata in the Bitcoin blockchain. Woodsy Owl says "Give a Hoot! Don't Pollute!"
suprfsat 11y ago
Maybe you should regulate the problem away.
zekevermillion 11y ago
Bitcoin is permissionless. Whether you, or Gavin, or Nick Szabo, thinks it should be used for a particular application is not relevant, if it in fact works for that application. Maybe your use of bitcoin as a value transfer protocol is burdening me by driving up the costs of storing objects for multiplayer games! Altcoins are often not suitable because there is not a well-maintained software implementation of the protocol for a particular alt, or because there is insufficient mining on that chain to foster adequate security. As it becomes more expensive to send tx on bitcoin, of course, there are opportunities for alts to fill certain niches. Just as there are opportunities for people to find ways like sidechains, or other hacks, to get more out of bitcoin itself.
- ryanlol 11y ago
  Until all of your data gets pruned from the blockchain in an update...
  - zekevermillion 11y ago
    Well, that is also fair game...
  - adrianmacneil 11y ago
    Pruning is/will be a client feature, so your data would never be pruned out of existence, it's simply up to clients whether or not they want to keep it around.
    - ryanlol 11y ago
      What's not a client feature? It'll be pretty annoying when you've only got a couple of custom clients hosting your version of the 10TB blockchain.
      - adrianmacneil 11y ago
        Well, you can't prune the blockchain until you have fully downloaded and verified it anyway. So if no clients are hosting the full, complete blockchain then you won't be able to sync at all, which will be bad for everyone. You could theoretically get around this by starting sync from a known (hard coded) recent block + UXTO set, but at that point you are losing _all_ transaction history, not just your OP_RETURNs. So, I don't think OP_RETURN data added to the blockchain today will be going anywhere fast.
arihant 11y ago
That's shortsighted. Sometimes such practices eliminate the need for additional protocols for related tasks. Maybe instruction slips could be passed on here? This kind of approach can be very helpful in the wild. For delivery of SMS, for example.
- ryanlol 11y ago
  How so? Blockchains would probably be the worst possible way to deliver SMS from a resource consumption point of view... And from a security POV it'd just be silly.

indutny 11y ago

Of course, no one remembers: https://github.com/indutny/bthread https://github.com/indutny/bthread

jonpaul 11y ago
Your statement reads as if you're bothered by the lack of usage / credit towards `bthread`? (Sincere question...) You write excellent code, most notably `elliptic`; your work on Node.js/io.js is much appreciated! I think the problem is that you don't prioritize documentation. The reality is that consumers of libraries need to know two things: 1) how said library can help save time / be a hero, etc 2) how to use said library. Keep writing the awesome code that you do. Next time, maybe spend bit more time on documentation, and I think you'll see more traction :)

spikexxx 11y ago

If only they could have found room for 60 more bytes...

aakilfernandes 11y ago

I would have loved this a few months ago. I've since switched to Ethereum and have been really happy with it. If you're trying to cram lots of data into OP_RETURNs, you should try looking at other blockchains.

bduerst 11y ago
How does Ethereum store data in a blockchain and what's the size?
- NH_2 11y ago
  I'm curious about this too. Also, what's the retrieval rate of the data? How much of it can you query at once? Is it different/faster on Ethereum?
  - aakilfernandes 11y ago
    I don't think there's a meaningful difference between the two, they're both essentially instant. Bitcoin doesnt make the data queryable in any way. You need to know the transactions where the data is stored ahead of time. With Ethereum, you can create a "contract" that emits "events" which automatically get added to a queryable database.
    - williamcotton 11y ago
      You're right, Bitcoin doesn't natively have a way to query a database, but here's an example of a how to read Blockcast transactions from the blockchain: https://github.com/blockai/blockcast-state-engine/blob/master/test.js#L32 https://github.com/blockai/blockcast-state-engine/blob/maste... The next step would be storing the results in a SQL database as we do with our production servers that maintain a queryable database of Open Publish registrations and transactions (Open Publish uses Blockcast for storing metadata on digital media assets). https://github.com/blockai/openpublish-state https://github.com/blockai/openpublish-state This is definitely a more piecemeal approach to creating smart contract applications than with Ethereum's one-stop-shop, but there are benefits to building on top of Bitcoin, especially for Open Publish, which uses native Bitcoin transactions for payments and royalties.

ck2 11y ago

Bitcoin is still around? March 2013 it was worth less than $100 Currently it has dropped to $200

grandalf 11y ago
the price is not the measure of bitcoin's success. It suffered/benefitted from a highly speculative market.. but currencies are generally most effective when the price is relatively stable, which it has been lately.
adrianmacneil 11y ago
At the risk of feeding your trolling... Bitcoin the currency isn't seeing a huge amount of adoption (or even speculation) lately, leading to a fairly stable price. Bitcoin the protocol (a permissionless, decentralized database, or as the media calls it: "blockchain technology") is seeing a huge amount of innovation and a wide variety of applications coming out. I suggest you read up on it.
brighton36 11y ago
Bitcoin's price has been going down ever since it was $1. Similary, there's yet to be any competitor that's worth a damn - so it seems here to stay around for at least a bit longer.

jkot 11y ago

Bitcoin transaction logs already have over 60GB. If people start using it as a storage, its size will skyrocket and it will be unsustainable.

aw3c2 11y ago
No, Bitcoin is the solution to all financial transaction woes, totally decentralised and the way for a better world!
IkmoIkmo 11y ago
There's a price on storage. If you want to spend $0.05 for a few hundred bytes, you'll run out of money very quickly if you want to actually use it for typical file storage (e.g. your photo album or music library). But it's fine if you want to store or record for posterity something valuable, e.g. a $10m loan transaction [0] [0] http://www.wired.com/2015/10/hedge-fund-borrows-10m-in-stock-via-the-bitcoin-blockchain/ http://www.wired.com/2015/10/hedge-fund-borrows-10m-in-stock... There's been a lot of work going into a market for transaction fees based on a supply&demand model that'd make spam not just costly, but exponentially more costly if you want to scale it. It's not perfect but it's looking pretty sensible atm.
ceejayoz 11y ago
> If people start using it as a storage, its size will skyrocket and it will be unsustainable. If a misguided - not even malicious! - use case can make it unusable, maybe it's not destined to live all that long.

xirdstl 11y ago

Soon: cloud backup via the Bitcoin blockchain

asciimo 11y ago

Namecoin (http://namecoin.org http://namecoin.org) gives you 520 bytes out-of-the-box. It was designed to store arbitrary data on a blockchain. Being merge-mined with Bitcoin, it enjoys the highest security against a 51% attack of any altcoin.

williamcotton 11y ago
Namecoin doesn't match all specific use-cases as documents expire after ~200 days. It also lacks the infrastructure of exchanges, APIs, tools, and software that support Bitcoin. Ultimately we feel that Bitcoin sidechains are a better approach to crypto-currencies than having competing alt-coins. https://github.com/blockai/blockcast#what-about-an-alternative-currency-like-namecoin https://github.com/blockai/blockcast#what-about-an-alternati... Still waiting on those sidechains...