6 ms·

Toshi – An Open Source Bitcoin Node for Developers

115 points by adrianmacneil 12y ago

lechuga 12y ago
Been working on this with Adrian for the last 3.5 months. Happy to answer any questions if I can.
- skorgu 12y ago
  Why Ruby? (Nothing against it, just curious)
  - adrianmacneil 12y ago
    We're mostly a ruby shop here - so using what we know is more efficient and allows more engineers to contribute (both internally, and from an open source perspective).
- zmanian 12y ago
  It is great to have more examples of building a indexed fast query representations of the the current ledger state. What did you guys go with as a policy for handling or avoiding orphan blocks...
  - lechuga 12y ago
    Currently we're saving every block we see. And if we find an orphan's parent which is on the main chain we try to connect them both. We aren't doing anything to explicitly avoid them. If you're referring to side chain blocks (which some people refer to as orphan blocks) we'll extend them to be the main chain should they win or leave them forever marked as side chain blocks. We aren't doing anything to explicitly avoid these either.
derefr 12y ago
What advantages does running a Toshi node hold over using one of the many cloud services that allow free public API access to basically the same endpoints?
- pipeep 12y ago
  You can host it yourself.
  - morenoh149 12y ago
    so if using the bitpay api adds too much cost overhead you should self host. Am I right?
- barmstrong 12y ago
  This is another cloud service that allows free public API access, but it's the first one I'm aware of that is also open source. Much better to build a service or business on something you are in control of.
  - patcon 12y ago
    - https://helloblock.io/ https://helloblock.io/ - https://github.com/helloblock/helloblock-app https://github.com/helloblock/helloblock-app :)
    - marcell 12y ago
      Don't forget https://github.com/bitpay/insight https://github.com/bitpay/insight
- deleted 12y ago
  [deleted]
kyledrake 12y ago
I told the Coinbase people to use Postgres for this two years ago, and they treated me like a crazy person for it. Assuming they arrived at this through my advice and not the inevitable catastrophe, glad they finally took my advice to heart. The next problem is that they are still using EventMachine. Still think I'm a crazy person? Oh look, it's like I'm a well-received field expert on this topic or something http://www.slideshare.net/mobile/KyleDrake/hybrid-concurrency-patterns http://www.slideshare.net/mobile/KyleDrake/hybrid-concurrenc... Incase anyone's interested in being a developer at Coinbase, I'll save you some time: they're more interested in pedigree and whiteboard puzzles than science and code. Most disrespectful interview I have ever had in my 13 years of coding, and probably ever will have.
- adrianmacneil 12y ago
  Thanks for your feedback. To be honest, I wasn't here when you gave your advice, but it was clearly well founded :) We haven't run any issues with using EventMachine yet, but will consider other alternatives here if the need arises.
  - kyledrake 12y ago
    You have an issue right now. By running EventMachine, you are blocking Ruby's ability to do concurrent IO using Thread, which means that you've reduced the entire ruby infrastructure (except the EM async code style that nobody programs for anymore) to a single blocking IO thread, dependent on 10-15k+ lines of awful C code that haven't been under active development in over four years, which breaks the premise of ruby's synchronous development methodology, is extremely difficult to debug, and which everyone agrees (including the maintainers) needs to be obsoleted. Indeed, there are a few maintainers on EM, for the sole purpose of preventing it from breaking every single time the ruby team releases an update, because if it broke, it would cause a problem with legacy code in the ruby space. If the EM maintainers walked away, Coinbase would be stuck on an old version of ruby until they themselves patched EM to solve it. EventMachine is not something people should be basing new projects on, especially projects that involve money. Tony Arcieri wrote nio4r (https://github.com/celluloid/nio4r https://github.com/celluloid/nio4r), which solves this problem in a far better way, and then used it to implement cellluloid-io (https://github.com/celluloid/celluloid-io https://github.com/celluloid/celluloid-io), which doesn't choke ruby's ability to do concurrent IO using it's internal threading mechanism. Combined with JRuby, you get even more than concurrent IO here like with MRI, you can also get real threading, which is important in a space where CPUs aren't getting faster, they're getting more parallel. Node.js has the same problem, but unlike Ruby/EM, JavaScript is actually designed to be async, so at least everything is designed to expect it. Because Sinatra (like all non-EM ruby code) uses the threading style for concurrency, as soon as you start running EventMachine, everything will block for a single request, even if it's slow IO bound. You can demonstrate this very easily by putting this into the code: get '/blocking' do sleep(10) end get '/notblocking' do 'ok' end Hit /blocking with curl, and then hit /nonblocking with a second curl command. The second request is blocked until the first one completes. Now try the same thing without eventmachine, and it serves the second request concurrently without a problem. That's what you lose in this exchange, and it's a big thing to lose. I'm using sleep here, but the same blocking will occur for database queries, remote cache hits, HTTPS requests (to get things like, say, the exchange rate), everything involving IO. The only way to get around this problem and still use ruby synchronously is to resort to crazy hacks, including a fairly popular project I created a while ago called sinatra-synchrony (https://github.com/kyledrake/sinatra-synchrony https://github.com/kyledrake/sinatra-synchrony), which wraps async code using fibers. But in order for that to work, every library you're using has to support it, so this path would require a massive refactor of the entire ruby ecosystem, and that's simply not going to happen. It's also incredibly difficult to debug, and there's a lot of bugs with EM - sometimes things would just lock up or segfault, or ruby would spit out an error stack that had nothing to do with the actual exception that occurred. I tried very politely to explain this problem to Coinbase about two years ago during an interview, and their reaction was to ask about my college degree, give me a bunch of weird whiteboard problems that had nothing to do with actual programming, and then show me the door, one day after I helped fix all of their client libraries when they switched SSL CA providers and didn't even realize it, and then provided a pull request to coinbase-ruby providing concurrency support using this very same threading mechanism, on the two-hour flight there (proof: https://github.com/coinbase/coinbase-ruby/pull/6 https://github.com/coinbase/coinbase-ruby/pull/6)
marcell 12y ago
- Can this full node be used for mining, if one is so inclined? - What is the user agent string for this node? (https://getaddr.bitnodes.io/nodes/ https://getaddr.bitnodes.io/nodes/)
- adrianmacneil 12y ago
  It doesn't support any RPC calls so standard mining software won't play well with it yet due to the lack of getwork/getblocktemplate. User agent: Toshi:0.1.0
swandriven 12y ago
You're sooo good! Come on, Toshi!
lappa 12y ago
I guess Coinbase didn't learn their lesson last time they tried using reimplemented consensus code. https://bitcointalk.org/index.php?topic=260595.0 https://bitcointalk.org/index.php?topic=260595.0
- lechuga 12y ago
  That's completely fair. Consensus systems are very difficult to (re-)implement correctly but I would argue there is value in doing so. There is also great risk and we appreciate that. This time we took a slightly different approach. Whenever possible, we directly ported the validation code from bitcoind. Granted, the script runner is the most fragile part and for that we are relying on bitcoin-ruby which does not directly port it from bitcoind. While we are passing Matt Corallo's brilliant test suite, it is not currently recommended that anyone use Toshi in production for the purpose of managing bitcoin. And if you are going to run it in your environment we highly suggest running it behind trusted bitcoind nodes. There are many pieces still missing -- like all of the DoS penalty code. We currently have the demo running in the wild for the very purpose of giving helpful people the opportunity to break it.
  - marcell 12y ago
    > it is not currently recommended that anyone use Toshi in production for the purpose of managing bitcoin Does Coinbase use Toshi in production?
    - lechuga 12y ago
      The only instances of it we having running are the ones listed in the blog post (and just perhaps a Litecoin version that we're trying to get to sync.) They are in no way currently integrated with the web site.
- patio11 12y ago
  I would encourage people to really read this post and try to reason about the implication for the Bitcoin "protocol" the claim that the Bitcoin Core software project is, and must always be, the only usable instantiation of it. (n.b. While this claim is extraordinary I don't believe it is actually wrong.)
kolev 12y ago
Project's homepage: https://toshi.io/ https://toshi.io/

115 points by adrianmacneil 12y ago

lechuga 12y ago

Been working on this with Adrian for the last 3.5 months. Happy to answer any questions if I can.

skorgu 12y ago
Why Ruby? (Nothing against it, just curious)
- adrianmacneil 12y ago
  We're mostly a ruby shop here - so using what we know is more efficient and allows more engineers to contribute (both internally, and from an open source perspective).
zmanian 12y ago
It is great to have more examples of building a indexed fast query representations of the the current ledger state. What did you guys go with as a policy for handling or avoiding orphan blocks...
- lechuga 12y ago
  Currently we're saving every block we see. And if we find an orphan's parent which is on the main chain we try to connect them both. We aren't doing anything to explicitly avoid them. If you're referring to side chain blocks (which some people refer to as orphan blocks) we'll extend them to be the main chain should they win or leave them forever marked as side chain blocks. We aren't doing anything to explicitly avoid these either.

derefr 12y ago

What advantages does running a Toshi node hold over using one of the many cloud services that allow free public API access to basically the same endpoints?

pipeep 12y ago
You can host it yourself.
- morenoh149 12y ago
  so if using the bitpay api adds too much cost overhead you should self host. Am I right?
barmstrong 12y ago
This is another cloud service that allows free public API access, but it's the first one I'm aware of that is also open source. Much better to build a service or business on something you are in control of.
- patcon 12y ago
  - https://helloblock.io/ https://helloblock.io/ - https://github.com/helloblock/helloblock-app https://github.com/helloblock/helloblock-app :)
  - marcell 12y ago
    Don't forget https://github.com/bitpay/insight https://github.com/bitpay/insight
deleted 12y ago
[deleted]

kyledrake 12y ago

I told the Coinbase people to use Postgres for this two years ago, and they treated me like a crazy person for it. Assuming they arrived at this through my advice and not the inevitable catastrophe, glad they finally took my advice to heart. The next problem is that they are still using EventMachine. Still think I'm a crazy person? Oh look, it's like I'm a well-received field expert on this topic or something http://www.slideshare.net/mobile/KyleDrake/hybrid-concurrency-patterns http://www.slideshare.net/mobile/KyleDrake/hybrid-concurrenc... Incase anyone's interested in being a developer at Coinbase, I'll save you some time: they're more interested in pedigree and whiteboard puzzles than science and code. Most disrespectful interview I have ever had in my 13 years of coding, and probably ever will have.

adrianmacneil 12y ago
Thanks for your feedback. To be honest, I wasn't here when you gave your advice, but it was clearly well founded :) We haven't run any issues with using EventMachine yet, but will consider other alternatives here if the need arises.
- kyledrake 12y ago
  You have an issue right now. By running EventMachine, you are blocking Ruby's ability to do concurrent IO using Thread, which means that you've reduced the entire ruby infrastructure (except the EM async code style that nobody programs for anymore) to a single blocking IO thread, dependent on 10-15k+ lines of awful C code that haven't been under active development in over four years, which breaks the premise of ruby's synchronous development methodology, is extremely difficult to debug, and which everyone agrees (including the maintainers) needs to be obsoleted. Indeed, there are a few maintainers on EM, for the sole purpose of preventing it from breaking every single time the ruby team releases an update, because if it broke, it would cause a problem with legacy code in the ruby space. If the EM maintainers walked away, Coinbase would be stuck on an old version of ruby until they themselves patched EM to solve it. EventMachine is not something people should be basing new projects on, especially projects that involve money. Tony Arcieri wrote nio4r (https://github.com/celluloid/nio4r https://github.com/celluloid/nio4r), which solves this problem in a far better way, and then used it to implement cellluloid-io (https://github.com/celluloid/celluloid-io https://github.com/celluloid/celluloid-io), which doesn't choke ruby's ability to do concurrent IO using it's internal threading mechanism. Combined with JRuby, you get even more than concurrent IO here like with MRI, you can also get real threading, which is important in a space where CPUs aren't getting faster, they're getting more parallel. Node.js has the same problem, but unlike Ruby/EM, JavaScript is actually designed to be async, so at least everything is designed to expect it. Because Sinatra (like all non-EM ruby code) uses the threading style for concurrency, as soon as you start running EventMachine, everything will block for a single request, even if it's slow IO bound. You can demonstrate this very easily by putting this into the code: get '/blocking' do sleep(10) end get '/notblocking' do 'ok' end Hit /blocking with curl, and then hit /nonblocking with a second curl command. The second request is blocked until the first one completes. Now try the same thing without eventmachine, and it serves the second request concurrently without a problem. That's what you lose in this exchange, and it's a big thing to lose. I'm using sleep here, but the same blocking will occur for database queries, remote cache hits, HTTPS requests (to get things like, say, the exchange rate), everything involving IO. The only way to get around this problem and still use ruby synchronously is to resort to crazy hacks, including a fairly popular project I created a while ago called sinatra-synchrony (https://github.com/kyledrake/sinatra-synchrony https://github.com/kyledrake/sinatra-synchrony), which wraps async code using fibers. But in order for that to work, every library you're using has to support it, so this path would require a massive refactor of the entire ruby ecosystem, and that's simply not going to happen. It's also incredibly difficult to debug, and there's a lot of bugs with EM - sometimes things would just lock up or segfault, or ruby would spit out an error stack that had nothing to do with the actual exception that occurred. I tried very politely to explain this problem to Coinbase about two years ago during an interview, and their reaction was to ask about my college degree, give me a bunch of weird whiteboard problems that had nothing to do with actual programming, and then show me the door, one day after I helped fix all of their client libraries when they switched SSL CA providers and didn't even realize it, and then provided a pull request to coinbase-ruby providing concurrency support using this very same threading mechanism, on the two-hour flight there (proof: https://github.com/coinbase/coinbase-ruby/pull/6 https://github.com/coinbase/coinbase-ruby/pull/6)

marcell 12y ago

- Can this full node be used for mining, if one is so inclined? - What is the user agent string for this node? (https://getaddr.bitnodes.io/nodes/ https://getaddr.bitnodes.io/nodes/)

adrianmacneil 12y ago
It doesn't support any RPC calls so standard mining software won't play well with it yet due to the lack of getwork/getblocktemplate. User agent: Toshi:0.1.0

swandriven 12y ago

You're sooo good! Come on, Toshi!

lappa 12y ago

I guess Coinbase didn't learn their lesson last time they tried using reimplemented consensus code. https://bitcointalk.org/index.php?topic=260595.0 https://bitcointalk.org/index.php?topic=260595.0

lechuga 12y ago
That's completely fair. Consensus systems are very difficult to (re-)implement correctly but I would argue there is value in doing so. There is also great risk and we appreciate that. This time we took a slightly different approach. Whenever possible, we directly ported the validation code from bitcoind. Granted, the script runner is the most fragile part and for that we are relying on bitcoin-ruby which does not directly port it from bitcoind. While we are passing Matt Corallo's brilliant test suite, it is not currently recommended that anyone use Toshi in production for the purpose of managing bitcoin. And if you are going to run it in your environment we highly suggest running it behind trusted bitcoind nodes. There are many pieces still missing -- like all of the DoS penalty code. We currently have the demo running in the wild for the very purpose of giving helpful people the opportunity to break it.
- marcell 12y ago
  > it is not currently recommended that anyone use Toshi in production for the purpose of managing bitcoin Does Coinbase use Toshi in production?
  - lechuga 12y ago
    The only instances of it we having running are the ones listed in the blog post (and just perhaps a Litecoin version that we're trying to get to sync.) They are in no way currently integrated with the web site.
patio11 12y ago
I would encourage people to really read this post and try to reason about the implication for the Bitcoin "protocol" the claim that the Bitcoin Core software project is, and must always be, the only usable instantiation of it. (n.b. While this claim is extraordinary I don't believe it is actually wrong.)

kolev 12y ago

Project's homepage: https://toshi.io/ https://toshi.io/