9 ms·

I told the Coinbase people to use Postgres for this two years ago, and they treated me like a crazy person for it. Assuming they arrived at this through my advi

I told the Coinbase people to use Postgres for this two years ago, and they treated me like a crazy person for it. Assuming they arrived at this through my advice and not the inevitable catastrophe, glad they finally took my advice to heart. The next problem is that they are still using EventMachine. Still think I'm a crazy person? Oh look, it's like I'm a well-received field expert on this topic or something http://www.slideshare.net/mobile/KyleDrake/hybrid-concurrency-patterns http://www.slideshare.net/mobile/KyleDrake/hybrid-concurrenc... Incase anyone's interested in being a developer at Coinbase, I'll save you some time: they're more interested in pedigree and whiteboard puzzles than science and code. Most disrespectful interview I have ever had in my 13 years of coding, and probably ever will have.

adrianmacneil 12y ago
Thanks for your feedback. To be honest, I wasn't here when you gave your advice, but it was clearly well founded :) We haven't run any issues with using EventMachine yet, but will consider other alternatives here if the need arises.
- kyledrake 12y ago
  You have an issue right now. By running EventMachine, you are blocking Ruby's ability to do concurrent IO using Thread, which means that you've reduced the entire ruby infrastructure (except the EM async code style that nobody programs for anymore) to a single blocking IO thread, dependent on 10-15k+ lines of awful C code that haven't been under active development in over four years, which breaks the premise of ruby's synchronous development methodology, is extremely difficult to debug, and which everyone agrees (including the maintainers) needs to be obsoleted. Indeed, there are a few maintainers on EM, for the sole purpose of preventing it from breaking every single time the ruby team releases an update, because if it broke, it would cause a problem with legacy code in the ruby space. If the EM maintainers walked away, Coinbase would be stuck on an old version of ruby until they themselves patched EM to solve it. EventMachine is not something people should be basing new projects on, especially projects that involve money. Tony Arcieri wrote nio4r (https://github.com/celluloid/nio4r https://github.com/celluloid/nio4r), which solves this problem in a far better way, and then used it to implement cellluloid-io (https://github.com/celluloid/celluloid-io https://github.com/celluloid/celluloid-io), which doesn't choke ruby's ability to do concurrent IO using it's internal threading mechanism. Combined with JRuby, you get even more than concurrent IO here like with MRI, you can also get real threading, which is important in a space where CPUs aren't getting faster, they're getting more parallel. Node.js has the same problem, but unlike Ruby/EM, JavaScript is actually designed to be async, so at least everything is designed to expect it. Because Sinatra (like all non-EM ruby code) uses the threading style for concurrency, as soon as you start running EventMachine, everything will block for a single request, even if it's slow IO bound. You can demonstrate this very easily by putting this into the code: get '/blocking' do sleep(10) end get '/notblocking' do 'ok' end Hit /blocking with curl, and then hit /nonblocking with a second curl command. The second request is blocked until the first one completes. Now try the same thing without eventmachine, and it serves the second request concurrently without a problem. That's what you lose in this exchange, and it's a big thing to lose. I'm using sleep here, but the same blocking will occur for database queries, remote cache hits, HTTPS requests (to get things like, say, the exchange rate), everything involving IO. The only way to get around this problem and still use ruby synchronously is to resort to crazy hacks, including a fairly popular project I created a while ago called sinatra-synchrony (https://github.com/kyledrake/sinatra-synchrony https://github.com/kyledrake/sinatra-synchrony), which wraps async code using fibers. But in order for that to work, every library you're using has to support it, so this path would require a massive refactor of the entire ruby ecosystem, and that's simply not going to happen. It's also incredibly difficult to debug, and there's a lot of bugs with EM - sometimes things would just lock up or segfault, or ruby would spit out an error stack that had nothing to do with the actual exception that occurred. I tried very politely to explain this problem to Coinbase about two years ago during an interview, and their reaction was to ask about my college degree, give me a bunch of weird whiteboard problems that had nothing to do with actual programming, and then show me the door, one day after I helped fix all of their client libraries when they switched SSL CA providers and didn't even realize it, and then provided a pull request to coinbase-ruby providing concurrency support using this very same threading mechanism, on the two-hour flight there (proof: https://github.com/coinbase/coinbase-ruby/pull/6 https://github.com/coinbase/coinbase-ruby/pull/6)
  - sillysaurus3 12y ago
    I tried very politely to explain this problem to Coinbase about two years ago during an interview, and their reaction was to ask about my college degree, give me a bunch of weird whiteboard problems that had nothing to do with actual programming, and then show me the door, one day after I helped fix all of their client libraries when they switched SSL CA providers and didn't even realize it, and then provided a pull request to coinbase-ruby providing concurrency support using this very same threading mechanism, on the two-hour flight there (proof: https://github.com/coinbase/coinbase-ruby/pull/6 https://github.com/coinbase/coinbase-ruby/pull/6). Are you being fair to Coinbase? There are two sides to every story. But if this is true at face value, then it's disappointing to hear that a YC startup would care at all about a degree or about whiteboard problems, of all things. If a company cares about what a programming candidate can do, then caring about a degree or whiteboard problems is caring about the wrong thing. Thought experiment: How much better or worse would YC's results be if they had candidates solve whiteboard problems during the interview? (Admittedly, YC cares about degrees though.)
    - wslh 12y ago
      > If a company cares about what a programming candidate can do, then caring about a degree or whiteboard problems is caring about the wrong thing. Yes, it is like being interviewed for making a time machine, you show a working time machine and they continue to ask you questions about physics...
    - xcubed 12y ago
      Not surprising to me about Coinbase. A friend interviewed there and decided not to continue with the process as he was so turned off by their team. In particular he made a suggestion for improvement and they got very defensive.
  - acoyfellow 12y ago
    I sure hope you landed a better gig than what they could have offered you. Thanks for the free education-- I'd be kicking myself if I was the dude who didn't hire you!
  - 12y ago
jonpaul 12y ago
For those that don't know, Kyle was responsible for helping to resurrect bitcoinjs-lib (https://github.com/orgs/bitcoinjs/people https://github.com/orgs/bitcoinjs/people) and developing one of the first self-hosted Bitcoin (HTML) wallets - Coinpunk (https://github.com/kyledrake/coinpunk https://github.com/kyledrake/coinpunk), so he's clearly a talented developer.
- bitcoiner2010 12y ago
  Web-wallets should not be trusted, no matter what. See: mybitcoin.com, instawallet.com etc. -> https://en.bitcoin.it/wiki/EWallet https://en.bitcoin.it/wiki/EWallet
  - tomlongson 12y ago
    BitGo is a good web wallet.
    - tomlongson 12y ago
      Not sure why I'm being downvoted, but it's one of the few multisig web wallets available. It's not the same as other sites.
      - johnhenry 12y ago
        I think you've been downvoted because that comment has little to do with the topic at hand and seems like a shameless promotion of BitGo. Alternatively, you could create a separate "Show HN : BitGO -- one of the few multisig web wallets available" thread and that would be totally acceptable.
deleted 12y ago
[deleted]
- beaner 12y ago
  If multisig applies to a future product and not the one that was released then it sounds like he was being factual with you. Not sure what you expect - for him to be non-factual? As for high risk cancellations, do you expect them to approve transactions that they expect will actively damage them? Of course the cancellation serves Coinbase - nothing else is reasonable. (It's also funny how you only hear about complaints of that when the price is going up - when the price is declining and they cancel due to high risk, nobody complains because Coinbase ended up saving them money.) It's funny how everybody who didn't get a job there comes out of the woodwork to post critical comments on something completely unrelated.
  - bitcoiner2010 12y ago
    >As for high risk cancellations, do you expect them to approve transactions that they expect will actively damage them? The point is to make clear is that Coinbase is not an exchange like Bitstamp et al. Coinbase will cancel your transaction /only when it benefits Coinbase/. Buy bitcoin and then USD/BTC crashes? Coinbase is Johnny on the spot to complete your transaction. But, when the opposite happens? Sorry: You are High-Risk. Regardless. Olaf did not know about Vault including multi-sig. This is /after Coinbase announced publicly that Vault would include multi-sig in the near future./ To me that signifies a disconnect between those in the company.
    - beaner 12y ago
      That doesn't really signify anything. If you made it sound like multisig was in the current version, and he rightfully didn't know anything about multisig in the current version - because, in fact, it was not the case - then you are wrong. Obviously no one can validate your statements but you, so we've got a one-sided story from a 1-hour old account from a person who didn't get hired. Clearly this account will represent the pinacle of objectivity. Also, since when does every employee know about all details of all products? If "olaf" is "head of risk" then he's probably not an eng and in the know of all that. At Dell a couple months ago people were like, "Oh, we make that tablet? I didn't even know." And that's hardware. Everyone knows Coinbase isn't an exchange because there are not limit orders and an open book. This is dummy-obvious and they don't try to obfuscate it. > Buy bitcoin and then USD/BTC crashes? Coinbase is Johnny on the spot to complete your transaction. This is factually false. I'm not sure if you read my comment. Coinbase has an equal amount of high-risk cancelations when the price is falling. You never hear about them because people don't have a reason to complain about it. I mean if you feel jilted because you didn't land a spot, go be grumpy about it if you want. Not sure why they would hire someone for a support role who is coming in with a grudge about their cancelation policy. How would such a person be able to be trusted helping others with that issue? Bitterness doesn't justify FUD.

adrianmacneil 12y ago

Thanks for your feedback. To be honest, I wasn't here when you gave your advice, but it was clearly well founded :) We haven't run any issues with using EventMachine yet, but will consider other alternatives here if the need arises.

kyledrake 12y ago
You have an issue right now. By running EventMachine, you are blocking Ruby's ability to do concurrent IO using Thread, which means that you've reduced the entire ruby infrastructure (except the EM async code style that nobody programs for anymore) to a single blocking IO thread, dependent on 10-15k+ lines of awful C code that haven't been under active development in over four years, which breaks the premise of ruby's synchronous development methodology, is extremely difficult to debug, and which everyone agrees (including the maintainers) needs to be obsoleted. Indeed, there are a few maintainers on EM, for the sole purpose of preventing it from breaking every single time the ruby team releases an update, because if it broke, it would cause a problem with legacy code in the ruby space. If the EM maintainers walked away, Coinbase would be stuck on an old version of ruby until they themselves patched EM to solve it. EventMachine is not something people should be basing new projects on, especially projects that involve money. Tony Arcieri wrote nio4r (https://github.com/celluloid/nio4r https://github.com/celluloid/nio4r), which solves this problem in a far better way, and then used it to implement cellluloid-io (https://github.com/celluloid/celluloid-io https://github.com/celluloid/celluloid-io), which doesn't choke ruby's ability to do concurrent IO using it's internal threading mechanism. Combined with JRuby, you get even more than concurrent IO here like with MRI, you can also get real threading, which is important in a space where CPUs aren't getting faster, they're getting more parallel. Node.js has the same problem, but unlike Ruby/EM, JavaScript is actually designed to be async, so at least everything is designed to expect it. Because Sinatra (like all non-EM ruby code) uses the threading style for concurrency, as soon as you start running EventMachine, everything will block for a single request, even if it's slow IO bound. You can demonstrate this very easily by putting this into the code: get '/blocking' do sleep(10) end get '/notblocking' do 'ok' end Hit /blocking with curl, and then hit /nonblocking with a second curl command. The second request is blocked until the first one completes. Now try the same thing without eventmachine, and it serves the second request concurrently without a problem. That's what you lose in this exchange, and it's a big thing to lose. I'm using sleep here, but the same blocking will occur for database queries, remote cache hits, HTTPS requests (to get things like, say, the exchange rate), everything involving IO. The only way to get around this problem and still use ruby synchronously is to resort to crazy hacks, including a fairly popular project I created a while ago called sinatra-synchrony (https://github.com/kyledrake/sinatra-synchrony https://github.com/kyledrake/sinatra-synchrony), which wraps async code using fibers. But in order for that to work, every library you're using has to support it, so this path would require a massive refactor of the entire ruby ecosystem, and that's simply not going to happen. It's also incredibly difficult to debug, and there's a lot of bugs with EM - sometimes things would just lock up or segfault, or ruby would spit out an error stack that had nothing to do with the actual exception that occurred. I tried very politely to explain this problem to Coinbase about two years ago during an interview, and their reaction was to ask about my college degree, give me a bunch of weird whiteboard problems that had nothing to do with actual programming, and then show me the door, one day after I helped fix all of their client libraries when they switched SSL CA providers and didn't even realize it, and then provided a pull request to coinbase-ruby providing concurrency support using this very same threading mechanism, on the two-hour flight there (proof: https://github.com/coinbase/coinbase-ruby/pull/6 https://github.com/coinbase/coinbase-ruby/pull/6)
- sillysaurus3 12y ago
  I tried very politely to explain this problem to Coinbase about two years ago during an interview, and their reaction was to ask about my college degree, give me a bunch of weird whiteboard problems that had nothing to do with actual programming, and then show me the door, one day after I helped fix all of their client libraries when they switched SSL CA providers and didn't even realize it, and then provided a pull request to coinbase-ruby providing concurrency support using this very same threading mechanism, on the two-hour flight there (proof: https://github.com/coinbase/coinbase-ruby/pull/6 https://github.com/coinbase/coinbase-ruby/pull/6). Are you being fair to Coinbase? There are two sides to every story. But if this is true at face value, then it's disappointing to hear that a YC startup would care at all about a degree or about whiteboard problems, of all things. If a company cares about what a programming candidate can do, then caring about a degree or whiteboard problems is caring about the wrong thing. Thought experiment: How much better or worse would YC's results be if they had candidates solve whiteboard problems during the interview? (Admittedly, YC cares about degrees though.)
  - wslh 12y ago
    > If a company cares about what a programming candidate can do, then caring about a degree or whiteboard problems is caring about the wrong thing. Yes, it is like being interviewed for making a time machine, you show a working time machine and they continue to ask you questions about physics...
  - xcubed 12y ago
    Not surprising to me about Coinbase. A friend interviewed there and decided not to continue with the process as he was so turned off by their team. In particular he made a suggestion for improvement and they got very defensive.
- acoyfellow 12y ago
  I sure hope you landed a better gig than what they could have offered you. Thanks for the free education-- I'd be kicking myself if I was the dude who didn't hire you!
- 12y ago

jonpaul 12y ago

For those that don't know, Kyle was responsible for helping to resurrect bitcoinjs-lib (https://github.com/orgs/bitcoinjs/people https://github.com/orgs/bitcoinjs/people) and developing one of the first self-hosted Bitcoin (HTML) wallets - Coinpunk (https://github.com/kyledrake/coinpunk https://github.com/kyledrake/coinpunk), so he's clearly a talented developer.

bitcoiner2010 12y ago
Web-wallets should not be trusted, no matter what. See: mybitcoin.com, instawallet.com etc. -> https://en.bitcoin.it/wiki/EWallet https://en.bitcoin.it/wiki/EWallet
- tomlongson 12y ago
  BitGo is a good web wallet.
  - tomlongson 12y ago
    Not sure why I'm being downvoted, but it's one of the few multisig web wallets available. It's not the same as other sites.
    - johnhenry 12y ago
      I think you've been downvoted because that comment has little to do with the topic at hand and seems like a shameless promotion of BitGo. Alternatively, you could create a separate "Show HN : BitGO -- one of the few multisig web wallets available" thread and that would be totally acceptable.

deleted 12y ago

[deleted]

beaner 12y ago
If multisig applies to a future product and not the one that was released then it sounds like he was being factual with you. Not sure what you expect - for him to be non-factual? As for high risk cancellations, do you expect them to approve transactions that they expect will actively damage them? Of course the cancellation serves Coinbase - nothing else is reasonable. (It's also funny how you only hear about complaints of that when the price is going up - when the price is declining and they cancel due to high risk, nobody complains because Coinbase ended up saving them money.) It's funny how everybody who didn't get a job there comes out of the woodwork to post critical comments on something completely unrelated.
- bitcoiner2010 12y ago
  >As for high risk cancellations, do you expect them to approve transactions that they expect will actively damage them? The point is to make clear is that Coinbase is not an exchange like Bitstamp et al. Coinbase will cancel your transaction /only when it benefits Coinbase/. Buy bitcoin and then USD/BTC crashes? Coinbase is Johnny on the spot to complete your transaction. But, when the opposite happens? Sorry: You are High-Risk. Regardless. Olaf did not know about Vault including multi-sig. This is /after Coinbase announced publicly that Vault would include multi-sig in the near future./ To me that signifies a disconnect between those in the company.
  - beaner 12y ago
    That doesn't really signify anything. If you made it sound like multisig was in the current version, and he rightfully didn't know anything about multisig in the current version - because, in fact, it was not the case - then you are wrong. Obviously no one can validate your statements but you, so we've got a one-sided story from a 1-hour old account from a person who didn't get hired. Clearly this account will represent the pinacle of objectivity. Also, since when does every employee know about all details of all products? If "olaf" is "head of risk" then he's probably not an eng and in the know of all that. At Dell a couple months ago people were like, "Oh, we make that tablet? I didn't even know." And that's hardware. Everyone knows Coinbase isn't an exchange because there are not limit orders and an open book. This is dummy-obvious and they don't try to obfuscate it. > Buy bitcoin and then USD/BTC crashes? Coinbase is Johnny on the spot to complete your transaction. This is factually false. I'm not sure if you read my comment. Coinbase has an equal amount of high-risk cancelations when the price is falling. You never hear about them because people don't have a reason to complain about it. I mean if you feel jilted because you didn't land a spot, go be grumpy about it if you want. Not sure why they would hire someone for a support role who is coming in with a grudge about their cancelation policy. How would such a person be able to be trusted helping others with that issue? Bitterness doesn't justify FUD.