3 ms·

Show HN: I Built 0xFast Stream – 100x Faster Ethereum Block Downloads

13 points by mrjn 4y ago

Getting web3 data out of Ethereum nodes or node services is excruciatingly slow (at least days if not weeks). I had wanted a way to just download the blocks quickly since I started exploring the web3 space. Took me a bunch of trial and errors to get to a way where I can parse all the blocks since genesis within 12 hours. But, instead of moving on to the next thing, I put this service together over the last few days, so everyone can benefit.

tarunmuvvala 4y ago
Curious how you managed compute cost. Are you running a node locally?
- mrjn 4y ago
  Things are cheaper when you put together your own hardware. For example: https://manishrjain.com/building-a-2-5-gbps-load-balancer-server-for-500 https://manishrjain.com/building-a-2-5-gbps-load-balancer-se...
kevinsimper 4y ago
Are you exposing full node or archive node? Because Archive node is several terabytes :)
- mrjn 4y ago
  It doesn't have the states yet. So, equivalent of full nodes -- block, txns, receipts, logs (no traces, states). But, could be extended to include those, if needed.
high_byte 4y ago
wow can't believe just yesterday I thought how cool it would be to have that - and today you made it! :D downloading... btw did you verify (PoW) blocks using the nonce? just to be on the safe side :)
- high_byte 4y ago
  also if you could use binary encoding rather than json you could probably cut the size by half. I already started download, but perhaps it's worth noting in your website to use "--compressed" flag like so: curl --compressed -H "Accept-Encoding: gzip" https://stream.0xfast.com/beta/eth-mainnet/1-latest https://stream.0xfast.com/beta/eth-mainnet/1-latest
  - mrjn 4y ago
    Oh, --compressed is is a great point, thanks. Updating the blog. I wanted to stick to JSON -- it's just the easiest for everyone to work with. And it compresses well, so all good.
- mrjn 4y ago
  Glad it's useful! :-) > btw did you verify (PoW) blocks using the nonce? I've validated the data and checksummed while exporting from Geth. But, perhaps can validate data when serving it as well. Any pointers / articles on the best practices here?

13 points by mrjn 4y ago

tarunmuvvala 4y ago

Curious how you managed compute cost. Are you running a node locally?

mrjn 4y ago
Things are cheaper when you put together your own hardware. For example: https://manishrjain.com/building-a-2-5-gbps-load-balancer-server-for-500 https://manishrjain.com/building-a-2-5-gbps-load-balancer-se...

kevinsimper 4y ago

Are you exposing full node or archive node? Because Archive node is several terabytes :)

mrjn 4y ago
It doesn't have the states yet. So, equivalent of full nodes -- block, txns, receipts, logs (no traces, states). But, could be extended to include those, if needed.

high_byte 4y ago

wow can't believe just yesterday I thought how cool it would be to have that - and today you made it! :D downloading... btw did you verify (PoW) blocks using the nonce? just to be on the safe side :)

high_byte 4y ago
also if you could use binary encoding rather than json you could probably cut the size by half. I already started download, but perhaps it's worth noting in your website to use "--compressed" flag like so: curl --compressed -H "Accept-Encoding: gzip" https://stream.0xfast.com/beta/eth-mainnet/1-latest https://stream.0xfast.com/beta/eth-mainnet/1-latest
- mrjn 4y ago
  Oh, --compressed is is a great point, thanks. Updating the blog. I wanted to stick to JSON -- it's just the easiest for everyone to work with. And it compresses well, so all good.
mrjn 4y ago
Glad it's useful! :-) > btw did you verify (PoW) blocks using the nonce? I've validated the data and checksummed while exporting from Geth. But, perhaps can validate data when serving it as well. Any pointers / articles on the best practices here?