2 ms·

> Ethereum [..] is getting dangerously close to this point. Not any more. Ethereum (PoW) sync time and storage size have improved a lot since you measured the

> Ethereum [..] is getting dangerously close to this point. Not any more. Ethereum (PoW) sync time and storage size have improved a lot since you measured them. (I've worked on algorithms to improve both these areas so I know them well.) Since we're talking about scaling issues, rather than what's readily available in the well known implementations, I'll describe the best implementations that I know about. Sync time: Syncing a full history archive node has been done in less than 1 day. It's reduced a lot from the multi-month syncs before, even though the chain is larger now. Syncing just the state for recent history is commonly done in under 6 hours. Both can be reduced further. Storage size: The entire full history archive state fits in about 400 GiB. With all blocks but just recent history (i.e. a typical node) it's about 200 GiB, and when reduced to 1 year of blocks (EIP-4444) the total storage required for an Ethereum node fits into about 100 GiB. Storage I/O: I don't have hard figures for this, but both the sync algorithms and storage compression techniques reduce IOPS load significantly. Storage of full history in particular has been hugely improved over the 9-10 TiB required by Geth or OpenEthereum shown at https://etherscan.io/chartsync/chainarchive https://etherscan.io/chartsync/chainarchive You can fit a full history archive node on a Raspberry Pi and small consumer SSD now if you want.

3np 5y ago
Oh, hi Jamie :) Great to hear about the substantial improvements! Some follow-up questions if you don't mind: > the best implementations that I know about Would that be Erigon (nee turbo-geth)? I think work was just going public on that around the time we were messing around with this and I haven't taken it for a spin yet. > Syncing a full history archive node has been done in less than 1 day. Am I right in assuming that this is some form of snapshot sync which doesn't include actual validation/verification of the blocks and integrity of the resulting state? Because otherwise that sounds (almost) impossibly short. If so, do you have any idea what one should expect to be able to achieve for that? > The entire full history archive state fits in about 400 GiB. Similar question here - if we disregard indexing (and I guess tracing?) etc which would have to be done separately, would the information contained therein be enough to derive past block states and events and, say, replicate a block explorer like etherscan? Regardless, keep up the good work. Nimbus seems like a great eth2 client. EDIT: Ooooh, I see now that I'm more out of the loop than I realized and Nimbus is also doing an eth1 implementation now - is that what you're referring to?
- jlokier 5y ago
  >> the best implementations that I know about > Would that be Erigon (nee turbo-geth)? No, but Erigon is very good. Erigon2 (the research branch), Besu (bonsai mode), Akula and Silkworm are other small-storage implementations. >> Syncing a full history archive node has been done in less than 1 day. > Am I right in assuming that this is some form of snapshot sync which doesn't include actual validation/verification of the blocks and integrity of the resulting state? No. Geth snapshot sync takes 4-6 hours. The verification of all blocks from genesis is done by Akula in less than 1 day according to their own tests. (They evidently have a faster machine than I do, but extrapolating from other data I expect < 2 days on mine.) > > The entire full history archive state fits in about 400 GiB. > Similar question here - if we disregard indexing (and I guess tracing?) etc which would have to be done separately, would the information contained therein be enough to derive past block states and events and, say, replicate a block explorer like etherscan? Yes, that 400 GiB is enough to provide fast lookup of any data point in the state history, sufficient for a block explorer like etherscan. A bit more space may be required for indexes such as lookup by transaction hash, as you say. Technically just the blocks is enough to derive past states and replicate a block explorer :-) But I know what you mean. > I see now that I'm more out of the loop than I realized and Nimbus is also doing an eth1 implementation now - is that what you're referring to? No, because a Nimbus-eth1 node takes 24,000 GB and months to sync :-) Nimbus-eth1 has its good points but it's behind other implementations in performance at the moment. There have been some good performance proof of concept tests, but I don't work on Nimbus any more and can't comment on its future.
  - deleted 5y ago
    [deleted]

3np 5y ago

Oh, hi Jamie :) Great to hear about the substantial improvements! Some follow-up questions if you don't mind: > the best implementations that I know about Would that be Erigon (nee turbo-geth)? I think work was just going public on that around the time we were messing around with this and I haven't taken it for a spin yet. > Syncing a full history archive node has been done in less than 1 day. Am I right in assuming that this is some form of snapshot sync which doesn't include actual validation/verification of the blocks and integrity of the resulting state? Because otherwise that sounds (almost) impossibly short. If so, do you have any idea what one should expect to be able to achieve for that? > The entire full history archive state fits in about 400 GiB. Similar question here - if we disregard indexing (and I guess tracing?) etc which would have to be done separately, would the information contained therein be enough to derive past block states and events and, say, replicate a block explorer like etherscan? Regardless, keep up the good work. Nimbus seems like a great eth2 client. EDIT: Ooooh, I see now that I'm more out of the loop than I realized and Nimbus is also doing an eth1 implementation now - is that what you're referring to?

jlokier 5y ago
>> the best implementations that I know about > Would that be Erigon (nee turbo-geth)? No, but Erigon is very good. Erigon2 (the research branch), Besu (bonsai mode), Akula and Silkworm are other small-storage implementations. >> Syncing a full history archive node has been done in less than 1 day. > Am I right in assuming that this is some form of snapshot sync which doesn't include actual validation/verification of the blocks and integrity of the resulting state? No. Geth snapshot sync takes 4-6 hours. The verification of all blocks from genesis is done by Akula in less than 1 day according to their own tests. (They evidently have a faster machine than I do, but extrapolating from other data I expect < 2 days on mine.) > > The entire full history archive state fits in about 400 GiB. > Similar question here - if we disregard indexing (and I guess tracing?) etc which would have to be done separately, would the information contained therein be enough to derive past block states and events and, say, replicate a block explorer like etherscan? Yes, that 400 GiB is enough to provide fast lookup of any data point in the state history, sufficient for a block explorer like etherscan. A bit more space may be required for indexes such as lookup by transaction hash, as you say. Technically just the blocks is enough to derive past states and replicate a block explorer :-) But I know what you mean. > I see now that I'm more out of the loop than I realized and Nimbus is also doing an eth1 implementation now - is that what you're referring to? No, because a Nimbus-eth1 node takes 24,000 GB and months to sync :-) Nimbus-eth1 has its good points but it's behind other implementations in performance at the moment. There have been some good performance proof of concept tests, but I don't work on Nimbus any more and can't comment on its future.
- deleted 5y ago
  [deleted]